Commit Graph
68 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 e05d5d48ee Do not place on a fleet that has not reported in yet
CI / test (push) Has been cancelled
A control plane that has just started knows nothing, and its first
reconcile tick runs before any agent has registered. "needs 2 nodes on one
fabric, but no node reports a fabric" in the log is that, and it looked
harmless because adoption catches up.

What is not harmless is placing on the fraction of the fleet that happened
to report first. After one restart both replicas of qwen2.5-0.5b landed on
node04 — the only node registered at that instant — when four were coming,
and nothing ever moved them, because the desired count was met.

Placement now waits, and leaves by either of two doors, because either
alone is wrong. Waiting for the window to expire would cost that much
downtime on every restart even when the whole fleet reconnected in a
second; waiting only for the node count to stop changing would hang forever
on a cluster whose nodes never all come back. So: the count holds steady for
a quarter of the window, or the window expires.

Only placement waits. Node expiry, stuck-replica expiry, draining and
reaping all stay correct on a partial fleet, and adoption in particular must
not wait — it is how a restart recovers what is already running, and
delaying it would leave live engines unaccounted for exactly while the
reconciler decides what to build.

The window is paid once per process. A node joining a running cluster is an
ordinary event, not a reason to stop placing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:18:29 +02:00
Ilker ManapandClaude Opus 5 db448a9e87 Stop using a liveness timeout as a generation budget
CI / test (push) Successful in 1m23s
response_header sounds like a check that the endpoint is alive, and for a
streaming request it is: headers arrive before generation starts. For a
non-streaming request the engine sends nothing until the completion is
finished, so the same setting is the total generation budget wearing a
liveness timeout's name.

At 60s that was invisible on GPUs and a hard cap of roughly a hundred
tokens on qwen2.5-coder-14b at 1.7 tokens a second. A request for 120
tokens failed at exactly one minute, and neither the setting's name nor the
error said why.

Two transports now: the short header timeout for streaming, where it still
catches a sick endpoint, and a generation budget for non-streaming,
defaulting to ten minutes. One number could not mean both things.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:15:00 +02:00
Ilker ManapandClaude Opus 5 3081c6f69b Place on nodes that can actually hold the model
CI / test (push) Successful in 1m23s
Placement had no idea how much memory a node has or a model needs. A CPU
profile has no card to be free, so eligibleNodes accepted every ready node:
Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed
just as happily on one of the 8 GB nodes, which reports itself perfectly
healthy right up until the OOM killer arrives.

node_selector was yesterday's stopgap and it is the operator restating by
hand something the system can know. Agents now report MemTotal, profiles
declare min_memory_gb, and placement filters on it, naming what was short —
"has 8 GB of memory, needs 24" is actionable where "no candidates were
considered" is not. The label selector stays, because "this node is for
coding models" is a real thing to want to say that memory does not express.

Two deliberate non-exclusions. A profile with no requirement fits anywhere,
which is every profile that exists today. And a node that reports no memory
is still usable: an agent older than the field says nothing, and letting a
control plane upgrade quietly empty half a fleet is a worse failure than the
one this prevents.

ctl nodes grows a MEM column showing "-" rather than 0 for a node that did
not say, because an old agent is not a small machine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:12:44 +02:00
Ilker ManapandClaude Opus 5 e9fe3ad379 Know how big a model's context is, and what it can do
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which
surfaced this week as somebody else's bug.

Context length. Nothing read it, /v1/models did not publish it, and an
over-long prompt was not refused: it was admitted, scheduled, given a
replica and a slot, and rejected by the engine in whatever shape that
engine chose. A client sizing prompts from /v1/models had nothing to read.
It is now declared per model, published when known, and checked before
authentication work or a queue slot is spent — with a ten percent margin,
because the estimate is bytes over four and being a few percent out must
not turn a valid request into a refusal.

Capabilities. The gateway forwards a tools array to any model, and one that
cannot use them answers in prose; every OpenAI-compatible client then
behaves as though the model is broken, and the only way to find out was to
try it. Declared rather than discovered, because no engine will say — which
is why these are claims, and why the next commit makes engine-check verify
the tool-calling one instead of trusting it.

And a bug found while wiring the first: the token estimate was taken from
the routing prefix. That prefix stops at 2 KB by design, so every prompt
above it estimated as 512 tokens however long it really was, and TPM limits
undercounted exactly the requests that consume the most. Prefix routing
wants a bounded head of the conversation; a token estimate wants all of it.
They are now separate functions, and structured multimodal content counts
rather than measuring zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:07:49 +02:00
Ilker ManapandClaude Opus 5 ea87021b5c Correct what the previous commit claimed about --jinja
CI / test (push) Successful in 1m30s
9fc098f said passing --jinja fixes tool calling. It does not. llama.cpp
build 10991 has jinja enabled by default — "--jinja, --no-jinja ... (default:
enabled)" — so the flag changed nothing, and tool calls came back as text
before and after it in exactly the same way.

The flag is worth keeping for a different reason, and the comment now says
that one: it pins how prompts are built, so an engine upgrade cannot change
it underneath a running fleet. Older builds do default to the generic
template.

What actually happens is that llama.cpp renders the tools correctly — the
prompt from /apply-template carries the <tools> block and the instruction to
reply in <tool_call> tags — and qwen2.5-coder answers with a markdown JSON
block instead. 3B and 7B both, and tool_choice "required" does not constrain
it either. That is the model, not the plumbing, and no flag here will fix
it.

Recording it because a wrong explanation in a commit message is worse than
no explanation: the next person to see tool calls arriving as text would
have checked --jinja, found it already set, and had nowhere to go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:30:22 +02:00
Ilker ManapandClaude Opus 5 9fc098fd5b Start llama.cpp with the model's own chat template
CI / test (push) Successful in 1m25s
Tool calling did not work, and it looked like the model's fault. Asked to
use a tool, qwen2.5-coder produced exactly the right call —
{"name": "list_files", "arguments": {"path": "."}} — as plain reply text,
with tool_calls null and finish_reason "stop". Every OpenAI-compatible
client, opencode among them, sees a model that ignores the tools it was
given.

The cause is that llama-server falls back to a generic chat template unless
--jinja is passed, and the generic one neither formats the tool definitions
the way the model was trained to read them nor parses what comes back into
tool_calls. It affects ordinary chat too, for the same reason.

A profile can still pass --no-jinja, because a model whose bundled template
is broken has to be serviceable without editing the adapter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:20:55 +02:00
Ilker ManapandClaude Opus 5 c174e46902 Read the filename from a path comment inside the block
CI / test (push) Successful in 1m35s
The commonest convention of all, and the one the extractor missed. Asked
for a two-file Go server, qwen2.5-coder-3b opened a plain ```go fence and
wrote "// cmd/server/main.go" as the first line of the file — not on the
fence, not above it. Both files downloaded as snippet-1.go and
snippet-2.go, which is the feature failing at the first real use of it.

Found by running it rather than by reading it: the unit tests covered the
conventions I had thought of, and passed throughout.

Only the first line or two, only when the line is a comment holding one
token that looks like a path. "// this reads config.json at startup" is a
sentence, "// Package main implements the server." is a doc comment, and a
path comment further down the file is just a comment. The markers cover the
languages people actually ask for, including block comments closed on the
same line.

The comment stays in the body. People write a path at the top of a file
anyway, and editing model output to make a filename look tidier is a
surprise nobody asked for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:12:00 +02:00
Ilker ManapandClaude Opus 5 45d0dee470 Let a reply full of code be downloaded as a zip
CI / test (push) Successful in 1m25s
Asking a coding model for anything real produces a small tree, and the only
way to get it out of the page was to select each block and paste it
somewhere. A reply containing fenced code now offers a zip, with the
directory structure the model described.

Server-side, because the interesting part is the filenames. They are written
by a language model and they go into an archive that somebody extracts with
a tool that trusts archives, so "../../.ssh/authorized_keys" and
"/etc/cron.d/x" are inputs to expect rather than hope against. archive/zip
and a sanitiser with tests beat string handling in a browser. An absolute
path is made relative — it cannot escape once the leading separator is gone,
and refusing a file over a leading slash throws away work. Anything that
still escapes after cleaning is rejected rather than repaired, because
silently turning ../../evil.go into evil.go puts a file somewhere the reader
did not expect; it gets a generated name instead.

Naming follows what models actually do: on the fence (```go:cmd/main.go,
```main.go, ```go main.go, title="x"), or on the line above it as a heading
or comment. Only the two lines immediately above, so a sentence mentioning
main.go is not read as a filename. A block nobody named still comes back
under a generated name, because losing code because it was unlabelled is
worse than naming it badly, and two blocks claiming one path both survive —
models show a file and then show it again with one line changed.

Nothing is stored: the reply arrives, the zip goes back, neither is kept.
The endpoint sits behind the same session guard and CSRF check as
/chat/send.

Two of these tests asserted the wrong thing when first written — that an
absolute path must be refused, and that hyphens are stripped from the
archive name. Both were corrected to what the code should do rather than
the code changed to match them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:06:07 +02:00
Ilker ManapandClaude Opus 5 0a98d01361 Say when a chat reply was cut off, and stop cutting it so early
CI / test (push) Successful in 1m20s
Reported as the model not completing a request and stopping. It did
complete — it ran out of room. The chat page caps a turn at max_tokens and
then never looked at finish_reason, so a reply that hit the cap ended
mid-word and looked exactly like one that was finished.

The silence was the real defect. A reader has no way to tell "the model had
nothing more to say" from "the page stopped it", and the only clue is that
the last sentence ends abruptly.

512 was also too low. It was chosen when every model here was a 0.5B
answering in a paragraph; it is about a screenful of code, so a coding
model gets cut off inside the first function it writes. 2048 now, which is
still a bound — the reason for having one has not changed.

The notice is appended beside the reply, never into it: that element holds
model output placed with textContent, and building markup in the same node
is how the careful handling everywhere else gets undone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 21:03:07 +02:00
Ilker ManapandClaude Opus 5 634f87ba15 Provision nodes with the host CPU model
CI / test (push) Successful in 1m26s
Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar
baseline — instead of the backend built for the host's instruction set, and
an inference node is not slightly slower but unusable.

Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
thread count, the only change being --cpu host:

  Qwen2.5-Coder     kvm64   host
  1.5B q4_K_M        5.80  39.09  tok/s
  3B   q4_K_M        2.90  20.51
  7B   q4_K_M        1.33   9.88

Prompt processing went from 7.6 to 143.8, a factor of nineteen.

This cost an afternoon because nothing looks broken. The node registers, the
engine starts, the replies are correct; every number is just small, and the
conclusion that suggests itself is that CPU inference is hopeless rather
than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a
second was nearly written off on that basis.

The trade is live migration, which a VM pinned to the host CPU cannot do.
For a node whose job is arithmetic that is the right way round, and --cpu
overrides it where it is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 20:42:48 +02:00
Ilker ManapandClaude Opus 5 8a894af1ca Let a profile say which nodes it is for
CI / test (push) Successful in 1m35s
Agents have reported labels since the beginning and nothing read them. That
was survivable while every CPU profile was a 0.5B model that fits anywhere,
and stops being survivable with a 14B one: a CPU profile has no card to be
free, so eligibleNodes accepts every ready node, and placing Qwen2.5-Coder
14B on one of the 8 GB nodes would OOM a machine that reported itself
perfectly healthy.

Nothing in the inventory says how much memory a node has, and inferring it
would be guessing at the wrong layer — the amount that matters is the
model's working set, which the scheduler does not know either. So the
operator says which nodes a profile is for, with the labels that already
exist:

  node_selector:
    ram: 32g

A rejection names the label it did not find, because "no candidates were
considered" is not something anyone can act on and "does not carry ram=32g"
is. An absent selector constrains nothing, which is every profile that
exists today, and there is a test that says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 19:36:10 +02:00
Ilker ManapandClaude Opus 5 aa4da3264d Make adding a node one command, and check the engine actually runs
CI / test (push) Successful in 1m26s
Building node05 by hand turned up three things worth fixing, in the order
they bit.

Onboarding died on "Host key verification failed" — an error about ssh, in
the middle of a script whose entire job is to talk to a machine created
sixty seconds ago. It now uses accept-new, which learns an unknown host key
and still refuses a changed one; the risk being guarded against is a
substituted machine, not a new one.

Then ssh started refusing connections on a node that had answered a moment
before: cloud-init regenerates the host keys and restarts sshd partway
through a new machine's first minute. The agent was registered and healthy
throughout. Onboarding now waits for ssh rather than failing on it, which
is also what makes running the two halves back to back safe.

Then llama-server: present, executable, and unable to start for want of
libgomp1. The node registered ready and would have stayed ready until a
replica was placed on it and the shard died with a missing shared library
three logs away. The docs already warned about this shape of failure;
nothing checked for it. Onboarding now runs a configured engine and says
plainly when it will not start.

add-node.sh drives both halves. The reason is not keystrokes: the VM name,
the hostname, the node id and the certificate's CN must all be the same
string, and the address appears in three places. Split across two commands
run minutes apart, that agreement is held together by whoever is typing.
Here it is stated once, and the VMID defaults to the last octet of the
address so qm list and ctl nodes stay in the same order. The two halves
remain separately runnable, because needing the hypervisor and needing the
CA really are different access.

Verified by building a throwaway node06 with it: 45 seconds from nothing to
registered, and the engine check caught that node06 had no llama-server.
Destroyed afterwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 19:07:56 +02:00
Ilker ManapandClaude Opus 5 082d52bc97 Stop shipping a placeholder that looks like a value, and say what is served
CI / test (push) Successful in 1m23s
A user copied the curl from the profile page and sent it as it stood. Every
other field in that snippet is filled in for them — the base URL, the key
they had just been issued — so "model": "MODEL" reads as a value rather
than a blank, and they got:

  no endpoint serves "MODEL"

which is true and useless. It reads the same whether the name was a typo,
a placeholder left in from the documentation, or a real outage, and it
never says what would have worked.

The message now lists what is being served, capped at ten with a count for
the rest, and says plainly when nothing is. That list is what /v1/models
already returns to the same caller, so it discloses nothing new.

The profile page's example now names a model that is actually running,
real ones before simulated — the point of an example someone can run is
that it runs, and sending a first request to the sim engine would answer
in Greek letters. The landing and claim pages keep a placeholder, because
they are served without a key and deliberately say nothing about what is
here; theirs is <model-name>, which cannot be mistaken for a value, with a
line saying where to get the real one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 18:23:19 +02:00
Ilker ManapandClaude Opus 5 6f3bde099e Let a multi-node replica be adopted one agent at a time
CI / test (push) Successful in 1m20s
A replica spanning two nodes is reported by two agents, registering
separately about a second apart. Adoption assumed one call carried a whole
replica, so the first half was adopted with WantShards=2 and immediately
failed for "replica has 1 of 2 shards" — and when the second node
registered, its half was condemned as unclaimed because the replica was
already written off.

big-model therefore could not survive a control plane restart. Seen after
today's deploy: adopted, failed, both halves stopped, rebuilt from
scratch, backing off 9s, 20s, 38s, and never becoming ready. The first
reconcile tick also runs before any agent has reconnected, which is the
harmless-looking "no node reports a fabric" that precedes it.

Two changes. Adopt merges a group into a replica it is still assembling
instead of treating it as a competing claim, and takes the endpoint from
whichever group carries rank 0 — the leader's agent is not necessarily the
first to register. deriveState leaves a replica that is short of shards
while still assembling in Starting rather than Failed, which puts it under
the startup timeout that already exists: a sibling that never arrives
still fails, one second later than it used to rather than one second
before its sibling reports in. AdoptedAt is cleared once the replica is
whole, so losing a shard after that is a failure again.

The existing multi-node adoption test passed throughout, because it hands
both shards to a single Adopt call — which no caller makes. Same shape as
the agentsvc bug in c97b671: a test that calls the function under test
directly cannot tell you how the caller calls it. The new test registers
one node at a time, and fails without this change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:15:00 +02:00
Ilker ManapandClaude Opus 5 72b03a1a1c Say which model is simulated instead of letting it look broken
CI / test (push) Successful in 1m31s
big-model runs on the sim engine, whose whole vocabulary is the Greek
alphabet, cycled at a fixed rate until max_tokens. That is what it is for:
with no H100s it is the only thing exercising the TP8 two-node path, the
fabric invariant and rebuild-after-node-loss. The output is the point.

What was wrong is that nothing said so. /v1/models returned it with
owned_by "llmgw", identical to the real model, and the chat picker sorted
purely by name — so "big-model" led the list, was preselected by the
browser, and anyone who typed a question and pressed Send got Greek
letters back with no explanation. Reported as the model spitting out crap,
which is a fair reading of what it looks like from outside.

Simulated is derived from the profiles rather than configured: an operator
who adds a sim profile and forgets a flag would be shipping placeholder
text under a real model's name. A model counts as simulated only when
every profile is, because one real profile means the request can still
come back as inference.

The marker on /v1/models is owned_by "llmgw-sim", which is already what
the sim engine calls itself on its own /v1/models — the gateway was
flattening a distinction that existed one layer down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:11:56 +02:00
Ilker ManapandClaude Opus 5 cbf276a41b Render maths in the chat, without letting the model write markup
CI / test (push) Successful in 1m20s
Replies are plain text, so a model asked about an integral answers with
\int_0^1 x^2 dx and the reader gets the source.

KaTeX is vendored into the binary rather than loaded from a CDN. This gateway
makes no outbound requests of its own and is expected to run where it could
not make one; a chat page that silently loses its maths on an air-gapped
cluster would be worse than one that never had it. 600 KB of JS, CSS and WOFF2
fonts, which the stripped binary absorbs without changing its rounded size.

The ordering is the safety argument, and there is a test for it rather than a
comment alone. The reply is put in place with textContent exactly as before,
and only then walked for maths: renderMathInElement replaces the spans it
recognises, so nothing the model wrote is ever parsed as HTML and the markup
that appears is KaTeX's. trust stays false — KaTeX's default, and what keeps
\href and \includegraphics out of a reply — and throwOnError is off so one
malformed expression does not take the rest with it.

Rendered once the reply is complete. Typesetting each chunk would try to set
half an expression on every token and flicker through the whole answer.

The static handler now serves a small embedded asset tree rather than one
file, so it moved to its own embed: a file server over the tree that also
carries the templates would serve those too. Directory listings are refused —
nothing links to one, so the only requests are from somebody looking around.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:03:46 +02:00
Ilker ManapandClaude Opus 5 3b67b1fe42 Stop the dashboard linking to pages that are not where it is
CI / test (push) Successful in 1m25s
Clicking "metrics" on the web dashboard answered "no api key was provided".

The page is served twice and its navigation assumed one of them. Metrics,
pprof and the management API exist on the admin listener; account
administration, the profile and the chat exist on the client-facing port. The
nav offered the first set in both places, so on the public port those links
fell through to the API-key middleware — which is a far more confusing way to
say "that is not here" than a 404 would have been.

Each page now offers what is actually reachable from it, and the public one
says where the rest lives rather than pretending it does not exist, including
why that port is not exposed: pprof can dump the process heap, and the heap
holds every API key in plaintext.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:57:58 +02:00
Ilker ManapandClaude Opus 5 f68fbf7c49 Check llmgw's engine flags against real vLLM and llama.cpp
CI / test (push) Successful in 1m25s
The launch arguments had been read from documentation and never from an
engine. Now they have been:

  vllm 0.29.0    --host --port --tensor-parallel-size --served-model-name
                 --distributed-executor-backend      all accepted
  llama.cpp b10991  --model --host --port --metrics --alias --parallel
                 --n-gpu-layers                      all accepted

Getting there found three faults in the checker rather than in llmgw.

It checked every command's flags against the engine's help, but a shard can be
several programs — vLLM under Ray is `ray start` and then `vllm serve` — and
they have different vocabularies. It reported --head and --num-gpus as
missing, which is true of vllm and irrelevant, because ray is what receives
them. Each command is now checked against its own help, and one that cannot be
run is reported as skipped rather than failed.

vLLM does not give up its whole help for --help: it prints a summary of
section names and expects --help=all. Checked against the summary, all five
flags looked missing. The warning about truncated help did not fire, because
the summary happens to mention --help= while listing how to ask for more.

And vLLM refuses to print anything at all on a machine with no accelerator
unless VLLM_TARGET_DEVICE says what to assume. Asking an engine about its
flags should not require the hardware it runs on.

Two things remain unchecked and are written down as such: the Ray sidecar,
because vLLM 0.29 does not install Ray, and SGLang, because it is not
installed anywhere. `launch_mode: mp` avoids Ray entirely and is worth
preferring until someone checks it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:52:11 +02:00
Ilker ManapandClaude Opus 5 5b425c32e5 Ship binaries with releases, and ask engines whether our flags still exist
CI / test (push) Successful in 1m31s
A release was a git tag. Installing llmgw on a node meant having a Go
toolchain, and the onboarding script copied a binary from wherever it was run —
which works and quietly makes whoever runs it the build server, with the fleet
ending up on whatever was in their working tree. Tags now build static
linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is
the point: it has to run on a node whose libc is older than the builder's, and
llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so
this stays true. Verified: "not a dynamic executable", runs on a node, reports
its version.

`ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch
arguments are read from each engine's documentation, and engines rename flags
between versions; when that happens the failure arrives minutes into a model
load, in the engine's own error message, on a node nobody was watching. It
builds the widest command llmgw would ever produce — multi-node, since a flag
that only appears there is the one nobody notices has gone — and checks each
flag against the engine's own --help. No GPU: a CLI prints its help anywhere.
Verified against llama.cpp b10991, all seven flags accepted.

Flags are matched on a word boundary, so --port is not satisfied by
--port-range. Without that, a renamed flag would sail through the check that
exists to catch renamed flags.

It also warns when the help text looks truncated, which is how the first run
went: llama.cpp's help cut to 400 of its 725 lines reported five flags as
missing that the running cluster uses.

TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky —
its wait loop fell through on timeout, so a slow machine turned the ordering
assertion into a no-op and the failure surfaced later as something else. It now
asserts, with a deadline that suits a machine running every other package at
the same time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +02:00
Ilker ManapandClaude Opus 5 7b4acd9284 A replica missing a shard is not ready
CI / test (push) Successful in 1m23s
Restarting the control plane over a running two-node replica adopted half of
it. Every shard it still had was healthy, so it was reported Ready — with one
shard of two and no endpoint at all, because the missing one was rank 0. The
router was handed a replica nothing could route to.

The cause is that a replica had no idea how many shards it should have. It
could tell "every shard I have is ready" and nothing else, which is the same
sentence whether or not half of it is gone. It now carries the count from its
profile, set at placement and again at adoption, and a replica short of shards
fails rather than pretending.

Onboarding also wrote no engine paths, so a node onboarded by the script could
not find llama-server and failed every shard placed on it with "executable
file not found". The script now takes them, and says so when none are given —
a node with no engine paths can run the sim engine and nothing else.

The cluster now runs both at once: two-node simulated replicas on each fabric,
and real llama.cpp inference on the CPU, which is a better test bed than
either alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:43:51 +02:00
Ilker ManapandClaude Opus 5 c97b671ef9 Prove the multi-node path on a simulated cluster
CI / test (push) Successful in 1m29s
Phase 4 had been written and never run. The hardware is not coming, so the
cluster is simulated instead — four nodes across two fabrics, eight GPUs each,
built with the onboarding scripts from the previous commit.

`sim` is now an engine adapter, not just a standalone command. Its leader binds
a rendezvous port and does not serve until every worker has joined; its workers
serve nothing at all. That is the part stub processes could never exercise, and
it is why this path survived four phases untested: the command line was
checked, the sequence was not.

Measured on the cluster:

- a 2-node replica on each fabric, node01+node02 on ib-a and node03+node04 on
  ib-b, four GPUs from each
- a request served through one of them
- only the leader listening — node02 had no port open at all, which is
  ServesHTTP working end to end rather than in a unit test
- node02 killed; at t+60s it was declared lost, the ib-a replica failed and was
  reaped; at t+80s a replacement was ready on ib-b, the fabric that could still
  host it, while node01 sat idle rather than being paired across fabrics

The invariant held under the one condition that tests it: when a replica could
not be rebuilt where it was, it moved to a fabric that could hold all of it
rather than spanning two.

What this does not prove is written into the plan next to what it does. The
GPUs are an inventory, the fabric is a sysfs-shaped directory, and the engine
speaks no NCCL — so the launch sequence is now exercised while the launch
arguments for vLLM and SGLang are still read from documentation rather than
from a running engine.

docs/node-onboarding.md is the procedure, including how to run all of this on
machines that have neither GPUs nor InfiniBand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:34:23 +02:00
Ilker ManapandClaude Opus 5 c1ac2f022f Onboard a node with a script, and find two examples that never worked
New nodes had been joined by hand, four times, by me. There was no procedure,
so there was nothing to be wrong — and nothing to be right either. Two scripts
now: provision-node.sh makes the VM on Proxmox, onboard-node.sh gives it an
identity. They are separate because they are run by different people with
different access, and the second must run where the certificate authority is,
which is the one part that cannot be delegated.

Running them turned up why phase 4 was never verified. FabricAttachment
carried no koanf tags, so the `name:` key the shipped example documents bound
to nothing and `rate_gbps` was never read at all. An agent configured exactly
as instructed refused to start with "fabrics[0]: name is required" — naming
the key the operator had just written. Every fabric in this project had been
configured in a Go test, where the tags do not matter.

The server example did not load either: its models had no source, so it failed
validation the moment anything tried to parse it.

Both examples are now parsed by a test. A published example that has never
been loaded is a guess, and these two had been guesses since they were
written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:28:36 +02:00
Ilker ManapandClaude Opus 5 a075ff4120 Add approximate prefix routing, and find out it did nothing
CI / test (push) Successful in 1m19s
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which
replica has already prefilled one. Bytes, not tokens — needing the engine's
exact tokenizer is what makes tier 2 a separate project.

It went in inert, and the decision metric is what caught it: 144 lookups, zero
hits. The cause was a rule I had written and tested for. A node's label is the
first request that created it, so the second request for the same system
prompt diverges in its last few bytes — a new question after the same
preamble, the single most common shape there is. Requiring the whole label to
match threw away the 700 bytes they shared. What matters is the shared head,
because that is what the engine cached, and a partial label match now counts.
Two unit tests asserted the wrong rule and have been inverted.

With that fixed, measured end to end against engines whose caches are too
small for the working set: 22.5% cache hits without, 81.7% with, and the
breakdown shows the misses are exactly the first round, when the tree is
empty.

The scenario matters as much as the number. With a handful of prompts and a
large cache, scattering is free — every replica ends up holding everything —
and prefix routing can only add constraint. The first version of the test
measured that case and showed a loss, which was a true statement about a
situation the feature is not for.

Two things keep it from being the disaster prefix routing usually is. The
imbalance guard, which needs both an absolute and a relative gap before it
fires, because either alone misfires — absolute on an idle cluster, relative
on a busy one. And it is a scorer rather than a filter: affinity is a vote the
queue and inflight scorers can outvote, not a rule.

The tree records the replica the picker actually chose, not the one the scorer
proposed, and a node prefers the replica it has seen most rather than most
recently — a count is stable under a randomizing picker, where recency just
chases whatever the picker last did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:13:19 +02:00
Ilker ManapandClaude Opus 5 de21649ad6 Add an in-browser chat, without putting a key in the browser
CI / test (push) Successful in 1m13s
So someone can try a model without setting up a client, and — more often —
tell whether a problem is the gateway or their code.

The page carries a session, and the server turns that into the identity an API
key would have produced, resolved from the person's project. The request then
goes down the ordinary path by being handed to the same handler /v1 uses:
admission, model allowlist, rate limits, budget, accounting. There is exactly
one request path, so there is no second place for its rules to drift — and
without that, a playground is a hole in every limit the rest of the gateway
enforces. The allowlist test posts a forbidden model and expects 403 for that
reason.

The body is rebuilt rather than forwarded, so the page cannot set fields it has
no business setting: a stream:false that would hang the reader, a max_tokens
large enough to occupy a replica for an hour. One turn is capped at 512 tokens.

The picker offers what is actually served and allowed, not what is catalogued,
because offering a model with no replica produces a 503 the reader cannot
interpret.

Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG.
It streams, because on CPU inference ten seconds is normal and a chat that
hangs and then dumps a paragraph feels broken. Replies are written with
textContent throughout: model output and user input have no business being
parsed as markup. Its POSTs carry the CSRF token in a header, which a
cross-site form cannot set at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:05:30 +02:00
Ilker ManapandClaude Opus 5 859f08da9b Approve accounts from the browser, and draw the traffic
CI / test (push) Successful in 1m19s
Two things the dashboard was missing: it could not act, and it was a column of
numbers.

Account administration moves to /admin/users — approving into a project,
suspending, reinstating. That is the work an operator does constantly and it
should not need a terminal. Three things stay on the CLI, and the line is not
arbitrary: promoting someone to admin, resetting a password, and issuing a key
on another person's behalf each turn a stolen admin session into more than it
was — permanent access, or somebody else's account. Approving and suspending
do not. Suspending signs the account out everywhere, or suspension is advisory
until the session happens to expire, and an admin cannot suspend themselves,
which is not a thing an interface should let you do by misclicking.

The charts are inline SVG generated in Go. A charting library would mean a
JavaScript toolchain in CI and a megabyte of bundle for what is a list of
rectangles, and the single-binary goal rules that out. Hourly over a day: fine
enough to show a working day, coarse enough that one slow minute is not an
incident. Empty hours are filled in, because a chart with hours missing where
nothing happened reads as a chart with hours missing from the data.

The SVG is marked as trusted HTML, so everything that reaches it is either a
number this package computed or escaped explicitly, and a bar's colour can only
be one of four names — a colour that could be anything is a way to write CSS
into every page the chart appears on. There are tests for both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:01:42 +02:00
Ilker ManapandClaude Opus 5 e519dc587c Serve the cluster dashboard over the web, without serving pprof with it
CI / test (push) Successful in 1m21s
Reaching it meant an SSH tunnel, because the page lived on the admin listener
and that listener is bound to loopback. Opening the port was never the way to
fix that: it also serves pprof, and pprof can dump the process heap, where
every API key sits in plaintext. Exposing the port would have been exposing
the key store.

So the page moves rather than the port. The same dashboard is now mounted at
/admin on the client-facing port, behind the session and CSRF machinery that
already guards /profile, gated on a flag on the account. The admin listener
keeps pprof and metrics and stays on loopback.

One bit, not a role system: it grants looking. Everything that changes the
cluster or issues a credential stays on the CLI, where it needs a shell rather
than a session — which keeps the worst case of a stolen admin cookie to "saw
the cluster" rather than "drained your fleet".

Someone without the flag gets 404, not 403. A 403 confirms the page is there
and that admin accounts exist here, which is free reconnaissance; 404 is what
every other unknown path returns. The profile page does not show a link to a
door the visitor cannot open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:46:45 +02:00
Ilker ManapandClaude Opus 5 058ee382fa An agent reporting nothing is reporting something
CI / test (push) Successful in 1m18s
The previous commit taught the reconciler to let go of shards an agent says
it is not running. It made no difference on the cluster, because the service
returned early when the agent's list was empty — which is precisely the case
that carries the news.

So the fix was dead code in production while its unit tests passed, and they
passed because they called the reconciler directly. The test added here goes
through the real stream instead, and asserts only that an agent running
nothing reaches the reconciler at all.

That is the lesson worth keeping: a test that calls the function under test
directly cannot tell you the caller never calls it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:36:35 +02:00
Ilker ManapandClaude Opus 5 aef1c143c5 Let go of shards an agent says it is not running
CI / test (push) Successful in 1m19s
Adoption handled "the agent has something we do not know about". The mirror
image was missing: "we think the agent has something it does not".

Both follow from the same rule — the fleet is authoritative about what is
running (plan §6.5) — and without the second half an engine that dies while
its agent survives leaves the control plane believing a replica is Ready
forever. Measured on the cluster: the engine was killed, the agent restarted
and registered with nothing running, and the replica stayed Ready pointing at
a dead port for two and a half minutes until I stopped watching. Traffic
survived, because the breaker took the endpoint out and retries covered it —
24 requests, none failed — so the only symptom was a cluster quietly running a
third short.

A registration follows a fresh connection, so anything dispatched over a
previous one is not running by definition; there is no window where a shard is
legitimately started but unreported.

The previous commit's disown rule was too broad, which this turned up: when a
known replica had a live shard it disowned everything the agent reported for
it, including the shards we already knew. A reconnecting agent would have been
told to stop the replica that was serving. It now disowns only what it does
not recognise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:30:49 +02:00
Ilker ManapandClaude Opus 5 555e4678cb Ship the systemd units, and stop the plan lying about what is done
CI / test (push) Successful in 1m16s
Phase 3 had a block duplicated into itself, including an unchecked
'EndpointSource becomes dynamic' sitting directly under the checked line
saying it was wired. Several phase 5 items had shipped without being ticked.

The units were only ever on the machines and in a scratch directory, which
makes them undocumented operational knowledge rather than part of the project.
KillMode=process in particular is not obvious and not optional: without it
systemd takes the engines down with the agent, turning a thirty-second upgrade
into a model reload on every node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:04:44 +02:00
Ilker ManapandClaude Opus 5 378383dd69 Align the bench table, which a tabwriter cannot
CI / test (push) Successful in 1m16s
Rows are printed as each concurrency level finishes, and a tabwriter lays out
only what has been flushed — so it aligned the header and the first row to
each other and every row after that to nothing. Fixed columns instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:59:55 +02:00
Ilker ManapandClaude Opus 5 5c837c531b Pin down where CPU replicas land, and where they deliberately do not
CI / test (push) Successful in 1m15s
Two tests for behaviour that looked wrong on the cluster and was not. Three
replicas all landed on one node, which is correct when it is the only node
registered: spreading is a preference and a model left unplaced to preserve a
preference would be worse. Over successive ticks with somewhere to go, they
spread.

What neither test asserts, because it is not true: nothing rebalances them
afterwards. A replica that moved on its own would interrupt the traffic it is
serving, and v1 has no autoscaler to decide it is worth it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:57:12 +02:00
Ilker ManapandClaude Opus 5 8a062fec35 Stop reusing replica ids, and stop handing out ports strays still hold
The previous commit's adoption fix made a deeper problem visible instead of
fixing it. Replica ids were the model name and a counter, and the counter
restarted with the process — so a control plane that came back placed a
replica called llama-1 while an engine started before the restart, still
running on a different node, also called itself llama-1.

Two nodes then claimed the same shard id. Adoption handed it back and forth
between them, every placement landed on a port an orphan was still bound to,
and the engine failed to bind, which failed the replica, which placed another
one on the same port. On the real cluster that loop ran for minutes:

    couldn't bind HTTP server socket, hostname: 0.0.0.0, port: 20002

Ids now carry four random bytes, which costs nothing and makes the collision
impossible. The "already known" branch in adoption goes back to meaning what
it says: an agent reconnecting, not a name clash across restarts.

Ports a stray still holds are also reserved now. Stopping is not instant and
an engine keeps its socket until it exits, so the allocator has to know about
sockets no replica claims — otherwise the same loop reappears in the window
between telling an agent to stop something and it actually stopping.

Two of the tests written for the previous commit had quietly become vacuous
under the new ids, asserting things about a literal llama-1 that no longer
exists. They capture the id now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:54:57 +02:00
Ilker ManapandClaude Opus 5 05bdc7d6d4 Add ctl bench, and fix two holes it walked into
`ctl bench` measures what a replica can actually serve, so weight,
target_waiting and kv_ceiling stop being numbers somebody typed. Concurrency
climbs until the SLO breaks and the last level that met it is the capacity:
peak throughput arrives long after latency has become unusable, so throughput
alone overstates what the hardware can serve. Queue depth and KV usage are
read from the engine while it is still meeting the target, which is exactly
what admission should treat as healthy.

Prompts are unique per request by default. With one repeated prompt the
engine's cache answers most of the prefill and the measurement describes the
cache rather than the hardware; --shared-prefix asks that question
deliberately, and it now produces byte-identical prompts, which the first
version did not — it dropped the random header and still picked words at
random, so the cache missed and the flag quietly measured the thing it exists
to avoid.

Running it against the cluster turned up two real defects:

A replica stuck in Starting never timed out. Every other way out of that state
needs the agent to report something, and when it never does the replica sits
there holding GPUs and counting towards the model's replica count while
serving nothing. Two of them had been like that for eight minutes past a
two-minute startup timeout, and nothing would ever have moved them.

They were stuck because adoption lost to a replica we had only placed. Replica
ids come from the model name and a counter, so a control plane that restarts
and places before its agents reconnect builds a different replica under the
same id on a freshly allocated port; the agent's report of the engine that was
actually serving was then discarded as already-known. The cluster pointed at
port 20004 while the engine answered on 20000. The fleet is authoritative
about what is running, so what we merely placed now loses to what an agent
reports — unless our copy has a shard that has actually come up, in which case
two things really are running and the stray is the one we did not ask for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:51:37 +02:00
Ilker ManapandClaude Opus 5 ec943f9587 Stop a one-in-sixty-two test from failing once in a while
CI / test (push) Successful in 1m14s
TestRejections built a "wrong" key by replacing a valid one's last character
with X. Keys are base62, so one in sixty-two already ends in X and the wrong
key was the right one, which authenticated and failed the assertion.

It passed locally all day and failed in CI, which is how long a flake of that
shape takes to look like a real bug — long enough that the instinct on seeing
it is to go looking in the authenticator.

The replacement is now chosen to differ from what it replaces, with a test on
the helper itself: a mutation that can return its input turns the test it
serves into one that quietly checks nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:42:20 +02:00
Ilker ManapandClaude Opus 5 79e459d9ff Let people change their own password, and operators reset a lost one
CI / test (push) Failing after 1m11s
The account system shipped without either, which made the first forgotten
password an unanswerable support request.

Changing it requires the current one. A session is what an attacker gets from
a borrowed laptop or a stolen cookie; the password is what they do not have,
and without the check the first thing they would do is take the account.

Every session is closed, including the browser doing the changing — leaving
the others open would mean a password change does nothing about the reason
someone is changing it. A new session is then opened for the browser in front
of us, so the person who just proved their password is not thrown back to the
sign-in page. That response also carries a new CSRF cookie, so the forms it
renders take the new token rather than the one the request arrived with; there
is a test that posts a form from that exact page.

Guessing the current password is bounded the same way sign-in is, because
someone holding a stolen session should not get unlimited attempts at the
password that would let them keep it.

There is no self-service reset: this gateway sends no email, and a reset link
with nowhere to send it is not a reset. `llmgw users reset-password` is the
operator's answer, and it closes their sessions too — if someone lost their
password, the reason might be that somebody else has it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:39:11 +02:00
Ilker ManapandClaude Opus 5 216fc9bb03 Let people sign up, be approved, and issue their own keys
CI / test (push) Successful in 1m9s
Every key needed an operator with a shell. This adds the login system that
made that necessary, which is the thing I argued against building until it was
asked for — so it is built as if it will be attacked, because a public login
form is.

Signing up produces an account that can do nothing. Approval is what makes it
work and stays on the CLI: having a shell on the control plane is what makes
someone an administrator, and that is a smaller thing to get right than roles
and an admin UI that mints credentials.

The details that matter:

Passwords are argon2id, where a slow hash is correct — the input is chosen by
a human and has a dictionary, which is the opposite of an API key. Both kinds
of authentication now live in this codebase and they are in separate packages
so neither set of assumptions leaks into the other. Length is the only rule:
composition rules push people towards Password1! and cost more entropy than
they buy.

Signup never says whether an address already has an account, and sign-in
answers the same way for a wrong password and a missing account — verifying
against a dummy hash when there is no user, so a stopwatch cannot enumerate
accounts. Account state is checked after the password, not before, so guessing
an address reveals nothing about it. Guessing is bounded per address rather
than per IP: an attacker has many addresses to come from and one to guess
against, and limiting by IP locks out an office behind one NAT.

Sessions are server-side, so signing out and suspending take effect
immediately rather than whenever a token expires; suspension is checked in the
same query that resolves the session, because a second round trip is a check
somebody eventually forgets. Every state-changing form carries a CSRF token
verified against a hash in the session — a cookie authenticates the browser,
not the page that asked.

Key ownership is enforced in the UPDATE rather than by reading first, so a
guessed key id from another account changes nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:32:35 +02:00
Ilker ManapandClaude Opus 5 8d35faf6f8 Give the landing page somewhere to send people
CI / test (push) Successful in 56s
The page explained the API, said keys come from an operator, and left the
reader there. Someone who reaches the address without a key had no route to
getting one, which is where onboarding stopped.

`landing.contact` is that route, because the gateway cannot guess who to ask.
An address becomes a mailto link, an http(s) URL stays a link, and anything
else — a channel, a queue, a person — is shown as text. Other schemes are
never linked: this string comes from configuration, configuration is not
always written by whoever runs the gateway, and a javascript: href on a page
served to every visitor is a short path to a stored XSS. Escaping the text
does not make a scheme safe.

Unconfigured, it now says something rather than nothing, and the page explains
that a key arrives as a one-time link so the reader knows what to expect
before it lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:22:42 +02:00
Ilker ManapandClaude Opus 5 3ed211b2f4 Onboard users with one-time links instead of pasted keys
CI / test (push) Successful in 2m43s
Issuing a key worked; getting it to the person it was issued for did not. The
operator had a secret on their terminal and no way to send it that did not
leave a bearer credential in a chat log forever.

`keys create --claim` prints a link instead. GET shows a button and consumes
nothing; POST reveals the key and burns the link. That split is not politeness
— chat clients, mail scanners and link previewers fetch URLs the moment they
are posted, so a GET that consumed the claim would mean the first thing to
unfurl the link destroys the key before anyone reads it.

A pending claim has to hold the key somewhere, which would otherwise put a
working credential in the database and undo what hashing bought. So the key is
sealed with a key derived from the link's own token, and the token is never
stored — only its hash. A database leak yields two useless hashes and a blob
nothing in the database can open. HKDF rather than the raw token, so the value
used for lookup can never become the value that decrypts.

Consuming is one UPDATE moving claimed_at off NULL, so two people opening the
same link produce one key rather than two copies. A used link, an expired one
and a token that was never real all render identically, because saying which
confirms that a token was once valid. The page is no-store, no-referrer and
noindex: the URL is itself the credential.

`GET /me` reports a key's own limits and spend, so nobody has to ask an
operator to read a dashboard they cannot reach. Outside /v1, because it is not
part of the OpenAI API.

docs/onboarding.md is written for whoever operates this, including the parts
that are deliberately absent — no self-service, no SSO, and why.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:11:28 +02:00
Ilker ManapandClaude Opus 5 cd3233ffe8 Do not leave the key store world-readable
CI / test (push) Successful in 1m1s
SQLite creates its file with the process umask, which on a stock system means
every account on the machine can read it. What is in there is not a working
credential — keys are stored hashed — but it is the whole tenancy tree and
every request anyone made, and that should not depend on an operator
remembering to chmod it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:01:41 +02:00
Ilker ManapandClaude Opus 5 1839832b8b Add a landing page and a read-only cluster dashboard
CI / test (push) Successful in 1m2s
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.

Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.

The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.

Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.

Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:58:05 +02:00
Ilker ManapandClaude Opus 5 0b93fb2883 Enforce budgets and rate limits, and write usage off the request path
CI / test (push) Successful in 1m7s
The second half of phase 5. A key now has a cost, a quota and a bill.

Rate limits use a sliding window rather than a fixed one. A fixed window lets
a caller send its whole minute's allowance in the last second of one window
and again in the first second of the next, so the observed rate is twice the
configured limit exactly when the system is least able to take it. Checking
and charging are one operation, or two concurrent requests both see room and
both take it.

Token limits are checked against an estimate, because the real count only
exists once the request has run; the difference is charged afterwards, so a
caller who consistently underestimates is still bounded, one window late. A
request with no max_tokens gets a modest assumption rather than zero —
assuming zero would let an unbounded request through every token limit.

Limits are checked before admission, not after: a request over its quota
should be refused rather than queued, where it would occupy a band's capacity
to no purpose. All refusals are 429, but a spent budget carries no Retry-After
because waiting does not help until the period rolls over, and saying
otherwise turns one client into a retry loop.

Usage goes to a buffered channel and is written in batches. Record never
blocks: if the buffer fills — a database stalled long enough to lose eight
thousand records — records are dropped and counted, because losing accounting
is bad and turning a stalled database into an outage is worse. A clean
shutdown drains what is queued; those are requests that were already served.
Failures are logged at error, since silently losing billing data is the kind
of thing discovered at the end of a month.

Requests are accounted whatever their outcome. A request that used a GPU and
then failed still used it, and a key that could spend nothing by failing would
be a way around a budget.

Budgets are counted in memory and restored from the store at startup, so a
restart does not hand every key a fresh allowance. The budget period travels
with the reservation rather than being re-derived, because a monthly budget
credited to a daily counter is a budget that never binds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:52:36 +02:00
Ilker ManapandClaude Opus 5 45c8cd23a3 Close the gateway: API keys, tenancy and model allowlists
CI / test (push) Successful in 2m34s
Until now anything that could reach the client port could use every model in
the catalog. A key now resolves to an identity, and that identity decides what
it may do.

Authentication wraps the whole mux rather than checking inside each handler,
so adding a route cannot add an unauthenticated one by omission. Health stays
open, because a load balancer has no key and a health check that needs one
reports the gateway down.

Nothing on the request path reads the database. A key is resolved once and
answered from memory until the entry expires; rejections are cached too, so a
client looping with a bad key cannot turn into a database read per request.
The price is that a revocation takes up to cache_ttl to take effect, which is
the trade stated in the configuration next to the knob. A store failure is
not an authentication answer: it is never cached as one and returns 503, not
401 — sending someone hunting for a key problem that does not exist wastes the
one thing an incident is short of. A revoked key is refused with the same
message as an unknown one, because saying otherwise confirms the key was real.

With auth on the band comes from the key and the band header is ignored. The
header was trusted only while there was nothing to consult, and it let any
client promote itself into the top priority band. Tenant fairness within a
band is now per key rather than per session, so one caller cannot take a band
by opening many sessions.

Allowlists are decided per model, not per string: a model's canonical name and
every alias name the same thing, so either form works in an allowlist and no
alias can reach a model that was not allowed. The first version of that test
passed without the resolution being there at all, which is why it now checks
all four combinations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:44:33 +02:00
Ilker ManapandClaude Opus 5 29617f8cb1 Add the store: tenancy, keys and usage on SQLite
Phase 5 begins. This is the persistence layer for the things the cluster
cannot reconstruct — who may call, what they are allowed, what they used.
Everything else llmgw knows is still derived from the fleet, which cannot be
wrong about itself the way a database can (§6.5).

SQLite behind an interface. The plan justified that by write volume, which was
the wrong argument: the write path is batched anyway, and a 500-row
transaction every 200ms is nothing for either engine. The real argument is the
failure mode. Postgres is a second service, and a gateway that stops serving
because its accounting database is unreachable is worse than one that buffers
some accounting. Embedded means the store shares the gateway's fate instead of
adding an independent way to be down. That reasoning stops holding at two
instances, where budgets have to be counted somewhere shared — so the
interface is here from the start, the SQL is portable, and migrations are per
dialect with shared version numbers.

Keys are stored as sha256 and nothing else, so a database leak hands over no
working credential. The fast hash is deliberate: the input is 256 random bits
with no dictionary to attack, so argon2 would buy nothing and charge tens of
milliseconds per request. That assumption is written down next to the code
that depends on it.

Two things the schema now says that it did not: budgets are integer
micro-units, because repeated addition of floats drifts; and an empty model
allowlist forbids everything while an absent one permits everything, which the
resolver has to keep distinct.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:37:40 +02:00
Ilker ManapandClaude Opus 5 41cf894c72 Tick the phase 1 and 1.5 boxes their releases already earned
The code shipped in v0.2.0 and v0.3.0 and both exit criteria were measured;
only the checkboxes were left behind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:27:03 +02:00
Ilker ManapandClaude Opus 5 f84e3b642a Record the llama.cpp work in the plan and changelog
CI / test (push) Successful in 58s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:19:55 +02:00
Ilker ManapandClaude Opus 5 f19db242ae Stop naming a metric llama.cpp does not have, and say so when it happens
Deployed against real llama.cpp, every scrape failed. The mapping named
llamacpp:kv_cache_usage_ratio; build b10991 serves fifteen metric families
and none is a cache ratio. Because KV usage was mandatory, that one absent
name threw away the two metrics the engine does publish, and the router
scheduled to all three replicas blind while the cluster looked healthy.

A mapping that leaves a metric unnamed now means the engine does not publish
it, and the field stays zero. Naming one the engine does not serve is still a
violation, so this does not become a way to tolerate a typo. Running and
waiting can never be omitted: an engine that reports neither what it is doing
nor what is queued cannot be scheduled against at all. For llama.cpp this
costs the KV half of the saturation signal, which is a real loss softened by
its fixed slot count — queue depth already says when it is full.

The second half is why it took a deployment to notice. A contract violation
was logged per scrape at debug level, so at the default level nothing said
anything. It is now a warning, once, with a matching line when it clears.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:18:27 +02:00
Ilker ManapandClaude Opus 5 501db3d3e8 Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same
OpenAI API on CPU, so the whole path — placement, supervision, scraping,
scheduling, admission — can be exercised end to end on ordinary machines.

Three things had to give way for it.

A profile may now omit its GPU class, and that is how it says it runs on the
CPU. Only engines that can actually serve without a card may omit it; for
vLLM an omitted class stays an error, because otherwise a typo becomes a
replica placed on a machine with no hardware and diagnosed minutes later
during model loading. Placement spreads CPU replicas instead of bin packing
them, since unlike GPUs there is no allocation to stop two of them sharing a
machine's cores.

Metric mappings are now per engine rather than per cluster. llama.cpp calls
the same three numbers requests_processing, requests_deferred and
kv_cache_usage_ratio, and a single global mapping would leave half a mixed
fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that
names its own Running metric is taken as a complete statement, so the fields
it leaves blank stay blank rather than being filled with names that engine
cannot have.

The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits
across local GPUs by a different mechanism than this adapter drives, and
accepting either would produce a replica that starts and then serves from one
machine while the catalog believes it has several.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:14:05 +02:00
Ilker ManapandClaude Opus 5 ec1b6b04df Fix an agent that could neither explain itself nor stop
CI / test (push) Successful in 54s
Two bugs found while verifying mTLS against real machines.

session cancelled its own derived context and then asked that context
whether the agent was shutting down. The answer was always yes, so every
stream error read as a clean close: the log said err=<nil>, the backoff
reset, and an agent whose certificate the control plane refused hammered it
forever at the base interval with nothing to diagnose. It now keeps the
parent context to answer that question, and a refusal reaches the operator
naming both the certificate and the claim.

Cancelling the context also did not stop the agent. HTTP/2 watches the
context while the request body is being written, and a bidirectional stream
spends its life blocked on the read side instead, so the read never returned.
Measured on a deployed node: `systemctl stop llmgw-agent` took 90 seconds and
ended in SIGKILL, on every restart. Closing the response when the context
ends is what actually unblocks it.

TestAgentStopsWhenContextEnds hangs reliably without the second fix, which is
the only reason to trust that it tests it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:03:21 +02:00
Ilker ManapandClaude Opus 5 1e3cafdc86 Add a private CA and mutual TLS for the agent stream
A node's identity was whatever the agent said it was. With mutual TLS the
control plane compares the client certificate's common name against the
claimed node id and refuses the connection when they disagree, so identity
becomes something a node proves rather than something it picks.

A public authority cannot supply these certificates: Let's Encrypt and its
peers stopped issuing the clientAuth usage, and Go verifies client chains
against it. Hence internal/tlsconf/ca.go, a small authority whose key lives
on one machine and whose root is trusted everywhere, and `llmgw ctl cert`
over it so issuing is reproducible instead of remembered openssl.

The server certificate's hosts matter more than they look: hostname
verification reads subject alternative names and ignores the common name
entirely, so a certificate without the address agents dial — bare IP
included — is rejected no matter what its subject says. There is a test for
exactly that, because it is the failure everyone hits once.

requireIdentity stays wired to false until the configuration lands, so this
commit changes no running behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 07:28:56 +02:00
ilkerandClaude Opus 5 e8d90cf375 Treat a zombie as dead, and test fairness where it is deterministic
CI / test (push) Successful in 52s
CI went red on Linux with two failures that never appear on macOS. Both were
worth the trip.

The first is a real bug. processAlive used kill(pid, 0), which succeeds on a
zombie — a process that has exited and is only waiting to be reaped. That
happens for real whenever an engine's parent dies before it does and nothing
reaps the orphan, which in a container is the normal case: PID 1 is usually the
workload rather than an init. The supervisor would then report a dead adopted
engine as running forever, and hold its GPUs against every future placement.
Zombies are now detected through procfs on Linux, where the answer exists, and
the check degrades to its old behaviour elsewhere rather than to something worse.

The second was a bad test. TestTenantFairnessWithinBand asserted on the order
goroutines sent to a channel after being admitted, which is the order the Go
scheduler woke them, not the order the dispatcher chose. Under load it measured
the runtime. The rotation itself lives in bandQueue.pop, which is pure, so it is
tested there — including what happens when a tenant runs out mid-rotation and
when a cancelled waiter is removed. The controller-level test now asserts the
thing that is actually stable: nobody is starved.

Verified with three consecutive `go test -race ./...` runs in a Linux container.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:34:08 +02:00