45e913bd10a57e304484b7ca72f2ddc4e9baf8e2
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5
|
21a502035f |
Document conversations, starting with what is not kept
CI / test (push) Successful in 1m35s
The section opens on the default rather than on the feature, because the default is the part someone deploying this needs to be sure of: nothing anyone types is stored until a project asks, and the command to ask is an act rather than a setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1b7e11b91e |
Document reclaiming disk, and what it deliberately will not reclaim
CI / test (push) Successful in 1m49s
A command that deletes data needs its limits written down more than its syntax. The section says plainly that llmgw only takes back files it fetched itself, and then says the uncomfortable half out loud: that this means it will not reclaim the storage most likely to be filling your disk, because weights you placed yourself are yours to remove. Also records the dash in the DISK column as "not asked" rather than "no space", which is the reading that matters when a node will not schedule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b58d67602d |
Stop the install guide contradicting itself about weights
CI / test (push) Successful in 1m41s
It said "there is no distribution mechanism yet" and then, two paragraphs later, documented the distribution mechanism. The known-gaps list — the part someone reads at 2am — still said "no cache, no fetch, no progress". Fetching landed in this cycle; both sentences were written before it and left standing. node-onboarding.md carried the same claim. What is actually still missing is narrower and worth naming precisely: a fetch reports no progress, so a first start on a large model looks like a hang, and every node fetches its own copy, so eight nodes on a new model are eight downloads from one origin. Also: the dashboard now shows who is using the cluster and what they are running, which the sentence describing it did not mention; and one sentence about idle timeouts had lost the words that made it a sentence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c232f41a66 |
Put the fleet view in the dashboard, so nothing else is needed to see it
CI / test (push) Successful in 1m32s
The metrics were exported and the only way to look at them was to stand up Prometheus beside the binary. For a project whose whole shape is one static binary with no dependencies, that is the wrong default: the common case is somebody wanting to know how full their cluster is, and that should not cost a monitoring stack. /admin now shows capacity per GPU class with a usage bar, free whole nodes per fabric — the number that decides whether a replica spanning nodes can land, since free cards on the wrong fabric are not capacity for a model that needs this one — and which replicas are holding hardware nobody is asking for. Nodes gain their memory and how many model files they can see. Taken from the same Summary that ctl status uses rather than derived in the page, because two implementations of the same arithmetic drift apart and the one on the screen is the one people would believe. A faulty card counts as neither used nor free, matching the scheduler. Counting it free is how a cluster looks like it has room it does not, and a page that disagreed with placement about that would be worse than no page. Prometheus stays supported and is now the answer to a narrower question: history, alerting, several clusters at once. The dashboard shows what is true now and keeps nothing, which is the honest division between them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1ea067d08e |
Give metrics a door that does not also open onto pprof
CI / test (push) Successful in 1m38s
The metrics existed and nothing could record them. They shared a listener with pprof, the management API and the dashboard, and pprof can dump the process heap where API keys sit in plaintext — so a fleet could be monitored or keep its heap private, and not both. That is not a choice anyone should have to make, and it is why every previous answer to "how do I scrape this" was a tunnel. metrics_listen serves metrics and health and nothing else, so it can be reached from wherever Prometheus lives. Metrics stay on the admin listener too: adding a door is not the same as moving one, and anything already scraping keeps working. While there: admin_listen defaulted to :9090, every interface, with a comment beside it saying not to expose it. A default that contradicts its own documentation is a footgun, so it is loopback now and someone who wants otherwise says so. The handler is nil until the registry is built, and says so rather than serving an empty page. Ordering has already caught this exact area once — a collector registered after AdminHandler published nothing at all, silently — and an empty scrape target looks like a working one. deploy/monitoring/prometheus.yml has a starting configuration and the alerts worth having: a node that has stopped reporting, a model that should be serving and is not, a card kept out of placement by a fault, and a fleet holding models nobody is asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
9b32531ce3 |
Stop an idle model to make room for one somebody asked for
CI / test (push) Successful in 1m29s
On-demand loading worked right up to the moment the cluster was full and then stopped meaning anything: the model somebody had just asked for could not be placed, and the cards were held by a model nobody had asked for in an hour. That is the only situation in which any of it matters. Deliberately narrow. Only idle replicas of other on-demand models — a model at min 1 is a promise that it stays up, and breaking it to serve a different request would make every guarantee in the catalog conditional on load. Only single-node replicas, since evicting a multi-node one frees several machines and costs tens of minutes to rebuild. One at a time, because freeing everything that qualifies would empty a cluster for a request that needed one node. And never a replica with no reported usage: after a router restart that is all of them, and unknown is not idle. Writing the test found a gap the feature had rather than a bug in the test. Two on-demand models on a cluster with room for one would take turns evicting each other, at a model load per cycle — worse for both than one of them waiting. A replica is now safe from eviction for a few minutes after it starts serving, which bounds that. Which in turn found a latent bug: deriveState stamped ReadyAt with time.Now() inside a package that has an injectable clock, so the new lifetime rule could not be tested at all — the fake clock said three minutes and the wall clock said eight months. Invisible while nothing compared ReadyAt to anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b619546516 |
Stop trusting hardware facts that were never checked
CI / test (push) Successful in 1m32s
Three places where the system took the infrastructure's word for something and would have failed silently on real hardware. A declared fabric adapter that is not on the node. Fabric membership has to be configuration — OpenSM's default subnet prefix is identical on every fabric, so a matching prefix proves nothing — but taking it entirely on trust was a different mistake. A typo, or a card that enumerated differently after a reboot, produced NCCL_IB_HCA naming an adapter that does not exist. An adapter with no ACTIVE port. Physically linked with no subnet manager sits in INIT: it looks healthy to anything counting adapters, carries no data, survives a reboot, and looks like a software problem. Discovery had recorded port state all along and placement only ever asked whether a fabric had any adapters listed. Both end the same way — the replica starts, passes its health checks, and hangs on its first all-reduce, which is the failure the fabric invariant exists to prevent arriving through a door nobody was watching. The agent now compares declared against discovered, where both are visible for the only time, and does not advertise a fabric that cannot carry traffic. The reason is logged on the machine that knows it, because the operator who typed the adapter name is the only person who can fix it. The fabric's rate is the slowest active link, since a collective runs at the speed of its slowest. And NVLink topology, which no published binary could see. NVML is its only source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the portable binary runs on an older libc. The cost was silent: tensor groups fall back to the lowest free indices, which is right on an SXM board with NVSwitch and wrong on a PCIe box where bridges pair specific cards — a group straddling two of them runs several times slower and nothing says so. Releases now also publish an nvml build for those nodes, and llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a fleet means the topology is unknown rather than absent. A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither had been passing, because nothing compared the two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
943c23a412 |
Fetch a model's weights when the simple arrangement needs it
CI / test (push) Successful in 1m26s
The default arrangement is the simple one: every node keeps its own copy. So a source naming a remote file is now fetched before the engine is launched, rather than left to fail as a missing path minutes into a load. Eight shards of a tensor-parallel replica start at the same instant and want the same file; they produce one download. An interrupted fetch resumes instead of starting over, which matters at forty gigabytes, and the download is written beside its target and renamed only when complete — a partial file under the model's name would be found by the next scan, reported as present, and handed to an engine that fails on it. The exotic arrangements need no code, which is the point. Weights on a Lustre mount or an image exported over the fabric are simply present, and present means nothing is fetched. The escape hatch from a distribution mechanism is a distribution mechanism that checks first. Wiring it up caught a regression in my own design. A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang resolve themselves, and treating every non-remote source as a local path made every existing vLLM profile fail. Only an absolute or explicitly relative path is now checked as a file; anything else is passed through untouched. Three agent tests failed on the same change for a better reason: they used "/models/x", and the new check is exactly the one that catches that. Whole repositories of shards are deliberately not fetched here. vLLM and SGLang do that well already, and a second snapshot downloader would be work spent arriving back where we started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ea4dd37f80 |
Document installing from nothing
CI / test (push) Successful in 1m24s
Written for whoever is standing up the cluster with root on some servers and nothing installed. Control plane, certificate authority, first node, first model, first request. It leads with what llmgw does not install — drivers, CUDA, engines, weights — because a node missing any of them registers, reports ready, and fails every replica placed on it, and the reason is three logs away. The useful order is to make a node able to serve before onboarding it, which is the opposite of what the tooling's convenience suggests. The failures it warns about are ones this cluster actually hit: llama-server present and unable to start for want of libgomp1 while the node reported healthy; a Proxmox VM on the default kvm64 CPU, where inference is six to nine times slower and nothing looks broken because every number is simply small; requests dying at exactly sixty seconds on a liveness timeout that is a generation budget in disguise for non-streaming calls. Verified rather than recalled. Every command it prints was checked against the binary, every config key against the struct tags, and its own verification and troubleshooting commands were run against the live cluster. That caught two errors in my draft: the health endpoint is /health and returns an empty body, not /healthz, and the Go version needed is 1.26. The known-gaps section is there so a reader finds them on a Tuesday afternoon rather than at 2am: weights and engines are theirs to distribute, there is one control plane and no HA, and multi-node replicas have never run on real InfiniBand — only against simulated hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
634f87ba15 |
Provision nodes with the host CPU model
CI / test (push) Successful in 1m26s
Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2 from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar baseline — instead of the backend built for the host's instruction set, and an inference node is not slightly slower but unusable. Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same thread count, the only change being --cpu host: Qwen2.5-Coder kvm64 host 1.5B q4_K_M 5.80 39.09 tok/s 3B q4_K_M 2.90 20.51 7B q4_K_M 1.33 9.88 Prompt processing went from 7.6 to 143.8, a factor of nineteen. This cost an afternoon because nothing looks broken. The node registers, the engine starts, the replies are correct; every number is just small, and the conclusion that suggests itself is that CPU inference is hopeless rather than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a second was nearly written off on that basis. The trade is live migration, which a VM pinned to the host CPU cannot do. For a node whose job is arithmetic that is the right way round, and --cpu overrides it where it is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
aa4da3264d |
Make adding a node one command, and check the engine actually runs
CI / test (push) Successful in 1m26s
Building node05 by hand turned up three things worth fixing, in the order they bit. Onboarding died on "Host key verification failed" — an error about ssh, in the middle of a script whose entire job is to talk to a machine created sixty seconds ago. It now uses accept-new, which learns an unknown host key and still refuses a changed one; the risk being guarded against is a substituted machine, not a new one. Then ssh started refusing connections on a node that had answered a moment before: cloud-init regenerates the host keys and restarts sshd partway through a new machine's first minute. The agent was registered and healthy throughout. Onboarding now waits for ssh rather than failing on it, which is also what makes running the two halves back to back safe. Then llama-server: present, executable, and unable to start for want of libgomp1. The node registered ready and would have stayed ready until a replica was placed on it and the shard died with a missing shared library three logs away. The docs already warned about this shape of failure; nothing checked for it. Onboarding now runs a configured engine and says plainly when it will not start. add-node.sh drives both halves. The reason is not keystrokes: the VM name, the hostname, the node id and the certificate's CN must all be the same string, and the address appears in three places. Split across two commands run minutes apart, that agreement is held together by whoever is typing. Here it is stated once, and the VMID defaults to the last octet of the address so qm list and ctl nodes stay in the same order. The two halves remain separately runnable, because needing the hypervisor and needing the CA really are different access. Verified by building a throwaway node06 with it: 45 seconds from nothing to registered, and the engine check caught that node06 had no llama-server. Destroyed afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
f68fbf7c49 |
Check llmgw's engine flags against real vLLM and llama.cpp
CI / test (push) Successful in 1m25s
The launch arguments had been read from documentation and never from an
engine. Now they have been:
vllm 0.29.0 --host --port --tensor-parallel-size --served-model-name
--distributed-executor-backend all accepted
llama.cpp b10991 --model --host --port --metrics --alias --parallel
--n-gpu-layers all accepted
Getting there found three faults in the checker rather than in llmgw.
It checked every command's flags against the engine's help, but a shard can be
several programs — vLLM under Ray is `ray start` and then `vllm serve` — and
they have different vocabularies. It reported --head and --num-gpus as
missing, which is true of vllm and irrelevant, because ray is what receives
them. Each command is now checked against its own help, and one that cannot be
run is reported as skipped rather than failed.
vLLM does not give up its whole help for --help: it prints a summary of
section names and expects --help=all. Checked against the summary, all five
flags looked missing. The warning about truncated help did not fire, because
the summary happens to mention --help= while listing how to ask for more.
And vLLM refuses to print anything at all on a machine with no accelerator
unless VLLM_TARGET_DEVICE says what to assume. Asking an engine about its
flags should not require the hardware it runs on.
Two things remain unchecked and are written down as such: the Ray sidecar,
because vLLM 0.29 does not install Ray, and SGLang, because it is not
installed anywhere. `launch_mode: mp` avoids Ray entirely and is worth
preferring until someone checks it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Ilker ManapandClaude Opus 5
|
5b425c32e5 |
Ship binaries with releases, and ask engines whether our flags still exist
CI / test (push) Successful in 1m31s
A release was a git tag. Installing llmgw on a node meant having a Go toolchain, and the onboarding script copied a binary from wherever it was run — which works and quietly makes whoever runs it the build server, with the fleet ending up on whatever was in their working tree. Tags now build static linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is the point: it has to run on a node whose libc is older than the builder's, and llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so this stays true. Verified: "not a dynamic executable", runs on a node, reports its version. `ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch arguments are read from each engine's documentation, and engines rename flags between versions; when that happens the failure arrives minutes into a model load, in the engine's own error message, on a node nobody was watching. It builds the widest command llmgw would ever produce — multi-node, since a flag that only appears there is the one nobody notices has gone — and checks each flag against the engine's own --help. No GPU: a CLI prints its help anywhere. Verified against llama.cpp b10991, all seven flags accepted. Flags are matched on a word boundary, so --port is not satisfied by --port-range. Without that, a renamed flag would sail through the check that exists to catch renamed flags. It also warns when the help text looks truncated, which is how the first run went: llama.cpp's help cut to 400 of its 725 lines reported five flags as missing that the running cluster uses. TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky — its wait loop fell through on timeout, so a slow machine turned the ordering assertion into a no-op and the failure surfaced later as something else. It now asserts, with a deadline that suits a machine running every other package at the same time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7b4acd9284 |
A replica missing a shard is not ready
CI / test (push) Successful in 1m23s
Restarting the control plane over a running two-node replica adopted half of it. Every shard it still had was healthy, so it was reported Ready — with one shard of two and no endpoint at all, because the missing one was rank 0. The router was handed a replica nothing could route to. The cause is that a replica had no idea how many shards it should have. It could tell "every shard I have is ready" and nothing else, which is the same sentence whether or not half of it is gone. It now carries the count from its profile, set at placement and again at adoption, and a replica short of shards fails rather than pretending. Onboarding also wrote no engine paths, so a node onboarded by the script could not find llama-server and failed every shard placed on it with "executable file not found". The script now takes them, and says so when none are given — a node with no engine paths can run the sim engine and nothing else. The cluster now runs both at once: two-node simulated replicas on each fabric, and real llama.cpp inference on the CPU, which is a better test bed than either alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c97b671ef9 |
Prove the multi-node path on a simulated cluster
CI / test (push) Successful in 1m29s
Phase 4 had been written and never run. The hardware is not coming, so the cluster is simulated instead — four nodes across two fabrics, eight GPUs each, built with the onboarding scripts from the previous commit. `sim` is now an engine adapter, not just a standalone command. Its leader binds a rendezvous port and does not serve until every worker has joined; its workers serve nothing at all. That is the part stub processes could never exercise, and it is why this path survived four phases untested: the command line was checked, the sequence was not. Measured on the cluster: - a 2-node replica on each fabric, node01+node02 on ib-a and node03+node04 on ib-b, four GPUs from each - a request served through one of them - only the leader listening — node02 had no port open at all, which is ServesHTTP working end to end rather than in a unit test - node02 killed; at t+60s it was declared lost, the ib-a replica failed and was reaped; at t+80s a replacement was ready on ib-b, the fabric that could still host it, while node01 sat idle rather than being paired across fabrics The invariant held under the one condition that tests it: when a replica could not be rebuilt where it was, it moved to a fabric that could hold all of it rather than spanning two. What this does not prove is written into the plan next to what it does. The GPUs are an inventory, the fabric is a sysfs-shaped directory, and the engine speaks no NCCL — so the launch sequence is now exercised while the launch arguments for vLLM and SGLang are still read from documentation rather than from a running engine. docs/node-onboarding.md is the procedure, including how to run all of this on machines that have neither GPUs nor InfiniBand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
de21649ad6 |
Add an in-browser chat, without putting a key in the browser
CI / test (push) Successful in 1m13s
So someone can try a model without setting up a client, and — more often — tell whether a problem is the gateway or their code. The page carries a session, and the server turns that into the identity an API key would have produced, resolved from the person's project. The request then goes down the ordinary path by being handed to the same handler /v1 uses: admission, model allowlist, rate limits, budget, accounting. There is exactly one request path, so there is no second place for its rules to drift — and without that, a playground is a hole in every limit the rest of the gateway enforces. The allowlist test posts a forbidden model and expects 403 for that reason. The body is rebuilt rather than forwarded, so the page cannot set fields it has no business setting: a stream:false that would hang the reader, a max_tokens large enough to occupy a replica for an hour. One turn is capped at 512 tokens. The picker offers what is actually served and allowed, not what is catalogued, because offering a model with no replica produces a 503 the reader cannot interpret. Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG. It streams, because on CPU inference ten seconds is normal and a chat that hangs and then dumps a paragraph feels broken. Replies are written with textContent throughout: model output and user input have no business being parsed as markup. Its POSTs carry the CSRF token in a header, which a cross-site form cannot set at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
859f08da9b |
Approve accounts from the browser, and draw the traffic
CI / test (push) Successful in 1m19s
Two things the dashboard was missing: it could not act, and it was a column of numbers. Account administration moves to /admin/users — approving into a project, suspending, reinstating. That is the work an operator does constantly and it should not need a terminal. Three things stay on the CLI, and the line is not arbitrary: promoting someone to admin, resetting a password, and issuing a key on another person's behalf each turn a stolen admin session into more than it was — permanent access, or somebody else's account. Approving and suspending do not. Suspending signs the account out everywhere, or suspension is advisory until the session happens to expire, and an admin cannot suspend themselves, which is not a thing an interface should let you do by misclicking. The charts are inline SVG generated in Go. A charting library would mean a JavaScript toolchain in CI and a megabyte of bundle for what is a list of rectangles, and the single-binary goal rules that out. Hourly over a day: fine enough to show a working day, coarse enough that one slow minute is not an incident. Empty hours are filled in, because a chart with hours missing where nothing happened reads as a chart with hours missing from the data. The SVG is marked as trusted HTML, so everything that reaches it is either a number this package computed or escaped explicitly, and a bar's colour can only be one of four names — a colour that could be anything is a way to write CSS into every page the chart appears on. There are tests for both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e519dc587c |
Serve the cluster dashboard over the web, without serving pprof with it
CI / test (push) Successful in 1m21s
Reaching it meant an SSH tunnel, because the page lived on the admin listener and that listener is bound to loopback. Opening the port was never the way to fix that: it also serves pprof, and pprof can dump the process heap, where every API key sits in plaintext. Exposing the port would have been exposing the key store. So the page moves rather than the port. The same dashboard is now mounted at /admin on the client-facing port, behind the session and CSRF machinery that already guards /profile, gated on a flag on the account. The admin listener keeps pprof and metrics and stays on loopback. One bit, not a role system: it grants looking. Everything that changes the cluster or issues a credential stays on the CLI, where it needs a shell rather than a session — which keeps the worst case of a stolen admin cookie to "saw the cluster" rather than "drained your fleet". Someone without the flag gets 404, not 403. A 403 confirms the page is there and that admin accounts exist here, which is free reconnaissance; 404 is what every other unknown path returns. The profile page does not show a link to a door the visitor cannot open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
79e459d9ff |
Let people change their own password, and operators reset a lost one
CI / test (push) Failing after 1m11s
The account system shipped without either, which made the first forgotten password an unanswerable support request. Changing it requires the current one. A session is what an attacker gets from a borrowed laptop or a stolen cookie; the password is what they do not have, and without the check the first thing they would do is take the account. Every session is closed, including the browser doing the changing — leaving the others open would mean a password change does nothing about the reason someone is changing it. A new session is then opened for the browser in front of us, so the person who just proved their password is not thrown back to the sign-in page. That response also carries a new CSRF cookie, so the forms it renders take the new token rather than the one the request arrived with; there is a test that posts a form from that exact page. Guessing the current password is bounded the same way sign-in is, because someone holding a stolen session should not get unlimited attempts at the password that would let them keep it. There is no self-service reset: this gateway sends no email, and a reset link with nowhere to send it is not a reset. `llmgw users reset-password` is the operator's answer, and it closes their sessions too — if someone lost their password, the reason might be that somebody else has it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
216fc9bb03 |
Let people sign up, be approved, and issue their own keys
CI / test (push) Successful in 1m9s
Every key needed an operator with a shell. This adds the login system that made that necessary, which is the thing I argued against building until it was asked for — so it is built as if it will be attacked, because a public login form is. Signing up produces an account that can do nothing. Approval is what makes it work and stays on the CLI: having a shell on the control plane is what makes someone an administrator, and that is a smaller thing to get right than roles and an admin UI that mints credentials. The details that matter: Passwords are argon2id, where a slow hash is correct — the input is chosen by a human and has a dictionary, which is the opposite of an API key. Both kinds of authentication now live in this codebase and they are in separate packages so neither set of assumptions leaks into the other. Length is the only rule: composition rules push people towards Password1! and cost more entropy than they buy. Signup never says whether an address already has an account, and sign-in answers the same way for a wrong password and a missing account — verifying against a dummy hash when there is no user, so a stopwatch cannot enumerate accounts. Account state is checked after the password, not before, so guessing an address reveals nothing about it. Guessing is bounded per address rather than per IP: an attacker has many addresses to come from and one to guess against, and limiting by IP locks out an office behind one NAT. Sessions are server-side, so signing out and suspending take effect immediately rather than whenever a token expires; suspension is checked in the same query that resolves the session, because a second round trip is a check somebody eventually forgets. Every state-changing form carries a CSRF token verified against a hash in the session — a cookie authenticates the browser, not the page that asked. Key ownership is enforced in the UPDATE rather than by reading first, so a guessed key id from another account changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
8d35faf6f8 |
Give the landing page somewhere to send people
CI / test (push) Successful in 56s
The page explained the API, said keys come from an operator, and left the reader there. Someone who reaches the address without a key had no route to getting one, which is where onboarding stopped. `landing.contact` is that route, because the gateway cannot guess who to ask. An address becomes a mailto link, an http(s) URL stays a link, and anything else — a channel, a queue, a person — is shown as text. Other schemes are never linked: this string comes from configuration, configuration is not always written by whoever runs the gateway, and a javascript: href on a page served to every visitor is a short path to a stored XSS. Escaping the text does not make a scheme safe. Unconfigured, it now says something rather than nothing, and the page explains that a key arrives as a one-time link so the reader knows what to expect before it lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3ed211b2f4 |
Onboard users with one-time links instead of pasted keys
CI / test (push) Successful in 2m43s
Issuing a key worked; getting it to the person it was issued for did not. The operator had a secret on their terminal and no way to send it that did not leave a bearer credential in a chat log forever. `keys create --claim` prints a link instead. GET shows a button and consumes nothing; POST reveals the key and burns the link. That split is not politeness — chat clients, mail scanners and link previewers fetch URLs the moment they are posted, so a GET that consumed the claim would mean the first thing to unfurl the link destroys the key before anyone reads it. A pending claim has to hold the key somewhere, which would otherwise put a working credential in the database and undo what hashing bought. So the key is sealed with a key derived from the link's own token, and the token is never stored — only its hash. A database leak yields two useless hashes and a blob nothing in the database can open. HKDF rather than the raw token, so the value used for lookup can never become the value that decrypts. Consuming is one UPDATE moving claimed_at off NULL, so two people opening the same link produce one key rather than two copies. A used link, an expired one and a token that was never real all render identically, because saying which confirms that a token was once valid. The page is no-store, no-referrer and noindex: the URL is itself the credential. `GET /me` reports a key's own limits and spend, so nobody has to ask an operator to read a dashboard they cannot reach. Outside /v1, because it is not part of the OpenAI API. docs/onboarding.md is written for whoever operates this, including the parts that are deliberately absent — no self-service, no SSO, and why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |