Commit Graph
41 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 f13b18149a Refuse weights the nodes' llama.cpp cannot load, before fetching them
kev-0.5b declared architecture "ggmlc", which llama.cpp does not know.
It was fetched, catalogued and placed, and failed on every node — while
the answer was in the first kilobytes of the file.

internal/gguf reads a GGUF header (v1-v3) up to general.architecture,
over HTTP with a Range request, and judges it against the architectures
llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at
that tag); a vision projector is not a model on its own. An unknown
build is not refused, only not vouched for, and a header that cannot be
fetched does not refuse anything.

Checked in three places: the registry listing (from the architecture
Hugging Face reports with expand[]=gguf, so one request per repository)
shows it and withholds the fetch button; a fetch reads the header and
refuses unless "fetch anyway"; and the model service refuses to add
such a model, which covers the serve form and ctl model add.

Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3
(bert), and Qwen2.5-0.5B on Hugging Face through its redirect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:44:36 +02:00
Ilker ManapandClaude Opus 5.5 bf94cd1667 Set how a model runs from the mirror page
Every mirrored version was in use on the cluster, so the serve link — hidden for a version already served — appeared nowhere, and changing a model between always on and on demand needed the CLI. Each model using a file now shows how it runs and offers always on, on demand (15 minutes idle) or stopped, through the model service's Scale; models declared in server.yaml show their mode and where to change it. The serve form asks the same question.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:34:41 +02:00
Ilker ManapandClaude Opus 5.5 e493b2f53d Give an organization's managers a page of their own
A member of an organization can be its manager (users join --org X
--role manager, or the star on the people page). /org shows a manager
their organization: usage over time by team, project or person, its
teams and projects with limits and 30-day spend, and its members, each
once with every place they belong. Read-only; administrators see every
organization. Only what was spent in the organization's projects is
shown — UsageBucketsIn filters by where a request was spent.

Found on the way:
- Usage by org, team and project, and the quotas page's spend, joined
  through api_keys and so dropped every chat-page request, which has no
  key row. They now read usage.project_id; keys no longer move between
  projects, so the two agree.
- /org was not exempt from API-key auth and answered browsers with a
  JSON error. A router test now drives the session pages through the
  middleware; it fails without the fix.

Membership queries, org-scoped usage and the reworked joins pass the
PostgreSQL store tests on 16.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 19:23:12 +02:00
Ilker ManapandClaude Opus 5.5 098886e0e4 Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:

- memberships: a person is a member of any number of organizations,
  teams and projects, or of none; a member of a team or organization
  may use every project in it. users join/leave, and the people page
  adds and takes out from either side. Approval no longer needs a
  project.
- personal keys: a key may have an owner and no project. /profile asks
  what a new key spends against; keys create --for does the same.
  auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
  project and the chat page; --per-member on an org, team or project
  gives each member their own share there. Resolution folds the tree,
  each per-member limit for the key's owner, and the owner's own, each
  in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
  personal use and chat included; restore no longer drops rows with no
  project.

Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.

Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.

users move is gone; join and leave replace it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 19:03:46 +02:00
Ilker ManapandClaude Opus 5.5 429c05a319 Show on the quotas page the limits that actually apply
A branch with no limit of its own read "—" in every column while its
organization held it to sixty requests a minute. Each cell now shows
the strictest value from above, muted and marked with an arrow, naming
its source on hover and in the form's placeholders. A larger value set
under a stricter one says what caps it, for rate and token limits as
well as budgets; only budgets were checked before.

The table follows the people page: indented tree, tinted organizations,
spend with a meter that turns red past 90%, a form that opens and saves
without reloading (htmx, falling back to plain links), and the
explanations folded under "How limits combine".

Rows are built from the store's Tree, so a project sits under its team
by id. Matching the start of its path put projects under the wrong team
when two organizations shared a name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 18:19:49 +02:00
Ilker ManapandClaude Opus 5.5 c9cd25efc2 Draw one menu on every page
Each page wrote its own nav: the same links in a different order, some
missing, the current page sometimes listed and sometimes not. A shared
"nav" template now draws it everywhere — Cluster, People, Quotas, Usage,
Mirror for an administrator, then Chat, Your keys and Sign out — with
the current page marked by aria-current. Pages rendered outside the
account pages (the dashboard, node logs) get it through the request
context their admin wrapper sets. The admin listener's own page keeps
its metrics/pprof/api links.

The chat and key pages said "project proj-3kq7…"; they now show the
project's path, and the key page no longer repeats the email.

A test renders several admin pages and requires the same menu on each.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 17:55:36 +02:00
Ilker ManapandClaude Opus 5.5 c22c3a309b Describe v0.9.2
CI / test (push) Successful in 2m14s
Release / build (push) Successful in 3m1s
Release / notes (push) Successful in 3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:50:12 +02:00
Ilker ManapandClaude Opus 5.5 42fa602247 Make a new control plane serve a model by itself
server init now also:
- runs a model mirror beside the control plane (/srv/models, bytes on
  :8081 at an address nodes reach, management API on loopback so it
  needs no certificate) and adds it to server.yaml;
- makes the machine a node of its own control plane: issues its agent
  certificate, writes agent.yaml labelled as the control plane, starts
  llmgw-agent, and waits for it to register and install llama.cpp;
- checks for NVIDIA GPUs with nvidia-smi, and reports a card on the PCI
  bus whose driver does not answer rather than ignoring it;
- fetches Qwen2.5 0.5B (Q4_K_M) into the mirror and adds it to the
  catalog pinned to this node: a llama.cpp profile per GPU class found,
  with a CPU profile as the fallback, then waits for it to serve.

--no-mirror, --no-node and --no-model leave these out. Every step looks
before it acts, so a rerun finishes what an interrupted one started.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:49:55 +02:00
Ilker ManapandClaude Opus 5.5 5501f77805 Describe v0.9.1
CI / test (push) Successful in 2m22s
Release / build (push) Successful in 3m1s
Release / notes (push) Successful in 2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:37:43 +02:00
Ilker ManapandClaude Opus 5.5 75d69a6b9c Show the mirror one line per model version
Each held file took two table rows — seven columns and its address
beneath — so one sharded model filled the screen. A version is now one
line: model and version, the file or "N files", total size, and the
catalog models that use it. The address is behind a "copy address"
button (a set's is its directory, trailing slash included) and the
checksum is in the file's tooltip.

The model service reports which models name each source, so a file in
use shows who uses it and is not offered for serving a second time;
removing one warns that the catalog uses it. Downloads and failures
stay one line per file, since each has its own progress or retry. The
header counts versions, not files, and the nav calls /admin/users
"people" everywhere.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:36:36 +02:00
Ilker ManapandClaude Opus 5.5 beae60a76e Describe v0.9.0
CI / test (push) Successful in 2m22s
Release / build (push) Successful in 3m1s
Release / notes (push) Successful in 2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:27:54 +02:00
Ilker ManapandClaude Opus 5.5 3104f4b706 Manage people and the tenancy tree, from the web and the CLI
Nothing in the tree could be renamed, moved or removed, and an approved
person stayed in the project they were first put in — from the web or
the CLI. Now:

- /admin/users is the tree, each project opening onto its people and
  keys, with approve, move, suspend, reinstate and remove for people,
  revoke for keys, and add, rename, move and remove at every level,
  plus a search over every account. Forms post and get the page back,
  so it works with no script; htmx 2.0.11 (0BSD, embedded, checked
  against npm's integrity hash) swaps only the changed part, keeping
  open projects open.
- llmgw org tree/add/rename/move/remove, users move/delete, and
  keys list --all.
- Projects, teams and organizations are named by id, path
  (Acme/platform/gateway) or unique name; users by id or email. With a
  single project --project can be left out, instead of defaulting to
  "default", which a tree built from the web does not have.

Moving a person moves the keys they made; operator keys stay with the
project. Only empty things are removed, and never a project that had a
key, since usage history refers to it.

Also: keys init no longer reparents an existing team or project;
reinstating an account with no project is refused (a rejected signup
used to come back active and fail at its first key); rejecting a
signup removes it; keys limits --budget takes whole units like the
quotas page.

Checked in headless Chrome: a move from an open project updates the
page in place, keeps the project open, and the search still filters.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:26:39 +02:00
Ilker ManapandClaude Opus 5.5 5141e4b138 Describe v0.8.8
CI / test (push) Successful in 2m26s
Release / build (push) Successful in 3m1s
Release / notes (push) Successful in 3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 14:47:55 +02:00
Ilker ManapandClaude Opus 5.5 2e0465f10a Describe v0.8.7
CI / test (push) Successful in 2m21s
Release / build (push) Successful in 3m1s
Release / notes (push) Successful in 3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 14:35:28 +02:00
Ilker ManapandClaude Opus 5.5 40c3d83110 Describe v0.8.6
CI / test (push) Successful in 2m8s
Release / build (push) Successful in 2m56s
Release / notes (push) Successful in 2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:55:46 +02:00
Ilker ManapandClaude Opus 5.5 4865a96ec2 Describe v0.8.5
Release / build (push) Successful in 2m57s
CI / test (push) Successful in 2m14s
Release / notes (push) Successful in 3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:44:32 +02:00
Ilker ManapandClaude Opus 5.5 746d4852af Describe v0.8.4
CI / test (push) Failing after 2m6s
Release / build (push) Successful in 2m56s
Release / notes (push) Successful in 2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:27:47 +02:00
Ilker ManapandClaude Opus 5.5 a821a05840 Describe v0.8.3
Release / build (push) Successful in 2m56s
CI / test (push) Successful in 2m13s
Release / notes (push) Successful in 3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:14:22 +02:00
Ilker ManapandClaude Opus 5.5 8097a5616d Describe v0.8.2
CI / test (push) Successful in 2m11s
Release / build (push) Successful in 2m56s
Release / notes (push) Successful in 3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 11:50:12 +02:00
Ilker ManapandClaude Opus 5.5 f90daa1ee9 Describe v0.8.1
Release / build (push) Successful in 2m57s
CI / test (push) Successful in 2m21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:22:23 +02:00
Ilker ManapandClaude Opus 5 3a4260c8c5 Cut v0.8.0
CI / test (push) Successful in 2m6s
Release / build (push) Successful in 2m56s
The store stops being a file on one machine, which is what stood between this
and more than one control plane.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 07:26:54 +02:00
Ilker ManapandClaude Opus 5 b9c2be6cc2 Record what failover actually did
CI / test (push) Successful in 1m36s
Three seconds to move leadership, six for every agent to arrive, and every
replica adopted rather than restarted. Also what it did not do: the public
address returned 502 while the standby served correctly on its own port,
because nothing in front of the two moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 19:39:55 +02:00
Ilker ManapandClaude Opus 5 fa60b81f37 Document moving to PostgreSQL, and when it is worth it
CI / test (push) Successful in 1m36s
The section leads with why SQLite is still the default rather than with how to
leave it: one file, no server, and a request path that barely touches it. Two
things change that, and both are recent — conversations and notes read and
write on every turn, and a second control plane is impossible while the store
is a file on one machine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 16:29:42 +02:00
Ilker ManapandClaude Opus 5 7dd1de6537 Cut v0.7.0
Release / build (push) Successful in 2m55s
CI / test (push) Successful in 3m19s
Seventeen commits since v0.6.0, and not a phase of the roadmap — which is why
the table leaves the phase column empty rather than inventing one. What landed
is what a cluster needs to be lived with rather than demonstrated: somewhere
to keep a conversation, somewhere to keep a note, and an honest answer about
the disk underneath both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 14:20:43 +02:00
deniz 2eb79f3400 agenta memory eklendi
CI / test (pull_request) Successful in 2m31s
2026-09-24 04:04:03 +03:00
Ilker ManapandClaude Opus 5 21a502035f Document conversations, starting with what is not kept
CI / test (push) Successful in 1m35s
The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:58:53 +02:00
Ilker ManapandClaude Opus 5 1b7e11b91e Document reclaiming disk, and what it deliberately will not reclaim
CI / test (push) Successful in 1m49s
A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.

Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 09:59:16 +02:00
Ilker ManapandClaude Opus 5 a2b3f63a0d Say what v0.6.0 is, and let the release actually publish it
CI / test (push) Successful in 1m56s
Release / build (push) Failing after 2m31s
The changelog had two "Unreleased" headings. The second sat below v0.5.0 and
was the detail behind it — phases 2 and 3, written while they were in progress
and never folded in when the release was cut — so it claimed as unreleased
work that had been shipped for two days. It is now a section of v0.5.0. The
status table said v0.5.0 was planned, which its own release notes disprove.

The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the
headline and the rest is what a cluster needs before anyone but its operator
is let near it.

Two things would have stopped the release workflow publishing anything:

Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on
the instance's default mode, and under the restricted default it is read-only
on releases. The job now asks for `releases: write`, which is correct under
either default. An upload that is refused now fails the job — without `-f`,
curl treats a 403 body as a successful transfer, so the release would end up
with some of its files, or none, and a green tick either way.

`workflow_dispatch` is offered but the upload assumed a tag. Fired from a
branch it would have created a release named "main".

README's status paragraph still said authentication, budgets and the control
plane were missing. It now says what is actually left, including the part
worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on
a machine that has any of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 00:29:15 +02:00
Ilker ManapandClaude Opus 5 f84e3b642a Record the llama.cpp work in the plan and changelog
CI / test (push) Successful in 58s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:19:55 +02:00
ilkerandClaude Opus 5 e1eb1d4149 Phase 3: ctl, and make draining a rolling move
CI / test (push) Failing after 50s
ctl talks to a management API on the admin listener, not the client-facing one.
The two have different audiences: one serves inference to whoever holds an API
key, the other lets a caller cordon nodes and read the cluster's shape.

- nodes surfaces unhealthy GPUs and why they are excluded. That is usually the
  answer when a node looks idle but nothing will schedule onto it, and it was
  invisible before.
- explain-placement prints every candidate considered and why it was rejected.
- replicas shows the shards a replica is made of, with per-shard errors.
- cordon, uncordon and drain are one endpoint with different targets, because
  the difference between cordon and drain is exactly what an operator needs to
  be sure of before running either.

Draining had to be fixed first to make ctl drain worth having. A drained replica
was stopped on the next pass, which cuts in-flight requests — the thing draining
exists to avoid. Now a node marked Draining has its replicas drained, the model
falls below its desired count, a replacement is placed elsewhere, and the
original is stopped only once the replacement is serving. A grace period bounds
it, because draining the only node that can host a model would otherwise never
finish and the node could never be taken out of service.

Verified against a two-node cluster of real binaries: draining moved a replica
to the surviving node and traffic kept returning 200 throughout.

ctl logs is deliberately absent. It needs a request/response pair the agent
protocol does not have, and the failure path already ships an engine log tail
with every status report — so the case that matters most is covered, and
reading a healthy shard's output on demand is not worth a protocol change yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:27:21 +02:00
ilkerandClaude Opus 5 1d1a6aebb0 Phase 4: make a multi-node replica able to start
CI / test (push) Failing after 53s
Placement and environment generation had landed early, which made multi-node
look further along than it was. A replica spanning nodes could not come up at
all, and all three reasons came down to assumptions that only hold on one node.

Not every rank serves HTTP. The agent health-checked every shard at its own
port, but only the leader answers; workers timed out, reported Failed and took
the replica with them. Engine.ServesHTTP now says which ranks serve, and a
worker is judged by staying alive. That is deliberately weaker evidence: the
replica's real readiness is the leader answering, which only happens once every
worker has joined.

A shard is not always one process. Engine.Commands returns a sequence — every
command but the last is a sidecar kept alive for the shard's lifetime and torn
down with it, and the last one's exit is the shard's. vLLM under Ray is why: the
leader runs a Ray head and then the server against it, while a worker runs only
the join and no vLLM process at all. RayCommand had been written and tested but
never called, so launch_mode: ray would have run vllm serve against a cluster
that did not exist. Sidecars are killed with the shard, because a Ray head left
behind holds the port the next replica needs.

Start order was backwards. Workers went first, on the theory that the head
should wait for them. Both engines put the rendezvous point on rank 0, so a
worker started first has nothing to connect to. vLLM's server blocks until the
Ray cluster has the GPUs it needs, so the leader can start immediately and waits
on its own. A command can now also require that rendezvous point to be listening
before the next starts, bounded by a timeout so a head that never appears fails
the shard instead of hanging it.

Stop was also killing only the engine, leaving sidecars behind.

None of this is verified against real engines — it is tested with stub
processes. The Ray sequencing, and which launch mode a given vLLM version wants,
can only be confirmed on hardware.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:20:03 +02:00
ilkerandClaude Opus 5 75a8e76706 Survive a control plane restart without orphaning live engines
CI / test (push) Successful in 1m44s
Not persisting replica state left one hole: restarting the control plane made it
forget every replica while the engines kept running. They held GPUs the
inventory believed were free, so the next placement failed to allocate, with an
error naming the new process rather than the old one.

The fix belongs on the agent, not in a database. A row can say "running" about a
process that died; the fleet cannot.

- the supervisor writes the full shard spec, not just a pid and command line. A
  pid identifies a process; the spec is what lets it be described back to the
  control plane and matched to a replica.
- Adopt takes over an engine a previous agent left running. It is not this
  process's child, so it cannot be waited on: it is watched by polling and
  stopped by signalling its group directly. Its earlier output is gone and the
  log says so rather than presenting emptiness as silence.
- every Register reports what the agent is running, not only the first, because
  a reconnect may be a control plane restart and has to recover the same way.
- the reconciler rebuilds replicas from those reports and reclaims their GPUs.
  Anything whose model or profile left the catalog is stopped; nothing will
  route to it and it is holding cards.
- an engine the agent cannot identify is killed. A process holding GPUs that
  nothing can describe is worse than no process, because placement hands those
  GPUs out again.

Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they
do not have, and reported them stopped without killing anything.

TestAgentReconnects had been passing for no reason. httptest's
CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c
hijacks the connection and httptest stops tracking hijacked ones — so the agent
never reconnected, and the assertion on the connection map passed on a stale
entry. Tests now sever connections through their own listener and count opened
streams.

Plan §11 also records what multi-node still needs, since placement landing early
makes it look further along than it is: workers never become Ready because the
agent health-checks every shard while only rank 0 serves HTTP; RayCommand is
written and tested but never called; and startup is fire-and-forget.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:15:04 +02:00
ilkerandClaude Opus 5 2373726efd Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.

- registration carries the whole hardware inventory, so there is no separate
  discovery call and a reconnecting agent refreshes what the scheduler knows.
  The fabric labels travel with it; they cannot be inferred from the wire, so
  losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
  it holds the open streams and is therefore the only thing that can deliver a
  command to a node.
- the agent validates a profile before launching anything, since an engine that
  refuses a profile does so after minutes of loading, then supervises the
  process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
  require touching every GPU node, and a fleet must not reconnect in lockstep
  and knock it over again. Engines keep serving throughout, because the router
  talks to them directly.

llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.

Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.

config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.

Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:48:21 +02:00
ilkerandClaude Opus 5 c1b9ea1bc2 Phase 3 (3/3): wire the control plane to the router
CI / test (push) Successful in 50s
Ready replicas become router endpoints with no static configuration involved.
Endpoints that survive an update keep their live state — inflight counts,
breaker state, drain rate — so a replica that was already serving does not start
from cold because another one moved. The sink is an interface, so when the
router and control plane split into separate processes it becomes a watch stream
and nothing above it changes.

Phase 3's exit criterion now holds end to end. A node hosting a live replica is
killed while four clients stream continuously; the control plane marks it
unreachable, then lost, rebuilds the replica on a survivor and republishes.
Across three runs ~370 requests completed with zero failures and no replica was
left on the dead node. This proves the control logic, not the transport: the
agent fleet is simulated and the real path still needs the stream.

The race detector caught a genuine bug while this went in. startReplica read the
replica's fields after publishing it to the map, and scaleUp walked replica
pointers outside the lock; both raced with agent status reports, which arrive
from another goroutine and mutate exactly those fields. Against the simulated
fleet the reports came back fast enough to hit the window on every run. State is
snapshotted under the lock now and those functions work from values.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:16:35 +02:00
ilkerandClaude Opus 5 f0134a2f95 Phase 3 (2/3): catalog and reconciler
CI / test (push) Successful in 24s
The desired-versus-actual loop, with a Dispatcher interface where the agent
stream will go. The loop's logic is where the interesting failures live, not the
transport, so it is tested against a fake dispatcher and a synthetic clock.

- catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and
  fallbacks resolve at load rather than at request time, because a typo in a
  fallback should stop a deployment instead of surfacing during the outage when
  the fallback is finally needed. The node/GPU/TP arithmetic is checked here
  too, so a bad profile fails at config load rather than minutes into loading.
- reconciler: one replica placed per pass, since placement depends on what is
  already allocated and several from one snapshot would be handed the same GPUs.
  Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray
  needs its workers attached before the head serves.
- a failed shard fails the whole replica. The others are waiting on collectives
  that will never complete while holding GPUs, so every shard is stopped and the
  replica rebuilt, with a jittered exponential backoff carried on the model
  rather than on the replica that was just destroyed.
- node expiry has two stages, Unreachable then Lost, deliberately far apart: a
  network blip must not tear down replicas that are still serving, and
  rebuilding a multi-node replica costs tens of minutes of model loading.
- scale-down drains before stopping, so in-flight requests finish.

The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the
reconciliation half end to end. The router continuing to serve throughout needs
the EndpointSource wiring and the agent stream.

Writing the tests caught a real property of the design worth recording: advancing
the clock makes every node look silent, so placement correctly stops. The test
harness now heartbeats nodes as it advances time, except where a node going
silent is the point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:12:06 +02:00
ilkerandClaude Opus 5 06672d1b85 Phase 3 (1/3): cluster inventory and placement
CI / test (push) Successful in 23s
The two hard constraints from plan §1.1 are enforced structurally rather than
checked after the fact: there is no code path that can combine nodes from two
fabrics, and only GPUs of the profile's class are ever considered. Getting
either wrong produces a replica that starts, passes health checks, and then
hangs or crawls under load — which looks like success until traffic arrives.

- inventory: nodes, hardware, fabric attachments, allocations. A re-registering
  agent cannot clear an operator's cordon, since a cordon is usually in place
  because something is wrong with the node. A GPU with an ECC fault or another
  process on it is not free in any useful sense. Replicas are released whole,
  because they fail whole.
- placement: highest-priority profile that fits, with every rejected candidate
  explained. "Unschedulable" on its own tells an operator nothing about which
  of six constraints bit.
- bin packing fills partly used nodes first so whole nodes stay free for large
  models that cannot be split; multi-node replicas get the emptiest machines
  instead, being the least movable thing in the cluster.
- NVLink-aware GPU selection keeps a tensor-parallel group inside one domain
  where the topology is known, falling back to lowest free indices otherwise.

Property-based tests generate 400 random clusters and assert the invariants on
every placement produced: one fabric per multi-node replica, only that fabric's
HCAs per shard, one GPU class, no unhealthy or already-allocated GPU. The
table-driven tests cover the cases someone thought of; these cover the rest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:07:29 +02:00
ilkerandClaude Opus 5 55417546fc Phase 2 (2/2): NVML backend behind a build tag
CI / test (push) Successful in 1m37s
NVML is the only source of NVLink topology, and that topology is what keeps a
tensor-parallel group off the slow path inside a node. It needs cgo, so it sits
behind the nvml tag and the default build keeps using nvidia-smi.

Discover() picks the backend and falls back to nvidia-smi when NVML fails,
logging the fallback rather than performing it silently: on a node that is meant
to have working NVML, a quiet downgrade hides a broken driver installation and
costs the scheduler its topology.

CI now compiles the tagged build. That immediately paid for itself: ErrNoNVML
was declared only in the untagged file, so the tests did not compile under
-tags nvml. It is now declared where both builds can see it.

Not verified: the NVML path has never run against a driver, because there is no
GPU machine available. The NVLink domains it reports should be treated as
unverified until someone runs `llmgw agent inventory` on a real node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:05:25 +02:00
ilkerandClaude Opus 5 6a13e3dddf Phase 2 (1/2): node discovery, engine adapters, process supervisor
CI / test (push) Successful in 45s
Everything here works without GPUs and is covered by tests. The control plane
connection is not built yet, so llmgw agent itself is still a stub and the phase
exit criterion is not met.

- discovery: IB from /sys/class/infiniband, parsing the formats sysfs actually
  uses and telling native InfiniBand from RoCE; GPUs via nvidia-smi, the
  cgo-free path, with health flags for uncorrected ECC and memory already held
  by another process. A node without IB is not an error, it is a single-node
  island.
- engines: vLLM and SGLang command and environment construction, kept free of
  process handling so it stays testable. The NCCL environment is the part the
  plan marks critical: NCCL_IB_HCA lists only the chosen fabric's adapters,
  because leaving it open lets NCCL pick an adapter on a fabric where the peer
  is unreachable, and the job then hangs instead of failing.
- validation before launch: no mixed GPU types in a tensor-parallel group, the
  node/GPU/TP arithmetic has to add up, and fp8 is refused below compute
  capability 8.9 rather than silently falling back to a slower path.
- supervisor: process groups, because engines fork workers that hold GPU memory
  and outlive a signal sent only to the leader; a bounded log tail, since engine
  output is the only diagnosis when loading fails; Starting separate from
  Loading, because a 70B model legitimately takes minutes; and orphan adoption
  through pid files fingerprinted with the command line, since pids get reused
  and adopting an unrelated process is worse than leaking the original.
- readiness is two questions: /health says the process is up, /v1/models says
  the weights loaded. Routing on /health alone sends traffic into a loading model.
- llmgw agent inventory reports what the scheduler would see on a node.

One plan correction: GPU class sizes come from the driver in GiB, which does not
always match the marketing name. An L40S is sold as 48GB and reports 44.99 GiB,
so its class is l40s-45g, not l40s-48g. Consistency matters more than matching
the box, because that number decides whether a model fits. Plan §2 and §5.1
updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:01:59 +02:00
ilkerandClaude Opus 5 dc38dd6096 Phase 1.5: admission control
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.

- saturation detector: one number per pool, the worse of queue depth and KV
  occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
  the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
  one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
  queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.

Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.

Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.

One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:53:38 +02:00
ilkerandClaude Opus 5 a3f71aa5ee Phase 1: the router serves traffic
CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.

- proxy: SSE relayed event by event with a flush after each, cancellation
  propagated upstream, and separate deadlines for dial, headers, first token,
  inter-token idle and total. The idle watchdog is what distinguishes a wedged
  engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
  nothing has reached the client; after that, splicing two responses together
  would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
  Validation rejects duplicate endpoint ids and aliases that shadow a model,
  because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
  per-filter drops on a separate admin listener with health and pprof.

Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.

The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:41:13 +02:00
ilkerandClaude Opus 5 40c884a50a Add a changelog and map releases onto roadmap phases
CI / test (push) Successful in 18s
Versions track the phases in the plan: a phase is released once its exit
criterion is met, not when its code lands. That keeps "released" meaning
something — v0.1.0 ships sim-engine and the routing libraries, and says plainly
that the gateway does not serve traffic yet.

scripts/gitea/milestones.py creates the matching Gitea milestones. Idempotent
and never destructive, so it can be re-run after the roadmap shifts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:29:41 +02:00