Commit Graph
16 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 92733317a9 A batch of embeddings is checked text by text against the model's context
llmgw-kb sends many passages in one /v1/embeddings request. Their sizes
were added together and compared with the context, so a batch of passages
that each fit was refused as one over-long prompt. The longest input is
what has to fit; the total still counts for usage and limits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 11:43:21 +02:00
Ilker ManapandClaude Opus 5.5 ba5df90602 llmgw-kb: sync a person's documents into a knowledge base from their own machine
Its own binary for Windows, macOS and Linux, carrying no server code. It reads
the files there, cuts them, has the gateway embed them with the person's key
and sends text and vectors for what changed only: a SQLite ledger on the
machine remembers each document's file, size, time, digest and what happened,
so a rerun reads nothing unchanged, a file skipped stays skipped until
--retry, and --delete removes only what this machine sent under the paths
given. --dry-run says what would leave the machine. Commands: login,
collections, sync, status, history, formats.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 13:43:47 +02:00
Ilker ManapandClaude Opus 5.5 7245dbe1dd llmgw stop, start and restart act on the whole system
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:

- llmgw stop sets serving.stopped in the cluster's settings, live: the
  reconciler wants every model at no replicas, freeing every GPU, and the
  gateway answers every POST under /v1/ with 503 system_stopped. The control
  planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
  the other control planes over ssh, standbys before a leader, then this
  one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.

One machine's services moved to llmgw service status|start|stop|restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 18:01:45 +02:00
Ilker ManapandClaude Opus 5.5 ba5158d463 List in /v1/models only the models the key may use
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:49:54 +02:00
Ilker ManapandClaude Opus 5.5 20ebd21243 Skip capacity fallbacks the key may not use, and charge what the model that answered costs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:13:59 +02:00
Ilker ManapandClaude Opus 5.5 62657f4c2a Release v0.10.4: CPU threads, tracing, and the rest of the review
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m57s
Release / build (push) Successful in 4m11s
CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.

Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.

Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.

reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 04:00:35 +02:00
Ilker ManapandClaude Opus 5.5 91fe24a105 Close what a second security review found still open
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m50s
A review of v0.10.4-rc.4, and a look at the live network, found:

- Alertmanager listening on every interface: from any node its API took
  silences, with no login. Bound to loopback on the monitoring host, and the
  monitoring README now installs it that way; Prometheus beside it is its only
  client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
  and nothing limited how many ran at once; the per-address throttle kept one
  entry per address ever tried. Hashing now takes one of a few slots and
  waits for one otherwise. The throttle expires entries whose window has
  passed when it fills, and if it is still full it refuses new addresses:
  evicting live ones instead would let a flood of made-up addresses reset the
  throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
  name it asked for, so any key holder could mint a series per request.
  Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
  default mode in the listable backup directory, a full copy of the store
  readable while the backup ran; it is now made in a private directory, and
  the unit sets UMask=0077. And archive links were checked by name only, so
  "a -> ." then "a/a/b -> ../.." passed and a file written through them landed
  above the install directory, as root. Archives now unpack through os.Root,
  and a link may only point at a file beside it — which every llama.cpp
  b10991 archive, CPU and CUDA, was checked to follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:25:07 +02:00
Ilker ManapandClaude Opus 5.5 37a80a9b44 Park a model that fails every time it starts, and say so everywhere
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m34s
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:21:57 +02:00
Ilker ManapandClaude Opus 5.5 b7e8e1a3a1 Let the chat be stopped, timed, tuned and kept across a reload
While a reply streams the Send button is Stop, which aborts the request
- and with it the one to the engine - keeping what had arrived. Under
each reply: its tokens, from the engine's own count (the page's turn now
asks for include_usage), tokens a second, time to the first token and in
all. A settings fold takes a system prompt, sent before every turn, and
a temperature, which the gateway holds to 0-2 whatever the page sends.

The conversation, the prompt and the temperature are kept in the
browser's storage, so a reload puts them back - the model's turns
rendered again by the server as when they came - and "New conversation"
clears them. The gateway still keeps no conversation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:54:23 +02:00
Ilker ManapandClaude Opus 5 28967edab3 Serve a spent budget something it can afford
CI / test (push) Successful in 1m38s
The accounting layer has always exempted a model that costs nothing, with a
comment saying it is "what lets a key with no budget left still be served a
cheap fallback". That described a caller nobody had written: the budget was
checked once, against the requested model, and a refusal was the end of it,
so the exemption could never fire.

A caller that has run out is now offered the first model in its fallbacks it
can still afford. The allowlist still applies — running out of money is not
a way to reach a model a key was never allowed — and a model with no
fallback is refused exactly as before, or a budget would stop meaning
anything. Only a spent budget falls back: a rate limit is answered by
waiting a second, and moving the request elsewhere would spend someone
else's capacity to avoid a delay nobody asked to avoid.

Which uncovered a header that has never worked. x-llmgw-served-model was set
in finish, and finish runs after the proxy has written the response — so the
one mechanism for telling a client that something else answered has been
announcing it to nobody since it was written. Nothing failed, because the
only thing asserting it was a comment. It is now set before dispatch, for
the capacity fallback as well as this one.

Finding that took bisecting a test I had written wrong: it polled for a 429
that never came, because the budget had run out and the fallback had served
the request — the feature working, reported as the feature failing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:26:03 +02:00
Ilker ManapandClaude Opus 5 7fe59bd5fb Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs
can now hold every model in its catalog and only the wanted ones in its
cards: min 0 with a max above it means load on demand, and once nobody has
asked for idle_timeout the replicas stop and the cards go back.

No new field, because min 0 max 1 already reads correctly — keep none
running, allow up to one. Which made min 0 max 0 meaningful for the first
time: configured and deliberately not running. normalize() used to fill a
zero max in, on the reasonable ground that a model with no maximum cannot
run; that would now silently turn every switched-off model into an
on-demand one, including the ones already written that way in this
cluster's own configuration. A model with no scaling stanza stays off for
the same reason — defaulting it to on-demand would let any client request
start a 70B nobody intended to run.

The one real decision is what happens to the request that triggers a load.
Refusing makes the feature useless, since the first request for every model
fails; waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds holds a connection for ten minutes. So it waits
briefly and then says what is happening — 503 with Retry-After and "the
model is loading", which is a different answer from "no endpoint serves
this" and produces different client behaviour. An unknown model still gets
the unknown-model error, or on-demand would turn every typo into "loading"
and the useful message would disappear.

Use keeps a model alive rather than only the request that started it,
without which the timer measures time since the load and a busy model is
evicted on schedule however hard it is working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:44:00 +02:00
Ilker ManapandClaude Opus 5 e9fe3ad379 Know how big a model's context is, and what it can do
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which
surfaced this week as somebody else's bug.

Context length. Nothing read it, /v1/models did not publish it, and an
over-long prompt was not refused: it was admitted, scheduled, given a
replica and a slot, and rejected by the engine in whatever shape that
engine chose. A client sizing prompts from /v1/models had nothing to read.
It is now declared per model, published when known, and checked before
authentication work or a queue slot is spent — with a ten percent margin,
because the estimate is bytes over four and being a few percent out must
not turn a valid request into a refusal.

Capabilities. The gateway forwards a tools array to any model, and one that
cannot use them answers in prose; every OpenAI-compatible client then
behaves as though the model is broken, and the only way to find out was to
try it. Declared rather than discovered, because no engine will say — which
is why these are claims, and why the next commit makes engine-check verify
the tool-calling one instead of trusting it.

And a bug found while wiring the first: the token estimate was taken from
the routing prefix. That prefix stops at 2 KB by design, so every prompt
above it estimated as 512 tokens however long it really was, and TPM limits
undercounted exactly the requests that consume the most. Prefix routing
wants a bounded head of the conversation; a token estimate wants all of it.
They are now separate functions, and structured multimodal content counts
rather than measuring zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:07:49 +02:00
Ilker ManapandClaude Opus 5 082d52bc97 Stop shipping a placeholder that looks like a value, and say what is served
CI / test (push) Successful in 1m23s
A user copied the curl from the profile page and sent it as it stood. Every
other field in that snippet is filled in for them — the base URL, the key
they had just been issued — so "model": "MODEL" reads as a value rather
than a blank, and they got:

  no endpoint serves "MODEL"

which is true and useless. It reads the same whether the name was a typo,
a placeholder left in from the documentation, or a real outage, and it
never says what would have worked.

The message now lists what is being served, capped at ten with a count for
the rest, and says plainly when nothing is. That list is what /v1/models
already returns to the same caller, so it discloses nothing new.

The profile page's example now names a model that is actually running,
real ones before simulated — the point of an example someone can run is
that it runs, and sending a first request to the sim engine would answer
in Greek letters. The landing and claim pages keep a placeholder, because
they are served without a key and deliberately say nothing about what is
here; theirs is <model-name>, which cannot be mistaken for a value, with a
line saying where to get the real one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 18:23:19 +02:00
Ilker ManapandClaude Opus 5 72b03a1a1c Say which model is simulated instead of letting it look broken
CI / test (push) Successful in 1m31s
big-model runs on the sim engine, whose whole vocabulary is the Greek
alphabet, cycled at a fixed rate until max_tokens. That is what it is for:
with no H100s it is the only thing exercising the TP8 two-node path, the
fabric invariant and rebuild-after-node-loss. The output is the point.

What was wrong is that nothing said so. /v1/models returned it with
owned_by "llmgw", identical to the real model, and the chat picker sorted
purely by name — so "big-model" led the list, was preselected by the
browser, and anyone who typed a question and pressed Send got Greek
letters back with no explanation. Reported as the model spitting out crap,
which is a fair reading of what it looks like from outside.

Simulated is derived from the profiles rather than configured: an operator
who adds a sim profile and forgets a flag would be shipping placeholder
text under a real model's name. A model counts as simulated only when
every profile is, because one real profile means the request can still
come back as inference.

The marker on /v1/models is owned_by "llmgw-sim", which is already what
the sim engine calls itself on its own /v1/models — the gateway was
flattening a distinction that existed one layer down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:11:56 +02:00
ilkerandClaude Opus 5 2373726efd Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.

- registration carries the whole hardware inventory, so there is no separate
  discovery call and a reconnecting agent refreshes what the scheduler knows.
  The fabric labels travel with it; they cannot be inferred from the wire, so
  losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
  it holds the open streams and is therefore the only thing that can deliver a
  command to a node.
- the agent validates a profile before launching anything, since an engine that
  refuses a profile does so after minutes of loading, then supervises the
  process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
  require touching every GPU node, and a fleet must not reconnect in lockstep
  and knock it over again. Engines keep serving throughout, because the router
  talks to them directly.

llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.

Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.

config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.

Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:48:21 +02:00
ilkerandClaude Opus 5 a3f71aa5ee Phase 1: the router serves traffic
CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.

- proxy: SSE relayed event by event with a flush after each, cancellation
  propagated upstream, and separate deadlines for dial, headers, first token,
  inter-token idle and total. The idle watchdog is what distinguishes a wedged
  engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
  nothing has reached the client; after that, splicing two responses together
  would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
  Validation rejects duplicate endpoint ids and aliases that shadow a model,
  because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
  per-filter drops on a separate admin listener with health and pprof.

Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.

The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:41:13 +02:00