Commit Graph
185 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 abaf086fd5 A standby's portal is the leader's; the changelog as the release notes read it
CI / test (push) Successful in 2m42s
Release / notes (push) Successful in 3s
Release / build (push) Successful in 3m20s
A proxy left pointing at a standby showed an empty cluster: the portal and
the admin pages read the standby's own memory. A standby now passes its
client port to the leader's, all but /health, /ready and /leader, which
must describe the machine a load balancer is choosing; and its admin port
passes the dashboard as well as the API. The leader trusts the control
planes' X-Forwarded-For, so clients keep their own address.

The changelog's candidate section is "Unreleased" again: release notes
for an rc read that section, and v0.11.3-rc.1 got none.

The e2e wait for two endpoints is 30s: under the race detector on a busy
runner, 15s was missed once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 06:34:57 +02:00
Ilker ManapandClaude Opus 5.5 c5b34978f5 Guardrails in the console
CI / test (push) Successful in 2m52s
The console never mentioned them. A Guardrails view shows each policy with
its mode, fail behaviour and checks, whether its judge models have replicas
ready, where policies are attached (attach and detach from it), and what
they did in a window. The admin API gains GET /api/v1/guard/events, and GET
/api/v1/guard says what each policy does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 03:36:17 +02:00
Ilker ManapandClaude Opus 5.5 c5e388807a ctl upgrade in waves, and the other control plane first
One node at a time was hours on a fleet of hundreds. Nodes now go one, then
two, four, up to --parallel at once, each wave waited on whole; a wave that
does not come back stops it. The other control plane is upgraded first over
ssh, so a leader restarted last hands over to a standby on the release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:43:33 +02:00
Ilker ManapandClaude Opus 5.5 2158a933e0 v0.11.2
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:30:37 +02:00
Ilker ManapandClaude Opus 5.5 803d1d5fb6 CHANGELOG and BACKLOG for the rest of C24 and the engine fixes; README
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:30:01 +02:00
Ilker ManapandClaude Opus 5.5 a13a519386 CHANGELOG and BACKLOG for C17-C23 and the first parts of C24; README lists the new docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:17:16 +02:00
Ilker ManapandClaude Opus 5.5 30f7f3474a Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:27:31 +02:00
Ilker ManapandClaude Opus 5.5 35657c1417 Secrets in Vault: env: and vault: references wherever a secret file is named, and the store's own secrets sealed under a master key — Vault transit or a local key
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:21:44 +02:00
Ilker ManapandClaude Opus 5.5 bce54f0941 Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:16:56 +02:00
Ilker ManapandClaude Opus 5.5 5b92ae0277 Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:13:18 +02:00
Ilker ManapandClaude Opus 5.5 10466416df server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:09:59 +02:00
Ilker ManapandClaude Opus 5.5 e502996198 Anthropic's Messages API: /v1/messages translated to chat completions and back, streamed or not, with tools and images; keys in x-api-key
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:59:55 +02:00
Ilker ManapandClaude Opus 5.5 9136bc71c2 More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:56:25 +02:00
Ilker ManapandClaude Opus 5.5 32ff1dbd6f Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:51:21 +02:00
Ilker ManapandClaude Opus 5.5 841ee21c12 Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 21:00:01 +02:00
Ilker ManapandClaude Opus 5.5 889be70b8f Canary on the console's model menu; docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 20:33:59 +02:00
Ilker ManapandClaude Opus 5.5 29a7347df2 Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 19:45:37 +02:00
Ilker ManapandClaude Opus 5.5 c553f2041a The baseline fails closed with two judges, and a judge not answering is a notification
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 19:19:49 +02:00
Ilker ManapandClaude Opus 5.5 ddb0472f9b A hardened install by default: baseline guardrails with Llama Guard 3 1B judging on a CPU, keys at most 90 days
server init (and the setup screens) write guard.default [baseline] —
secrets refused, cards masked, injection recorded, the judge refusing
S1-S4, S7, S9-S12 in prompts and answers, fail open after 5s — fetch the
judge into the mirror and run it on a node's CPU. --no-guard-model leaves
it out. The closing text names what init cannot do: TLS on the client
port and backup recipients.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:45:31 +02:00
Ilker ManapandClaude Opus 5.5 1b71c38f89 Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates
notify.webhooks sends signed JSON or Slack-compatible messages, queued,
retried, de-duplicated hourly; ctl notify test checks each webhook.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:58:11 +02:00
Ilker ManapandClaude Opus 5.5 57739d736a Usage export as CSV: by period and by place, person, key and model, from the rollups
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:49:04 +02:00
Ilker ManapandClaude Opus 5.5 8866a1ea0f Key lifecycle: lifetime limits by scope, rotation with overlap, kinds and expiry in the listing
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:45:05 +02:00
Ilker ManapandClaude Opus 5.5 14ea59d83f Provisioning through SCIM 2.0: accounts and groups kept in step by the identity provider
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:40:15 +02:00
Ilker ManapandClaude Opus 5.5 4cf40644c9 Single sign-on through OpenID Connect: accounts, staff roles and memberships from the provider
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:36:15 +02:00
Ilker ManapandClaude Opus 5.5 b60cf45adf Staff roles (viewer, operator, billing) and the console's Audit view
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:30:56 +02:00
Ilker ManapandClaude Opus 5.5 d55472c00a Guardrails in the console: attach and detach on a branch, where each applies from, and a column
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:18:42 +02:00
Ilker ManapandClaude Opus 5.5 43f6a585a6 A judge model on the cluster: role guard in the catalog, judge in a guard policy
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:09:31 +02:00
Ilker ManapandClaude Opus 5.5 2f07b2ba0b Document guardrails, correct what data.md said leaves the cluster, and regenerate the reference
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:36:37 +02:00
Ilker ManapandClaude Opus 5.5 979aa44804 Hook a guard into the router: judge requests before dispatch and filter answers as they are relayed
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:20:24 +02:00
Ilker ManapandClaude Opus 5.5 eb10aee93b Document TLS between the store's machines, and close section A of the backlog
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:12:35 +02:00
Ilker ManapandClaude Opus 5.5 efb4b06fc5 Report a ready engine dying from every connection, adopted ones included
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 11:50:09 +02:00
Ilker ManapandClaude Opus 5.5 7b6276901a Keep llmgw's internal ports to the cluster's networks with an nftables table of its own
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 11:03:46 +02:00
Ilker ManapandClaude Opus 5.5 7e57863edd Hide the mirror's bookkeeping, bound GGUF array nesting, and forget old serve jobs and downloads
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:56:16 +02:00
Ilker ManapandClaude Opus 5.5 fdd4dcf841 Serve every page under a hash-based Content-Security-Policy, and update KaTeX to 0.16.47
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:54:04 +02:00
Ilker ManapandClaude Opus 5.5 fd9f665ef8 End a session unused for a day, and purge dead sessions and claims hourly
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:50:56 +02:00
Ilker ManapandClaude Opus 5.5 ba5158d463 List in /v1/models only the models the key may use
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:49:54 +02:00
Ilker ManapandClaude Opus 5.5 f9493aadb9 Guard a GPU-less node's problems, and bound thread ids
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:49:10 +02:00
Ilker ManapandClaude Opus 5.5 a56ad8e4c4 Keep the control plane's fetches off link-local addresses, and see through NAT64, 6to4 and proxies
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:48:01 +02:00
Ilker ManapandClaude Opus 5.5 fbac1aa664 Take only torch from PyTorch's index, and lock its builds with every file's address
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:45:55 +02:00
Ilker ManapandClaude Opus 5.5 12efb7d085 Encrypt backups to backup_recipients with age
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:38:47 +02:00
Ilker ManapandClaude Opus 5.5 3b05ec0538 Have engines listen on the management address, and take SGLang's key off its command line
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:36:42 +02:00
Ilker ManapandClaude Opus 5.5 9b61bbc983 Revoke a removed node's certificate, and require --api-client with a mirror's mutual TLS
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:34:35 +02:00
Ilker ManapandClaude Opus 5.5 fe5b7f88f0 Believe X-Forwarded-* only from router.trusted_proxies
A new internal/clientip says who sent a request and how: the connection's
address, scheme and Host, or through a configured proxy what it forwarded.
The base URL in pages and setup scripts, isHTTPS for cookies, and the
per-address limits of auth and the sign-in forms use it. /setup scripts are
no-store.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:29:02 +02:00
Ilker ManapandClaude Opus 5.5 44cd58005a Count sign-in attempts per address and client, guard the sign-in form with a token, ration both forms per address
A wrong password from anywhere counted against the address, so ten locked
its owner out. Attempts are now counted per address and client, with a
looser bound per address alone. The sign-in form carries a token matched
by a cookie (login CSRF). POST /login and /signup are rationed per client
address, a hash waits at most ten seconds for a turn, and server init
writes signup_open: false.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:26:29 +02:00
Ilker ManapandClaude Opus 5.5 f1b49d0251 Bound idle time, header size and request reading on every listener, and connections on the client port
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:21:15 +02:00
Ilker ManapandClaude Opus 5.5 d3fb217b31 Bound the key cache and ration lookups of unknown keys per client address
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:19:38 +02:00
Ilker ManapandClaude Opus 5.5 ebf437a36a Have the setup's sudo grant remove itself within the hour, and keep a target from reading as an ssh option
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:17:52 +02:00
Ilker ManapandClaude Opus 5.5 1f406247e4 Expire join codes on the setup's side, bind the join port to its address, trust the peer address
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:16:53 +02:00
Ilker ManapandClaude Opus 5.5 27c49d0af9 Refuse the keys of an account that is not active
Resolve now checks the owner's state for a personal key; the authenticator
treats an inactive owner as a revoked key. Suspension stops the keys within
the cache TTL, reinstatement brings them back.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:15:32 +02:00
Ilker ManapandClaude Opus 5.5 20ebd21243 Skip capacity fallbacks the key may not use, and charge what the model that answered costs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:13:59 +02:00