Commit Graph
71 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 6289b01ed6 The cluster's settings live in the store, and every change takes effect
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.

Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.

llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.

A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.

Retires appendGuardIfMissing: the import does it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 15:25:23 +02:00
Ilker ManapandClaude Opus 5.5 33334c8f33 Merge C24 federation: another llmgw cluster as a provider
Conflicts with facility-aware placement (config, control, admin API routes,
store test tables) all keep both; the generated reference and OpenAPI
document regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:29:35 +02:00
Ilker ManapandClaude Opus 5.5 9e46ee83e4 Merge C24 KV-index routing from vLLM's cache events
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:24:25 +02:00
Ilker ManapandClaude Opus 5.5 40e9fd2ab1 Federation: another llmgw cluster as a provider, its models as PEER/MODEL
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:30 +02:00
Ilker ManapandClaude Opus 5.5 117f417ef5 Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:11 +02:00
Ilker ManapandClaude Opus 5.5 bd3eca4092 Merge C24 LoRA adapters on demand for vLLM, with affinity
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:15:12 +02:00
Ilker ManapandClaude Opus 5.5 2a1e0e6c4c LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:14:38 +02:00
Ilker ManapandClaude Opus 5.5 e8a62b23b7 Merge features-c2: the docs portal and its links in English and Turkish
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:15:25 +02:00
Ilker ManapandClaude Opus 5.5 11231d8c2b The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates
Every template and handler message in internal/web goes through a JSON
catalogue under internal/web/locales/<code>/, with English as the fallback.
The language is the person's choice (a cookie set by /_lang/{code}), else
Accept-Language, else landing.language. Tests refuse a key some language
lacks, a translation with other arguments than English, and template text
left in English.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:12:53 +02:00
Ilker ManapandClaude Opus 5.5 44f3a9e31d Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:50:53 +02:00
Ilker ManapandClaude Opus 5.5 30f7f3474a Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:27:31 +02:00
Ilker ManapandClaude Opus 5.5 5b92ae0277 Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:13:18 +02:00
Ilker ManapandClaude Opus 5.5 9136bc71c2 More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:56:25 +02:00
Ilker ManapandClaude Opus 5.5 32ff1dbd6f Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:51:21 +02:00
Ilker ManapandClaude Opus 5.5 841ee21c12 Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 21:00:01 +02:00
Ilker ManapandClaude Opus 5.5 0a4076928a Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 19:56:16 +02:00
Ilker ManapandClaude Opus 5.5 29a7347df2 Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 19:45:37 +02:00
Ilker ManapandClaude Opus 5.5 14ea59d83f Provisioning through SCIM 2.0: accounts and groups kept in step by the identity provider
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:40:15 +02:00
Ilker ManapandClaude Opus 5.5 4cf40644c9 Single sign-on through OpenID Connect: accounts, staff roles and memberships from the provider
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:36:15 +02:00
Ilker ManapandClaude Opus 5.5 bd4736125b Do not name a judge model in the error a client gets for a model nobody serves
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:37:52 +02:00
Ilker ManapandClaude Opus 5.5 02aac84ca5 The /admin/guard page, a Guardrails link, and the policies in force on the profile page
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:15:38 +02:00
Ilker ManapandClaude Opus 5.5 47e847423d Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:11:50 +02:00
Ilker ManapandClaude Opus 5.5 43f6a585a6 A judge model on the cluster: role guard in the catalog, judge in a guard policy
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:09:31 +02:00
Ilker ManapandClaude Opus 5.5 ec737cffb5 no_store: refuse conversations, notes and models outside the cluster under a policy that keeps nothing
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:36:02 +02:00
Ilker ManapandClaude Opus 5.5 9b9349183b gofmt the guard hooks
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:23:02 +02:00
Ilker ManapandClaude Opus 5.5 979aa44804 Hook a guard into the router: judge requests before dispatch and filter answers as they are relayed
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:20:24 +02:00
Ilker ManapandClaude Opus 5.5 fd9f665ef8 End a session unused for a day, and purge dead sessions and claims hourly
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:50:56 +02:00
Ilker ManapandClaude Opus 5.5 20ebd21243 Skip capacity fallbacks the key may not use, and charge what the model that answered costs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:13:59 +02:00
Ilker ManapandClaude Opus 5.5 36cdbce125 Merge ha: replicated store, second control plane, join codes, setup and console screens
Release / build (push) Successful in 4m26s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 5m14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 22:59:49 +02:00
Ilker ManapandClaude Opus 5.5 aa3efcebbe Add a standby control plane with server add, and draw the screens as windows
server add gives another machine this llmgw, a certificate, the
configuration and store address, lets it into the store through Patroni, and
starts it; --vip keeps a floating address on the leader with keepalived,
following a new /leader endpoint. Nodes learn every control plane from the
leader and notice a dead one by HTTP/2 ping instead of after the kernel's
fifteen minutes. The setup, console, node and init screens are drawn full
screen in one frame: title bar, sidebar, bordered panel, key bar, dialogs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 22:29:12 +02:00
Ilker ManapandClaude Opus 5.5 ce006a6e6f Publish per-tenant request and budget series, and alert on them
Requests by outcome, tokens and first-token time per tenant, at the level
router.tenant_label names (team by default), and every budget's share spent
this period. Rules alert on budgets at 80% and 100%, tenants hitting limits,
failing, spiking past their weekly average, or waiting on first tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 19:55:24 +02:00
Ilker ManapandClaude Opus 5.5 4454e1322c Read usage from sums kept as it is written, not from every request
Charts, totals and budgets summed the request table on every view. Usage is
now also summed in the writing transaction into five-minute, hourly and daily
tables per key, person, project and model, with latency and first-token
histograms. Charts read the coarsest tier their slices allow; totals and
budgets compose rows, slices, hours and days, and stay exact. Request records
can be given a retention without moving any total.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 19:43:10 +02:00
Ilker ManapandClaude Opus 5.5 1b89d5cdea Say on the chat page whether a GPU or a CPU answered
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m54s
A model with replicas on a card and on CPUs answered at either speed with no
way to tell which. The picker names where each model runs, each reply names
the hardware that served it, and every answer carries x-llmgw-hardware.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 18:57:50 +02:00
Ilker ManapandClaude Opus 5.5 cf53904cc5 Serve LoRA adapters on llama.cpp models, asked for by name
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m11s
CI / test (push) Successful in 3m47s
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:40:47 +02:00
Ilker ManapandClaude Opus 5.5 94eec557ad Serve a model from an external OpenAI-compatible provider
CI / test (push) Successful in 3m47s
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:29:01 +02:00
Ilker ManapandClaude Opus 5.5 3d3c96e422 Displace the newest lowest-band request when the gateway queue is full
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m9s
CI / test (push) Successful in 3m50s
A total bound across bands (admission.queue_capacity_total, default half of
all bands' capacity together); reached, a request is not refused while a
lower band has requests queued, and the newest of the lowest band gets a 503
with Retry-After instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:07:53 +02:00
Ilker ManapandClaude Opus 5.5 62657f4c2a Release v0.10.4: CPU threads, tracing, and the rest of the review
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m57s
Release / build (push) Successful in 4m11s
CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.

Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.

Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.

reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 04:00:35 +02:00
Ilker ManapandClaude Opus 5.5 adb1b3cf2a Add replicas under load, up to a model's max, and take them back when quiet
Release / build (push) Successful in 1m2s
Release / notes (push) Successful in 2s
CI / test (push) Successful in 3m48s
A model's max never did anything: desiredFor returned the minimum, or one
for an on-demand model someone was asking for, whatever the queue. Six small
CPU nodes run out of slots quickly, and a model with room to grow stayed at
its minimum while requests waited.

The router already scrapes each engine's running and waiting requests. The
control plane now reads them per model on every tick, alongside when each
replica was last used, and the reconciler keeps how many replicas load has
added above a model's base. One or more requests waiting per ready replica,
held for a minute, adds one; ten minutes with nothing waiting removes one of
those again. It never goes above max or below the base, moves one step at a
time, and does not add while the last addition is still placing or starting
— on this fleet a replica is minutes of weights and gigabytes of memory, and
a spike should not leave a pile of replicas behind it. Decisions are logged
with the load behind them.

Nothing live changes until a model's max is raised above its min.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:18:07 +02:00
Ilker ManapandClaude Opus 5.5 37a80a9b44 Park a model that fails every time it starts, and say so everywhere
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m34s
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:21:57 +02:00
Ilker ManapandClaude Opus 5.5 9853936e0b Give every engine a key only the router holds
Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.

Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.

The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.

Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.

Also formats two files that went in unformatted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 00:38:29 +02:00
Ilker ManapandClaude Opus 5.5 5aa5675a9c Show a model loading in the chat, and send once it is up
A message to a model that was not running waited 45 seconds and came
back as "the model is loading; retry shortly". The page now asks for
the model with /chat/wake, shows where it stands from
/chat/model-status - waiting for a node, downloading with bytes and a
percentage, starting the engine - and sends the message when it answers.

Also passes the nodes' llama.cpp build to the pages, which the mirror's
checks for loadable weights needed and were never given.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:52:31 +02:00
Ilker ManapandClaude Opus 5.5 098886e0e4 Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:

- memberships: a person is a member of any number of organizations,
  teams and projects, or of none; a member of a team or organization
  may use every project in it. users join/leave, and the people page
  adds and takes out from either side. Approval no longer needs a
  project.
- personal keys: a key may have an owner and no project. /profile asks
  what a new key spends against; keys create --for does the same.
  auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
  project and the chat page; --per-member on an org, team or project
  gives each member their own share there. Resolution folds the tree,
  each per-member limit for the key's owner, and the owner's own, each
  in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
  personal use and chat included; restore no longer drops rows with no
  project.

Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.

Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.

users move is gone; join and leave replace it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 19:03:46 +02:00
Ilker ManapandClaude Opus 5 c6df0659af Put the cluster's own charts on its own admin page
CI / test (push) Successful in 1m50s
The dashboard charted what the cluster had been billed for — requests, tokens,
spend, all from llmgw's own store — and nothing about whether it was healthy.
Saturation, queue depth, cache occupancy and replica counts were numbers in a
table at the instant the page loaded, with no way to see that one of them had
been climbing for an hour.

Four line charts now, read back from wherever the metrics were scraped to.

Read back rather than kept twice. llmgw publishes these already; what a gauge
cannot do is remember, and a ring buffer in this process would be a rival copy
of the same numbers with its own retention, lost on every restart and
disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus
is empty by default and the panels are then absent rather than empty.

One axis per chart, always. Two measures of different scale belong in two
charts: a second y-axis lets any two lines be made to cross wherever the author
likes and the reader cannot tell it happened.

Four colours, handed out in fixed order, never cycled. A fifth series is summed
into "other" and drawn in grey — dropping it would make a chart that does not
add up, and inventing a hue for it would put two lines on the page that nobody
can tell apart. The fold is by timestamp, not by position: the tail's series do
not share sample times, and adding them by index slides points sideways onto
each other's clock.

A legend for every chart with more than one line, because colour alone is not
an encoding a reader can rely on.

A flat zero is drawn along the bottom rather than left blank: nothing happening
is a real answer, and an empty frame reads as a broken chart. A single sample
is not a line and says so instead of putting a dot in the corner.

One failed query fails all four panels. Three drawn and one blank invites
reading the blank one as "nothing is happening there".

NaN is dropped rather than plotted. Prometheus sends it as a string for a
sample it has no value for, Go's ParseFloat accepts it happily, and one NaN
coordinate is an SVG path that does not draw at all — taking the rest of the
line with it.

Axis maxima are rounded to a number a person would pick. An axis topping out at
0.8347 makes every gridline unreadable, and the shape is what a chart is for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 18:25:56 +02:00
Ilker ManapandClaude Opus 5 e034862b79 Serve a mirrored model from the list, instead of writing YAML for it
CI / test (push) Successful in 1m35s
Fetching a model put it on the mirror and left the operator to hand-write a
catalogue entry and run `ctl model add` on the control plane — for a source the
page was already displaying, with an engine decided by the file extension it
had just shown them.

A present row now offers to serve it. The form opens on that row with the name,
the mirror's own URL and the engine filled in, and asks for the two things
nobody can infer: the context pool and how many replicas.

What it refuses is the point.

A version of several files is refused with the reason. A catalogue entry names
one source, so an entry for a sharded model is an entry for one shard, which
loads as a broken model rather than failing to load. Nothing said so before —
the mirror happily holds a four-shard model that no catalogue entry can
describe.

Safetensors on a fleet with no GPU is warned about before anything is added.
This is the failure we hit today: fifteen gigabytes of shards arrived, and only
then did looking at a node show no card, no vLLM and no SGLang. The control
plane is not told which engines a node has installed, so the check is on cards,
which it does know — a warning rather than a refusal, because a card can be
added tomorrow.

A name already declared in configuration is refused here too, matching what the
model service would say anyway, but before the form is filled in rather than
after.

Without a store there is no serve action at all: a model added at runtime would
not survive a restart, and offering it then refusing is worse than not offering.

The mirror page's dependencies became a struct. Three are optional interfaces
and two are the same shape, which at a call site is a swap nobody notices until
a page shows the wrong thing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 15:54:20 +02:00
Ilker ManapandClaude Opus 5 ed530133b6 Find a model in the registry instead of pasting a URL into a box
CI / test (push) Successful in 1m41s
Mirroring anything meant knowing a repository existed, knowing which of its
thirty files were the weights, and assembling a resolve URL by hand. The mirror
page had a text box for that URL and no way to find one.

It now searches Hugging Face and lists a repository's weight files with their
sizes and published checksums, each with a button that fills the fetch form.

Only weight files. A repository is mostly a README and a tokenizer, and listing
those buries the four entries somebody came for.

The size shown is the LFS size, not the plain one. For an LFS-stored file —
every weights file — the plain field is the size of the pointer, so the other
one reports a gigabyte model as 133 bytes, and that is the number somebody
would then plan disk against.

The source URL is pinned to the commit the listing came from rather than to a
branch, so what the mirror fetches is what was on screen. A repository
re-uploaded under the same branch otherwise leaves two different sets of
weights with identical provenance and no way to tell them apart. The version
defaults to that revision for the same reason.

The registry's own failures are reported on their own line. It is somebody
else's server on the other side of the internet, and it being down says nothing
about the mirror — which is what the page is for and has to keep working
regardless. With no registry configured there is no search box at all, rather
than one that can only time out.

Gating decodes from a bool or a string, because the field is documented as the
first and is the second in practice: strict decoding fails the whole response,
so one gated model would hide the other nineteen in a page of results.

And a bug this made obvious: downloads were keyed by model and version, so the
second shard of a sharded model was refused as "already fetching" until the
first had finished. A fifteen-shard model had to be pulled one file at a time,
each waiting on the last. Keyed by file now — which is what the single-flight
was ever protecting, since the race is two writers on one .part. A version made
of several files reports as the worst of them: it is not present until every
shard is, and one shard failing makes the version unusable however many others
arrived.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 15:18:57 +02:00
Ilker ManapandClaude Opus 5 ebce41e22d Stop calling the PostgreSQL store SQLite, and stop guessing where it is
Two things that were only ever confusing, and one of them stopped a command
working on the cluster.

The store type was named SQLite and kept the name when PostgreSQL was added, so
OpenPostgres returned a *SQLite and all seventy-odd methods on the production
store were spelled after the database it was not running on. Nothing broke;
people did. Earlier today I added three methods to *SQLite for a table that on
this cluster only ever lives in PostgreSQL, and the name did not once look
wrong. It is a field, which is all it ever was.

`llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a
deployment whose store is PostgreSQL every one of them failed with "unable to
open database file" while the right answer sat in auth.store in the config file
beside them; the conclusion an operator draws is that the tool is broken, and
they are not far off. They now ask the server's own configuration before
falling back, and the error names which of the three sources chose the address
— the failure was never the path, it was which one picked it.

The single-machine default still applies last, so a deployment with no config
file keeps working with no flags.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 12:20:40 +02:00
Ilker ManapandClaude Opus 5 cd7eedf09e Give the mirror a page, so managing it is not a curl command
CI / test (push) Successful in 1m49s
/admin/mirror, behind the same admin session as the other admin pages. What is
held, what is arriving with its byte count, what failed and why; start a fetch,
retry a failed one, remove a version.

Not mounted at all without a mirror configured. The right page for a deployment
with no mirror is no page, rather than an empty one pointing at nothing.

It reloads itself only while something is downloading. A reload at any other
moment takes with it whatever is half-typed into the fetch form, and there is
nothing to watch when nothing is moving.

A failed row offers Retry, not Remove. A fetch that failed left nothing on disk,
so the remove could only answer "the mirror holds nothing for this" — a button
whose single outcome is a confusing error. Starting the same download again is
what somebody looking at that row wants, and the mirror allows it: only a
download still running is refused.

An unreachable mirror is said outright instead of rendering as an empty table.
The two look nearly identical and mean opposite things: one is a mirror waiting
to be filled, the other is a machine to go and look at.

Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a
call site get swapped silently and are discovered by somebody clicking the
wrong link.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 10:17:12 +02:00
Ilker ManapandClaude Opus 5 a76ebdb79a Run the store on PostgreSQL, chosen by what the address looks like
CI / test (push) Successful in 3m11s
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing
configuration keeps working untouched and moving is one line changed rather
than a new setting to find. An address of the form @/path/to/file reads the
real one out of that file, which is how a DSN carrying a password stays out of
a configuration that gets copied, diffed and pasted into issues.

The driver is pgx, which is pure Go. That is not a preference: the whole binary
is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the
builder's, and a cgo driver would take that away from every deployment,
including the ones still on SQLite.

Tested against a real PostgreSQL, which found two things nothing else could.
The schema_migrations table is created in Go rather than in a migration file,
so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in
PostgreSQL, and a millisecond timestamp does not fit, so the very first
migration failed to record that it had run. And the memory migration uses an
FTS5 virtual table with three triggers keeping it in step, which has no
counterpart: PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget.

That leaves exactly one query that is not the same sentence in both dialects —
the note search, MATCH against @@ — and it is written out twice rather than
hidden behind a shared spelling that would mean something subtly different in
each. Everything else is one set of queries.

The text search uses the 'simple' configuration rather than English. These are
an agent's own notes, in whatever language it chose, and stemming English
rules over Turkish would quietly stop finding things.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 16:18:10 +02:00
deniz 2eb79f3400 agenta memory eklendi
CI / test (pull_request) Successful in 2m31s
2026-09-24 04:04:03 +03:00
Ilker ManapandClaude Opus 5 d26a30471e Let people forget a conversation, and make retention actually expire things
CI / test (push) Successful in 1m38s
Two halves of keeping the promise that storing makes.

DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.

And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:52:06 +02:00