Commit Graph
11 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 3a3db8cd20 Answers to similar questions, not only the same one
CI / test (push) Successful in 8m52s
llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.

The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:47:11 +01:00
Ilker ManapandClaude Opus 5.5 d1bf26528b keys limits changes only the limits it is given
CI / test (push) Successful in 5m25s
Setting one limit at a scope wrote the whole row, so a tool grant given
alone cleared the scope's rate, budget and models. --replace writes it
whole, as the console's form, which shows every field, now asks for.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 17:32:43 +02:00
Ilker ManapandClaude Opus 5.5 4453f469e1 Company MCP servers are registered with ctl mcp, and tools granted with keys limits --tools
CI / test (push) Failing after 5m4s
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 12:50:55 +02:00
Ilker ManapandClaude Opus 5.5 6289b01ed6 The cluster's settings live in the store, and every change takes effect
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.

Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.

llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.

A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.

Retires appendGuardIfMissing: the import does it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 15:25:23 +02:00
Ilker ManapandClaude Opus 5.5 44f3a9e31d Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:50:53 +02:00
Ilker ManapandClaude Opus 5.5 30f7f3474a Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:27:31 +02:00
Ilker ManapandClaude Opus 5.5 32ff1dbd6f Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:51:21 +02:00
Ilker ManapandClaude Opus 5.5 4979e08d64 Console: rotate a key, cap key lifetimes, test the webhooks; README, onboarding and console docs caught up
The README's status stopped at v0.9 and listed gaps since closed; onboarding
still said there was no SSO and no roles.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:28:24 +02:00
Ilker ManapandClaude Opus 5.5 b60cf45adf Staff roles (viewer, operator, billing) and the console's Audit view
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 16:30:56 +02:00
Ilker ManapandClaude Opus 5.5 d55472c00a Guardrails in the console: attach and detach on a branch, where each applies from, and a column
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:18:42 +02:00
Ilker ManapandClaude Opus 5.5 eb7dac8a67 The console manages people, keys, organizations and the mirror, and scrolls a tall sidebar
Accounts, API keys and Organizations views read the store and change it by
running the users, keys and org commands in-process, so their checks and
messages are the ones shown. A Mirror view lists what the mirror holds and
which models use it, downloads into it and removes from it. Models gain
adding from a definition file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:30:24 +02:00