e502996198
Anthropic's Messages API: /v1/messages translated to chat completions and back, streamed or not, with tools and images; keys in x-api-key
Ilker ManapandClaude Opus 5.52026-10-04 22:59:55 +02:00
9136bc71c2
More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge
Ilker ManapandClaude Opus 5.52026-10-04 22:56:25 +02:00
32ff1dbd6f
Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay
Ilker ManapandClaude Opus 5.52026-10-04 22:51:21 +02:00
841ee21c12
Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it
Ilker ManapandClaude Opus 5.52026-10-04 21:00:01 +02:00
a1dfd4df1d
Canary of a model's newer weights: ctl model canary, promote and canaries, rolled back by itself when it serves worse
Ilker ManapandClaude Opus 5.52026-10-04 20:31:26 +02:00
0a4076928a
Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared
Ilker ManapandClaude Opus 5.52026-10-04 19:56:16 +02:00
29a7347df2
Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names
Ilker ManapandClaude Opus 5.52026-10-04 19:45:37 +02:00
e59ff52932
Merge features-c: section C of the backlog, items 1-5, and a hardened install
Ilker ManapandClaude Opus 5.52026-10-04 19:20:41 +02:00
c553f2041a
The baseline fails closed with two judges, and a judge not answering is a notification
Ilker ManapandClaude Opus 5.52026-10-04 19:19:49 +02:00
07af03aa93
server init: the firewall and the judge before the starter model, and a model that fits nowhere said within two minutes
Ilker ManapandClaude Opus 5.52026-10-04 19:11:12 +02:00
ddb0472f9b
A hardened install by default: baseline guardrails with Llama Guard 3 1B judging on a CPU, keys at most 90 days
Ilker ManapandClaude Opus 5.52026-10-04 17:45:31 +02:00
6abd510628
Docs: one way in — quick-install starts at the setup screens, quickstart becomes development, install.md no longer says engines are yours to install
Ilker ManapandClaude Opus 5.52026-10-04 17:32:54 +02:00
4979e08d64
Console: rotate a key, cap key lifetimes, test the webhooks; README, onboarding and console docs caught up
Ilker ManapandClaude Opus 5.52026-10-04 17:28:24 +02:00
1b71c38f89
Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates
Ilker ManapandClaude Opus 5.52026-10-04 16:58:11 +02:00
57739d736a
Usage export as CSV: by period and by place, person, key and model, from the rollups
Ilker ManapandClaude Opus 5.52026-10-04 16:49:04 +02:00
15a7237a02
Match a mapped project however its path is spaced: the store writes "Acme / ml / train"
Ilker ManapandClaude Opus 5.52026-10-04 16:45:47 +02:00
8866a1ea0f
Key lifecycle: lifetime limits by scope, rotation with overlap, kinds and expiry in the listing
Ilker ManapandClaude Opus 5.52026-10-04 16:45:05 +02:00
14ea59d83f
Provisioning through SCIM 2.0: accounts and groups kept in step by the identity provider
Ilker ManapandClaude Opus 5.52026-10-04 16:40:15 +02:00
4cf40644c9
Single sign-on through OpenID Connect: accounts, staff roles and memberships from the provider
Ilker ManapandClaude Opus 5.52026-10-04 16:36:15 +02:00
f8fcecd640
Audit log: every change through the admin API and the administration pages, with ctl audit
Ilker ManapandClaude Opus 5.52026-10-04 16:25:56 +02:00
bd4736125b
Do not name a judge model in the error a client gets for a model nobody serves
Ilker ManapandClaude Opus 5.52026-10-04 15:37:52 +02:00
dbcecd1e85
Judge an answer with Llama Guard 3's own prompt, as a completion, not a conversation ending on the answer
Ilker ManapandClaude Opus 5.52026-10-04 15:35:57 +02:00
742d6633e2
Read a judge's verdict after the answer llama.cpp goes on from, and log a verdict that cannot be read
Ilker ManapandClaude Opus 5.52026-10-04 15:33:23 +02:00
d55472c00a
Guardrails in the console: attach and detach on a branch, where each applies from, and a column
Ilker ManapandClaude Opus 5.52026-10-04 15:18:42 +02:00
02aac84ca5
The /admin/guard page, a Guardrails link, and the policies in force on the profile page
Ilker ManapandClaude Opus 5.52026-10-04 15:15:38 +02:00
7e18b8ab53
Show guardrail refusals and stopped answers on the chat page, and the verdict in ctl request
Ilker ManapandClaude Opus 5.52026-10-04 15:13:15 +02:00
2f07b2ba0b
Document guardrails, correct what data.md said leaves the cluster, and regenerate the reference
Ilker ManapandClaude Opus 5.52026-10-04 12:36:37 +02:00
ec737cffb5
no_store: refuse conversations, notes and models outside the cluster under a policy that keeps nothing
Ilker ManapandClaude Opus 5.52026-10-04 12:36:02 +02:00
13e6f19301
Attach guard policies to scopes: store, admin API and ctl guard; run the gate when server.yaml defines policies
Ilker ManapandClaude Opus 5.52026-10-04 12:34:47 +02:00
b4bdd250af
The gate: judge requests and answers by the policies in force, buffered or a sentence at a time
Ilker ManapandClaude Opus 5.52026-10-04 12:31:55 +02:00
ccdc07fe0c
Guard policies: a library in server.yaml, merged so nothing below loosens what is above
Ilker ManapandClaude Opus 5.52026-10-04 12:27:23 +02:00
ee33524599
Find personal data and secrets by validated pattern, and prompt injection through disguises
Ilker ManapandClaude Opus 5.52026-10-04 12:25:20 +02:00
979aa44804
Hook a guard into the router: judge requests before dispatch and filter answers as they are relayed
Ilker ManapandClaude Opus 5.52026-10-04 12:20:24 +02:00
e701f92806
Add the guard interface, and filter hooks in the proxy for buffered and streamed answers
Ilker ManapandClaude Opus 5.52026-10-04 12:16:00 +02:00
191c65e3db
Put etcd and Patroni's API on TLS, and move a running cluster there in maintenance mode
Ilker ManapandClaude Opus 5.52026-10-04 12:07:17 +02:00
bd155c18ac
Give the secrets directory its mode when init hands it to root, so the service can still read the store's address
Ilker ManapandClaude Opus 5.52026-10-04 11:39:49 +02:00
7b6276901a
Keep llmgw's internal ports to the cluster's networks with an nftables table of its own
Ilker ManapandClaude Opus 5.52026-10-04 11:03:46 +02:00
7e57863edd
Hide the mirror's bookkeeping, bound GGUF array nesting, and forget old serve jobs and downloads
Ilker ManapandClaude Opus 5.52026-10-04 10:56:16 +02:00
fdd4dcf841
Serve every page under a hash-based Content-Security-Policy, and update KaTeX to 0.16.47
Ilker ManapandClaude Opus 5.52026-10-04 10:54:04 +02:00
a56ad8e4c4
Keep the control plane's fetches off link-local addresses, and see through NAT64, 6to4 and proxies
Ilker ManapandClaude Opus 5.52026-10-04 10:48:01 +02:00
fbac1aa664
Take only torch from PyTorch's index, and lock its builds with every file's address
Ilker ManapandClaude Opus 5.52026-10-04 10:45:55 +02:00
3b05ec0538
Have engines listen on the management address, and take SGLang's key off its command line
Ilker ManapandClaude Opus 5.52026-10-04 10:36:42 +02:00
9b61bbc983
Revoke a removed node's certificate, and require --api-client with a mirror's mutual TLS
Ilker ManapandClaude Opus 5.52026-10-04 10:34:35 +02:00
44cd58005a
Count sign-in attempts per address and client, guard the sign-in form with a token, ration both forms per address
Ilker ManapandClaude Opus 5.52026-10-04 10:26:29 +02:00
f1b49d0251
Bound idle time, header size and request reading on every listener, and connections on the client port
Ilker ManapandClaude Opus 5.52026-10-04 10:21:15 +02:00
ebf437a36a
Have the setup's sudo grant remove itself within the hour, and keep a target from reading as an ssh option
Ilker ManapandClaude Opus 5.52026-10-04 10:17:52 +02:00
1f406247e4
Expire join codes on the setup's side, bind the join port to its address, trust the peer address
Ilker ManapandClaude Opus 5.52026-10-04 10:16:53 +02:00
20ebd21243
Skip capacity fallbacks the key may not use, and charge what the model that answered costs
Ilker ManapandClaude Opus 5.52026-10-04 10:13:59 +02:00
2b29016d5b
Ask for a password on Patroni's API and give the store TLS, new clusters and existing ones in place
Ilker ManapandClaude Opus 5.52026-10-04 10:11:28 +02:00
2d47427cdc
Run engines as llmgw-engine rather than root, and make the user when a node is added
Ilker ManapandClaude Opus 5.52026-10-04 09:57:41 +02:00
0bd5305e9a
Install only releases signed with the key built in; fetch a node's release on the control plane
Ilker ManapandClaude Opus 5.52026-10-04 09:48:03 +02:00
e01107d525
Write down what the review of 2026-10-04 found to do: security first, then guardrails, then features
Ilker ManapandClaude Fable 5.12026-10-04 09:38:01 +02:00
a1b1663e9d
A btop-like monitor in the console, from Prometheus; node exporters discovered and installed
Ilker ManapandClaude Opus 5.52026-10-04 01:51:05 +02:00