Compare commits
No files matched your search
@@ -9,3 +9,5 @@ coverage.out
|
||||
/terraform-provider-llmgw/terraform-provider-llmgw
|
||||
/terraform-provider-llmgw/examples/.terraform*
|
||||
/terraform-provider-llmgw/examples/terraform.tfstate*
|
||||
# What a development server run from the checkout writes beside it.
|
||||
/secrets/
|
||||
+153
-31
@@ -15,40 +15,51 @@ Conventions this codebase keeps, which every item here must keep too:
|
||||
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
|
||||
cut and nothing is deployed to the PoC cluster without being told to.
|
||||
|
||||
## Open, in order (2026-10-16, after v0.14.7)
|
||||
## Open, in order (2026-10-20)
|
||||
|
||||
What is left, most needed first. The detail is in the item each names.
|
||||
|
||||
1. ~~**J1**~~ — done 2026-10-16: a restart no longer refused for room
|
||||
another restart was giving back.
|
||||
2. ~~**J5**~~ — done 2026-10-16: an upgrade refits an earlier install's
|
||||
judge by itself.
|
||||
3. ~~**I3**~~ — done 2026-10-16: `ctl bench overhead`, figures in the
|
||||
README.
|
||||
4. ~~**I4**~~ — done 2026-10-17: several keys for one outside provider.
|
||||
5. ~~**Active Directory and Entra ID**~~ — done 2026-10-18: `auth.ldap`
|
||||
sign-in and sync, Entra's 200-group overflow kept from taking roles.
|
||||
Still to do: try it against a real domain controller (a Samba AD DC
|
||||
on Proxmox): LDAPS, nested groups, a disabled user suspended by the
|
||||
sync.
|
||||
6. **J2** — a judge the leader adds by itself is sized as init's is. (S)
|
||||
7. **J3** — HA setup puts the judges on nodes of their own. (S)
|
||||
8. **J8** — a judge budget that fits the machine, measured at install. (S)
|
||||
9. **F5** — large imports limited by the embedding engine's slots. (M)
|
||||
10. **I1** — answers to similar questions from the cache. (M)
|
||||
11. **J4** — the model mirror on more than one machine. (M)
|
||||
12. **I2** — the customer's MCP servers behind llmgw. (L)
|
||||
13. Smaller leftovers: audit before/after values, a tenant-admin role and
|
||||
forms hidden from viewers (C1); Parquet and a scheduled push for usage
|
||||
export (C4); English errors on Turkish pages (C24); H1 tried on the PoC;
|
||||
licences in video and audio (H5); nginx in front of several control
|
||||
planes (J7).
|
||||
**Now: what goes wrong where it is installed.**
|
||||
1. ~~**J8**~~: done 2026-10-20. `ctl guard judge check`, and init times the judge. Left of **J3**: its load test
|
||||
against judges on the control plane, which was stopped half way. Judges on a 4-core node of their own refused
|
||||
0/60 at 2 at once, 13/60 at 4 and 42/60 at 10.
|
||||
2. ~~**I2 P2**~~: done 2026-10-20: the guardrails both ways, a rate per caller and server, a breaker. A portal page
|
||||
for the servers is left, with P3 (stdio, per-person OAuth).
|
||||
3. **K**: fixes from the fabric test the operator runs with docs/large-models.md.
|
||||
|
||||
Not code, the operator's to decide or do: the PoC on v0.14.7 (it runs a
|
||||
v0.14.5 dev build; no hurry); where Alertmanager sends (its receiver is
|
||||
"none"). Settled 2026-10-16: Apertus 8B is on demand (min 0); the HA test
|
||||
VMs 157–159, 165, 166 and the e2e VMs 151, 153–156 (a v0.11 HA test from
|
||||
2026-10-03, idle since, holding 10.0.0.160 with VRRP id 51) were removed.
|
||||
**Soon: the customer's experience, and HA.**
|
||||
4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
|
||||
forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
|
||||
5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M)
|
||||
6. **F5**: large imports limited by the embedding engine's slots. (M)
|
||||
|
||||
**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's.
|
||||
|
||||
**Governance.**
|
||||
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||
|
||||
**When there is room.**
|
||||
9. **I1**: answers to similar questions from the cache. (M)
|
||||
10. **H1**: tried on the PoC.
|
||||
11. **C4**: Parquet, and a scheduled push for usage export.
|
||||
|
||||
**When needed.**
|
||||
12. **H5**: licences in video and audio. 13. **H3**: replicas shared, the catalog side. 14. **J7**: nginx in front
|
||||
of several control planes. 15. **J6**: rate limits per process (a note).
|
||||
|
||||
**Done since 2026-10-16:**
|
||||
- J1, J5, I3, I4, J2.
|
||||
- Active Directory and Entra ID, tried against a Samba DC, with the console's Sign-in screen.
|
||||
- J3's code.
|
||||
- K's code and docs/large-models.md.
|
||||
- I2 P1 and the cluster's own read-only MCP server (llmgw:admin).
|
||||
- The node console's Overview; a node's page showing its version and the models on each GPU.
|
||||
|
||||
**Not code, the operator's to decide or do:**
|
||||
- Where Alertmanager sends (its receiver is "none"): with a judge down, nobody hears.
|
||||
- The fabric test for K.
|
||||
- The PoC's control planes run a dev build of main (v0.14.9-11-g4eb49d0); its nodes run v0.14.8.
|
||||
|
||||
---
|
||||
|
||||
@@ -1485,7 +1496,13 @@ Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
|
||||
routing by what a request holds. Not taken: a realtime (WebSocket audio)
|
||||
API, prompt management, an `npx` install.
|
||||
|
||||
### I1. Answers to similar questions, not only the same one (M)
|
||||
### I1. Answers to similar questions, not only the same one (M) — done 2026-10-25
|
||||
`keys cache --similar`, `router.cache.embedding_model`, `internal/router/similar.go`. Changed
|
||||
from the plan: the vectors are in the gateway's memory beside the exact answers, not a pgvector
|
||||
table. The exact cache keeps answer text out of the store on purpose (data.md), and a table of
|
||||
answers would undo that; it also works on SQLite. The usage row marks a similar hit as a hit
|
||||
(no column of its own). Open: the 40-rewordings and eval-suite runs on the PoC.
|
||||
The original item follows.
|
||||
The answer cache (`internal/router/cache.go`, C17) answers a request whose
|
||||
body is the same byte for byte. A help desk asked "how do I reset my
|
||||
password" in forty wordings asks the model forty times.
|
||||
@@ -1509,6 +1526,14 @@ password" in forty wordings asks the model forty times.
|
||||
point of the run with it off.
|
||||
|
||||
### I2. MCP gateway: the customer's tool servers behind llmgw (L)
|
||||
P1 done 2026-10-20: HTTP servers registered (`ctl mcp`, `/api/v1/mcp/servers`,
|
||||
store `mcp_servers`), tools as `server.tool` in `/mcp`, `allowed_tools`
|
||||
grants in the tree (deny by default), credentials in the store or by
|
||||
reference, the caller named in `X-Llmgw-*` headers, time and size bounds,
|
||||
progress passed on, audit (Source `mcp`) and metrics. Left: P2, the
|
||||
guardrails both ways, a per-tool rate limit, a breaker, and a portal page;
|
||||
P3, stdio servers and per-person OAuth. The "done when" with Claude Code
|
||||
against a real server is still to try.
|
||||
`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
|
||||
the file server, a database — each an MCP server, each today configured
|
||||
in every client with its own credentials and nobody seeing what was called.
|
||||
@@ -1662,6 +1687,103 @@ default: two cores are close to it, and a busy machine passes it.
|
||||
- Done when: an install on a machine too slow for its judge says so before
|
||||
its first refused request.
|
||||
|
||||
## L. Taken from Kong's AI Gateway (written 2026-10-25)
|
||||
|
||||
From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
|
||||
3.9, and only six of its AI plugins are open (proxy, prompt template,
|
||||
decorator, regex prompt guard, request and response transformer); the rest
|
||||
is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
|
||||
plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
|
||||
prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
|
||||
unshown), prompt templates and decorators (no one has asked), Konnect's
|
||||
entity model. MCP items are llmgw-80's (I2); they are listed for it.
|
||||
|
||||
### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
|
||||
`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
|
||||
operation, request and response model, max_tokens, temperature, top_p, the
|
||||
session as conversation id, input/output/cached tokens, the engine as
|
||||
provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
|
||||
Prometheus already counted tokens by direction, cached apart. Left: the MCP
|
||||
calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
|
||||
|
||||
### L2. Redacted personal data put back in the answer (S) — done 2026-10-25
|
||||
`restore: true` (`guard.Rules.Restore`), `Verdict.Restore`, `internal/router/restore.go`
|
||||
ahead of the output filter. The original item follows.
|
||||
The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
|
||||
`placeholders`), the same value always the same placeholder. A model that
|
||||
answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
|
||||
- Per policy, `restore: true`: the placeholders of one request kept in memory
|
||||
for its life, and swapped back in the answer, streamed too (a placeholder
|
||||
split across two chunks held until it is whole).
|
||||
- Only into the answer to the request that held them; never stored, never in
|
||||
usage rows or traces.
|
||||
- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
|
||||
policy reaches the model as placeholders and the client reads the real name
|
||||
and address back.
|
||||
|
||||
### L3. A guardrail that is someone else's service (S) — done 2026-10-25
|
||||
`guard.Service` (`internal/guard/service.go`), policy `services`, asked
|
||||
before the judge on input and once on the whole answer. The original item follows.
|
||||
Customers may be required to use one: Azure Content Safety, Lakera, NeMo
|
||||
Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
|
||||
templated HTTP call with an expression for "block".
|
||||
- A detector kind `http`: URL, headers from secret files, a body template
|
||||
with the text, a JSON path that says blocked and one that says why;
|
||||
timeout and `fail` as the judge's. Checked on input, output or both.
|
||||
- Done when: a policy with an `http` detector against a test service blocks
|
||||
what it says to, and a service that does not answer is treated as `fail`
|
||||
says.
|
||||
|
||||
### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
|
||||
`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
|
||||
what `fallbacks` already does (local, then an outside model when no room). Left:
|
||||
routing by cost between several outside providers. The original item follows.
|
||||
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
|
||||
and KV cache. Kong also balances by measured latency, by tokens or cost
|
||||
used, and in priority tiers.
|
||||
- A `latency` scorer: a moving average of time to first token per endpoint.
|
||||
- Fallbacks in tiers: local replicas first, then an outside provider, with
|
||||
the tier said in the usage row.
|
||||
- Done when: with two replicas of one model, one on a slower card, `latency`
|
||||
sends most of the traffic to the faster and the slower still gets some.
|
||||
|
||||
### L5. Prices by kind and by cache (S–M) — done 2026-10-25
|
||||
`cached_per_1k`, charged on the answer's `cached_tokens`. A budget in money was already
|
||||
there (`keys limits --budget`). Not done: cache-write and audio/video token prices —
|
||||
no engine llmgw runs reports them, and no provider it reaches is billed by them yet.
|
||||
The original item follows.
|
||||
`catalog` prices are per million input/output tokens and per picture/mesh.
|
||||
Kong prices cached reads and cache writes apart, and audio, image and video
|
||||
tokens apart, and limits by cost.
|
||||
- Price fields for cached input and by modality; the usage row's cost from
|
||||
them; a budget limit in money as well as tokens (`keys limits --cost`).
|
||||
- Done when: a request with 90% cached prompt costs what the cached rate says.
|
||||
|
||||
### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
|
||||
Kong turns OpenAPI operations into MCP tools. A company with an internal
|
||||
REST API and no MCP server could offer it to agents through `/mcp`:
|
||||
`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
|
||||
operation a tool, under the same grants, audit, guardrails and rate as I2.
|
||||
|
||||
### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
|
||||
Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
|
||||
resource metadata, tokens checked by JWKS against the company's identity
|
||||
provider), and a token exchanged for a short-lived one per upstream server
|
||||
rather than one long-lived secret.
|
||||
|
||||
### L8. A2A (S, when asked)
|
||||
Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
|
||||
llmgw's, calls audited and traced. The standard is young; wait for a
|
||||
customer.
|
||||
|
||||
### L9. llmgw behind Kong, written down (S) — done 2026-10-25
|
||||
docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on;
|
||||
Kong 3.9 counts no completion tokens of a stream. The original item follows.
|
||||
A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
|
||||
`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
|
||||
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
|
||||
container, and what Kong's token accounting sees.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
+213
-1
@@ -18,12 +18,224 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## v0.15.0 — 2026-10-25
|
||||
|
||||
The company's tools and editors through the gateway, and guardrails that
|
||||
can lean on others. Its own MCP servers sit behind llmgw's `/mcp` under the
|
||||
guardrails, a rate and a breaker. VS Code gets the gateway's models from an
|
||||
extension the gateway serves. A guardrail policy can ask an outside
|
||||
service, and can put redacted values back in the answer. Chat goes to the
|
||||
replica answering soonest. A similar question can get the answer already
|
||||
given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
|
||||
GenAI conventions, and Kong is written down from trying it. Also an install
|
||||
guide with every screen, at `/install`.
|
||||
|
||||
**Upgrading.**
|
||||
- Migrations 0049 (`mcp_servers`), 0050 (their rate and breaker) and 0051
|
||||
(`projects.cache_similar`), applied when the control plane starts.
|
||||
- `llmgw server init` now refuses a machine that would serve models without
|
||||
the memory for them: the starter model and the judges need 12 GB on one
|
||||
machine, where 8 GB was said before. `--no-model`, `--no-guard-model` or
|
||||
`--no-node` make it fit. A running cluster is not checked again.
|
||||
- Traces carry the GenAI conventions' `gen_ai.*` attributes beside the
|
||||
`llmgw.*` ones, which stay; docs/install.md, "Tracing", lists them.
|
||||
- Nothing else changes until it is turned on: `services` and `restore` in a
|
||||
guardrail policy, `keys cache --similar` with
|
||||
`router.cache.embedding_model`, `cached_per_1k` in a price.
|
||||
|
||||
### Added
|
||||
|
||||
- **Answers to similar questions.** `llmgw keys cache --similar 0.95` also
|
||||
gives a chat question the answer already given to one like it in the same
|
||||
conversation, compared by `router.cache.embedding_model`, as
|
||||
`x-llmgw-cache: similar` with the similarity. Held in memory beside the
|
||||
exact answers; nothing written to the store but the setting.
|
||||
- **A cached prompt costs the cached rate.** `cached_per_1k` in a price
|
||||
charges the prompt tokens an engine or provider read from its cache (the
|
||||
answer's `cached_tokens`) at their own rate; without it they cost what any
|
||||
prompt token does, as before.
|
||||
- **Chat goes to the replica answering soonest.** The router keeps each
|
||||
replica's time to first token, smoothed over its last answers, and the
|
||||
`chat` profile scores by it beside the queue and the KV cache: of two
|
||||
replicas of one model, the one on the slower card or the busier host gets
|
||||
less, and a replica not yet measured is tried.
|
||||
- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9
|
||||
in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw
|
||||
key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
|
||||
no completion tokens of a streamed answer, so bill from llmgw's usage.
|
||||
- **A guardrail that is someone else's service.** A policy's `services`
|
||||
asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
|
||||
HTTP: a body template, keys from files, dotted paths to where the answer
|
||||
says unsafe and why, on input, output or both, block or flag, and the
|
||||
policy's `fail` when it does not answer.
|
||||
- **Redacted values put back in the answer.** `restore: true` on a guardrail
|
||||
policy's input: the model reads `[EMAIL_1]`, the client reads its own
|
||||
address in the reply, streamed or not. Only the request's own values, held
|
||||
in memory for its life; the output rules read the answer as the client
|
||||
will.
|
||||
- **Traces in the OpenTelemetry GenAI conventions.** An inference request
|
||||
and each attempt at an engine carry `gen_ai.operation.name`, the requested
|
||||
and the answering model, `max_tokens`, `temperature` and `top_p` when set,
|
||||
the session as `gen_ai.conversation.id`, and the input, output and cached
|
||||
tokens, and an attempt the engine that answered as `gen_ai.provider.name`.
|
||||
Tools that read the conventions show llmgw's calls as model calls with no
|
||||
dashboard of their own. No prompt or answer text is ever in a span.
|
||||
- **llmgw for VS Code.** An extension of llmgw's own, served by every gateway
|
||||
at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from
|
||||
the binary itself: plain JavaScript, no dependencies, no build, and so no
|
||||
marketplace needed at a site with no internet. It offers the gateway's chat
|
||||
models to VS Code's language-model API — the chat's model picker, agent
|
||||
mode, and every extension that asks VS Code for a model — with the sizes
|
||||
and the tool and vision capabilities the catalog says, streamed, with tool
|
||||
calls and their results both ways. **VS Code** is a tool on the account
|
||||
page's **Connect your editor**: a key, the one-line install, and an **Open
|
||||
in VS Code** link that hands the key to the extension, which keeps it in
|
||||
VS Code's secret storage. **llmgw: Sign In** opens that page with a state
|
||||
of its own; a link without it is confirmed first, the gateway named.
|
||||
- **An install guide with every screen, at `/install`.** From three bare
|
||||
Ubuntu machines to a first answer: what each machine needs and how to
|
||||
check it, the release fetched and checked, the setup screens one by one,
|
||||
the system's state, and a first request with a key from the account page.
|
||||
The screens are a real three-machine install, recorded and kept as
|
||||
terminal frames — sharp at any size and a few kilobytes — with each
|
||||
session played back on request; the administrator's password and every
|
||||
key were masked before it was kept. `/about` says more of what llmgw does
|
||||
(documents, guardrails, sign-in, high availability, pictures and 3D), and
|
||||
its install steps are the ones of today, leading to the guide.
|
||||
- **The company's MCP tools under the guardrails, a rate and a breaker.**
|
||||
- **The answer:** the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
|
||||
ticket is withheld as a pasted one would be, and personal data is redacted.
|
||||
- **The arguments:** they are read before they leave, so a secret pasted into one does not reach the company system.
|
||||
- **Rate:** `ctl mcp add --rate N` allows each caller N calls a minute to a server.
|
||||
- **Breaker:** five failures in a row take a server out for a minute, its tools off the list.
|
||||
|
||||
- **How fast the judge is, measured.** `llmgw ctl guard judge check` (and `POST /api/v1/guard/judges/check`) works
|
||||
out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
|
||||
and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
|
||||
and what to do: more cores, more judges, a GPU judge, a longer wait, or `fail: open`. `server init` runs it once the
|
||||
judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.
|
||||
|
||||
- **The company's own MCP servers through llmgw's `/mcp`.** `llmgw ctl mcp add jira --url … --secret-file
|
||||
Authorization=FILE` registers one; `/mcp` then lists its tools as `jira.search` beside llmgw's own. Only keys granted
|
||||
them see them: `llmgw keys limits --tools 'jira.*'`, patterns at any level of the tenancy tree. None granted anywhere
|
||||
is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
|
||||
server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
|
||||
on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
|
||||
admin API has `/api/v1/mcp/servers`.
|
||||
- **The cluster as a read-only MCP server.** Registered as `llmgw ctl mcp add llmgw --url llmgw:admin` and granted
|
||||
as `llmgw.*`, its tools say what `llmgw ctl` does:
|
||||
- `cluster_status`, `list_nodes`, `list_models`, `list_replicas`, `explain_placement`, `node_logs`;
|
||||
- nothing that changes the cluster.
|
||||
|
||||
It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
|
||||
The admin API serves the same at `POST /api/v1/mcp`. `ctl mcp add --bearer-file` keeps a token, sent as
|
||||
`Authorization: Bearer`.
|
||||
|
||||
- **Large models across cards and nodes.** For a model of experts on more than one card, the plan now adds vLLM's
|
||||
`--enable-expert-parallel`. It reads the expert count from `config.json` under any of its names, and sizes a model
|
||||
that also reads pictures by its `text_config`. Several nodes are offered in any number, not only powers of two. Each
|
||||
way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
|
||||
shows in `ctl model plan`, in the console's Serve a model, and in a serve job's first step. The serve form checks that
|
||||
a longer context still fits. `ctl model serve --env KEY=VALUE` puts engine variables such as `NCCL_DEBUG=INFO` on
|
||||
every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.
|
||||
|
||||
- **Machines of the judges' own in a three-machine install.** The setup screen asks for up to two judge machines after
|
||||
the store's two; `server init` takes `--judge-node user@host` for each. Each is added as a node labelled
|
||||
`llmgw.role: judge`, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
|
||||
control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
|
||||
of their own, 2 were. A node with an `llmgw.role` label takes only profiles whose node selector asks for that role,
|
||||
so no other model is placed on the judges' machine.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A machine too small for the models it would serve is said so first.**
|
||||
The control plane that is also a node, serving the starter model and two
|
||||
judges, needs about 10.5 GB. On 8 GB the install finished every step and
|
||||
then had no room for the starter. Now the setup's checks fail its memory
|
||||
with what it would serve, and `server init` refuses before it installs
|
||||
anything, naming `--no-model`, `--no-guard-model` and `--no-node`.
|
||||
|
||||
- **Turkish pages are Turkish to the end.**
|
||||
- **Errors:** those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
|
||||
catalog", the settings changed meanwhile, and others) are now in the page's language.
|
||||
- **Percent:** a percentage is written the language's way: `%93` in Turkish, `93%` in English.
|
||||
- **Footer:** the time at the foot of a page is in the language's date form.
|
||||
- **Stale forms:** they are refused with a sentence in that language rather than "bad request".
|
||||
|
||||
- **`llmgw keys limits` no longer clears the limits it was not given.** Setting one limit at a scope wrote the whole
|
||||
row, so `--tools 'jira.*'` alone took away that scope's rate, budget and model list. Only what is given changes now.
|
||||
`--replace` writes every limit, as the console's limits form does.
|
||||
|
||||
- **The plan refused tensor parallelism past a model's KV heads.** Qwen3-235B, with four KV heads, had no way to run
|
||||
on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
|
||||
offered, as is one that divides them.
|
||||
- **A big model on a slow mirror was failed while it downloaded.** The startup timeout counted the download, so
|
||||
hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
|
||||
The timeout now runs from when a node's download last moved.
|
||||
|
||||
- **The judge init installs is no longer replaced by the leader's own.** Just after the control plane started, the
|
||||
leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
|
||||
a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
|
||||
after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.
|
||||
- **A judge the leader serves by itself is sized as init's is.** When a guard policy names a judge the catalog lacks,
|
||||
the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
|
||||
3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and `fail: closed` refused the
|
||||
request. It now gets what `server init` gives its judges: 2 GB, two cores each (fewer on a small node), and no
|
||||
prompt cache. Every judge served through `modelserve` runs with `--cache-ram 0`.
|
||||
|
||||
## v0.14.9 — 2026-10-19
|
||||
|
||||
Sign-in set up from the console, with Active Directory tried before it is
|
||||
kept. A node's console now shows what the machine is doing, and a node's web
|
||||
page shows what it runs. Also fixes found trying Active Directory against a
|
||||
Samba domain controller on the test cluster. Sign-in through it, nested
|
||||
groups and a disabled account suspended by the sync all worked there.
|
||||
|
||||
**Upgrading.**
|
||||
- No migration.
|
||||
- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
|
||||
block written into `server.yaml` on a cluster with a store was never
|
||||
read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
|
||||
again from the console's Sign-in screen.
|
||||
- A Samba DC's own autogenerated certificate is refused (a negative serial
|
||||
and no SAN). Give it one from a CA.
|
||||
|
||||
### Added
|
||||
|
||||
- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
|
||||
Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
|
||||
tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
|
||||
reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
|
||||
password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
|
||||
domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
|
||||
has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
|
||||
- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
|
||||
and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
|
||||
GPUs.
|
||||
- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
|
||||
version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
|
||||
here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
|
||||
itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
|
||||
machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
|
||||
Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
|
||||
- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
|
||||
ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
|
||||
control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
|
||||
- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
|
||||
cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
|
||||
domain controller needs.
|
||||
|
||||
## v0.14.8 — 2026-10-18
|
||||
|
||||
Sign-in with an on-premises Active Directory, and Entra ID made safe for
|
||||
|
||||
@@ -40,7 +40,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -183,6 +184,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
cluster that exists
|
||||
- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
|
||||
Kubernetes with the Helm chart, and why GPU nodes stay outside it
|
||||
- [docs/kong.md](docs/kong.md) — llmgw behind Kong Gateway, or Kong behind
|
||||
llmgw: the configuration, what Kong counts, and which to choose
|
||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||
when a replica spanning nodes will not run
|
||||
@@ -311,8 +314,9 @@ within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
||||
admission off.
|
||||
|
||||
Routing profiles select how endpoints are scored: `chat` favours session
|
||||
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
||||
cache. Pick one per model in the config, or per request with the
|
||||
affinity and the replica that has been answering soonest (its time to first
|
||||
token, smoothed), `batch` favours the shortest queue, `long-context` favours
|
||||
free KV cache. Pick one per model in the config, or per request with the
|
||||
`x-llmgw-profile` header.
|
||||
|
||||
## sim-engine
|
||||
|
||||
@@ -23,6 +23,9 @@
|
||||
{
|
||||
"name": "audit"
|
||||
},
|
||||
{
|
||||
"name": "auth"
|
||||
},
|
||||
{
|
||||
"name": "canaries"
|
||||
},
|
||||
@@ -62,6 +65,9 @@
|
||||
{
|
||||
"name": "keys"
|
||||
},
|
||||
{
|
||||
"name": "mcp"
|
||||
},
|
||||
{
|
||||
"name": "metrics"
|
||||
},
|
||||
@@ -226,6 +232,129 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/auth": {
|
||||
"get": {
|
||||
"operationId": "getAuth",
|
||||
"summary": "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
|
||||
"tags": [
|
||||
"auth"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/AuthView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/auth/ldap/test": {
|
||||
"post": {
|
||||
"operationId": "postAuthLdapTest",
|
||||
"summary": "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
|
||||
"tags": [
|
||||
"auth"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/LDAPTestRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/LDAPTestView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/auth/secrets/{name}": {
|
||||
"put": {
|
||||
"operationId": "putAuthSecretsName",
|
||||
"summary": "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
|
||||
"tags": [
|
||||
"auth"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "ldap-bind, oidc-client-secret or scim-token.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SecretRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/canaries": {
|
||||
"get": {
|
||||
"operationId": "getCanaries",
|
||||
@@ -1155,6 +1284,48 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/guard/judges/check": {
|
||||
"post": {
|
||||
"operationId": "postGuardJudgesCheck",
|
||||
"summary": "How a judge keeps up: three judgments in turn, then a burst at once, against the budget",
|
||||
"tags": [
|
||||
"guard"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/JudgeCheckRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/JudgeCheckView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/jobs": {
|
||||
"get": {
|
||||
"operationId": "getJobs",
|
||||
@@ -1810,6 +1981,215 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/mcp": {
|
||||
"post": {
|
||||
"operationId": "postMcp",
|
||||
"summary": "This cluster as a read-only MCP server: status, nodes, models, replicas, placement, logs",
|
||||
"description": "JSON-RPC: initialize, tools/list, tools/call. Registered with the gateway as llmgw:admin.",
|
||||
"tags": [
|
||||
"mcp"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/mcp/servers": {
|
||||
"get": {
|
||||
"operationId": "getMcpServers",
|
||||
"summary": "The company's MCP servers /mcp serves, and how each is reached",
|
||||
"tags": [
|
||||
"mcp"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/MCPServerView"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/mcp/servers/{name}": {
|
||||
"put": {
|
||||
"operationId": "putMcpServersName",
|
||||
"summary": "Add an MCP server, or change one: its address, headers and the secrets they carry",
|
||||
"tags": [
|
||||
"mcp"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The MCP server's name: its tools are NAME.tool in /mcp.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/MCPServerRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/MCPServerView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"operationId": "deleteMcpServersName",
|
||||
"summary": "Remove an MCP server: its tools leave /mcp",
|
||||
"tags": [
|
||||
"mcp"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The MCP server's name: its tools are NAME.tool in /mcp.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/mcp/servers/{name}/test": {
|
||||
"post": {
|
||||
"operationId": "postMcpServersNameTest",
|
||||
"summary": "Ask an MCP server for its tools, with its credentials, as /mcp would",
|
||||
"tags": [
|
||||
"mcp"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The MCP server's name: its tools are NAME.tool in /mcp.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/MCPTestView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/metrics/range": {
|
||||
"post": {
|
||||
"operationId": "postMetricsRange",
|
||||
@@ -5351,6 +5731,31 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"AuthView": {
|
||||
"type": "object",
|
||||
"description": "AuthView is how people sign in on this gateway.",
|
||||
"properties": {
|
||||
"signup_open": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"oidc": {
|
||||
"$ref": "#/components/schemas/OIDCView",
|
||||
"description": "OIDC is single sign-on through an identity provider: Entra ID, Keycloak, Okta."
|
||||
},
|
||||
"ldap": {
|
||||
"$ref": "#/components/schemas/LDAPView",
|
||||
"description": "LDAP is sign-in checked by a directory: Active Directory."
|
||||
},
|
||||
"scim": {
|
||||
"type": "boolean",
|
||||
"description": "SCIM is whether a provider may keep accounts in step."
|
||||
},
|
||||
"base_url": {
|
||||
"type": "string",
|
||||
"description": "BaseURL is the portal's address, when the settings say it: what a provider's redirect address is made from."
|
||||
}
|
||||
}
|
||||
},
|
||||
"CanaryPercentRequest": {
|
||||
"type": "object",
|
||||
"description": "CanaryPercentRequest changes the share of a model's requests its canary takes.",
|
||||
@@ -5421,6 +5826,21 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"CheckView": {
|
||||
"type": "object",
|
||||
"description": "CheckView is one step of trying a directory.",
|
||||
"properties": {
|
||||
"step": {
|
||||
"type": "string"
|
||||
},
|
||||
"ok": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"detail": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"ClassGPUs": {
|
||||
"type": "object",
|
||||
"description": "ClassGPUs is how many of a node's devices are of one class, and how many of those are free or kept out of placement.",
|
||||
@@ -5573,6 +5993,34 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"DirectorySyncView": {
|
||||
"type": "object",
|
||||
"description": "DirectorySyncView is the last time the leader read the directory.",
|
||||
"properties": {
|
||||
"at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"people": {
|
||||
"type": "integer"
|
||||
},
|
||||
"accounts": {
|
||||
"type": "integer"
|
||||
},
|
||||
"suspended": {
|
||||
"type": "integer"
|
||||
},
|
||||
"reinstated": {
|
||||
"type": "integer"
|
||||
},
|
||||
"roles_changed": {
|
||||
"type": "integer"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"DomainView": {
|
||||
"type": "object",
|
||||
"description": "DomainView is one domain: its nodes, its power and whether it is held.",
|
||||
@@ -6349,6 +6797,70 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"JudgeCheckRequest": {
|
||||
"type": "object",
|
||||
"description": "JudgeCheckRequest asks how a judge keeps up: Burst judgments at once after three in turn; zero is two for each ready replica.",
|
||||
"properties": {
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"burst": {
|
||||
"type": "integer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"JudgeCheckView": {
|
||||
"type": "object",
|
||||
"description": "JudgeCheckView is how long a judgment took, alone and among others, set against the time a request waits for one.",
|
||||
"properties": {
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"budget_ms": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"ready_replicas": {
|
||||
"type": "integer"
|
||||
},
|
||||
"took_ms": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"description": "TookMs are three judgments of the same text, one after another."
|
||||
},
|
||||
"slowest_ms": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"burst": {
|
||||
"type": "integer",
|
||||
"description": "Burst judgments at once: the slowest of them, how many gave no verdict, and how many a second the judges read."
|
||||
},
|
||||
"burst_slowest_ms": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"burst_failed": {
|
||||
"type": "integer"
|
||||
},
|
||||
"judgments_per_second": {
|
||||
"type": "number"
|
||||
},
|
||||
"verdict": {
|
||||
"type": "string",
|
||||
"description": "Verdict is ok, tight (past half the budget alone, or past it among others) or slow (past it alone); Advice what to do when not ok."
|
||||
},
|
||||
"advice": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"JudgeView": {
|
||||
"type": "object",
|
||||
"description": "JudgeView is one judge model the guardrail policies ask, and whether it can answer them.",
|
||||
@@ -6389,6 +6901,11 @@
|
||||
"wanted_replicas": {
|
||||
"type": "integer"
|
||||
},
|
||||
"budget_ms": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "BudgetMs is how long a request waits for it: the shortest of its policies' latency budgets."
|
||||
},
|
||||
"last_error": {
|
||||
"type": "string",
|
||||
"description": "LastError is its last failure to give a verdict, if any."
|
||||
@@ -6826,6 +7343,61 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"LDAPTestRequest": {
|
||||
"type": "object",
|
||||
"description": "LDAPTestRequest is an auth.ldap block to try, as YAML, and the service account's password when it is not kept yet.",
|
||||
"properties": {
|
||||
"settings": {
|
||||
"type": "string"
|
||||
},
|
||||
"password": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"LDAPTestView": {
|
||||
"type": "object",
|
||||
"description": "LDAPTestView is how trying the directory went.",
|
||||
"properties": {
|
||||
"ok": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"checks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/CheckView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"LDAPView": {
|
||||
"type": "object",
|
||||
"description": "LDAPView is auth.ldap as it is in force, and how its last sync went.",
|
||||
"properties": {
|
||||
"urls": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"base_dn": {
|
||||
"type": "string"
|
||||
},
|
||||
"bind_dn": {
|
||||
"type": "string"
|
||||
},
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"off": {
|
||||
"type": "string",
|
||||
"description": "Off is why this control plane could not use the settings, when it could not; local accounts still sign in."
|
||||
},
|
||||
"last_sync": {
|
||||
"$ref": "#/components/schemas/DirectorySyncView"
|
||||
}
|
||||
}
|
||||
},
|
||||
"Live": {
|
||||
"type": "object",
|
||||
"description": "Live is one of the cluster's adapters as its node last reported it.",
|
||||
@@ -6941,6 +7513,107 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"MCPServerRequest": {
|
||||
"type": "object",
|
||||
"description": "MCPServerRequest adds a server or changes one. Secrets are header values to keep in the store, sealed, each then read by reference; HeaderRefs name secrets kept elsewhere (a file, env:, vault:).",
|
||||
"properties": {
|
||||
"url": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"type": "string"
|
||||
},
|
||||
"headers": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"header_refs": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"secrets": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"timeout_seconds": {
|
||||
"type": "integer"
|
||||
},
|
||||
"rate_per_minute": {
|
||||
"type": "integer",
|
||||
"description": "RatePerMinute bounds each caller's calls to its tools; a negative number takes the bound away."
|
||||
}
|
||||
}
|
||||
},
|
||||
"MCPServerView": {
|
||||
"type": "object",
|
||||
"description": "MCPServerView is a server as the API shows it: header values only for the headers sent as written, and for the others the reference read.",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"url": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": {
|
||||
"type": "string"
|
||||
},
|
||||
"headers": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"header_refs": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"timeout_seconds": {
|
||||
"type": "integer"
|
||||
},
|
||||
"rate_per_minute": {
|
||||
"type": "integer"
|
||||
},
|
||||
"created_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"created_by": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"MCPTestView": {
|
||||
"type": "object",
|
||||
"description": "MCPTestView is what the server said when asked: its tools, or why not.",
|
||||
"properties": {
|
||||
"ok": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"tools": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"MembershipRequest": {
|
||||
"type": "object",
|
||||
"description": "MembershipRequest puts a person in an organization, team or project, or with DELETE takes them out.",
|
||||
@@ -7492,6 +8165,27 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"OIDCView": {
|
||||
"type": "object",
|
||||
"description": "OIDCView is auth.oidc as it is in force.",
|
||||
"properties": {
|
||||
"issuer": {
|
||||
"type": "string"
|
||||
},
|
||||
"client_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"create_accounts": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"no_passwords": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"OrgView": {
|
||||
"type": "object",
|
||||
"description": "OrgView is an organization with what is in it.",
|
||||
@@ -8182,6 +8876,15 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"SecretRequest": {
|
||||
"type": "object",
|
||||
"description": "SecretRequest is a secret for the store to keep.",
|
||||
"properties": {
|
||||
"value": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"SettingsRequest": {
|
||||
"type": "object",
|
||||
"description": "SettingsRequest is a new version of the cluster's settings, made from the version Base, which must still be the newest.",
|
||||
@@ -10237,6 +10940,20 @@
|
||||
"bits_per_weight": {
|
||||
"type": "number",
|
||||
"description": "BitsPerWeight is the artifact's precision, 0 when unknown."
|
||||
},
|
||||
"weights_gb": {
|
||||
"type": "number",
|
||||
"description": "What NeedGB is made of, a card's share (or, on the CPU, the node's RAM): the weights, the context's KV cache, and the engine's own."
|
||||
},
|
||||
"context_gb": {
|
||||
"type": "number"
|
||||
},
|
||||
"engine_gb": {
|
||||
"type": "number"
|
||||
},
|
||||
"expert_parallel": {
|
||||
"type": "boolean",
|
||||
"description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -10341,6 +11058,15 @@
|
||||
"context": {
|
||||
"type": "integer"
|
||||
},
|
||||
"kv_bytes_per_token": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "KVBytesPerToken is one token of context, the whole model: what an option needs for another context is worked out from it. 0 is unknown."
|
||||
},
|
||||
"experts": {
|
||||
"type": "integer",
|
||||
"description": "Experts is a mixture-of-experts model's routed experts; 0 for dense."
|
||||
},
|
||||
"gated": {
|
||||
"type": "boolean"
|
||||
},
|
||||
@@ -10490,6 +11216,17 @@
|
||||
"cpu_cores": {
|
||||
"type": "integer",
|
||||
"description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."
|
||||
},
|
||||
"env": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Env is set in the engine's environment, on every node of a replica: NCCL_DEBUG=INFO to see how its cards talk, NCCL_IB_* to steer them. Only the variables engines are tuned with; the catalog refuses others."
|
||||
},
|
||||
"memory_gb": {
|
||||
"type": "integer",
|
||||
"description": "MemoryGB is what a replica on CPUs reserves, in place of the figure worked out from its weights and context: a judge's, whose own sizing is known."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -579,6 +579,15 @@ claude mcp add --transport http llmgw https://llm.example.com/mcp \
|
||||
--header "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
**Your company's tools.** Where an administrator has registered the
|
||||
company's own MCP servers (Jira, a file server, a database) and your key is
|
||||
granted their tools, the same `/mcp` lists them too, each as `server.tool`
|
||||
(`jira.search`). Give your agent only llmgw's `/mcp` and your key: the
|
||||
servers' own credentials stay with llmgw. A tool your key is not granted is
|
||||
not listed, and calling it by name is refused. Each call is in the audit
|
||||
log under your key: which tool, the size of what was sent, how it ended and
|
||||
how long it took, never what it said.
|
||||
|
||||
### Licences you leave out
|
||||
|
||||
A model's licence is in `/v1/models` (`license`: its `id`, `name`, whether
|
||||
|
||||
+39
-4
@@ -2,8 +2,8 @@
|
||||
|
||||
`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
|
||||
with nothing installed it opens the setup screens; on a control plane, the
|
||||
console described here; on a node, the node's own console, which edits
|
||||
`agent.yaml`. Everything on these screens can also be done by command — each
|
||||
console described here; on a node, the node's own console, which shows
|
||||
what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
|
||||
screen says which — so a script never needs them and a person never needs
|
||||
to remember the commands.
|
||||
|
||||
@@ -246,6 +246,28 @@ long a key made there may live, and is the expiry of one made without any
|
||||
Limits are requests and tokens a minute, a budget per day or month, and the
|
||||
models allowed. Each level only narrows what is above it.
|
||||
|
||||
**Sign-in** — how people get in besides a password kept here: single
|
||||
sign-on, Active Directory and provisioning, each on or off, with the
|
||||
directory's last sync (how many people, accounts kept in step, suspended,
|
||||
reinstated).
|
||||
- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
|
||||
and the groups for administrators and operators. A provider other than
|
||||
Entra takes its issuer URL in place of the tenant. The form shows the
|
||||
redirect address to register.
|
||||
- **D** sets up Active Directory: the servers, where people are, the
|
||||
service account and its password, the domain's CA certificate as a PEM
|
||||
file on this machine, the group whose members may sign in, and groups
|
||||
for roles and teams.
|
||||
- Before it is kept, the directory is tried from the control plane: each
|
||||
server reached and bound, how many people may sign in, how many are in
|
||||
each group. If any step fails, nothing is kept and the steps are shown.
|
||||
**T** tries the settings in force.
|
||||
- On the provisioning row, Enter makes a SCIM token, shown once.
|
||||
- The password, the client secret and the token are kept in the store,
|
||||
sealed by the master key when one is set. The settings name them as
|
||||
`store:auth/…`, so each control plane has them with no file put on it.
|
||||
The CA is written into the settings themselves.
|
||||
|
||||
**Guardrails…** on a row lists the guardrail policies server.yaml defines.
|
||||
Each policy is marked in one of three ways:
|
||||
|
||||
@@ -335,5 +357,18 @@ to restart it.
|
||||
|
||||
## The node's console
|
||||
|
||||
On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
|
||||
manual. The cluster is watched and changed from a control plane.
|
||||
On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
|
||||
on the machine itself, so it works without an admin token and while the control
|
||||
plane is out of reach. Refreshed every three seconds, it shows
|
||||
|
||||
- llmgw-agent's state and since when, and the version of llmgw;
|
||||
- each control plane the agent dials, from `agent.yaml` and the list the
|
||||
leader last sent, and whether it can be reached;
|
||||
- each GPU: how busy, its memory, its heat, and the models on it;
|
||||
- the engines running here, from the agent's own records: replica, model,
|
||||
engine, GPUs, port, whether the process is still there, and for how long;
|
||||
- the free space where weights and engines are kept;
|
||||
- the agent's warnings and errors of the last hour.
|
||||
|
||||
Below it, `agent.yaml` in the same settings view, and the manual. The cluster
|
||||
is watched and changed from a control plane.
|
||||
+3
-1
@@ -72,7 +72,9 @@ each answer that may be reused, as the client got it, and a hash of the
|
||||
request it answers, in the gateway process's memory — never in the store,
|
||||
never on disk — for the project's TTL or until pushed out by newer ones
|
||||
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
||||
policy. Forgetting a person does not reach into it; the answers age out by
|
||||
policy. A project that reuses answers to similar questions (`--similar`)
|
||||
also has each question's last user message embedded, and the vector kept
|
||||
beside its answer, in the same memory and for as long. Forgetting a person does not reach into it; the answers age out by
|
||||
the TTL. The store keeps only the setting, and a usage record's mark that it
|
||||
was answered from the cache.
|
||||
|
||||
|
||||
@@ -91,6 +91,21 @@ guard:
|
||||
value is the same placeholder throughout a request).
|
||||
- **`block`** stops it.
|
||||
|
||||
**Putting them back.** With `restore: true` in a policy's `input`, the
|
||||
values a request had redacted are put back in its answer: the model reads
|
||||
"write to [PERSON_1] at [EMAIL_1]", the client reads the name and address
|
||||
it sent. Streamed answers too — a placeholder split across two events is
|
||||
held until it is whole. Only that request's own values, only into its
|
||||
answer; the map is kept in memory for the request and nowhere else. The
|
||||
output rules then read the answer as the client will, so a policy that
|
||||
redacts personal data in answers still does.
|
||||
|
||||
```yaml
|
||||
input:
|
||||
pii: {email: redact, phone: redact}
|
||||
restore: true
|
||||
```
|
||||
|
||||
**Shapes.**
|
||||
|
||||
- `system_prompt` is put first.
|
||||
@@ -258,6 +273,39 @@ themselves.
|
||||
context rather than overflowing it and going unjudged.
|
||||
- **Cache:** the same text is not asked about twice within five minutes.
|
||||
|
||||
## An outside service
|
||||
|
||||
A site required to use a particular guardrail — Azure Content Safety,
|
||||
Lakera Guard, NeMo Guardrails, a classifier of its own — names it in a
|
||||
policy's `services`. It is asked over HTTP beside the patterns and before
|
||||
the judge; its "unsafe" blocks or flags like theirs, and a service that does
|
||||
not answer is what the policy's `fail` says.
|
||||
|
||||
```yaml
|
||||
policies:
|
||||
strict:
|
||||
fail: closed
|
||||
services:
|
||||
- name: lakera
|
||||
url: https://api.lakera.ai/v2/guard
|
||||
header_files: {Authorization: /etc/llmgw/secrets/lakera} # "Bearer …", read each call
|
||||
body: '{"messages": [{"role": "user", "content": {{json .Text}}}]}'
|
||||
blocked: flagged # where the answer says unsafe
|
||||
reason: breakdown.0.detector_type
|
||||
on: both # input, output or both
|
||||
action: block # or flag
|
||||
timeout: 2s
|
||||
```
|
||||
|
||||
`body` is a template given `.Text` (`{{json .Text}}` writes it as a JSON
|
||||
string; empty sends `{"input": …}`). `blocked` and `reason` are dotted paths
|
||||
into the JSON answer, numbers for list items; true, a number above zero, a
|
||||
non-empty list or a string other than "false" is unsafe. On input the
|
||||
service reads the latest user turn; on output the whole answer, once, at its
|
||||
end — a streamed answer released a sentence at a time is then stopped or
|
||||
flagged as the judge's is. The rule recorded is `service:<name>:<reason>`.
|
||||
The service is told the text only, never who sent it.
|
||||
|
||||
## Where a policy applies
|
||||
|
||||
`guard.default` applies everywhere. Others are attached to a part of the
|
||||
|
||||
+179
-7
@@ -199,7 +199,31 @@ a busy one, or a VM without AVX2 (see
|
||||
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
||||
first thing to fail. The choices there are a longer `latency_budget`,
|
||||
`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
|
||||
or `--no-guard-model`. Defamation, specialized advice,
|
||||
or `--no-guard-model`.
|
||||
|
||||
init times the judge once it is up. If a judgment takes more than half the
|
||||
wait, or more than the wait when several come at once, it says so before the
|
||||
next steps, with what to do. The wait itself is not changed without being
|
||||
asked. `llmgw ctl guard judge check` does the same at any time:
|
||||
|
||||
```
|
||||
qwen3guard-0.6b, 2 replica(s) ready, a request waits 5.0 s for it
|
||||
alone: 2.4, 2.3, 2.3 s
|
||||
4 at once: slowest 6.1 s; about 0.7 judgments a second, so about 0.3 requests a second with short answers
|
||||
tight: a busy moment passes the budget
|
||||
```
|
||||
|
||||
In a three-machine install the judges are better on machines of their own.
|
||||
On a 4-core control plane that was also a node, 38 of 60 requests under load
|
||||
were refused for want of a judgment. With the judges on a node of their own
|
||||
(4 cores, two judges of two cores each), 2 were refused. The setup screen asks
|
||||
for up to two **judge machines** after the store's two, or give
|
||||
`--judge-node user@host` to `server init`, once for each. Each is added as a
|
||||
node labelled `llmgw.role: judge`, and the judge's profile selects that
|
||||
label, so the judges run only there and nothing else does. Four cores and
|
||||
8 GB are enough for two judges.
|
||||
|
||||
Defamation, specialized advice,
|
||||
intellectual property and elections are left out of what it refuses: they
|
||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||
|
||||
@@ -367,7 +391,11 @@ by itself. Three things are worth knowing before you edit it:
|
||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||
budgets mean anything. Without a price every request costs zero and a budget
|
||||
never binds; without the defaults, a project created without limits is
|
||||
metered by nothing at all.
|
||||
metered by nothing at all. A price may set `cached_per_1k` beside
|
||||
`prompt_per_1k`: the prompt tokens the engine or provider read from its
|
||||
cache (the answer's `cached_tokens`) are charged at it, the rest at the
|
||||
prompt rate. A provider that bills cached reads at a tenth is
|
||||
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
|
||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||
is empty. Without it, a file with no models is refused as having nothing to
|
||||
serve.
|
||||
@@ -1933,6 +1961,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
An inference request and each `upstream` attempt also carry the
|
||||
[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
|
||||
so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
|
||||
llmgw's calls as model calls without a dashboard made for llmgw:
|
||||
`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
|
||||
`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
|
||||
when the client set them, `gen_ai.conversation.id` (the session),
|
||||
`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
|
||||
`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
|
||||
engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
|
||||
what was answered are never in a span.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
@@ -2118,8 +2158,46 @@ router:
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
### Similar questions
|
||||
|
||||
A help desk asked "how do I reset my password" in forty wordings asks the
|
||||
model forty times under the exact cache. `--similar` also gives a chat
|
||||
question the answer to one like it:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
|
||||
```
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
embedding_model: bge-m3 # an embedding model in the catalog
|
||||
```
|
||||
|
||||
- **What is compared:** the last user message, embedded by
|
||||
`router.cache.embedding_model`, against the questions already answered
|
||||
in the same conversation — the same system prompt, the same earlier turns,
|
||||
tools, `response_format` and sampling settings, the same model and
|
||||
project. A like question in another conversation, or after a tool's
|
||||
result, is not taken for this one. The exact cache is asked first, and an
|
||||
exact hit embeds nothing.
|
||||
- **How alike:** the cosine similarity of the two embeddings, at least the
|
||||
project's `--similar`. 0.95 is strict; below 0.9 different questions start
|
||||
to meet. A question that differs only in a number or a name ("2+2" and
|
||||
"2+3") can be closer than that, so a project whose questions turn on
|
||||
figures should not use it. A question over 2,000 characters is reused only
|
||||
when it is the same.
|
||||
- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
|
||||
and `Age`. Recorded and charged as a hit. The same rules as a hit decide
|
||||
which answers are reused and kept; guardrails judge the request before.
|
||||
- **The embedding** is asked of the model by the gateway on its own account,
|
||||
as a judge is, within 2 seconds; every miss waits for it. Without the
|
||||
model, or when it does not answer, only the same request is reused. The
|
||||
vectors are kept in memory with their answers and counted in
|
||||
`router.cache.max_bytes`.
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
|
||||
and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
@@ -2484,6 +2562,79 @@ the same files and keys as `locales/en/`, and its `_meta.*` entries saying how
|
||||
it writes numbers and dates. The tests refuse a catalogue that lacks a key a
|
||||
page uses.
|
||||
|
||||
## The company's MCP servers
|
||||
|
||||
llmgw's `/mcp` can serve the company's own MCP servers beside its own tools.
|
||||
Each server, a ticket system, a file server or a database, is registered once.
|
||||
Its credentials stay on the control planes, and people's agents are given only
|
||||
`/mcp` and their llmgw key.
|
||||
|
||||
```
|
||||
llmgw ctl mcp add jira --url https://jira.internal/mcp \
|
||||
--description "Acme Jira" --secret-file Authorization=/root/jira-token
|
||||
llmgw ctl mcp test jira # its tools, asked with its credentials
|
||||
llmgw keys limits --scope team --id Acme/support --tools 'jira.*'
|
||||
```
|
||||
|
||||
- **Credentials:**
|
||||
- `--secret-file NAME=FILE` reads a header's value once and keeps it in
|
||||
the store, sealed by the master key, so no file has to be on each
|
||||
control plane.
|
||||
- `--header-ref NAME=REF` reads it at each call from a file, `env:`,
|
||||
`vault:` or `store:` instead.
|
||||
- `--header NAME=VALUE` is for what is not secret.
|
||||
- **Who may call what:** tools are named `server.tool` and granted as
|
||||
patterns (`jira.*`, `db.query`) at any level of the tenancy tree, as
|
||||
models are.
|
||||
- A level that names any narrows what is above it.
|
||||
- Unlike models, **none granted anywhere is none**: a tool reaches into a
|
||||
company system, so it is granted, never assumed.
|
||||
- The console's limits form has the same field.
|
||||
- **The server is told who calls:** `X-Llmgw-Key`, and `X-Llmgw-User` and
|
||||
`X-Llmgw-Project` where there are some. It is called with llmgw's
|
||||
credentials, never the caller's key.
|
||||
- **Bounds:** a call has 60 seconds (`--timeout`) and an answer 4 MB. A
|
||||
server's progress is passed on to a client that takes a stream. `--rate N`
|
||||
allows each caller N calls a minute to a server's tools.
|
||||
- **Guardrails, both ways:** the caller's policies read a tool's answer as
|
||||
the prompt it becomes.
|
||||
- An instruction planted in a ticket is caught as a pasted one would be
|
||||
(`injection: block`).
|
||||
- Personal data is redacted or refused as the policy says. A refused
|
||||
answer does not reach the agent.
|
||||
- The arguments are read before they leave, so a secret pasted into one
|
||||
does not reach the company system.
|
||||
- **A server that keeps failing:** five calls failing in a row take a server
|
||||
out for a minute. Its tools leave the list, and a call is answered at once.
|
||||
- **Record:** every call is in `llmgw ctl audit`, with its arguments' size
|
||||
and a digest, never their text. `llmgw_mcp_tool_calls_total` and
|
||||
`llmgw_mcp_tool_call_seconds` count them by server, tool and outcome.
|
||||
- **Changes:** a server added, changed or removed on one control plane is
|
||||
served by every one within half a minute; its tool list is read again
|
||||
every five minutes. `--disabled` keeps a server but serves none of its
|
||||
tools.
|
||||
|
||||
**This cluster's own, read only.** `llmgw ctl mcp add llmgw --url
|
||||
llmgw:admin` registers the cluster itself as a server: tools that say what
|
||||
`llmgw ctl` does, and change nothing. It is answered in the control plane's
|
||||
own process, with no address and no token to give.
|
||||
|
||||
| Tool | Says |
|
||||
|---|---|
|
||||
| `llmgw.cluster_status` | control planes, store, nodes, models against what they asked for, GPUs, what is unhealthy |
|
||||
| `llmgw.list_nodes` | each node's state, GPUs, load, fabrics, agent version and engines |
|
||||
| `llmgw.list_models` | each model's replicas ready against wanted, profiles, why one is parked |
|
||||
| `llmgw.list_replicas` | each replica's shards: node, GPUs, state (`model` for one model's) |
|
||||
| `llmgw.explain_placement` | why a model is not placed, node by node |
|
||||
| `llmgw.node_logs` | a node's last agent log lines, or an engine's (`source`) |
|
||||
|
||||
Grant it as any other: `llmgw keys limits --scope key --id KEY-ID --tools
|
||||
'llmgw.*'`. It is the same as an administrator's view, so grant it to the
|
||||
people who would have `ctl`.
|
||||
|
||||
Servers over HTTP only for now (the streamable HTTP transport). A server
|
||||
that runs as a local process (stdio) is planned.
|
||||
|
||||
## Single sign-on
|
||||
|
||||
People can sign in through your identity provider — Keycloak, Entra ID,
|
||||
@@ -2491,8 +2642,10 @@ Google, Okta, Authentik, anything that speaks OpenID Connect.
|
||||
|
||||
1. Register a client there with the redirect address
|
||||
`https://<gateway>/login/oidc/callback`.
|
||||
2. Put its secret in a file only root reads.
|
||||
3. Add to `server.yaml`:
|
||||
2. Put its secret in a file the llmgw user reads, on every control plane.
|
||||
3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
|
||||
cluster with a store, `auth` is the cluster's: the same block in
|
||||
`server.yaml` is not read, and the log says so.
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
@@ -2543,6 +2696,9 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
|
||||
|
||||
### Entra ID
|
||||
|
||||
The console sets this up: `sudo llmgw`, People → Sign-in, **E**. What
|
||||
follows is what it writes, and what to do in Entra.
|
||||
|
||||
Entra ID is an OpenID Connect provider like the others, with two things to
|
||||
set in the app registration:
|
||||
|
||||
@@ -2568,7 +2724,14 @@ the secret token to the contents of `auth.scim.token_file`.
|
||||
|
||||
With an on-premises Active Directory, people sign in on the gateway's own
|
||||
form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
|
||||
password. The gateway checks them with the domain controllers over LDAPS:
|
||||
password. The gateway checks them with the domain controllers over LDAPS.
|
||||
|
||||
The console sets this up and tries it first: `sudo llmgw`, People →
|
||||
Sign-in, **D**. It keeps the service account's password in the store
|
||||
(`bind_password_file: store:auth/ldap-bind`) and writes the CA into the
|
||||
settings (`ca:`), so nothing has to be copied to each control plane.
|
||||
By hand it is this block in the cluster's settings (`llmgw ctl config
|
||||
edit`), where files it names must then be on every control plane:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
@@ -2595,6 +2758,13 @@ auth:
|
||||
checked by binding as them, and is never stored.
|
||||
- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
|
||||
Plaintext LDAP is refused, since passwords would cross the network.
|
||||
- **Certificate:** the domain controller's certificate must name it in
|
||||
its subject alternative names: a name in the CN only is refused, as are
|
||||
certificates with a negative serial number. Samba's own autogenerated
|
||||
certificate is both, so a Samba DC needs one from a CA (`tls certfile`
|
||||
in smb.conf). A CA file a control plane cannot read turns directory
|
||||
sign-in off on that control plane, said in its log. Local accounts still
|
||||
sign in.
|
||||
- **Who may sign in:** enabled user objects under `base_dn` that match
|
||||
`user_filter`. The example admits members of `llmgw-users`, nested groups
|
||||
included.
|
||||
@@ -3188,6 +3358,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
|
||||
- `docs/choosing-models.md` — which models to run and how many, if the answer
|
||||
to "we have GPUs, now what" is not already obvious
|
||||
- `docs/large-models.md` — a model on several cards or several nodes: how it
|
||||
is split, what each way needs of a card, and bringing one up on a fabric
|
||||
- `docs/data.md` — what is stored and what leaves the machine, for the review
|
||||
that happens before this is allowed near anyone's documents
|
||||
- `docs/reference.md` — every configuration key, generated from the source, so
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
# llmgw and Kong
|
||||
|
||||
For a site that already runs Kong. The two meet over plain HTTP: Kong in
|
||||
front of llmgw, as one more API it governs, or Kong behind llmgw, as an
|
||||
OpenAI-compatible provider. Nothing of either is built into the other.
|
||||
|
||||
Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
|
||||
and later need a Kong licence) in DB-less mode, in front of llmgw.
|
||||
|
||||
## Kong in front of llmgw
|
||||
|
||||
Two ways, both tried.
|
||||
|
||||
**A plain route.** Kong forwards the request as it came. Tried with chat
|
||||
completions, streamed and not; llmgw's other APIs — Anthropic's
|
||||
`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
|
||||
since Kong does not read them. Kong adds its own authentication, rate limits
|
||||
and logging; llmgw still checks the key, the budget and the guardrails.
|
||||
|
||||
```yaml
|
||||
_format_version: "3.0"
|
||||
services:
|
||||
- name: llmgw
|
||||
url: https://llm.example.com # llmgw's client port
|
||||
routes:
|
||||
- name: llmgw
|
||||
paths: [/llmgw] # /llmgw/v1/chat/completions → /v1/chat/completions
|
||||
```
|
||||
|
||||
**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
|
||||
route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
- name: llmgw-chat
|
||||
url: https://llm.example.com
|
||||
routes:
|
||||
- name: chat
|
||||
paths: [/chat]
|
||||
plugins:
|
||||
- name: ai-proxy
|
||||
config:
|
||||
route_type: llm/v1/chat
|
||||
logging: {log_statistics: true, log_payloads: false}
|
||||
model:
|
||||
provider: openai
|
||||
name: qwen-coder # a model llmgw serves
|
||||
options:
|
||||
upstream_url: https://llm.example.com/v1/chat/completions
|
||||
```
|
||||
|
||||
**Keys.** Without an `auth` block, `ai-proxy` passes the client's
|
||||
`Authorization` header on: each team keeps its own llmgw key, and llmgw's
|
||||
budgets, limits, guardrails and usage stay per team. With
|
||||
`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
|
||||
sends one key of its own instead, and llmgw sees every caller as that key —
|
||||
then budgets per team are Kong's to keep.
|
||||
|
||||
**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
|
||||
records prompt and completion tokens as llmgw did. For a streamed answer
|
||||
Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
|
||||
even with `stream_options.include_usage` set — the usage chunk reaches it
|
||||
and is not read. llmgw's own accounting counts both; use its usage export
|
||||
(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
|
||||
|
||||
**Timeouts.** A long answer from a model still loading outlasts Kong's
|
||||
60-second defaults: set the service's `read_timeout` to the time a model may
|
||||
take to start, as for nginx in [install.md](install.md).
|
||||
|
||||
## Kong behind llmgw
|
||||
|
||||
Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
|
||||
it reaches any provider: a model whose source is `external`.
|
||||
|
||||
```yaml
|
||||
- name: kong-gpt
|
||||
source: external
|
||||
external:
|
||||
url: https://kong.example.com/openai # Kong's route, without /v1
|
||||
model: gpt-4o-mini
|
||||
api_key_file: /etc/llmgw/secrets/kong-consumer.key
|
||||
```
|
||||
|
||||
The key is the one Kong's own authentication asks of a consumer. As a
|
||||
fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
|
||||
room.
|
||||
|
||||
## Which to choose
|
||||
|
||||
Kong in front when the site's API policy says every API goes through Kong:
|
||||
llmgw is then one more upstream, and its per-team accounting still holds
|
||||
when each team keeps its own key. Kong behind when the site already reaches
|
||||
outside providers through Kong and llmgw should too. llmgw does not need
|
||||
Kong for anything it does itself — keys, budgets, guardrails, routing to
|
||||
its own replicas.
|
||||
@@ -0,0 +1,145 @@
|
||||
# Large models: several cards, several nodes
|
||||
|
||||
A model too big for one card runs on several. llmgw chooses how from the
|
||||
model's own figures and the cluster as it is; this page is the rules it
|
||||
follows, the commands, and what to look at when a first one is brought up
|
||||
on new hardware.
|
||||
|
||||
## How a model is split
|
||||
|
||||
In order, the first that fits:
|
||||
|
||||
1. **One card.** Nothing to split.
|
||||
2. **Several cards of one node, tensor parallel.** Every layer is cut
|
||||
across the cards, so they talk for every token: they must be on one
|
||||
NVLink domain. The agent reads `nvidia-smi topo -m` and gives a replica
|
||||
the cards with the most NVLink between them
|
||||
([install.md](install.md#nvlink-if-your-gpus-have-it)).
|
||||
3. **Several nodes, pipeline parallel between them.** Each node holds a
|
||||
run of layers on all its cards (tensor parallel inside it), and only a
|
||||
layer's output crosses to the next node. The nodes must share one
|
||||
InfiniBand or RoCE fabric: a replica never spans two, and never a node
|
||||
with none. Any number of nodes, not only two or four.
|
||||
|
||||
The plan takes the fewest cards, then the fewest nodes, then the better
|
||||
precision up to eight bits.
|
||||
|
||||
**Tensor width and the KV heads.** The cards split the model's KV heads
|
||||
among them, or past their number each card holds a copy of one. So the
|
||||
width divides the KV heads or is a multiple of them: a model with 8 KV
|
||||
heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8.
|
||||
|
||||
**Mixture of experts.** A model of experts (Qwen3-235B-A22B, DeepSeek-V3,
|
||||
Llama 4, Mixtral) on more than one card is served with vLLM's
|
||||
`--enable-expert-parallel`: each card holds whole experts rather than a
|
||||
slice of every one, which is less traffic per token. It is read from
|
||||
`config.json` (`num_experts`, `n_routed_experts`, `num_local_experts`).
|
||||
|
||||
**When it does not fit.** A smaller precision first: an FP8 or AWQ
|
||||
repository of the same model, or a GGUF quantisation on llama.cpp if one
|
||||
node's cards hold it. Then more nodes. `llmgw ctl model plan REPO
|
||||
--precision bf16,fp8` shows each precision's fit from the model's config.
|
||||
|
||||
## What a way needs
|
||||
|
||||
Each way the plan offers says what a replica needs of each card, part by
|
||||
part, and of all its cards together:
|
||||
|
||||
```
|
||||
$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768
|
||||
FILE SIZE BEST WAY A CARD OF ALL CARDS NOW
|
||||
original 131 GB 4 × h100-80g 37.4 GB 72 GB 150 GB free
|
||||
|
||||
Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards.
|
||||
The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another).
|
||||
```
|
||||
|
||||
- **weights:** the files, divided over the cards.
|
||||
- **context:** the KV cache for the context planned, divided likewise.
|
||||
- **engine:** the engine's own: 2 GB and a fiftieth of the weights for vLLM.
|
||||
- **usable:** 90% of a card; the rest is the driver's and fragmentation's.
|
||||
|
||||
The console's **Serve a model** shows the same for every way a file runs.
|
||||
A context longer than planned is checked against it before anything is
|
||||
fetched.
|
||||
|
||||
## Serving one
|
||||
|
||||
Without flags the plan's best way is taken:
|
||||
|
||||
```
|
||||
llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768
|
||||
```
|
||||
|
||||
Or a way given by hand, for instance two nodes of eight:
|
||||
|
||||
```
|
||||
llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \
|
||||
--gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO
|
||||
```
|
||||
|
||||
`--gpus` is the cards on each node and `--nodes` the nodes. The pipeline
|
||||
is the nodes and the tensor width the cards of each.
|
||||
|
||||
`--env KEY=VALUE` goes into the engine's environment on every node:
|
||||
`NCCL_DEBUG=INFO` while bringing a cluster up, `NCCL_IB_*` to steer it.
|
||||
Only the variables engines are tuned with are taken. llmgw already sets
|
||||
these:
|
||||
- `NCCL_IB_HCA` to the fabric's adapters;
|
||||
- `NCCL_IB_GID_INDEX` on RoCE;
|
||||
- `NCCL_SOCKET_IFNAME` to the management network;
|
||||
- for an NVLink domain spanning nodes, `NCCL_MNNVL_ENABLE`.
|
||||
|
||||
The job's first step says what it needs of each card. The weights then
|
||||
come to the mirror once and from it to each node. The startup timeout (20
|
||||
minutes unless the profile's `startup_timeout` says otherwise) runs from
|
||||
when a node's download last moved. So a model of hundreds of gigabytes is
|
||||
not failed while it is still arriving, and one whose download stops is
|
||||
failed that long after.
|
||||
|
||||
## Watching it come up
|
||||
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| The replica and its shards, one per node, each with its download | `llmgw ctl replicas` |
|
||||
| Why it is not placed | `llmgw ctl explain-placement MODEL` |
|
||||
| A node's engine output (NCCL's lines with `NCCL_DEBUG=INFO`) | `llmgw ctl logs NODE --source SHARD` |
|
||||
| The bandwidth between two nodes, rail by rail | `llmgw ctl fabric-test NODE NODE` |
|
||||
| Each node's GPUs, NVLink and adapters | `llmgw doctor` on the node; its page under /admin |
|
||||
| Every step of a start, across nodes | the `replica start` trace ([install.md](install.md)) |
|
||||
|
||||
With `NCCL_DEBUG=INFO`, NCCL says which transport it took between nodes:
|
||||
`NET/IB` is the fabric. `NET/Socket` means it fell back to TCP, and the
|
||||
model will run, slowly.
|
||||
|
||||
## Bringing it up on new hardware
|
||||
|
||||
A first run on a fabric, in order. Each step is worth its output if
|
||||
something goes wrong.
|
||||
|
||||
1. **The nodes.** Add each with its fabric:
|
||||
`llmgw node add host --fabric ib-a:mlx5_0,mlx5_1`. Then
|
||||
`llmgw ctl nodes` shows every node ready, on `ib-a`.
|
||||
2. **The fabric.** `llmgw ctl fabric-test node1 node2`: each rail at its
|
||||
rated speed (a 400 Gb/s rail near 45 GB/s).
|
||||
3. **One node, several cards.** A dense 70B at bf16 on four or eight
|
||||
cards of one node, by the plan:
|
||||
`llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors`.
|
||||
Ready, then a question answered.
|
||||
4. **Two nodes.** The same model forced over two:
|
||||
`--gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO`, under
|
||||
another `--name`. Ready, answering, and `NET/IB` in both shards' logs.
|
||||
5. **Experts.** A mixture-of-experts model on one node's eight cards,
|
||||
for instance `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`: its profile
|
||||
carries `--enable-expert-parallel` (`llmgw ctl models`, or its
|
||||
definition from the admin API).
|
||||
6. **A node lost.** With the two-node replica serving, stop one node's
|
||||
agent (`systemctl stop llmgw-agent`): the replica fails and is placed
|
||||
again once the node is back or another has room. Nothing is left
|
||||
holding cards.
|
||||
|
||||
For each, worth keeping:
|
||||
- the time from serve to ready;
|
||||
- tokens a second from `llmgw ctl bench`;
|
||||
- the plan's per-card figure against what `nvidia-smi` shows once loaded;
|
||||
- anything it said that was wrong.
|
||||
+13
-3
@@ -54,10 +54,20 @@ on the page that creates it and never again.
|
||||
|
||||
### Connecting an editor
|
||||
|
||||
The account page's **Connect your editor** makes a key for Continue, Codex or
|
||||
Cline and shows what the tool needs with that key, this gateway's address and a
|
||||
model already in it:
|
||||
The account page's **Connect your editor** makes a key for VS Code, Continue,
|
||||
Codex or Cline and shows what the tool needs with that key, this gateway's
|
||||
address and a model already in it:
|
||||
|
||||
- **VS Code** — llmgw's own extension, downloaded from the gateway at
|
||||
`/setup/vscode/llmgw.vsix` with this gateway's address already in it. It
|
||||
offers every chat model the key may use to VS Code itself: the chat's model
|
||||
picker, agent mode, and any extension that asks VS Code for a language model
|
||||
(Cline's "VS Code LM API" provider among them), with no setup of their own.
|
||||
**llmgw: Sign In** in VS Code opens this page with VS Code chosen; the page's
|
||||
**Open in VS Code** link hands the key back, and the extension keeps it in
|
||||
VS Code's secret storage. A link the extension did not ask for is confirmed
|
||||
first, with the gateway's address shown, since it decides where code is sent.
|
||||
**llmgw: Enter API Key** takes a pasted one instead.
|
||||
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
|
||||
so their `config.yaml` is not touched.
|
||||
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
|
||||
|
||||
@@ -8,7 +8,7 @@ explains every choice.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 12 GB RAM when it is also a node running the starter model and the two judges, which setup checks before it installs anything; four cores or more, or the judges on machines of their own, keep a judgment within its wait. |
|
||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
||||
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
||||
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
||||
|
||||
@@ -53,6 +53,7 @@ rather than repeated.
|
||||
| `retain_days` | `int` | [Batch](#batch) |
|
||||
| `max_bytes` | `int64` | [Cache](#cache) |
|
||||
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
||||
| `embedding_model` | `string` | [Cache](#cache) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
@@ -151,6 +152,7 @@ rather than repeated.
|
||||
| `urls` | `[]string` | [LDAP](#ldap) |
|
||||
| `start_tls` | `bool` | [LDAP](#ldap) |
|
||||
| `ca_file` | `string` | [LDAP](#ldap) |
|
||||
| `ca` | `string` | [LDAP](#ldap) |
|
||||
| `bind_dn` | `string` | [LDAP](#ldap) |
|
||||
| `bind_password_file` | `string` | [LDAP](#ldap) |
|
||||
| `base_dn` | `string` | [LDAP](#ldap) |
|
||||
@@ -256,6 +258,7 @@ rather than repeated.
|
||||
| `require_capabilities` | `bool` | [Policy](#policy) |
|
||||
| `validate_structured` | `bool` | [Policy](#policy) |
|
||||
| `judge` | `*JudgePolicy` | [Policy](#policy) |
|
||||
| `services` | `[]Service` | [Policy](#policy) |
|
||||
| `stream` | `string` | [Policy](#policy) |
|
||||
| `enabled` | `bool` | [Prefix](#prefix) |
|
||||
| `max_nodes` | `int` | [Prefix](#prefix) |
|
||||
@@ -319,6 +322,7 @@ rather than repeated.
|
||||
| `injection` | `string` | [Rules](#rules) |
|
||||
| `deny` | `[]Pattern` | [Rules](#rules) |
|
||||
| `max_chars` | `int` | [Rules](#rules) |
|
||||
| `restore` | `bool` | [Rules](#rules) |
|
||||
| `token_file` | `string` | [SCIM](#scim) |
|
||||
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
||||
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
||||
@@ -379,6 +383,15 @@ rather than repeated.
|
||||
| `facility` | `Facility` | [Server](#server) |
|
||||
| `federation` | `Federation` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `name` | `string` | [Service](#service) |
|
||||
| `url` | `string` | [Service](#service) |
|
||||
| `header_files` | `map[string]string` | [Service](#service) |
|
||||
| `body` | `string` | [Service](#service) |
|
||||
| `blocked` | `string` | [Service](#service) |
|
||||
| `reason` | `string` | [Service](#service) |
|
||||
| `on` | `string` | [Service](#service) |
|
||||
| `action` | `string` | [Service](#service) |
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
@@ -412,6 +425,7 @@ rather than repeated.
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `cached_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
@@ -450,6 +464,7 @@ rather than repeated.
|
||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `cached_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
@@ -755,6 +770,15 @@ Past it, the answers used longest ago go first.
|
||||
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
unless set.
|
||||
|
||||
### `embedding_model`
|
||||
|
||||
`string`
|
||||
|
||||
EmbeddingModel embeds the questions of projects that reuse answers to
|
||||
similar ones (llmgw keys cache --similar): an embedding model in the
|
||||
catalog. Unset, those projects reuse only the same request's answer.
|
||||
The gateway asks it on its own account, as it asks a judge.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
@@ -1475,6 +1499,14 @@ is refused.
|
||||
CAFile is the authority the servers' certificates are checked against,
|
||||
when it is not the system's: an AD Certificate Services root.
|
||||
|
||||
### `ca`
|
||||
|
||||
`string`
|
||||
|
||||
CA is the same authority written out, PEM: kept in the cluster's
|
||||
settings, so no file has to be put on every control plane. Used in
|
||||
place of CAFile when both are set.
|
||||
|
||||
### `bind_dn`
|
||||
|
||||
`string`
|
||||
@@ -2286,6 +2318,12 @@ Judge has a judge model on the cluster read the text too.
|
||||
|
||||
Takes a [JudgePolicy](#judgepolicy) block.
|
||||
|
||||
### `services`
|
||||
|
||||
`[]Service`
|
||||
|
||||
Services are outside guardrails asked over HTTP, beside the judge.
|
||||
|
||||
### `stream`
|
||||
|
||||
`string`
|
||||
@@ -2760,6 +2798,14 @@ Deny are patterns, matched against the text as a model reads it.
|
||||
|
||||
MaxChars refuses text longer than this; zero is no limit.
|
||||
|
||||
### `restore`
|
||||
|
||||
`bool`
|
||||
|
||||
Restore, on input, puts what redact rewrote as placeholders back into
|
||||
the answer: the model reads "[EMAIL_1]", the client reads its own
|
||||
address. Only the values that request sent, only into its answer.
|
||||
|
||||
## SCIM
|
||||
|
||||
### `token_file`
|
||||
@@ -3267,6 +3313,67 @@ Endpoints is the phase-1 static replica list. It is only consulted when
|
||||
no model declares a profile, which is what makes a router-only smoke test
|
||||
against sim-engines possible without a control plane.
|
||||
|
||||
## Service
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name says which service decided, in the verdict's rule.
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
URL is POSTed the body.
|
||||
|
||||
### `header_files`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
HeaderFiles are headers whose values are read from files each time,
|
||||
so a rotated key needs no restart: {Authorization: /etc/…/lakera}.
|
||||
|
||||
### `body`
|
||||
|
||||
`string`
|
||||
|
||||
Body is the request, a template given .Text; {{json .Text}} writes it
|
||||
as a JSON string. Empty is {"input": <text>}.
|
||||
|
||||
### `blocked`
|
||||
|
||||
`string`
|
||||
|
||||
Blocked is where in the answer the service says the text is unsafe:
|
||||
a dotted path, numbers for list items (results.0.flagged). True, a
|
||||
number above zero, a non-empty list or string other than "false" is
|
||||
unsafe.
|
||||
|
||||
### `reason`
|
||||
|
||||
`string`
|
||||
|
||||
Reason is where it says why, for the verdict; optional.
|
||||
|
||||
### `on`
|
||||
|
||||
`string`
|
||||
|
||||
On: input, output or both. Empty is input.
|
||||
|
||||
### `action`
|
||||
|
||||
`string`
|
||||
|
||||
Action on unsafe: block or flag. Empty is block.
|
||||
|
||||
### `timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Timeout bounds one call. Empty is 3s; past it the policy's fail says.
|
||||
|
||||
## ServingSettings
|
||||
|
||||
### `stopped`
|
||||
@@ -3505,6 +3612,15 @@ is sha256=<hex HMAC of the body>.
|
||||
|
||||
`int64`
|
||||
|
||||
### `cached_per_1k`
|
||||
|
||||
`int64`
|
||||
|
||||
CachedPer1k is what a prompt token costs that the engine or provider
|
||||
read from its cache instead of computing: the part of prompt_tokens
|
||||
the answer counts as cached_tokens. Zero charges those at
|
||||
PromptPer1k, as before there was a rate for them.
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
@@ -3779,6 +3895,14 @@ had rather wait behind people than be turned away.
|
||||
|
||||
`int64`
|
||||
|
||||
### `cached_per_1k`
|
||||
|
||||
`int64`
|
||||
|
||||
CachedPer1k is the cost of a prompt token the engine or provider read
|
||||
from its cache (the answer's cached_tokens, counted within the
|
||||
prompt's). Zero charges them at prompt_per_1k.
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CachedTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,20 +39,37 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0, 0, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
|
||||
if got := p.Cost(1000, 0, 1000, 0, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0, 0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACachedPromptCostsTheCachedRate(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, CachedPer1k: 100, CompletionPer1k: 3000}
|
||||
// 10,000 prompt tokens of which 9,000 were read from the cache: 1,000 at
|
||||
// the prompt rate and 9,000 at the cached one.
|
||||
if got, want := p.Cost(10000, 9000, 0, 0, 0), int64(1000+900); got != want {
|
||||
t.Errorf("a 90%% cached prompt cost %d, want %d", got, want)
|
||||
}
|
||||
// More cached than prompt is an engine's miscount, not a refund.
|
||||
if got := p.Cost(100, 500, 0, 0, 0); got != 10 {
|
||||
t.Errorf("cached beyond the prompt cost %d, want 10", got)
|
||||
}
|
||||
// No cached rate: cached tokens cost what any prompt token does.
|
||||
if got := (Price{PromptPer1k: 1000}).Cost(10000, 9000, 0, 0, 0); got != 10000 {
|
||||
t.Errorf("without a cached rate a cached prompt cost %d, want 10000", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
|
||||
if got := p.Cost(20, 0, 0, 3, 0); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
|
||||
@@ -18,6 +18,11 @@ import (
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// CachedPer1k is what a prompt token costs that the engine or provider
|
||||
// read from its cache instead of computing: the part of prompt_tokens
|
||||
// the answer counts as cached_tokens. Zero charges those at
|
||||
// PromptPer1k, as before there was a rate for them.
|
||||
CachedPer1k int64 `koanf:"cached_per_1k"`
|
||||
// PerImage is what one picture costs, for a model that makes them: a
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
@@ -33,15 +38,24 @@ type Price struct {
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool {
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.CachedPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
}
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
//
|
||||
// cachedTokens are counted within promptTokens, as OpenAI's usage counts
|
||||
// them, and are charged at the cached rate instead of the prompt rate.
|
||||
func (p Price) Cost(promptTokens, cachedTokens, completionTokens, images, meshes int64) int64 {
|
||||
cachedTokens = min(max(cachedTokens, 0), max(promptTokens, 0))
|
||||
cachedRate := p.CachedPer1k
|
||||
if cachedRate == 0 {
|
||||
cachedRate = p.PromptPer1k
|
||||
}
|
||||
return divUp((promptTokens-cachedTokens)*p.PromptPer1k+cachedTokens*cachedRate, 1000) +
|
||||
divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
||||
}
|
||||
|
||||
|
||||
@@ -74,6 +74,13 @@ func NewSensorSampler() *SensorSampler {
|
||||
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
||||
}
|
||||
|
||||
// SampleGPUs reads every card's heat, power and how busy it is now, without
|
||||
// a sampler's ten seconds of memory: for a screen that asks every few seconds
|
||||
// and wants what it asked for. Nil on a machine without nvidia-smi or a card.
|
||||
func SampleGPUs(ctx context.Context) []GPUSensor {
|
||||
return (&SensorSampler{smi: runSMI}).gpus(ctx)
|
||||
}
|
||||
|
||||
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
@@ -12,6 +14,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
project string
|
||||
ttl time.Duration
|
||||
anyAnswer bool
|
||||
similar float64
|
||||
off bool
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
@@ -28,15 +31,25 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
"a request continuing a stored conversation or under a guardrails no_store\n" +
|
||||
"policy. The guardrails judge a reused answer as they judge any other.\n\n" +
|
||||
"Off for every project until this is run. Without --ttl or --off, says how\n" +
|
||||
"it stands. The answers are held in memory, router.cache bounds how much.",
|
||||
"it stands. The answers are held in memory, router.cache bounds how much.\n\n" +
|
||||
"--similar also answers a chat question that is like one already answered:\n" +
|
||||
"its last user message embedded by router.cache.embedding_model, and given\n" +
|
||||
"the answer whose question reaches that cosine similarity (0.95 is strict,\n" +
|
||||
"0.9 looser), as x-llmgw-cache: similar. The system prompt, the earlier\n" +
|
||||
"turns, tools and the sampling settings must still be the same.",
|
||||
Example: " llmgw keys cache --project Acme/ml/train --ttl 1h\n" +
|
||||
" llmgw keys cache --project Acme/ml/eval --ttl 24h --any\n" +
|
||||
" llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95\n" +
|
||||
" llmgw keys cache --project Acme/ml/train --off",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
switch {
|
||||
case off && (ttl != 0 || anyAnswer):
|
||||
return fmt.Errorf("--off, or --ttl with --any: not both")
|
||||
case off && (ttl != 0 || anyAnswer || similar != 0):
|
||||
return fmt.Errorf("--off, or --ttl with --any or --similar: not both")
|
||||
case similar < 0 || similar > 1:
|
||||
return fmt.Errorf("--similar is a cosine similarity between 0 and 1, like 0.95")
|
||||
case similar != 0 && ttl == 0:
|
||||
return fmt.Errorf("--similar goes with --ttl: say how long answers are reused")
|
||||
case ttl < 0:
|
||||
return fmt.Errorf("--ttl is how long an answer is reused, like 1h")
|
||||
case ttl > 0 && ttl < time.Second:
|
||||
@@ -54,7 +67,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
return err
|
||||
}
|
||||
if off || ttl > 0 {
|
||||
if err := s.SetAnswerCache(cmd.Context(), p.ID, ttl, anyAnswer); err != nil {
|
||||
if err := s.SetAnswerCache(cmd.Context(), p.ID, store.AnswerCache{TTL: ttl, Any: anyAnswer, Similar: similar}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
@@ -71,6 +84,9 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s reuses answers for %s: those asked at temperature 0 or with a seed, "+
|
||||
"and embeddings, rerank and score.\n", p.Path, c.TTL)
|
||||
}
|
||||
if c.TTL > 0 && c.Similar > 0 {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Chat questions reaching %.2f similarity to one answered get its answer too.\n", c.Similar)
|
||||
}
|
||||
if off || ttl > 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Gateways see the change within 30 seconds.")
|
||||
}
|
||||
@@ -80,6 +96,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
|
||||
cmd.Flags().DurationVar(&ttl, "ttl", 0, "how long an answer is reused, like 1h")
|
||||
cmd.Flags().BoolVar(&anyAnswer, "any", false, "reuse sampled answers too, not only reproducible ones")
|
||||
cmd.Flags().Float64Var(&similar, "similar", 0, "also answer chat questions this similar to one answered, like 0.95")
|
||||
cmd.Flags().BoolVar(&off, "off", false, "stop reusing answers for this project")
|
||||
return cmd
|
||||
}
|
||||
@@ -186,6 +186,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newCapacityCmd(client, &asJSON),
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newGuardCmd(client, &asJSON),
|
||||
newMCPCmd(client, &asJSON),
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
func newMCPCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "mcp",
|
||||
Short: "The company's own MCP servers, served through llmgw's /mcp",
|
||||
Long: "An MCP server registered here — Jira, a file server, a database — is served\n" +
|
||||
"through llmgw's /mcp beside llmgw's own tools, its tools named SERVER.tool.\n" +
|
||||
"Its credentials stay on the control planes: a client gives only its llmgw\n" +
|
||||
"key. A key sees and calls only the tools granted it, as patterns, with\n" +
|
||||
"`llmgw keys limits --tools 'jira.*'` at any level of the tenancy tree;\n" +
|
||||
"none granted anywhere is none. Every call is in `llmgw ctl audit`.",
|
||||
}
|
||||
c.AddCommand(newMCPAddCmd(client, asJSON), newMCPListCmd(client, asJSON), newMCPRemoveCmd(client),
|
||||
newMCPTestCmd(client, asJSON))
|
||||
return c
|
||||
}
|
||||
|
||||
func newMCPAddCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var req adminapi.MCPServerRequest
|
||||
var headers, refs, secretFiles []string
|
||||
var bearerFile string
|
||||
var disabled bool
|
||||
c := &cobra.Command{
|
||||
Use: "add NAME --url URL",
|
||||
Short: "Register an MCP server, or change one",
|
||||
Example: " llmgw ctl mcp add jira --url https://jira.internal/mcp --bearer-file /root/jira-token\n" +
|
||||
" llmgw ctl mcp add llmgw --url llmgw:admin --description \"this cluster, read only\"\n" +
|
||||
" llmgw ctl mcp add wiki --url https://wiki.internal/mcp --header-ref Authorization=vault:secret/mcp/wiki#token\n" +
|
||||
" llmgw ctl mcp add jira --disabled # off, kept",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var err error
|
||||
if req.Headers, err = pairs(headers, "--header"); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.HeaderRefs, err = pairs(refs, "--header-ref"); err != nil {
|
||||
return err
|
||||
}
|
||||
files, err := pairs(secretFiles, "--secret-file")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// A secret read here and kept in the store, sealed: the file
|
||||
// need not be on every control plane, and is not read again.
|
||||
for header, file := range files {
|
||||
b, err := os.ReadFile(file) //nolint:gosec // a file the operator names
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if req.Secrets == nil {
|
||||
req.Secrets = map[string]string{}
|
||||
}
|
||||
req.Secrets[header] = strings.TrimSpace(string(b))
|
||||
}
|
||||
if bearerFile != "" {
|
||||
b, err := os.ReadFile(bearerFile) //nolint:gosec // a file the operator names
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if req.Secrets == nil {
|
||||
req.Secrets = map[string]string{}
|
||||
}
|
||||
req.Secrets["Authorization"] = "Bearer " + strings.TrimSpace(string(b))
|
||||
}
|
||||
if cmd.Flags().Changed("disabled") {
|
||||
on := !disabled
|
||||
req.Enabled = &on
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
var v adminapi.MCPServerView
|
||||
if err := client().put(cmd.Context(), "/api/v1/mcp/servers/"+args[0], body, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||
}
|
||||
state := "on"
|
||||
if !v.Enabled {
|
||||
state = "off"
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s, %s. Its tools are %s.TOOL in /mcp, to keys granted them "+
|
||||
"(llmgw keys limits --tools '%s.*'). Check it: llmgw ctl mcp test %s\n", v.Name, v.URL, state, v.Name, v.Name, v.Name)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := c.Flags()
|
||||
f.StringVar(&req.URL, "url", "", "the server's streamable HTTP address, https://…/mcp; llmgw:admin for this cluster, read only")
|
||||
f.StringVar(&req.Description, "description", "", "what it is, shown beside its tools")
|
||||
f.StringArrayVar(&headers, "header", nil, "NAME=VALUE sent as written, for what is not secret; repeatable")
|
||||
f.StringArrayVar(&refs, "header-ref", nil, "NAME=REF read at each call: a file on every control plane, env:, vault: or store:; repeatable")
|
||||
f.StringArrayVar(&secretFiles, "secret-file", nil, "NAME=FILE read now and kept in the store, sealed: a token; repeatable")
|
||||
f.StringVar(&bearerFile, "bearer-file", "", "a token FILE, read now and kept sealed, sent as Authorization: Bearer")
|
||||
f.BoolVar(&disabled, "disabled", false, "keep it but serve none of its tools")
|
||||
f.IntVar(&req.TimeoutSeconds, "timeout", 0, "seconds one call may take (default 60)")
|
||||
f.IntVar(&req.RatePerMinute, "rate", 0, "calls a minute each caller may make to its tools (default: no bound; -1 takes one away)")
|
||||
return c
|
||||
}
|
||||
|
||||
func newMCPListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "The MCP servers /mcp serves",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var list []adminapi.MCPServerView
|
||||
if err := client().get(cmd.Context(), "/api/v1/mcp/servers", &list); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(list)
|
||||
}
|
||||
if len(list) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "no MCP servers; add one: llmgw ctl mcp add NAME --url URL")
|
||||
return nil
|
||||
}
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "NAME\tSTATE\tURL\tCREDENTIALS\tDESCRIPTION")
|
||||
for _, s := range list {
|
||||
state := "on"
|
||||
if !s.Enabled {
|
||||
state = "off"
|
||||
}
|
||||
var creds []string
|
||||
for h := range s.HeaderRefs {
|
||||
creds = append(creds, h)
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", s.Name, state, s.URL, dash(strings.Join(creds, ",")), s.Description)
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newMCPRemoveCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "remove NAME",
|
||||
Short: "Remove an MCP server: its tools leave /mcp",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := client().delete(cmd.Context(), "/api/v1/mcp/servers/"+args[0], nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s removed; its tools leave /mcp within half a minute\n", args[0])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newMCPTestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "test NAME",
|
||||
Short: "Ask an MCP server for its tools, with its credentials, as /mcp would",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var v adminapi.MCPTestView
|
||||
if err := client().post(cmd.Context(), "/api/v1/mcp/servers/"+args[0]+"/test", nil, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||
}
|
||||
if !v.OK {
|
||||
return errors.New(args[0] + " did not answer: " + v.Error)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s answers with %d tool(s):\n", args[0], len(v.Tools))
|
||||
for _, t := range v.Tools {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " "+t)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// pairs reads NAME=VALUE flags.
|
||||
func pairs(list []string, flag string) (map[string]string, error) {
|
||||
if len(list) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
out := map[string]string{}
|
||||
for _, p := range list {
|
||||
k, v, ok := strings.Cut(p, "=")
|
||||
if !ok || strings.TrimSpace(k) == "" {
|
||||
return nil, fmt.Errorf("%s %q is not NAME=VALUE", flag, p)
|
||||
}
|
||||
out[strings.TrimSpace(k)] = v
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -33,7 +34,7 @@ func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
"then fetches and serves by itself: " + knownJudgeNames() + ".",
|
||||
}
|
||||
c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client),
|
||||
newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON))
|
||||
newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON), newGuardJudgeCheckCmd(client, asJSON))
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -383,3 +384,70 @@ func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command
|
||||
c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
|
||||
return c
|
||||
}
|
||||
|
||||
func newGuardJudgeCheckCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var model string
|
||||
var burst int
|
||||
c := &cobra.Command{
|
||||
Use: "check",
|
||||
Short: "How a judge keeps up: a judgment alone and among others, against the time a request waits",
|
||||
Long: "Times three judgments of a prompt of about 280 tokens, one after another, then\n" +
|
||||
"a burst of them at once (two for each ready replica unless --burst says),\n" +
|
||||
"and sets both against the policies' latency budget. A judgment past the\n" +
|
||||
"budget is refused under fail: closed; past half of it alone, or past it\n" +
|
||||
"among others, a busy moment refuses requests. Each request is judged once\n" +
|
||||
"for its prompt and again every 600 characters of its answer.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
v, err := checkJudge(cmd.Context(), client(), model, burst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(v)
|
||||
}
|
||||
fmt.Fprint(cmd.OutOrStdout(), judgeCheckText(v))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
|
||||
c.Flags().IntVar(&burst, "burst", 0, "judgments at once (default: two for each ready replica)")
|
||||
return c
|
||||
}
|
||||
|
||||
func checkJudge(ctx context.Context, c *ctlClient, model string, burst int) (adminapi.JudgeCheckView, error) {
|
||||
body, _ := json.Marshal(adminapi.JudgeCheckRequest{Model: model, Burst: burst})
|
||||
var v adminapi.JudgeCheckView
|
||||
err := c.patient(3*time.Minute).post(ctx, "/api/v1/guard/judges/check", body, &v)
|
||||
return v, err
|
||||
}
|
||||
|
||||
// judgeCheckText says a check in a few lines, and what to do when it is not
|
||||
// fine.
|
||||
func judgeCheckText(v adminapi.JudgeCheckView) string {
|
||||
var b strings.Builder
|
||||
took := make([]string, len(v.TookMs))
|
||||
for i, t := range v.TookMs {
|
||||
took[i] = fmt.Sprintf("%.1f", float64(t)/1000)
|
||||
}
|
||||
fmt.Fprintf(&b, "%s, %d replica(s) ready, a request waits %.1f s for it\n", v.Model, v.Ready, float64(v.BudgetMs)/1000)
|
||||
fmt.Fprintf(&b, " alone: %s s\n", strings.Join(took, ", "))
|
||||
fmt.Fprintf(&b, " %d at once: slowest %.1f s", v.Burst, float64(v.BurstSlowestMs)/1000)
|
||||
if v.BurstFailed > 0 {
|
||||
fmt.Fprintf(&b, ", %d gave no verdict", v.BurstFailed)
|
||||
}
|
||||
fmt.Fprintf(&b, "; about %.1f judgments a second, so about %.1f requests a second with short answers\n",
|
||||
v.PerSecond, v.PerSecond/2)
|
||||
switch v.Verdict {
|
||||
case "ok":
|
||||
b.WriteString("ok: well inside the budget\n")
|
||||
case "tight":
|
||||
b.WriteString("tight: a busy moment passes the budget\n")
|
||||
default:
|
||||
b.WriteString("slow: a judgment alone passes the budget\n")
|
||||
}
|
||||
for _, a := range v.Advice {
|
||||
b.WriteString(" - " + a + "\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -32,6 +32,12 @@ type judgeKeeper struct {
|
||||
// tried is when each judge was last asked for, so one that cannot be
|
||||
// fetched is not asked for again on every look.
|
||||
tried map[string]time.Time
|
||||
// grace is how long after its first look the keeper serves no judge
|
||||
// itself: init adds its own judge just after the control plane starts,
|
||||
// sized and placed as the install says, and one the leader added first
|
||||
// was what init then kept. Zero waits for nothing.
|
||||
grace time.Duration
|
||||
first time.Time
|
||||
|
||||
// What refitting an earlier install's judge needs: the catalog to change
|
||||
// its profile in, the replicas to restart, and the nodes' CPUs. Any nil
|
||||
@@ -51,6 +57,9 @@ type judgeKeeper struct {
|
||||
// asked for again.
|
||||
const judgeRetry = 10 * time.Minute
|
||||
|
||||
// judgeGrace is the keeper's grace on a control plane just started.
|
||||
const judgeGrace = 10 * time.Minute
|
||||
|
||||
func (k *judgeKeeper) run(ctx context.Context, every time.Duration) {
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
@@ -84,6 +93,12 @@ func (k *judgeKeeper) check() {
|
||||
have[m.Name] = true
|
||||
k.refit(m)
|
||||
}
|
||||
if k.first.IsZero() {
|
||||
k.first = now()
|
||||
}
|
||||
if now().Sub(k.first) < k.grace {
|
||||
return
|
||||
}
|
||||
busy := map[string]bool{}
|
||||
for _, j := range k.serve.Jobs() {
|
||||
if j.State == modelserve.Running {
|
||||
@@ -110,6 +125,15 @@ func (k *judgeKeeper) check() {
|
||||
}
|
||||
k.tried[name] = now()
|
||||
var opts judgeServeOptions
|
||||
if !kj.GPU {
|
||||
// The cores init would give the same number of judges.
|
||||
var cpus []int
|
||||
if k.nodeCPUs != nil {
|
||||
cpus = k.nodeCPUs()
|
||||
}
|
||||
opts.Replicas = 2
|
||||
opts.Cores = refitCores(opts.Replicas, cpus)
|
||||
}
|
||||
if kj.GPU {
|
||||
if opts.GPUClass = roomiestGPUClass(k.gpus()); opts.GPUClass == "" {
|
||||
k.log.Warn("a policy names a judge that runs on a GPU, and this cluster has none",
|
||||
@@ -119,6 +143,8 @@ func (k *judgeKeeper) check() {
|
||||
}
|
||||
req := kj.serveRequest(opts)
|
||||
req.By = "llmgw (a guard policy names it)"
|
||||
// init, or an operator, may add it while this fetches: theirs stays.
|
||||
req.IfAbsent = true
|
||||
job, err := k.serve.Start(req)
|
||||
if err != nil {
|
||||
k.log.Error("could not start serving a judge a policy names", "judge", name, "err", err)
|
||||
|
||||
@@ -52,7 +52,8 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
|
||||
// The GPU judge alone: the CPU one is there, and one llmgw does not know
|
||||
// is not guessed at.
|
||||
if len(fs.started) != 1 || fs.started[0].Name != "qwen3guard-4b" || fs.started[0].GPUClass != "6000-24g" ||
|
||||
fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU {
|
||||
fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU ||
|
||||
!fs.started[0].IfAbsent {
|
||||
t.Fatalf("started %+v", fs.started)
|
||||
}
|
||||
|
||||
@@ -91,8 +92,12 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
|
||||
lib = guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
|
||||
inCatalog = nil
|
||||
k.gpus = func() []adminapi.ClassUse { return nil }
|
||||
k.nodeCPUs = func() []int { return []int{8, 4} }
|
||||
k.check()
|
||||
if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 {
|
||||
// Sized as init sizes its two judges: 2 GB, and two cores each on a
|
||||
// cluster whose smallest node has four.
|
||||
if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 ||
|
||||
last.MemoryGB != judgeMemoryGB || last.CPUCores != refitCores(2, []int{8, 4}) || last.CPUCores == 0 {
|
||||
t.Errorf("CPU judge: %+v", last)
|
||||
}
|
||||
fs.err = errors.New("the mirror is down")
|
||||
@@ -145,7 +150,7 @@ func TestAnEarlierInstallsJudgeIsRefittedOnce(t *testing.T) {
|
||||
lib := guard.Config{Policies: map[string]guard.Policy{
|
||||
"baseline": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}},
|
||||
}}
|
||||
earlier := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 1)
|
||||
earlier := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 1, nil)
|
||||
earlier.Profiles[0].MinMemoryGB, earlier.Profiles[0].Args = 4, []string{"--ctx-size", "8192"}
|
||||
inCatalog := []catalog.Model{earlier}
|
||||
fp := &fakeProfiles{}
|
||||
@@ -174,7 +179,7 @@ func TestAnEarlierInstallsJudgeIsRefittedOnce(t *testing.T) {
|
||||
}
|
||||
|
||||
// Changed by somebody: an extra arg. Left as it is.
|
||||
mine := guardModel("http://x/g.gguf", 2, 3)
|
||||
mine := guardModel("http://x/g.gguf", 2, 3, nil)
|
||||
mine.Profiles[0].Args = []string{"--ctx-size", "16384"}
|
||||
inCatalog = []catalog.Model{mine}
|
||||
fp.set, k.looked = nil, nil
|
||||
@@ -203,3 +208,30 @@ func TestAJudgeIsRefittedToTheCoresInitWouldGive(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Just after the control plane starts, the judge init is adding is left to
|
||||
// init: the keeper serves one itself only after its grace.
|
||||
func TestTheKeeperLeavesAJustStartedInstallsJudgeToInit(t *testing.T) {
|
||||
lib := guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
|
||||
now := time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC)
|
||||
fs := &fakeServe{}
|
||||
k := &judgeKeeper{
|
||||
library: func() guard.Config { return lib },
|
||||
models: func() []catalog.Model { return nil },
|
||||
serve: fs, grace: judgeGrace,
|
||||
gpus: func() []adminapi.ClassUse { return nil },
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
now: func() time.Time { return now },
|
||||
}
|
||||
k.check()
|
||||
now = now.Add(judgeGrace / 2)
|
||||
k.check()
|
||||
if len(fs.started) != 0 {
|
||||
t.Fatalf("served within its grace: %+v", fs.started)
|
||||
}
|
||||
now = now.Add(judgeGrace)
|
||||
k.check()
|
||||
if len(fs.started) != 1 {
|
||||
t.Errorf("never served after its grace: %d", len(fs.started))
|
||||
}
|
||||
}
|
||||
@@ -142,7 +142,9 @@ func (k knownJudge) serveRequest(o judgeServeOptions) modelserve.Request {
|
||||
}
|
||||
return req
|
||||
}
|
||||
req.CPU, req.CPUCores = true, o.Cores
|
||||
// Sized as init sizes its judge: the memory it measured to need with
|
||||
// no prompt cache, and the cores its rule gives.
|
||||
req.CPU, req.CPUCores, req.MemoryGB = true, o.Cores, judgeMemoryGB
|
||||
if req.Replicas == 0 {
|
||||
req.Replicas = 2
|
||||
}
|
||||
|
||||
+27
-1
@@ -7,6 +7,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"path"
|
||||
"slices"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
@@ -301,7 +302,8 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
var perMember bool
|
||||
var rpm, tpm int
|
||||
var budget float64
|
||||
var models []string
|
||||
var models, tools []string
|
||||
var replace bool
|
||||
|
||||
cmd := &cobra.Command{
|
||||
Use: "limits",
|
||||
@@ -357,7 +359,19 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
return errors.New("--per-member is for --scope org, team or project")
|
||||
}
|
||||
}
|
||||
// Only what is given changes: setting a tool grant must not
|
||||
// clear the scope's rate or budget. --replace writes the row
|
||||
// whole, a flag not given then no limit, as the console's
|
||||
// form does with every field it shows.
|
||||
l := store.Limits{Scope: level, ScopeID: scopeID}
|
||||
if !replace {
|
||||
switch cur, err := s.LimitsFor(cmd.Context(), level, scopeID); {
|
||||
case err == nil:
|
||||
l = cur
|
||||
case !errors.Is(err, store.ErrNotFound):
|
||||
return err
|
||||
}
|
||||
}
|
||||
if cmd.Flags().Changed("rpm") {
|
||||
l.RPM = &rpm
|
||||
}
|
||||
@@ -379,6 +393,14 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
}
|
||||
l.AllowedModels = models
|
||||
}
|
||||
if cmd.Flags().Changed("tools") {
|
||||
for _, p := range tools {
|
||||
if _, err := path.Match(p, "x.y"); err != nil {
|
||||
return fmt.Errorf("--tools %q is not a pattern: %w", p, err)
|
||||
}
|
||||
}
|
||||
l.AllowedTools = tools
|
||||
}
|
||||
if err := s.PutLimits(cmd.Context(), l); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -395,6 +417,10 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
cmd.Flags().Float64Var(&budget, "budget", 0, "budget per period, in whole units as the quotas page shows them")
|
||||
cmd.Flags().StringVar(&period, "period", "month", "day | month")
|
||||
cmd.Flags().StringSliceVar(&models, "models", nil, "allowed models; empty allows none")
|
||||
cmd.Flags().BoolVar(&replace, "replace", false,
|
||||
"write every limit at the scope: one not given is then no limit there (default: only what is given changes)")
|
||||
cmd.Flags().StringSliceVar(&tools, "tools", nil,
|
||||
"the company MCP servers' tools granted here, as patterns: jira.*,db.query; none granted anywhere is none")
|
||||
return cmd
|
||||
}
|
||||
|
||||
|
||||
@@ -69,6 +69,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
|
||||
f.IntVar(&req.TP, "tp", 0, "the older name of --gpus")
|
||||
_ = f.MarkHidden("tp")
|
||||
f.StringToStringVar(&req.Env, "env", nil,
|
||||
"KEY=VALUE in the engine's environment on every node, e.g. NCCL_DEBUG=INFO; repeatable")
|
||||
f.BoolVar(&req.Fallbacks, "fallbacks", false, "also add the next best ways to run it, used when the first is busy")
|
||||
f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
|
||||
f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
|
||||
|
||||
@@ -291,14 +291,14 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
}
|
||||
fmt.Fprintln(out)
|
||||
w := tabTo(out)
|
||||
fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tHOLDS\tOF\tNOW")
|
||||
fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tA CARD\tOF\tALL CARDS\tNOW")
|
||||
for _, a := range p.Artifacts {
|
||||
name := a.Name
|
||||
if a.Shards > 1 && a.Format == "gguf" {
|
||||
name += fmt.Sprintf(" (%d parts)", a.Shards)
|
||||
}
|
||||
if a.Best == nil {
|
||||
fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\n", name, bytesShort(a.Bytes))
|
||||
fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\t\n", name, bytesShort(a.Bytes))
|
||||
continue
|
||||
}
|
||||
b := a.Best
|
||||
@@ -306,11 +306,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
if b.FreeNow {
|
||||
now = "free"
|
||||
}
|
||||
holds := fmt.Sprintf("%.1f GB a card", b.NeedGB)
|
||||
holds, all := fmt.Sprintf("%.1f GB", b.NeedGB), fmt.Sprintf("%.0f GB", b.TotalGB())
|
||||
if b.CPU {
|
||||
holds = fmt.Sprintf("%.1f GB of RAM", b.NeedGB)
|
||||
holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, now)
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
|
||||
}
|
||||
_ = w.Flush()
|
||||
rec, ok := p.Recommended()
|
||||
@@ -318,7 +318,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
|
||||
return errDoesNotFit
|
||||
}
|
||||
fmt.Fprintf(out, "\nBest: %s on %s.\n", rec.Artifact, rec.Label())
|
||||
fmt.Fprintf(out, "\nBest: %s on %s: %s.\n", rec.Artifact, rec.Label(), rec.MemoryText())
|
||||
if p.KVBytesPerToken > 0 {
|
||||
fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n",
|
||||
modelplan.GB(p.KVBytesPerToken*1000))
|
||||
}
|
||||
if len(p.Options) > 1 {
|
||||
fmt.Fprintln(out, "\nEvery way, best first:")
|
||||
w = tabTo(out)
|
||||
@@ -327,7 +331,7 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
if o.FreeNow {
|
||||
now = "free"
|
||||
}
|
||||
fmt.Fprintf(w, " %s\t%s\t%.1f of %.0f GB\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.NeedGB, o.HaveGB, o.Hosts, now)
|
||||
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now)
|
||||
}
|
||||
_ = w.Flush()
|
||||
}
|
||||
|
||||
@@ -131,3 +131,33 @@ func TestKeyBudgetsAreWholeUnits(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A limit given alone changes that limit alone: granting a tool keeps the
|
||||
// scope's rate. --replace writes the row whole, as the console's form does.
|
||||
func TestALimitGivenAloneLeavesTheOthersAtTheScope(t *testing.T) {
|
||||
db := filepath.Join(t.TempDir(), "s.db")
|
||||
mustRun(t, db, "org", "add", "org", "Acme")
|
||||
mustRun(t, db, "org", "add", "team", "platform")
|
||||
mustRun(t, db, "org", "add", "project", "gateway")
|
||||
mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--rpm", "30")
|
||||
mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--tools", "jira.*")
|
||||
read := func() store.Limits {
|
||||
s, err := store.Open(context.Background(), db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
all, _ := s.AllLimits(context.Background())
|
||||
for _, l := range all {
|
||||
return l
|
||||
}
|
||||
return store.Limits{}
|
||||
}
|
||||
if l := read(); l.RPM == nil || *l.RPM != 30 || len(l.AllowedTools) != 1 {
|
||||
t.Errorf("after a tool grant: rpm %v, tools %v", l.RPM, l.AllowedTools)
|
||||
}
|
||||
mustRun(t, db, "keys", "limits", "--replace", "--scope", "project", "--id", "gateway", "--tpm", "1000")
|
||||
if l := read(); l.RPM != nil || l.AllowedTools != nil || l.TPM == nil {
|
||||
t.Errorf("after --replace: %+v", l)
|
||||
}
|
||||
}
|
||||
+25
-15
@@ -76,6 +76,7 @@ func newServerCmd() *cobra.Command {
|
||||
|
||||
cmd := &cobra.Command{
|
||||
Use: "server",
|
||||
Args: cobra.NoArgs,
|
||||
Short: "Run the router (phase 1; the control plane arrives in phase 3)",
|
||||
Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
|
||||
"In this phase the replica list is static: give it in the config file, or\n" +
|
||||
@@ -138,6 +139,26 @@ func newServerCmd() *cobra.Command {
|
||||
_ = shutdownTracing(sctx)
|
||||
}()
|
||||
|
||||
// The ports before anything else: a second server on this machine,
|
||||
// whose ports are taken, must stop here, not go on to compete for
|
||||
// the lease and lead a cluster no request can reach.
|
||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||
}
|
||||
if cfg.MaxConnections > 0 {
|
||||
// Past this, new connections wait in the kernel's backlog
|
||||
// rather than each taking a goroutine and its buffers.
|
||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||
if err != nil {
|
||||
_ = apiLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||
}
|
||||
defer apiLn.Close()
|
||||
defer adminLn.Close()
|
||||
|
||||
rt, err := router.New(cfg, log)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -418,6 +439,9 @@ func newServerCmd() *cobra.Command {
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
api.SetMCP(st)
|
||||
rt.SetAdminMCP(http.HandlerFunc(api.MCP))
|
||||
api.SetAuth(&adminapi.AuthAdmin{Status: rt.AuthStatus, TestLDAP: rt.TestDirectory, Secrets: st, Log: log})
|
||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||
Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
|
||||
// And the same document on the portal, for administrators.
|
||||
@@ -585,6 +609,7 @@ func newServerCmd() *cobra.Command {
|
||||
// A judge a policy names and the catalog lacks is
|
||||
// fetched and served, on the leader.
|
||||
keeper := &judgeKeeper{
|
||||
grace: judgeGrace,
|
||||
library: gate.Library,
|
||||
models: func() []catalog.Model { return admin.Catalog().Models() },
|
||||
serve: serve,
|
||||
@@ -661,21 +686,6 @@ func newServerCmd() *cobra.Command {
|
||||
ReadTimeout: time.Minute,
|
||||
})
|
||||
|
||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||
}
|
||||
if cfg.MaxConnections > 0 {
|
||||
// Past this, new connections wait in the kernel's backlog
|
||||
// rather than each taking a goroutine and its buffers.
|
||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||
if err != nil {
|
||||
_ = apiLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||
}
|
||||
|
||||
// A metrics-only listener, when one is configured. Started after
|
||||
// AdminHandler, which is where the registry it serves is built.
|
||||
var metricsSrv *http.Server
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// `llmgw server check` once ran a whole second server: words after server
|
||||
// that name no subcommand are refused.
|
||||
func TestServerTakesNoWordsItDoesNotKnow(t *testing.T) {
|
||||
cmd := newServerCmd()
|
||||
cmd.SetArgs([]string{"check", "--config", "/nonexistent"})
|
||||
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||
if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "unknown command") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A second server whose port is taken stops at once, before the router or
|
||||
// any control plane is built.
|
||||
func TestAServerWhosePortIsTakenStopsFirst(t *testing.T) {
|
||||
taken, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer taken.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
cmd := newServerCmd()
|
||||
cmd.SetArgs([]string{"--listen", taken.Addr().String(), "--admin-listen", "127.0.0.1:0",
|
||||
"--endpoint", "a=http://127.0.0.1:1=m"})
|
||||
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||
if err := cmd.ExecuteContext(ctx); err == nil || !strings.Contains(err.Error(), "listening on") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
@@ -80,6 +80,13 @@ type serverInit struct {
|
||||
// setup screen does with the password it was given; when set, patroni
|
||||
// only names them.
|
||||
patroniNodes []pgha.Node
|
||||
// judgeMachines are machines for the guardrails' judges alone, added as
|
||||
// nodes labelled for them; judgeCPUs their cores, once added.
|
||||
judgeMachines []judgeMachine
|
||||
judgeCPUs []int
|
||||
// judgeNote is what timing the judge found, said before the next
|
||||
// steps when it is not fine.
|
||||
judgeNote string
|
||||
// noMirror, noNode and noModel turn off the parts of a first install that
|
||||
// make it serve something: a mirror, this machine as a node, and a small
|
||||
// model on it.
|
||||
@@ -234,6 +241,8 @@ func newServerInitCmd() *cobra.Command {
|
||||
f.StringVar(&judgeCats, "judge-categories", strings.Join(defaultJudgeCategories, ","),
|
||||
"what the judge refuses, comma-separated, of: "+strings.Join(judgeCategories, ", "))
|
||||
f.IntVar(&in.judge.Replicas, "judge-replicas", 2, "how many copies of the judge run, each on CPUs")
|
||||
f.Var(judgeMachineFlag{&in.judgeMachines}, "judge-node",
|
||||
"user@host of a machine for the guardrails' judges alone: added as a node, and the judges run only there; repeatable")
|
||||
f.DurationVar(&in.judge.Budget, "judge-budget", 5*time.Second, "how long a request waits for the judge")
|
||||
f.StringVar(&in.language, "language", "", "the portal's language for a browser that asks for neither: en or tr")
|
||||
f.BoolVar(&in.signupOpen, "signup-open", false, "let anyone ask for an account on the portal, to be approved by an administrator")
|
||||
@@ -257,6 +266,9 @@ func (in *serverInit) run(ctx context.Context) error {
|
||||
if err := in.checkBundle(); err != nil {
|
||||
return fmt.Errorf("offline bundle: %w", err)
|
||||
}
|
||||
if err := in.checkMemory(); err != nil {
|
||||
return err
|
||||
}
|
||||
steps := []struct {
|
||||
name string
|
||||
fn func(context.Context) error
|
||||
@@ -317,6 +329,11 @@ func (in *serverInit) run(ctx context.Context) error {
|
||||
if err := in.ensureWebhooks(ctx); err != nil {
|
||||
fmt.Fprintf(in.out, " !! webhooks: %v\n", err)
|
||||
}
|
||||
if hasSystemd && len(in.judgeMachines) > 0 {
|
||||
if err := in.ensureJudgeNodes(ctx); err != nil {
|
||||
return fmt.Errorf("judge nodes: %w", err)
|
||||
}
|
||||
}
|
||||
if in.judgeWanted() && in.configJudgesWith(guardModelName) {
|
||||
// Not fatal: without its judge the baseline policy still checks for
|
||||
// secrets and injection, and a judged request is served flagged.
|
||||
@@ -338,6 +355,9 @@ func (in *serverInit) run(ctx context.Context) error {
|
||||
return fmt.Errorf("standby control planes: %w", err)
|
||||
}
|
||||
}
|
||||
if hasSystemd && in.judgeWanted() && in.configJudgesWith(guardModelName) {
|
||||
in.checkJudge(ctx)
|
||||
}
|
||||
in.nextSteps()
|
||||
return nil
|
||||
}
|
||||
@@ -902,6 +922,9 @@ func (in *serverInit) waitHealthy(ctx context.Context) error {
|
||||
}
|
||||
|
||||
func (in *serverInit) nextSteps() {
|
||||
if in.judgeNote != "" {
|
||||
fmt.Fprint(in.out, "\n"+in.judgeNote)
|
||||
}
|
||||
fmt.Fprint(in.out, `
|
||||
The control plane is up. Next:
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"math"
|
||||
"net"
|
||||
"os"
|
||||
"path"
|
||||
"slices"
|
||||
"strconv"
|
||||
@@ -11,6 +12,8 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||
)
|
||||
|
||||
// What a first install lets a person choose beyond which parts it has: which
|
||||
@@ -154,6 +157,62 @@ func (s starterSpec) minMemoryGB() int {
|
||||
return max(2, int(math.Ceil(s.SizeGB+1.5)))
|
||||
}
|
||||
|
||||
// servingMemoryMB is what this machine needs, beyond what the control plane
|
||||
// does, for the models it will serve itself: the starter's and the judges'
|
||||
// reservations, and a gigabyte more because the scheduler counts a machine's
|
||||
// memory in whole gigabytes, rounded down — 7.75 GB is 7. Zero when it serves
|
||||
// none of them; what names those it does.
|
||||
//
|
||||
// Asked before anything is installed: the 8 GB machine the guide was first
|
||||
// recorded on finished every step and then had no room for the starter, the
|
||||
// one thing a person asks first.
|
||||
func (in *serverInit) servingMemoryMB(judgesElsewhere bool) (mb int, what string) {
|
||||
if in.noNode {
|
||||
return 0, ""
|
||||
}
|
||||
gb := 0
|
||||
var parts []string
|
||||
if !in.noModel && (!in.noMirror || in.mirrorURL != "") {
|
||||
gb += in.starterModelSpec().minMemoryGB()
|
||||
parts = append(parts, "the starter model")
|
||||
}
|
||||
if !in.noGuardModel && !judgesElsewhere {
|
||||
n := in.judgeFor().Replicas
|
||||
gb += n * judgeMemoryGB
|
||||
parts = append(parts, fmt.Sprintf("%d judges", n))
|
||||
}
|
||||
if gb == 0 {
|
||||
return 0, ""
|
||||
}
|
||||
return (gb + 1) * 1024, strings.Join(parts, " and ")
|
||||
}
|
||||
|
||||
// checkMemory refuses, before anything is installed, a machine too small for
|
||||
// the models it would serve as well as the control plane.
|
||||
func (in *serverInit) checkMemory() error {
|
||||
if in.unitPath == "" {
|
||||
return nil // a development run, served by nothing
|
||||
}
|
||||
meminfo, err := os.ReadFile("/proc/meminfo")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
total, ok := pgha.MemTotal(string(meminfo))
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
extra, what := in.servingMemoryMB(len(in.judgeMachines) > 0)
|
||||
if extra == 0 {
|
||||
return nil
|
||||
}
|
||||
have, need := int(total>>20), preflight.Needs[preflight.RoleControl].MemoryMB+extra
|
||||
if have >= need {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("memory: this machine has %d MB; the control plane and %s it would serve need %d. "+
|
||||
"Give it more, or leave them off it: --no-model, --no-guard-model, or --no-node to serve models only on nodes added later", have, what, need)
|
||||
}
|
||||
|
||||
// defaultContext is the context a client may use of the starter.
|
||||
const defaultContext = 4096
|
||||
|
||||
@@ -325,6 +384,9 @@ func (in *serverInit) initFlags() []string {
|
||||
for _, t := range in.patroni {
|
||||
add("--patroni", t)
|
||||
}
|
||||
for _, m := range in.judgeMachines {
|
||||
add("--judge-node", m.target)
|
||||
}
|
||||
switch {
|
||||
case in.mirrorURL != "":
|
||||
add("--mirror", in.mirrorURL)
|
||||
|
||||
@@ -202,7 +202,7 @@ func TestTheJudgesAndTheStarterShareTheCPUsRatherThanEachTakingAll(t *testing.T)
|
||||
t.Errorf("%+v: judge %d starter %d, want %d %d", c.in, got.judge, got.starter, c.judge, c.starter)
|
||||
}
|
||||
}
|
||||
m := guardModel("http://x/g.gguf", 2, 1)
|
||||
m := guardModel("http://x/g.gguf", 2, 1, nil)
|
||||
if p := m.Profiles[0]; p.CPUCores != 1 || p.MinMemoryGB != 2 || !slices.Contains(p.Args, "--cache-ram") {
|
||||
t.Errorf("judge profile: %d cores, %d GB, args %v", p.CPUCores, p.MinMemoryGB, p.Args)
|
||||
}
|
||||
|
||||
@@ -485,7 +485,7 @@ func starterModel(spec starterSpec, ctx, cores int, source string, gpuClasses []
|
||||
|
||||
// guardModel is the judge's catalog entry: CPU only, so it takes no card from
|
||||
// the models people use, on whichever node has room; replicas copies of it.
|
||||
func guardModel(source string, replicas, cores int) catalog.Model {
|
||||
func guardModel(source string, replicas, cores int, selector map[string]string) catalog.Model {
|
||||
return catalog.Model{
|
||||
Name: guardModelName,
|
||||
Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
|
||||
@@ -503,6 +503,8 @@ func guardModel(source string, replicas, cores int) catalog.Model {
|
||||
// that left a 7 GB machine room for only one of the two.
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: judgeMemoryGB, CPUCores: cores,
|
||||
Args: []string{"--ctx-size", "8192", "--cache-ram", "0"},
|
||||
// The judge machines, when the install was given some.
|
||||
NodeSelector: selector,
|
||||
}},
|
||||
}
|
||||
}
|
||||
@@ -524,7 +526,8 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
|
||||
func (in *serverInit) ensureGuardModel(ctx context.Context) error {
|
||||
return in.ensureServed(ctx, guardModelName, guardModelSource, guardModelVersion, false,
|
||||
func(url string, _ []string) catalog.Model {
|
||||
return guardModel(url, in.judgeFor().Replicas, in.cpuPlan().judge)
|
||||
cores, selector := in.judgePlacement()
|
||||
return guardModel(url, in.judgeFor().Replicas, cores, selector)
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -171,7 +171,7 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestTheGuardModelIsAValidJudge(t *testing.T) {
|
||||
m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 2)
|
||||
m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 2, nil)
|
||||
c, err := catalog.New([]catalog.Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("not a valid catalog entry: %v", err)
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// Machines of the judges' own. Under load on a 4-core control plane that was
|
||||
// also a node, 38 of 60 requests were refused for want of a judgment; with the
|
||||
// judges on a node of their own the same test refused 2. A judge machine is
|
||||
// added as a node carrying judgeRoleLabel, and the judge's profile selects
|
||||
// that label, so nothing else is placed beside it and it on nothing else.
|
||||
|
||||
// judgeRoleLabel marks a node as the judges'.
|
||||
const (
|
||||
judgeRoleLabel = placement.RoleLabel
|
||||
judgeRoleValue = "judge"
|
||||
)
|
||||
|
||||
// judgeMachine is a machine for the judges: user@host, and the connection the
|
||||
// setup screen already made to it, when it made one.
|
||||
type judgeMachine struct {
|
||||
target string
|
||||
remote onboard.Remote
|
||||
}
|
||||
|
||||
// ensureJudgeNodes adds each judge machine as a node labelled for the
|
||||
// judges, and says how many cores the smallest of them has.
|
||||
func (in *serverInit) ensureJudgeNodes(ctx context.Context) error {
|
||||
if len(in.judgeMachines) == 0 {
|
||||
return nil
|
||||
}
|
||||
na := nodeAdd{serverCfg: in.configPath(), plain: true}
|
||||
na.opts.PKI = in.pkiDir
|
||||
na.opts.NoFirewall = in.noFirewall
|
||||
if err := na.prepare(); err != nil {
|
||||
return err
|
||||
}
|
||||
in.judgeCPUs = nil
|
||||
for _, jm := range in.judgeMachines {
|
||||
o := na.target(jm.target)
|
||||
fmt.Fprintf(in.out, " .. adding %s as a node for the judges\n", o.Addr)
|
||||
r := onboard.New(o, nil)
|
||||
if jm.remote != nil {
|
||||
r.Remote = jm.remote
|
||||
}
|
||||
res, err := r.Run(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %w", o.Addr, err)
|
||||
}
|
||||
if err := na.watch(ctx, res.Node, func(onboard.Event) {}); err != nil && err != errNoAdmin {
|
||||
return fmt.Errorf("%s: %w", res.Node, err)
|
||||
}
|
||||
if out, err := r.Remote.Run(ctx, "nproc", nil); err == nil {
|
||||
if n, err := strconv.Atoi(strings.TrimSpace(string(out))); err == nil {
|
||||
in.judgeCPUs = append(in.judgeCPUs, n)
|
||||
}
|
||||
}
|
||||
if _, err := changeStoredSettings(in.configPath(), "a node for the judges, from init", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"agents", "nodes", res.Node, "labels", judgeRoleLabel}, judgeRoleValue)
|
||||
}); err != nil {
|
||||
return fmt.Errorf("labelling %s for the judges: %w", res.Node, err)
|
||||
}
|
||||
in.done("%s is a node for the judges alone (%s: %s)", res.Node, judgeRoleLabel, judgeRoleValue)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// judgePlacement is the judge's cores and where it may run: on the judge
|
||||
// machines when there are some, by init's rule over their cores; otherwise
|
||||
// on any node, with this machine's plan.
|
||||
func (in *serverInit) judgePlacement() (cores int, selector map[string]string) {
|
||||
if len(in.judgeMachines) == 0 {
|
||||
return in.cpuPlan().judge, nil
|
||||
}
|
||||
return refitCores(in.judgeFor().Replicas, in.judgeCPUs), map[string]string{judgeRoleLabel: judgeRoleValue}
|
||||
}
|
||||
|
||||
// judgeMachineFlag is --judge-node, given again for each machine.
|
||||
type judgeMachineFlag struct{ to *[]judgeMachine }
|
||||
|
||||
func (f judgeMachineFlag) String() string {
|
||||
if f.to == nil {
|
||||
return ""
|
||||
}
|
||||
var out []string
|
||||
for _, m := range *f.to {
|
||||
out = append(out, m.target)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
func (f judgeMachineFlag) Set(v string) error {
|
||||
if strings.TrimSpace(v) == "" {
|
||||
return fmt.Errorf("--judge-node wants user@host")
|
||||
}
|
||||
*f.to = append(*f.to, judgeMachine{target: strings.TrimSpace(v)})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (judgeMachineFlag) Type() string { return "user@host" }
|
||||
|
||||
// judgeReadyWait is how long init waits for its judge to load before timing
|
||||
// it: the weights come from the mirror, and llama.cpp loads in seconds.
|
||||
const judgeReadyWait = 15 * time.Minute
|
||||
|
||||
// checkJudge times the judge once it is up, and keeps what it found for the
|
||||
// closing text: a machine too slow for its judge is said before its first
|
||||
// refused request, not after. The budget is not changed without asking.
|
||||
func (in *serverInit) checkJudge(ctx context.Context) {
|
||||
c := in.adminAPI()
|
||||
deadline := time.Now().Add(judgeReadyWait)
|
||||
for {
|
||||
var judges []adminapi.JudgeView
|
||||
if err := c.get(ctx, "/api/v1/guard/judges", &judges); err == nil {
|
||||
ready := false
|
||||
for _, j := range judges {
|
||||
ready = ready || (j.Model == guardModelName && j.Ready > 0)
|
||||
}
|
||||
if ready {
|
||||
break
|
||||
}
|
||||
}
|
||||
if time.Now().After(deadline) || ctx.Err() != nil {
|
||||
in.judgeNote = " !! the judge was not up to be timed; once it is: llmgw ctl guard judge check\n"
|
||||
return
|
||||
}
|
||||
time.Sleep(10 * time.Second)
|
||||
}
|
||||
fmt.Fprintf(in.out, " .. timing the judge\n")
|
||||
v, err := checkJudge(ctx, c, guardModelName, 0)
|
||||
if err != nil {
|
||||
in.judgeNote = " !! the judge could not be timed (" + err.Error() + "); later: llmgw ctl guard judge check\n"
|
||||
return
|
||||
}
|
||||
if v.Verdict == "ok" {
|
||||
in.done("the judge reads a prompt in %.1f s, of the %.1f s a request waits; about %.1f requests a second",
|
||||
float64(v.SlowestMs)/1000, float64(v.BudgetMs)/1000, v.PerSecond/2)
|
||||
return
|
||||
}
|
||||
in.judgeNote = "The judge is " + v.Verdict + " on this cluster:\n" + indent(judgeCheckText(v), " ") + "\n"
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestJudgesGoOnTheirOwnMachinesWhenTheInstallHasSome(t *testing.T) {
|
||||
in := newDefaultServerInit()
|
||||
if _, sel := in.judgePlacement(); sel != nil {
|
||||
t.Errorf("no judge machines, yet a selector: %v", sel)
|
||||
}
|
||||
if err := (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.21"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.22")
|
||||
in.judge.Replicas, in.judgeCPUs = 2, []int{4, 4}
|
||||
cores, sel := in.judgePlacement()
|
||||
if sel[judgeRoleLabel] != judgeRoleValue || cores != refitCores(2, []int{4, 4}) {
|
||||
t.Errorf("cores %d, selector %v", cores, sel)
|
||||
}
|
||||
m := guardModel("http://x/g.gguf", 2, cores, sel)
|
||||
if m.Profiles[0].NodeSelector[judgeRoleLabel] != judgeRoleValue {
|
||||
t.Errorf("the judge's profile does not select its machines: %+v", m.Profiles[0])
|
||||
}
|
||||
// The same from a script.
|
||||
flags := strings.Join(in.initFlags(), " ")
|
||||
if !strings.Contains(flags, "--judge-node ubuntu@10.0.0.21") || !strings.Contains(flags, "--judge-node ubuntu@10.0.0.22") {
|
||||
t.Errorf("flags %s", flags)
|
||||
}
|
||||
if err := (judgeMachineFlag{&in.judgeMachines}).Set(" "); err == nil {
|
||||
t.Error("an empty --judge-node was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSetupAsksForJudgeMachinesWhenChosen(t *testing.T) {
|
||||
m := newSetupModel("ctl", []string{"10.0.0.5"})
|
||||
m.choice, m.page = 1, pMachines
|
||||
m.mfocus = m.shown() * mfCount // the judges' choice
|
||||
m = press(m, "right").(setupModel)
|
||||
if m.judges != 1 || m.shown() != 3 || m.mfocus != 3*mfCount {
|
||||
t.Fatalf("judges %d, shown %d, focus %d", m.judges, m.shown(), m.mfocus)
|
||||
}
|
||||
if v := m.machinesView(); !strings.Contains(v, "Judge machine 1") || !strings.Contains(v, "Check the 4 machines") {
|
||||
t.Errorf("the page lacks the judge machine:\n%s", v)
|
||||
}
|
||||
for i, addr := range []string{"10.0.0.11", "10.0.0.12", ""} {
|
||||
m.db[i].addr.SetValue(addr)
|
||||
m.db[i].user.SetValue("ubuntu")
|
||||
m.db[i].auth = authAgent
|
||||
}
|
||||
if err := m.validateMachines(); !strings.Contains(err, "judge machine 1 has no address") {
|
||||
t.Errorf("err %q", err)
|
||||
}
|
||||
m.db[2].addr.SetValue("10.0.0.21")
|
||||
if err := m.validateMachines(); err != "" {
|
||||
t.Errorf("err %q", err)
|
||||
}
|
||||
if cmd := m.command(); !strings.Contains(cmd, "--judge-node ubuntu@10.0.0.21") || strings.Contains(cmd, "--patroni ubuntu@10.0.0.21") {
|
||||
t.Errorf("command %s", cmd)
|
||||
}
|
||||
// None again: the judge machine is no longer asked for.
|
||||
m = press(m, "left").(setupModel)
|
||||
if m.judges != 0 || strings.Contains(m.machinesView(), "Judge machine 1") {
|
||||
t.Errorf("judges %d after choosing none", m.judges)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||
)
|
||||
|
||||
func TestTheMemoryAMachineNeedsCountsWhatItServes(t *testing.T) {
|
||||
in := &serverInit{}
|
||||
mb, what := in.servingMemoryMB(false)
|
||||
need := preflight.Needs[preflight.RoleControl].MemoryMB + mb
|
||||
if what == "" || need <= 7941 || need > 11961 {
|
||||
t.Errorf("node, starter and judges here: need %d MB (%q); an 8 GB machine must fail and a 12 GB one pass", need, what)
|
||||
}
|
||||
// Judges on machines of their own take their share off this one.
|
||||
if mb2, _ := in.servingMemoryMB(true); mb2 >= mb {
|
||||
t.Errorf("with judge machines %d MB, without %d", mb2, mb)
|
||||
}
|
||||
// A control plane that serves nothing needs only its own.
|
||||
if mb, what := (&serverInit{noNode: true}).servingMemoryMB(false); mb != 0 || what != "" {
|
||||
t.Errorf("no node: %d %q", mb, what)
|
||||
}
|
||||
if mb, _ := (&serverInit{noModel: true, noGuardModel: true}).servingMemoryMB(false); mb != 0 {
|
||||
t.Errorf("nothing served: %d", mb)
|
||||
}
|
||||
}
|
||||
+44
-15
@@ -53,6 +53,7 @@ const (
|
||||
cAccounts
|
||||
cKeys
|
||||
cOrgs
|
||||
cSignIn
|
||||
cAudit
|
||||
cManual
|
||||
cSettings
|
||||
@@ -69,7 +70,7 @@ const (
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
@@ -140,8 +141,8 @@ type consoleModel struct {
|
||||
detail bool
|
||||
scroll int
|
||||
|
||||
// node is the console of a node: its agent.yaml and nothing to watch,
|
||||
// since the cluster is watched from the control plane.
|
||||
// node is the console of a node: what runs on the machine and its
|
||||
// agent.yaml, since the cluster is watched from the control plane.
|
||||
node bool
|
||||
|
||||
// The settings.
|
||||
@@ -181,6 +182,7 @@ type consoleModel struct {
|
||||
quotas []adminapi.QuotaView
|
||||
quotasErr error
|
||||
guard guardState
|
||||
signIn signInState
|
||||
kb kbState
|
||||
|
||||
st *storeHolder
|
||||
@@ -190,6 +192,11 @@ type consoleModel struct {
|
||||
docs docViewer
|
||||
mon monitorState
|
||||
det monitorState
|
||||
|
||||
// nodeOv is a node's overview as last read; nodeBusy says a reading is
|
||||
// under way.
|
||||
nodeOv nodeOverview
|
||||
nodeBusy bool
|
||||
}
|
||||
|
||||
var usageGroupings = []string{"team", "project", "org"}
|
||||
@@ -198,9 +205,6 @@ func newConsoleModel(addr, file string) consoleModel {
|
||||
m := consoleModel{client: newCtlClient(addr), spin: newSpinner(), width: 100, height: 30, file: file,
|
||||
st: &storeHolder{file: file}, docs: newDocViewer()}
|
||||
m.node = strings.HasSuffix(file, "agent.yaml")
|
||||
if m.node {
|
||||
m.menu = cSettings
|
||||
}
|
||||
m.loadSettings()
|
||||
return m
|
||||
}
|
||||
@@ -257,7 +261,7 @@ func (m *consoleModel) loadSettings() {
|
||||
|
||||
func (m consoleModel) Init() tea.Cmd {
|
||||
if m.node {
|
||||
return m.spin.Tick
|
||||
return tea.Batch(m.fetchNodeOverview(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
|
||||
}
|
||||
return tea.Batch(m.fetch(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
|
||||
}
|
||||
@@ -320,7 +324,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
return m, cmd
|
||||
case consoleTickMsg:
|
||||
if m.node {
|
||||
return m, nil
|
||||
return m.nodeTick()
|
||||
}
|
||||
cmds := []tea.Cmd{m.fetch(), tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })}
|
||||
if m.menu == cUsage {
|
||||
@@ -332,6 +336,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
if m.menu == cGuard {
|
||||
cmds = append(cmds, m.fetchGuard())
|
||||
}
|
||||
if m.menu == cSignIn {
|
||||
cmds = append(cmds, m.fetchSignIn())
|
||||
}
|
||||
if m.menu == cKB {
|
||||
cmds = append(cmds, m.fetchKB())
|
||||
}
|
||||
@@ -351,6 +358,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
cmds = append(cmds, m.fetchLogs(m.logs.node))
|
||||
}
|
||||
return m, tea.Batch(cmds...)
|
||||
case nodeOverviewMsg:
|
||||
m.nodeOv, m.nodeBusy = nodeOverview(msg), false
|
||||
case consoleDataMsg:
|
||||
m.loaded = true
|
||||
if msg.err != nil {
|
||||
@@ -395,6 +404,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
m.quotas, m.quotasErr = msg.rows, msg.err
|
||||
case consoleKBMsg:
|
||||
m.kb = kbState{view: msg.view, err: msg.err, loaded: true}
|
||||
case consoleSignInMsg:
|
||||
m.signIn = signInState{view: msg.view, err: msg.err, loaded: true, ldap: msg.ldap, oidc: msg.oidc}
|
||||
case consoleGuardMsg:
|
||||
window := m.guard.window
|
||||
m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
|
||||
@@ -423,7 +434,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
|
||||
func (m consoleModel) entryOK(i int) bool {
|
||||
if m.node {
|
||||
return i == cSettings || i == cManual || i == cWipe || i == cLeave
|
||||
return i == cOverview || i == cSettings || i == cManual || i == cWipe || i == cLeave
|
||||
}
|
||||
return i != cReplicate || !m.storeReplicated()
|
||||
}
|
||||
@@ -553,6 +564,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.menuD = m.quotaMenu(m.quotas[m.row])
|
||||
case cGuard:
|
||||
m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
|
||||
case cSignIn:
|
||||
m.menuD = m.signInMenu(m.signInRowKey())
|
||||
case cKB:
|
||||
m.menuD = m.kbMenu(m.kb.view.Collections[m.row])
|
||||
case cMirror:
|
||||
@@ -649,6 +662,19 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.usageExportForm()
|
||||
return m, nil
|
||||
}
|
||||
if m.menu == cSignIn && m.signIn.loaded {
|
||||
m.entraForm()
|
||||
return m, nil
|
||||
}
|
||||
case "d":
|
||||
if m.menu == cSignIn && m.signIn.loaded {
|
||||
m.directoryForm()
|
||||
return m, nil
|
||||
}
|
||||
case "t":
|
||||
if m.menu == cSignIn && m.signIn.view.LDAP != nil {
|
||||
return m, m.tryDirectory()
|
||||
}
|
||||
case "right", "l":
|
||||
if m.menu == cUsage {
|
||||
m.usageBy = (m.usageBy + 1) % len(usageGroupings)
|
||||
@@ -675,6 +701,8 @@ func (m consoleModel) onEnter() tea.Cmd {
|
||||
return m.fetchQuotas()
|
||||
case cGuard:
|
||||
return m.fetchGuard()
|
||||
case cSignIn:
|
||||
return m.fetchSignIn()
|
||||
case cKB:
|
||||
return m.fetchKB()
|
||||
case cMirror:
|
||||
@@ -1053,6 +1081,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.quotas)
|
||||
case cGuard:
|
||||
return len(m.guard.view.Attachments)
|
||||
case cSignIn:
|
||||
return len(signInRows)
|
||||
case cKB:
|
||||
return len(m.kb.view.Collections)
|
||||
case cMirror:
|
||||
@@ -1080,9 +1110,6 @@ func (m consoleModel) rowCount() int {
|
||||
func (m consoleModel) View() string {
|
||||
host, _ := os.Hostname()
|
||||
nav := []navItem{{Label: "Watch", Heading: true}}
|
||||
if m.node {
|
||||
nav = nil
|
||||
}
|
||||
for i := 0; i < cCount; i++ {
|
||||
if m.node && !m.entryOK(i) {
|
||||
continue
|
||||
@@ -1177,6 +1204,9 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
}
|
||||
switch m.menu {
|
||||
case cOverview:
|
||||
if m.node {
|
||||
return m.nodeOverviewPanel(pw)
|
||||
}
|
||||
var b bytes.Buffer
|
||||
c := &cobra.Command{}
|
||||
c.SetOut(&b)
|
||||
@@ -1292,6 +1322,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return m.quotasPanel(pw)
|
||||
case cGuard:
|
||||
return m.guardPanel(pw)
|
||||
case cSignIn:
|
||||
return m.signInPanel(pw)
|
||||
case cKB:
|
||||
return m.kbPanel(pw)
|
||||
case cAccounts, cKeys, cOrgs:
|
||||
@@ -1648,9 +1680,6 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
|
||||
}
|
||||
}
|
||||
menu := cOverview
|
||||
if strings.HasSuffix(file, "agent.yaml") {
|
||||
menu = cSettings
|
||||
}
|
||||
for {
|
||||
m := newConsoleModel(addr, file)
|
||||
m.menu = menu
|
||||
|
||||
+14
-7
@@ -107,11 +107,7 @@ func optionLabel(o modelplan.Option) string {
|
||||
if o.FreeNow {
|
||||
now = sOK.Render("free")
|
||||
}
|
||||
need := fmt.Sprintf("%.1f of %.0f GB a card", o.NeedGB, o.HaveGB)
|
||||
if o.CPU {
|
||||
need = fmt.Sprintf("%.1f GB of RAM", o.NeedGB)
|
||||
}
|
||||
return fmt.Sprintf("%s · %s · %s", o.Label(), need, now)
|
||||
return fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
|
||||
}
|
||||
|
||||
// repoMenu offers a repository's files, each with the best way it runs here;
|
||||
@@ -197,8 +193,15 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
if p.Note != "" {
|
||||
note += "\n" + sWarn.Render("! ") + p.Note
|
||||
}
|
||||
note += fmt.Sprintf("\nPlanned for a %d-token context. The mirror fetches it and checks it, it joins the\n"+
|
||||
"catalog, and a replica is started and asked to say hello.", p.Context)
|
||||
note += fmt.Sprintf("\nMemory for a %d-token context, each way it runs:", p.Context)
|
||||
for _, o := range opts {
|
||||
note += "\n " + o.Label() + ": " + o.MemoryText()
|
||||
}
|
||||
if p.KVBytesPerToken > 0 {
|
||||
note += fmt.Sprintf("\nThe context costs %.2f GB per 1000 tokens, the whole model; a longer one is checked\n"+
|
||||
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
|
||||
}
|
||||
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
|
||||
ctxDefault := "4096"
|
||||
if a.Format == "safetensors" {
|
||||
ctxDefault = strconv.Itoa(p.Context)
|
||||
@@ -234,6 +237,10 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
req.File = a.Name
|
||||
}
|
||||
o := opts[slices.Index(where, v[4])]
|
||||
// Asked for a context other than planned: does it still fit?
|
||||
if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
|
||||
return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
|
||||
}
|
||||
if o.CPU {
|
||||
req.CPU = true
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,715 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
|
||||
// A node's overview: what a bare `sudo llmgw` opens on a machine that runs
|
||||
// only llmgw-agent. Everything on it is read here — the agent's own files,
|
||||
// systemd, the cards and the disks — because a node has no admin token, and
|
||||
// the control plane may be the very thing it cannot reach.
|
||||
|
||||
// nodeCmdTimeout bounds every command and check the overview runs: a screen
|
||||
// refreshed every three seconds cannot wait on a tool that hangs.
|
||||
const nodeCmdTimeout = 2 * time.Second
|
||||
|
||||
// nodeControlPlanesFile is where the agent keeps the control planes the
|
||||
// leader last named (controlPlanesFile in the agent package).
|
||||
var nodeControlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
|
||||
|
||||
// nodeProblemLines is how many of the agent's warnings the overview shows.
|
||||
const nodeProblemLines = 6
|
||||
|
||||
type nodeOverviewMsg nodeOverview
|
||||
|
||||
// nodeOverview is everything the overview shows, read in one go.
|
||||
type nodeOverview struct {
|
||||
at time.Time
|
||||
version string
|
||||
agent agentService
|
||||
planes []planeCheck
|
||||
gpus []nodeGPU
|
||||
// gpuNote says why there are no GPU rows: no card, or no answer.
|
||||
gpuNote string
|
||||
// names is the cards' names by index, which do not change and are asked
|
||||
// for once rather than every refresh.
|
||||
names map[int]string
|
||||
engines []engineRun
|
||||
enginesNote string
|
||||
disks []diskUse
|
||||
problems []string
|
||||
// problemsNote replaces the problems when the journal cannot be read.
|
||||
problemsNote string
|
||||
}
|
||||
|
||||
// agentService is what systemd says of llmgw-agent.
|
||||
type agentService struct {
|
||||
state, sub string
|
||||
since time.Time
|
||||
// note replaces the rest when systemd cannot say.
|
||||
note string
|
||||
}
|
||||
|
||||
// planeCheck is one control plane the agent dials, and whether it answers.
|
||||
type planeCheck struct {
|
||||
addr string
|
||||
// learned is an address the leader named rather than one in agent.yaml.
|
||||
learned bool
|
||||
err string
|
||||
}
|
||||
|
||||
type nodeGPU struct {
|
||||
index int
|
||||
name string
|
||||
// util is -1 where the driver does not report it.
|
||||
util float64
|
||||
memUsedMB, memTotalMB float64
|
||||
tempC float64
|
||||
models []string
|
||||
}
|
||||
|
||||
// engineRun is one engine the supervisor started here, from its state file.
|
||||
type engineRun struct {
|
||||
replica, model, engine string
|
||||
gpus []int
|
||||
port, pid int
|
||||
alive bool
|
||||
started time.Time
|
||||
}
|
||||
|
||||
type diskUse struct {
|
||||
what, path string
|
||||
free, total uint64
|
||||
err string
|
||||
}
|
||||
|
||||
// nodeProbe reads this machine. Every path and tool is a field, so a test
|
||||
// reads a directory of its own instead.
|
||||
type nodeProbe struct {
|
||||
configured []string
|
||||
planesFile string
|
||||
workDir string
|
||||
weightsDir string
|
||||
enginesDir string
|
||||
root bool
|
||||
names map[int]string
|
||||
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
dial func(ctx context.Context, addr string) error
|
||||
gpus func(ctx context.Context) []discovery.GPUSensor
|
||||
cards func(ctx context.Context) ([]discovery.GPU, error)
|
||||
alive func(pid int) bool
|
||||
statfs func(path string) (free, total uint64, err error)
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// newNodeProbe reads the machine agent.yaml describes, as the agent would.
|
||||
func newNodeProbe(cfg any, names map[int]string) nodeProbe {
|
||||
c, ok := cfg.(agent.Config)
|
||||
if !ok {
|
||||
c = agent.DefaultConfig()
|
||||
}
|
||||
var configured []string
|
||||
if c.Server != "" {
|
||||
configured = append(configured, c.Server)
|
||||
}
|
||||
configured = append(configured, c.Servers...)
|
||||
// The same defaults the agent applies as it starts.
|
||||
weights := c.WeightsCache
|
||||
if weights == "" && len(c.ModelDirs) > 0 {
|
||||
weights = c.ModelDirs[0]
|
||||
}
|
||||
if weights == "" {
|
||||
weights = agent.DefaultWeightsCache
|
||||
}
|
||||
enginesDir := c.EnginesDir
|
||||
if enginesDir == "" {
|
||||
enginesDir = provision.DefaultRoot
|
||||
}
|
||||
work := c.Superv.WorkDir
|
||||
if work == "" {
|
||||
work = "/var/lib/llmgw-agent"
|
||||
}
|
||||
return nodeProbe{
|
||||
configured: configured, planesFile: nodeControlPlanesFile, workDir: work,
|
||||
weightsDir: weights, enginesDir: enginesDir, root: os.Geteuid() == 0, names: names,
|
||||
run: runLocal, dial: dialTCP, gpus: discovery.SampleGPUs,
|
||||
cards: func(ctx context.Context) ([]discovery.GPU, error) { return discovery.DiscoverGPUsSMI(ctx, "") },
|
||||
alive: pidAlive, statfs: statfs, now: time.Now,
|
||||
}
|
||||
}
|
||||
|
||||
// fetchNodeOverview reads the machine in the background.
|
||||
func (m consoleModel) fetchNodeOverview() tea.Cmd {
|
||||
p := newNodeProbe(m.cfg, m.nodeOv.names)
|
||||
return func() tea.Msg { return nodeOverviewMsg(p.gather(context.Background())) }
|
||||
}
|
||||
|
||||
// nodeTick refreshes the overview while it is on screen, one reading at a
|
||||
// time; the other screens of a node watch nothing.
|
||||
func (m consoleModel) nodeTick() (consoleModel, tea.Cmd) {
|
||||
tick := tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })
|
||||
if m.menu != cOverview || m.nodeBusy {
|
||||
return m, tick
|
||||
}
|
||||
m.nodeBusy = true
|
||||
return m, tea.Batch(m.fetchNodeOverview(), tick)
|
||||
}
|
||||
|
||||
// gather reads everything at once, each part in its own goroutine, so the
|
||||
// slowest part bounds the whole.
|
||||
func (p nodeProbe) gather(ctx context.Context) nodeOverview {
|
||||
ov := nodeOverview{version: version.Version}
|
||||
var wg sync.WaitGroup
|
||||
do := func(f func()) {
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); f() }()
|
||||
}
|
||||
var engines []engineRun
|
||||
do(func() { ov.agent = p.agentState(ctx) })
|
||||
do(func() { ov.planes = p.checkPlanes(ctx) })
|
||||
do(func() { ov.gpus, ov.names, ov.gpuNote = p.readGPUs(ctx) })
|
||||
do(func() { engines, ov.enginesNote = readShardStates(p.workDir, p.alive) })
|
||||
do(func() { ov.disks = p.readDisks() })
|
||||
do(func() { ov.problems, ov.problemsNote = p.readProblems(ctx) })
|
||||
wg.Wait()
|
||||
ov.engines = engines
|
||||
ov.gpus = modelsOnGPUs(ov.gpus, engines)
|
||||
ov.at = p.now()
|
||||
return ov
|
||||
}
|
||||
|
||||
// --- the agent ---
|
||||
|
||||
func (p nodeProbe) agentState(ctx context.Context) agentService {
|
||||
out, err := p.run(ctx, "systemctl", "show", "llmgw-agent",
|
||||
"-p", "LoadState", "-p", "ActiveState", "-p", "SubState", "-p", "ActiveEnterTimestamp")
|
||||
if err != nil && len(out) == 0 {
|
||||
return agentService{note: "systemctl is not available: " + err.Error()}
|
||||
}
|
||||
return parseSystemctlShow(string(out))
|
||||
}
|
||||
|
||||
// parseSystemctlShow reads `systemctl show -p …`: one key=value a line.
|
||||
func parseSystemctlShow(out string) agentService {
|
||||
kv := map[string]string{}
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
|
||||
kv[k] = v
|
||||
}
|
||||
}
|
||||
if kv["LoadState"] == "not-found" {
|
||||
return agentService{note: "llmgw-agent is not installed as a service here"}
|
||||
}
|
||||
s := agentService{state: kv["ActiveState"], sub: kv["SubState"]}
|
||||
if s.state == "" {
|
||||
s.note = "systemd does not say"
|
||||
}
|
||||
s.since = parseSystemdTime(kv["ActiveEnterTimestamp"])
|
||||
return s
|
||||
}
|
||||
|
||||
// parseSystemdTime reads systemd's "Mon 2026-10-19 08:12:03 CEST", in this
|
||||
// machine's zone; zero when it is empty or n/a.
|
||||
func parseSystemdTime(v string) time.Time {
|
||||
v = strings.TrimSpace(v)
|
||||
if v == "" || v == "n/a" {
|
||||
return time.Time{}
|
||||
}
|
||||
for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05 -0700", "Mon 2006-01-02 15:04:05"} {
|
||||
if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
// --- the control planes ---
|
||||
|
||||
// readControlPlanes is every address the agent dials, in its order: the
|
||||
// configured ones, then those the leader named that agent.yaml lacks.
|
||||
func readControlPlanes(file string, configured []string) []planeCheck {
|
||||
var out []planeCheck
|
||||
seen := map[string]bool{}
|
||||
add := func(s string, learned bool) {
|
||||
hp := hostPort(s)
|
||||
if hp == "" || seen[hp] {
|
||||
return
|
||||
}
|
||||
seen[hp] = true
|
||||
out = append(out, planeCheck{addr: hp, learned: learned})
|
||||
}
|
||||
for _, s := range configured {
|
||||
add(s, false)
|
||||
}
|
||||
if b, err := os.ReadFile(file); err == nil {
|
||||
var told []string
|
||||
if json.Unmarshal(b, &told) == nil {
|
||||
for _, s := range told {
|
||||
add(s, true)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// hostPort is the host:port a control plane's address dials, whether it was
|
||||
// written as a URL or not.
|
||||
func hostPort(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
port := ""
|
||||
switch {
|
||||
case strings.HasPrefix(s, "https://"):
|
||||
s, port = strings.TrimPrefix(s, "https://"), "443"
|
||||
case strings.HasPrefix(s, "http://"):
|
||||
s, port = strings.TrimPrefix(s, "http://"), "80"
|
||||
}
|
||||
s, _, _ = strings.Cut(s, "/")
|
||||
if _, _, err := net.SplitHostPort(s); err != nil && port != "" && s != "" {
|
||||
s = net.JoinHostPort(strings.Trim(s, "[]"), port)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func (p nodeProbe) checkPlanes(ctx context.Context) []planeCheck {
|
||||
planes := readControlPlanes(p.planesFile, p.configured)
|
||||
var wg sync.WaitGroup
|
||||
for i := range planes {
|
||||
wg.Add(1)
|
||||
go func(c *planeCheck) {
|
||||
defer wg.Done()
|
||||
if err := p.dial(ctx, c.addr); err != nil {
|
||||
c.err = err.Error()
|
||||
}
|
||||
}(&planes[i])
|
||||
}
|
||||
wg.Wait()
|
||||
return planes
|
||||
}
|
||||
|
||||
func dialTCP(ctx context.Context, addr string) error {
|
||||
d := net.Dialer{Timeout: nodeCmdTimeout}
|
||||
c, err := d.DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return c.Close()
|
||||
}
|
||||
|
||||
// --- the GPUs ---
|
||||
|
||||
// readGPUs samples the cards, naming them from discovery the first time.
|
||||
func (p nodeProbe) readGPUs(ctx context.Context) ([]nodeGPU, map[int]string, string) {
|
||||
ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
|
||||
defer cancel()
|
||||
sensors := p.gpus(ctx)
|
||||
if len(sensors) == 0 {
|
||||
// Without nvidia-smi discovery says nothing and no error: a machine
|
||||
// without a card. With it and no answer, a driver in trouble.
|
||||
if _, err := p.cards(ctx); err != nil {
|
||||
return nil, p.names, "The GPUs do not answer: " + err.Error()
|
||||
}
|
||||
return nil, p.names, "No GPU on this machine."
|
||||
}
|
||||
names := p.names
|
||||
for _, s := range sensors {
|
||||
if names[s.Index] == "" {
|
||||
names = map[int]string{}
|
||||
if cards, err := p.cards(ctx); err == nil {
|
||||
for _, c := range cards {
|
||||
if c.MIG == nil {
|
||||
names[c.Index] = c.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
var out []nodeGPU
|
||||
for _, s := range sensors {
|
||||
g := nodeGPU{index: s.Index, name: names[s.Index], util: s.UtilPercent, tempC: s.TempC,
|
||||
memUsedMB: s.MemoryUsedMB, memTotalMB: s.MemoryTotalMB}
|
||||
if s.MemoryTotalMB == 0 {
|
||||
g.util = -1
|
||||
}
|
||||
out = append(out, g)
|
||||
}
|
||||
return out, names, ""
|
||||
}
|
||||
|
||||
// cardOf is the card a device number is on: itself, or a MIG instance's card.
|
||||
func cardOf(i int) int {
|
||||
if discovery.IsMIGIndex(i) {
|
||||
return (i - discovery.MIGIndex(0, 0)) / 100
|
||||
}
|
||||
return i
|
||||
}
|
||||
|
||||
// modelsOnGPUs names, on each card, the models whose engines hold it.
|
||||
func modelsOnGPUs(gpus []nodeGPU, runs []engineRun) []nodeGPU {
|
||||
for i := range gpus {
|
||||
gpus[i].models = nil
|
||||
for _, r := range runs {
|
||||
for _, d := range r.gpus {
|
||||
if cardOf(d) == gpus[i].index && !slices.Contains(gpus[i].models, r.model) {
|
||||
gpus[i].models = append(gpus[i].models, r.model)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return gpus
|
||||
}
|
||||
|
||||
// --- the engines ---
|
||||
|
||||
// shardStateFile is the part of the supervisor's shard-*.json the overview
|
||||
// reads (shardState in the supervisor package).
|
||||
type shardStateFile struct {
|
||||
ShardID string `json:"shard_id"`
|
||||
ReplicaID string `json:"replica_id"`
|
||||
PID int `json:"pid"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
Spec engines.ShardSpec `json:"spec"`
|
||||
}
|
||||
|
||||
// readShardStates is the engines the supervisor keeps state files for in
|
||||
// its work directory, with whether each process is still there.
|
||||
func readShardStates(dir string, alive func(int) bool) ([]engineRun, string) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
switch {
|
||||
case errors.Is(err, os.ErrPermission):
|
||||
return nil, "The engines' state is root's: run with sudo."
|
||||
case errors.Is(err, os.ErrNotExist):
|
||||
return nil, ""
|
||||
case err != nil:
|
||||
return nil, err.Error()
|
||||
}
|
||||
var runs []engineRun
|
||||
note := ""
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if !strings.HasPrefix(name, "shard-") || !strings.HasSuffix(name, ".json") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, name))
|
||||
if errors.Is(err, os.ErrPermission) {
|
||||
note = "The engines' state is root's: run with sudo."
|
||||
continue
|
||||
}
|
||||
var st shardStateFile
|
||||
if err != nil || json.Unmarshal(b, &st) != nil {
|
||||
continue
|
||||
}
|
||||
replica := st.ReplicaID
|
||||
if replica == "" {
|
||||
replica = st.Spec.ReplicaID
|
||||
}
|
||||
model := st.Spec.Profile.ServedModelName
|
||||
if model == "" {
|
||||
model = modelOfReplica(replica)
|
||||
}
|
||||
runs = append(runs, engineRun{replica: replica, model: model, engine: st.Spec.Profile.Engine,
|
||||
gpus: st.Spec.GPUIndices, port: st.Spec.HTTPPort, pid: st.PID, alive: st.PID > 0 && alive(st.PID),
|
||||
started: st.StartedAt})
|
||||
}
|
||||
sort.Slice(runs, func(i, j int) bool { return runs[i].replica < runs[j].replica })
|
||||
return runs, note
|
||||
}
|
||||
|
||||
// modelOfReplica is the model a replica id names: the control plane calls a
|
||||
// replica the model's name and a suffix.
|
||||
func modelOfReplica(id string) string {
|
||||
if i := strings.LastIndexByte(id, '-'); i > 0 {
|
||||
return id[:i]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func pidAlive(pid int) bool {
|
||||
err := syscall.Kill(pid, 0)
|
||||
return err == nil || errors.Is(err, syscall.EPERM)
|
||||
}
|
||||
|
||||
// --- the disks ---
|
||||
|
||||
func (p nodeProbe) readDisks() []diskUse {
|
||||
disks := []diskUse{{what: "weights", path: p.weightsDir}, {what: "engines", path: p.enginesDir}}
|
||||
for i := range disks {
|
||||
// In its own goroutine: statfs on a network mount that has gone away
|
||||
// does not return.
|
||||
type res struct {
|
||||
free, total uint64
|
||||
err error
|
||||
}
|
||||
ch := make(chan res, 1)
|
||||
go func(path string) {
|
||||
f, t, err := p.statfs(path)
|
||||
ch <- res{f, t, err}
|
||||
}(disks[i].path)
|
||||
select {
|
||||
case r := <-ch:
|
||||
disks[i].free, disks[i].total = r.free, r.total
|
||||
if r.err != nil {
|
||||
disks[i].err = r.err.Error()
|
||||
if errors.Is(r.err, os.ErrNotExist) {
|
||||
disks[i].err = "not there yet"
|
||||
}
|
||||
}
|
||||
case <-time.After(nodeCmdTimeout):
|
||||
disks[i].err = "does not answer"
|
||||
}
|
||||
}
|
||||
return disks
|
||||
}
|
||||
|
||||
func statfs(path string) (free, total uint64, err error) {
|
||||
var st syscall.Statfs_t
|
||||
if err := syscall.Statfs(path, &st); err != nil {
|
||||
return 0, 0, &os.PathError{Op: "statfs", Path: path, Err: err}
|
||||
}
|
||||
//nolint:unconvert // the field types differ between systems
|
||||
return uint64(st.Bavail) * uint64(st.Bsize), uint64(st.Blocks) * uint64(st.Bsize), nil
|
||||
}
|
||||
|
||||
// --- the problems ---
|
||||
|
||||
func (p nodeProbe) readProblems(ctx context.Context) ([]string, string) {
|
||||
out, err := p.run(ctx, "journalctl", "-u", "llmgw-agent", "-p", "warning", "--since", "-1h",
|
||||
"--no-pager", "-o", "cat", "-n", strconv.Itoa(nodeProblemLines))
|
||||
if err != nil && len(out) == 0 {
|
||||
return nil, "The journal cannot be read: " + err.Error()
|
||||
}
|
||||
lines := parseJournal(string(out), nodeProblemLines)
|
||||
if len(lines) == 0 && !p.root {
|
||||
return nil, "None that this user can see: run with sudo to read the agent's journal."
|
||||
}
|
||||
return lines, ""
|
||||
}
|
||||
|
||||
// parseJournal keeps the last n lines journalctl printed, without its own
|
||||
// remarks.
|
||||
func parseJournal(out string, n int) []string {
|
||||
var lines []string
|
||||
sc := bufio.NewScanner(strings.NewReader(out))
|
||||
sc.Buffer(make([]byte, 64*1024), 1024*1024)
|
||||
for sc.Scan() {
|
||||
l := strings.TrimSpace(sc.Text())
|
||||
if l == "" || strings.HasPrefix(l, "-- ") || strings.HasPrefix(l, "Hint: ") {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, l)
|
||||
}
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return lines
|
||||
}
|
||||
|
||||
// runLocal runs a tool on this machine, bounded by nodeCmdTimeout. A tool
|
||||
// that is not installed is said so, not waited for.
|
||||
func runLocal(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
if _, err := exec.LookPath(name); err != nil {
|
||||
return nil, fmt.Errorf("%s is not installed", name)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
|
||||
defer cancel()
|
||||
//nolint:gosec // fixed tools and arguments, from this file
|
||||
return exec.CommandContext(ctx, name, args...).Output()
|
||||
}
|
||||
|
||||
// --- the panel ---
|
||||
|
||||
func (m consoleModel) nodeOverviewPanel(pw int) (string, string, string) {
|
||||
keys := "↑↓ choose · PgUp/PgDn scroll · Q leave · refreshed every three seconds"
|
||||
if m.nodeOv.at.IsZero() {
|
||||
return "Overview", "\n " + m.spin.View() + " reading this machine…", keys
|
||||
}
|
||||
return "Overview", renderNodeOverview(m.nodeOv, pw, time.Now()), keys
|
||||
}
|
||||
|
||||
// renderNodeOverview draws the overview from what was read.
|
||||
func renderNodeOverview(ov nodeOverview, pw int, now time.Time) string {
|
||||
var b strings.Builder
|
||||
head := func(s string) { b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render(s) + "\n") }
|
||||
kv := func(k, v string) { fmt.Fprintf(&b, " %-14s %s\n", sDim.Render(k), v) }
|
||||
line := func(s string) { b.WriteString(" " + s + "\n") }
|
||||
|
||||
head("Agent")
|
||||
a := ov.agent
|
||||
switch {
|
||||
case a.note != "":
|
||||
kv("llmgw-agent", sWarn.Render(a.note))
|
||||
default:
|
||||
state := a.state
|
||||
if a.sub != "" {
|
||||
state += " (" + a.sub + ")"
|
||||
}
|
||||
if a.state == "active" {
|
||||
state = sOK.Render(state)
|
||||
} else {
|
||||
state = sBad.Render(state)
|
||||
}
|
||||
if !a.since.IsZero() {
|
||||
state += sDim.Render(" since " + a.since.Format("2006-01-02 15:04") + ", " + uptime(now.Sub(a.since)) + " ago")
|
||||
}
|
||||
kv("llmgw-agent", state)
|
||||
}
|
||||
kv("llmgw", ov.version)
|
||||
|
||||
head("Control planes")
|
||||
if len(ov.planes) == 0 {
|
||||
line(sWarn.Render("None configured: agent.yaml names no server."))
|
||||
}
|
||||
for _, c := range ov.planes {
|
||||
state := sOK.Render("reachable")
|
||||
if c.err != "" {
|
||||
state = sBad.Render("✗ ") + truncate(c.err, max(20, pw-30))
|
||||
}
|
||||
if c.learned {
|
||||
state += sDim.Render(" (named by the leader)")
|
||||
}
|
||||
fmt.Fprintf(&b, " %-22s %s\n", c.addr, state)
|
||||
}
|
||||
|
||||
head("GPUs")
|
||||
if len(ov.gpus) == 0 {
|
||||
line(sDim.Render(ov.gpuNote))
|
||||
} else {
|
||||
var rows [][]string
|
||||
for _, g := range ov.gpus {
|
||||
busy, mem, temp := "—", "—", "—"
|
||||
if g.util >= 0 {
|
||||
busy = fmt.Sprintf("%.0f%%", g.util)
|
||||
}
|
||||
if g.memTotalMB > 0 {
|
||||
mem = fmt.Sprintf("%.1f/%.0f GB", g.memUsedMB/1024, g.memTotalMB/1024)
|
||||
}
|
||||
if g.tempC > 0 {
|
||||
temp = fmt.Sprintf("%.0f°C", g.tempC)
|
||||
}
|
||||
models := strings.Join(g.models, ", ")
|
||||
if models == "" {
|
||||
models = sDim.Render("free")
|
||||
}
|
||||
rows = append(rows, []string{strconv.Itoa(g.index), g.name, busy, mem, temp, models})
|
||||
}
|
||||
b.WriteString(indent(table([]string{"GPU", "CARD", "BUSY", "MEMORY", "TEMP", "MODELS"}, rows, -1, pw-2, false), " ") + "\n")
|
||||
}
|
||||
|
||||
head("Engines running here")
|
||||
switch {
|
||||
case len(ov.engines) > 0:
|
||||
var rows [][]string
|
||||
for _, r := range ov.engines {
|
||||
// The process by its pid, marked when it is no longer there.
|
||||
proc := sOK.Render(strconv.Itoa(r.pid))
|
||||
if !r.alive {
|
||||
proc = sBad.Render(strconv.Itoa(r.pid) + " gone")
|
||||
}
|
||||
gpus := strings.Join(discovery.DeviceLabels(r.gpus), ",")
|
||||
if gpus == "" {
|
||||
gpus = sDim.Render("CPU")
|
||||
}
|
||||
up := "—"
|
||||
if !r.started.IsZero() {
|
||||
up = uptime(now.Sub(r.started))
|
||||
}
|
||||
port := "—"
|
||||
if r.port > 0 {
|
||||
port = strconv.Itoa(r.port)
|
||||
}
|
||||
rows = append(rows, []string{r.replica, r.model, r.engine, gpus, port, proc, up})
|
||||
}
|
||||
b.WriteString(indent(table([]string{"REPLICA", "MODEL", "ENGINE", "GPUS", "PORT", "PID", "UP"}, rows, -1, pw-2, false), " ") + "\n")
|
||||
if ov.enginesNote != "" {
|
||||
line(sWarn.Render(ov.enginesNote))
|
||||
}
|
||||
case ov.enginesNote != "":
|
||||
line(sWarn.Render(ov.enginesNote))
|
||||
default:
|
||||
line(sDim.Render("None: the control plane has placed nothing here."))
|
||||
}
|
||||
|
||||
head("Disk")
|
||||
for _, d := range ov.disks {
|
||||
use := sWarn.Render(d.err)
|
||||
if d.err == "" {
|
||||
use = fmt.Sprintf("%s free of %s", diskSize(d.free), diskSize(d.total))
|
||||
if d.total > 0 && d.free*10 < d.total {
|
||||
use = sWarn.Render(use)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, " %-14s %-28s %s\n", sDim.Render(d.what), d.path, use)
|
||||
}
|
||||
|
||||
head("Recent problems")
|
||||
switch {
|
||||
case ov.problemsNote != "":
|
||||
line(sDim.Render(ov.problemsNote))
|
||||
case len(ov.problems) == 0:
|
||||
line(sOK.Render("None in the last hour."))
|
||||
default:
|
||||
for _, l := range ov.problems {
|
||||
l = truncateWidth(l, max(20, pw-4))
|
||||
if strings.Contains(l, "level=ERROR") {
|
||||
l = sBad.Render(l)
|
||||
} else {
|
||||
l = sWarn.Render(l)
|
||||
}
|
||||
line(l)
|
||||
}
|
||||
}
|
||||
b.WriteString("\n " + sDim.Render("read at "+ov.at.Format("15:04:05")) + "\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// uptime is a duration as people say it: 3d4h, 2h15m, 40s.
|
||||
func uptime(d time.Duration) string {
|
||||
d = d.Round(time.Second)
|
||||
switch {
|
||||
case d < 0:
|
||||
return "0s"
|
||||
case d >= 24*time.Hour:
|
||||
return fmt.Sprintf("%dd%dh", d/(24*time.Hour), (d%(24*time.Hour))/time.Hour)
|
||||
case d >= time.Hour:
|
||||
return fmt.Sprintf("%dh%dm", d/time.Hour, (d%time.Hour)/time.Minute)
|
||||
case d >= time.Minute:
|
||||
return fmt.Sprintf("%dm%ds", d/time.Minute, (d%time.Minute)/time.Second)
|
||||
}
|
||||
return d.String()
|
||||
}
|
||||
|
||||
// diskSize is a byte count in the unit disks are sold in.
|
||||
func diskSize(n uint64) string {
|
||||
switch {
|
||||
case n >= 1<<40:
|
||||
return fmt.Sprintf("%.1f TB", float64(n)/(1<<40))
|
||||
case n >= 1<<30:
|
||||
return fmt.Sprintf("%.0f GB", float64(n)/(1<<30))
|
||||
}
|
||||
return fmt.Sprintf("%.0f MB", float64(n)/(1<<20))
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/charmbracelet/x/ansi"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
func writeNodeFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewDialsTheConfiguredControlPlanesThenTheLeadersOwn(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, "control-planes.json")
|
||||
writeNodeFile(t, file, `["https://10.0.0.150:7443","10.0.0.140:7443","https://cp3.example/agent"]`)
|
||||
got := readControlPlanes(file, []string{"10.0.0.140:7443", "http://10.0.0.141"})
|
||||
want := []planeCheck{{addr: "10.0.0.140:7443"}, {addr: "10.0.0.141:80"},
|
||||
{addr: "10.0.0.150:7443", learned: true}, {addr: "cp3.example:443", learned: true}}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %+v, want %+v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("%d: got %+v, want %+v", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
// Without the file, what agent.yaml says.
|
||||
if got := readControlPlanes(filepath.Join(dir, "none.json"), []string{"cp:7443"}); len(got) != 1 || got[0].addr != "cp:7443" {
|
||||
t.Errorf("without the file: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewReadsTheEnginesFromTheSupervisorsStateFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"shard_id":"qwen3-8b-ab12-0","replica_id":"qwen3-8b-ab12",
|
||||
"pid":4242,"started_at":"2026-10-19T08:00:00Z","spec":{"ReplicaID":"qwen3-8b-ab12","GPUIndices":[0,1],"HTTPPort":8101,
|
||||
"Profile":{"Name":"vllm-tp2","Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-bge-m3-7-0.json"), `{"replica_id":"bge-m3-7","pid":99,
|
||||
"spec":{"GPUIndices":[1002],"HTTPPort":8102,"Profile":{"Engine":"llamacpp"}}}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-x.json.tmp"), `{"replica_id":"half-written"}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "control-planes.json"), `[]`)
|
||||
runs, note := readShardStates(dir, func(pid int) bool { return pid == 4242 })
|
||||
if note != "" {
|
||||
t.Fatalf("note %q", note)
|
||||
}
|
||||
if len(runs) != 2 {
|
||||
t.Fatalf("got %+v", runs)
|
||||
}
|
||||
b, q := runs[0], runs[1]
|
||||
if q.replica != "qwen3-8b-ab12" || q.model != "qwen3-8b" || q.engine != "vllm" || q.port != 8101 || !q.alive ||
|
||||
len(q.gpus) != 2 || !q.started.Equal(time.Date(2026, 10, 19, 8, 0, 0, 0, time.UTC)) {
|
||||
t.Errorf("qwen: %+v", q)
|
||||
}
|
||||
// No served name: the model from the replica's id.
|
||||
if b.model != "bge-m3" || b.alive || b.engine != "llamacpp" {
|
||||
t.Errorf("bge: %+v", b)
|
||||
}
|
||||
if runs, note := readShardStates(filepath.Join(dir, "absent"), pidAlive); runs != nil || note != "" {
|
||||
t.Errorf("no work dir: %+v %q", runs, note)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewNamesTheModelOnEachGPU(t *testing.T) {
|
||||
gpus := []nodeGPU{{index: 0}, {index: 1}, {index: 2}}
|
||||
runs := []engineRun{
|
||||
{model: "qwen3-8b", gpus: []int{0, 1}},
|
||||
{model: "bge-m3", gpus: []int{discovery.MIGIndex(1, 2)}},
|
||||
{model: "qwen3-8b", gpus: []int{1}},
|
||||
}
|
||||
got := modelsOnGPUs(gpus, runs)
|
||||
if strings.Join(got[0].models, ",") != "qwen3-8b" || strings.Join(got[1].models, ",") != "qwen3-8b,bge-m3" ||
|
||||
len(got[2].models) != 0 {
|
||||
t.Errorf("got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewReadsWhatSystemdSaysOfTheAgent(t *testing.T) {
|
||||
s := parseSystemctlShow("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:12:03 UTC\n")
|
||||
if s.state != "active" || s.sub != "running" || s.note != "" || s.since.IsZero() || s.since.Minute() != 12 {
|
||||
t.Errorf("got %+v", s)
|
||||
}
|
||||
if s := parseSystemctlShow("LoadState=not-found\nActiveState=inactive\nActiveEnterTimestamp=\n"); s.note == "" {
|
||||
t.Errorf("a missing service: %+v", s)
|
||||
}
|
||||
if s := parseSystemctlShow("LoadState=loaded\nActiveState=failed\nSubState=failed\nActiveEnterTimestamp=n/a\n"); s.state != "failed" ||
|
||||
!s.since.IsZero() {
|
||||
t.Errorf("a failed service: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewKeepsTheLastWarningsOfTheJournal(t *testing.T) {
|
||||
var b strings.Builder
|
||||
for i := 1; i <= 9; i++ {
|
||||
b.WriteString("level=WARN msg=w" + string(rune('0'+i)) + "\n")
|
||||
}
|
||||
got := parseJournal(b.String(), 6)
|
||||
if len(got) != 6 || got[0] != "level=WARN msg=w4" || got[5] != "level=WARN msg=w9" {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := parseJournal("-- No entries --\n", 6); len(got) != 0 {
|
||||
t.Errorf("no entries: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// fakeProbe is a node with two cards, one engine and a broken second control
|
||||
// plane, and no tool of the machine running the test touched.
|
||||
func fakeProbe(t *testing.T) nodeProbe {
|
||||
dir := t.TempDir()
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"replica_id":"qwen3-8b-ab12","pid":4242,
|
||||
"started_at":"2026-10-19T08:00:00Z","spec":{"GPUIndices":[0],"HTTPPort":8101,"Profile":{"Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
|
||||
return nodeProbe{
|
||||
configured: []string{"10.0.0.140:7443", "10.0.0.150:7443"},
|
||||
planesFile: filepath.Join(dir, "control-planes.json"),
|
||||
workDir: dir, weightsDir: "/var/lib/llmgw-weights", enginesDir: "/opt/llmgw/engines", root: true,
|
||||
run: func(_ context.Context, name string, _ ...string) ([]byte, error) {
|
||||
switch name {
|
||||
case "systemctl":
|
||||
return []byte("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:00:00 UTC\n"), nil
|
||||
case "journalctl":
|
||||
return []byte("level=WARN msg=\"the control plane does not answer\" server=10.0.0.150:7443\n"), nil
|
||||
}
|
||||
return nil, errors.New(name + " is not installed")
|
||||
},
|
||||
dial: func(_ context.Context, addr string) error {
|
||||
if addr == "10.0.0.150:7443" {
|
||||
return errors.New("dial tcp 10.0.0.150:7443: connect: connection refused")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
gpus: func(context.Context) []discovery.GPUSensor {
|
||||
return []discovery.GPUSensor{
|
||||
{Index: 0, TempC: 61, UtilPercent: 87, MemoryUsedMB: 20480, MemoryTotalMB: 24576},
|
||||
{Index: 1, TempC: 35, UtilPercent: 0, MemoryUsedMB: 300, MemoryTotalMB: 24576},
|
||||
}
|
||||
},
|
||||
cards: func(context.Context) ([]discovery.GPU, error) {
|
||||
return []discovery.GPU{{Index: 0, Name: "NVIDIA RTX A5000"}, {Index: 1, Name: "NVIDIA RTX A5000"}}, nil
|
||||
},
|
||||
alive: func(int) bool { return true },
|
||||
statfs: func(string) (uint64, uint64, error) { return 120 << 30, 500 << 30, nil },
|
||||
now: func() time.Time { return time.Date(2026, 10, 19, 11, 12, 0, 0, time.UTC) },
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewShowsWhatRunsOnTheMachine(t *testing.T) {
|
||||
ov := fakeProbe(t).gather(context.Background())
|
||||
out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
|
||||
t.Log("\n" + out)
|
||||
for _, want := range []string{
|
||||
"llmgw-agent", "active (running)", "3h12m ago",
|
||||
"10.0.0.140:7443", "reachable", "connection refused",
|
||||
"NVIDIA RTX A5000", "87%", "20.0/24 GB", "61°C", "qwen3-8b", "free",
|
||||
"qwen3-8b-ab12", "vllm", "8101", "4242", "3h12m",
|
||||
"/var/lib/llmgw-weights", "120 GB free of 500 GB", "/opt/llmgw/engines",
|
||||
"the control plane does not answer",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("the overview lacks %q", want)
|
||||
}
|
||||
}
|
||||
if ov.names[0] != "NVIDIA RTX A5000" {
|
||||
t.Errorf("the names are not kept for the next reading: %v", ov.names)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeWithoutAGPUSaysSoInOneLine(t *testing.T) {
|
||||
p := fakeProbe(t)
|
||||
p.gpus = func(context.Context) []discovery.GPUSensor { return nil }
|
||||
p.cards = func(context.Context) ([]discovery.GPU, error) { return nil, nil }
|
||||
p.run = func(context.Context, string, ...string) ([]byte, error) { return nil, errors.New("not installed") }
|
||||
ov := p.gather(context.Background())
|
||||
out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
|
||||
if !strings.Contains(out, "No GPU on this machine.") {
|
||||
t.Errorf("no line saying there is no GPU:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "BUSY") {
|
||||
t.Errorf("an empty GPU table:\n%s", out)
|
||||
}
|
||||
// No systemctl or journalctl is said, not a crash.
|
||||
if !strings.Contains(out, "systemctl is not available") || !strings.Contains(out, "The journal cannot be read") {
|
||||
t.Errorf("missing tools are not said:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesConsoleOpensOnItsOverview(t *testing.T) {
|
||||
m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "agent.yaml"))
|
||||
if m.menu != cOverview || !m.entryOK(cOverview) {
|
||||
t.Fatalf("menu %d", m.menu)
|
||||
}
|
||||
_, body, _ := m.panel()
|
||||
if !strings.Contains(body, "reading this machine") {
|
||||
t.Errorf("before the first reading: %q", body)
|
||||
}
|
||||
}
|
||||
@@ -395,6 +395,7 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
|
||||
if l.AllowedModels != nil {
|
||||
models = strings.Join(l.AllowedModels, ",")
|
||||
}
|
||||
tools := strings.Join(l.AllowedTools, ",")
|
||||
m.openForm(&formDialog{title: "Limits on " + who,
|
||||
note: "Each level only narrows what is above it. Empty is no limit here.",
|
||||
fields: []formField{
|
||||
@@ -403,9 +404,11 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
|
||||
textF("budget", budget, "spend per period, in whole units"),
|
||||
choiceF("per", []string{"day", "month"}, period),
|
||||
textF("models", models, "allowed, separated by commas; \"every model\" for no list"),
|
||||
textF("company tools", tools, "MCP tools granted here, as jira.*,db.query; empty grants none here"),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
args := []string{"keys", "limits", "--scope=" + string(scope), "--id=" + id}
|
||||
// Every field the form shows, so an emptied one is cleared.
|
||||
args := []string{"keys", "limits", "--replace", "--scope=" + string(scope), "--id=" + id}
|
||||
for i, flag := range []string{"rpm", "tpm"} {
|
||||
if v[i] != "" {
|
||||
if _, err := strconv.Atoi(v[i]); err != nil {
|
||||
@@ -423,6 +426,9 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
|
||||
if v[4] != "every model" {
|
||||
args = append(args, "--models="+strings.ReplaceAll(v[4], " ", ""))
|
||||
}
|
||||
if t := strings.ReplaceAll(v[5], " ", ""); t != "" {
|
||||
args = append(args, "--tools="+t)
|
||||
}
|
||||
return m.runCLI("Limits on "+who, args...), nil
|
||||
}})
|
||||
}
|
||||
@@ -567,13 +573,20 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog {
|
||||
if c.Any {
|
||||
which = "any"
|
||||
}
|
||||
similar := "off"
|
||||
if c.Similar > 0 {
|
||||
similar = strconv.FormatFloat(c.Similar, 'f', -1, 64)
|
||||
}
|
||||
m.openForm(&formDialog{title: "Cache answers · " + r.path,
|
||||
note: "Off by default. On, the same request to the same model within the time below is\n" +
|
||||
"answered again from the gateway's memory, without the model, and costs nothing.\n" +
|
||||
"Reproducible: only temperature 0 or a seed, and embeddings. Any: sampled answers too.",
|
||||
"Reproducible: only temperature 0 or a seed, and embeddings. Any: sampled answers too.\n" +
|
||||
"Similar: a chat question this alike to one answered gets its answer (needs\n" +
|
||||
"router.cache.embedding_model).",
|
||||
fields: []formField{
|
||||
textF("reuse for", ttl, "like 1h; off reuses none"),
|
||||
choiceF("answers", []string{"reproducible", "any"}, which),
|
||||
textF("similar", similar, "like 0.95; off reuses only the same request"),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
args := []string{"keys", "cache", "--project=" + r.id}
|
||||
@@ -587,6 +600,12 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog {
|
||||
if v[1] == "any" {
|
||||
args = append(args, "--any")
|
||||
}
|
||||
if v[2] != "off" && v[2] != "" && v[2] != "0" {
|
||||
if f, err := strconv.ParseFloat(v[2], 64); err != nil || f <= 0 || f > 1 {
|
||||
return nil, fmt.Errorf("similar is a number like 0.95, or off")
|
||||
}
|
||||
args = append(args, "--similar="+v[2])
|
||||
}
|
||||
return m.runCLI("Cache answers", args...), nil
|
||||
}})
|
||||
return nil
|
||||
@@ -761,5 +780,8 @@ func limitsCell(l store.Limits) string {
|
||||
if l.AllowedModels != nil {
|
||||
p = append(p, fmt.Sprintf("%d models", len(l.AllowedModels)))
|
||||
}
|
||||
if len(l.AllowedTools) > 0 {
|
||||
p = append(p, fmt.Sprintf("%d tool grants", len(l.AllowedTools)))
|
||||
}
|
||||
return strings.Join(p, ", ")
|
||||
}
|
||||
+129
-37
@@ -89,14 +89,21 @@ func (d dbMachine) visible(f int) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// dbConn is a database machine as it is reached, whichever way.
|
||||
// dbConn is a database machine as it is reached, whichever way, or a judge
|
||||
// machine when judge is set.
|
||||
type dbConn struct {
|
||||
remote onboard.Remote
|
||||
addr string
|
||||
ssh *onboard.GoSSH
|
||||
joined *joinpkg.Remote
|
||||
judge bool
|
||||
target string
|
||||
}
|
||||
|
||||
// The machines the page asks for: the store's two, then up to maxJudges for
|
||||
// the guardrails' judges alone.
|
||||
const maxJudges = 2
|
||||
|
||||
func newDBMachine(n int) dbMachine {
|
||||
f := func(ph string, mask bool) textField {
|
||||
t := newTextField()
|
||||
@@ -119,6 +126,24 @@ func newDBMachine(n int) dbMachine {
|
||||
return m
|
||||
}
|
||||
|
||||
// newJudgeMachine is the fields of a machine for the judges.
|
||||
func newJudgeMachine(n int) dbMachine {
|
||||
d := newDBMachine(n + 2)
|
||||
d.addr.Placeholder = fmt.Sprintf("10.0.0.%d or judge%d.internal", 20+n, n)
|
||||
return d
|
||||
}
|
||||
|
||||
// shown is how many machines the page asks for.
|
||||
func (m setupModel) shown() int { return 2 + m.judges }
|
||||
|
||||
// machineTitle names machine i: the store's, or the judges'.
|
||||
func machineTitle(i int) string {
|
||||
if i < 2 {
|
||||
return fmt.Sprintf("Database machine %d", i+1)
|
||||
}
|
||||
return fmt.Sprintf("Judge machine %d", i-1)
|
||||
}
|
||||
|
||||
// field is the text field at a machine's position, nil for the selector.
|
||||
func (d *dbMachine) field(i int) *textField {
|
||||
switch i {
|
||||
@@ -156,8 +181,10 @@ type setupModel struct {
|
||||
|
||||
choices *installForm
|
||||
|
||||
db [2]dbMachine
|
||||
mfocus int // over both machines' fields, then the check button
|
||||
db [2 + maxJudges]dbMachine
|
||||
// judges is how many judge machines are asked for, none to maxJudges.
|
||||
judges int
|
||||
mfocus int // over the machines' fields, then the judges' choice, then the check button
|
||||
|
||||
conns []dbConn
|
||||
listener *joinpkg.Listener
|
||||
@@ -192,7 +219,8 @@ func newSetupModel(host string, ips []string) setupModel {
|
||||
in.judge.Replicas = 2
|
||||
return setupModel{
|
||||
ref: &progRef{}, host: host, ips: ips, in: &in, choices: newInstallForm(&in, host, ips, true),
|
||||
db: [2]dbMachine{newDBMachine(1), newDBMachine(2)}, spin: newSpinner(), width: 100, height: 30,
|
||||
db: [2 + maxJudges]dbMachine{newDBMachine(1), newDBMachine(2), newJudgeMachine(1), newJudgeMachine(2)},
|
||||
spin: newSpinner(), width: 100, height: 30,
|
||||
log: &[]string{}, logMu: &sync.Mutex{},
|
||||
}
|
||||
}
|
||||
@@ -327,7 +355,7 @@ func (m *setupModel) focusMachine() {
|
||||
}
|
||||
}
|
||||
}
|
||||
if m.mfocus < 2*mfCount {
|
||||
if m.mfocus < m.shown()*mfCount {
|
||||
if t := m.db[m.mfocus/mfCount].field(m.mfocus % mfCount); t != nil {
|
||||
t.Focus()
|
||||
}
|
||||
@@ -335,7 +363,8 @@ func (m *setupModel) focusMachine() {
|
||||
}
|
||||
|
||||
func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
n := 2*mfCount + 1
|
||||
sel := m.shown() * mfCount
|
||||
n := sel + 2
|
||||
switch k.Type {
|
||||
case tea.KeyTab, tea.KeyDown:
|
||||
m.step(1, n)
|
||||
@@ -344,7 +373,7 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.step(-1, n)
|
||||
return m, nil
|
||||
case tea.KeyEnter:
|
||||
if m.mfocus < 2*mfCount {
|
||||
if m.mfocus <= sel {
|
||||
m.step(1, n)
|
||||
return m, nil
|
||||
}
|
||||
@@ -361,7 +390,17 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
cmd := m.startCheck()
|
||||
return m, cmd
|
||||
}
|
||||
if m.mfocus == 2*mfCount {
|
||||
if m.mfocus == sel {
|
||||
switch k.String() {
|
||||
case "left", "h":
|
||||
m.judges = (m.judges + maxJudges) % (maxJudges + 1)
|
||||
case "right", "l", " ":
|
||||
m.judges = (m.judges + 1) % (maxJudges + 1)
|
||||
}
|
||||
m.mfocus = m.shown() * mfCount
|
||||
return m, nil
|
||||
}
|
||||
if m.mfocus > sel {
|
||||
return m, nil
|
||||
}
|
||||
d := &m.db[m.mfocus/mfCount]
|
||||
@@ -391,7 +430,7 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
func (m *setupModel) step(d, n int) {
|
||||
for range n {
|
||||
m.mfocus = (m.mfocus + d + n) % n
|
||||
if m.mfocus == 2*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
|
||||
if m.mfocus >= m.shown()*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
|
||||
break
|
||||
}
|
||||
}
|
||||
@@ -399,7 +438,7 @@ func (m *setupModel) step(d, n int) {
|
||||
}
|
||||
|
||||
func (m setupModel) allJoined() bool {
|
||||
for _, d := range m.db {
|
||||
for _, d := range m.db[:m.shown()] {
|
||||
if d.auth == authJoin && d.joined == nil {
|
||||
return false
|
||||
}
|
||||
@@ -411,7 +450,7 @@ func (m setupModel) allJoined() bool {
|
||||
// codes, and waits for them; it says whether there is anything to wait for.
|
||||
func (m *setupModel) startJoins() (tea.Cmd, bool) {
|
||||
var cmds []tea.Cmd
|
||||
for i := range m.db {
|
||||
for i := range m.db[:m.shown()] {
|
||||
d := &m.db[i]
|
||||
if d.auth != authJoin || d.joined != nil {
|
||||
continue
|
||||
@@ -429,7 +468,11 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
|
||||
m.listener = l
|
||||
}
|
||||
if d.arrive == nil {
|
||||
c, ch := m.listener.Code("database")
|
||||
role := "database"
|
||||
if i >= 2 {
|
||||
role = "judge"
|
||||
}
|
||||
c, ch := m.listener.Code(role)
|
||||
d.code, d.arrive = c.String(), ch
|
||||
}
|
||||
ch, i := d.arrive, i
|
||||
@@ -440,20 +483,20 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
|
||||
|
||||
func (m setupModel) validateMachines() string {
|
||||
seen := map[string]bool{m.thisAddr(): true, m.host: true}
|
||||
for i, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
if d.auth == authJoin {
|
||||
continue
|
||||
}
|
||||
addr := strings.TrimSpace(d.addr.Value())
|
||||
addr, title := strings.TrimSpace(d.addr.Value()), strings.ToLower(machineTitle(i)[:1])+machineTitle(i)[1:]
|
||||
switch {
|
||||
case addr == "":
|
||||
return fmt.Sprintf("database machine %d has no address", i+1)
|
||||
return title + " has no address"
|
||||
case seen[addr]:
|
||||
return fmt.Sprintf("%s is given twice, or is this machine: the three must be different machines", addr)
|
||||
return fmt.Sprintf("%s is given twice, or is this machine: each must be a different machine", addr)
|
||||
case strings.TrimSpace(d.user.Value()) == "":
|
||||
return fmt.Sprintf("database machine %d has no user to log in as", i+1)
|
||||
return title + " has no user to log in as"
|
||||
case d.auth != authAgent && d.secret.Value() == "":
|
||||
return fmt.Sprintf("database machine %d needs its %s", i+1, authNames[d.auth])
|
||||
return fmt.Sprintf("%s needs its %s", title, authNames[d.auth])
|
||||
}
|
||||
seen[addr] = true
|
||||
}
|
||||
@@ -469,15 +512,22 @@ func (m *setupModel) startCheck() tea.Cmd {
|
||||
}
|
||||
}
|
||||
m.conns = nil
|
||||
serves, what := m.in.servingMemoryMB(m.judges > 0)
|
||||
machines := []preflight.Machine{{
|
||||
Name: m.host + " (this machine)", Addr: m.thisAddr(), Local: true, Remote: pgha.Local{},
|
||||
Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
|
||||
Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
|
||||
ServesMB: serves, Serves: what,
|
||||
}}
|
||||
for _, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
roles, judge := m.dbRoles(), i >= 2
|
||||
if judge {
|
||||
roles = []preflight.Role{preflight.RoleNode}
|
||||
}
|
||||
if d.auth == authJoin {
|
||||
m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined})
|
||||
m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined,
|
||||
judge: judge, target: "root@" + d.joined.Addr})
|
||||
machines = append(machines, preflight.Machine{Name: d.joined.Hostname + " (" + d.joined.Addr + ", joined)",
|
||||
Addr: d.joined.Addr, Roles: m.dbRoles(), Remote: d.joined})
|
||||
Addr: d.joined.Addr, Roles: roles, Remote: d.joined})
|
||||
continue
|
||||
}
|
||||
g := &onboard.GoSSH{
|
||||
@@ -490,9 +540,8 @@ func (m *setupModel) startCheck() tea.Cmd {
|
||||
case authPassword:
|
||||
g.Password = d.secret.Value()
|
||||
}
|
||||
m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g})
|
||||
machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr,
|
||||
Roles: m.dbRoles(), Remote: g})
|
||||
m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g, judge: judge, target: g.User + "@" + g.Addr})
|
||||
machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr, Roles: roles, Remote: g})
|
||||
}
|
||||
m.checking = true
|
||||
m.report = preflight.Report{}
|
||||
@@ -535,7 +584,7 @@ func (m setupModel) dbRoles() []preflight.Role {
|
||||
// targets are the database machines as user@address, for the command line.
|
||||
func (m setupModel) targets() []string {
|
||||
var out []string
|
||||
for _, d := range m.db {
|
||||
for _, d := range m.db[:2] {
|
||||
if d.joined != nil {
|
||||
out = append(out, "root@"+d.joined.Addr)
|
||||
continue
|
||||
@@ -545,11 +594,25 @@ func (m setupModel) targets() []string {
|
||||
return out
|
||||
}
|
||||
|
||||
// judgeTargets are the judge machines as user@address.
|
||||
func (m setupModel) judgeTargets() []judgeMachine {
|
||||
var out []judgeMachine
|
||||
for _, d := range m.db[2:m.shown()] {
|
||||
if d.joined != nil {
|
||||
out = append(out, judgeMachine{target: "root@" + d.joined.Addr})
|
||||
continue
|
||||
}
|
||||
out = append(out, judgeMachine{target: strings.TrimSpace(d.user.Value()) + "@" + strings.TrimSpace(d.addr.Value())})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// command is what this setup is on the command line, for doing it again
|
||||
// from a script — with keys in place, since a password cannot be given there.
|
||||
func (m setupModel) command() string {
|
||||
in := *m.in
|
||||
in.patroni = m.targets()
|
||||
in.judgeMachines = m.judgeTargets()
|
||||
return strings.Join(append([]string{"sudo --preserve-env=SSH_AUTH_SOCK llmgw server init"}, in.initFlags()...), " \\\n ")
|
||||
}
|
||||
|
||||
@@ -558,6 +621,7 @@ func (m setupModel) command() string {
|
||||
func (m *setupModel) startRun() tea.Cmd {
|
||||
in := m.in
|
||||
in.patroni = m.targets()
|
||||
in.judgeMachines = nil
|
||||
m.run = newInitModel(in, nil, false)
|
||||
m.run.running, m.run.width, m.run.spin = true, m.width, m.spin
|
||||
ref, conns, logp, mu, listener := m.ref, m.conns, m.log, m.logMu, m.listener
|
||||
@@ -578,6 +642,11 @@ func (m *setupModel) startRun() tea.Cmd {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if c.judge {
|
||||
// Added as a node once the control plane is up.
|
||||
in.judgeMachines = append(in.judgeMachines, judgeMachine{target: c.target, remote: c.remote})
|
||||
continue
|
||||
}
|
||||
out, err := c.remote.Run(ctx, "hostname", nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %v", c.addr, err)
|
||||
@@ -720,8 +789,12 @@ func (m setupModel) machinesView() string {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n The two machines that keep the store: Ubuntu 22.04/24.04 or Debian 12.\n" +
|
||||
" PostgreSQL, Patroni and etcd are installed on them.\n\n")
|
||||
for i, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
base := i * mfCount
|
||||
if i == 2 {
|
||||
b.WriteString(" " + sDim.Render("Machines for the guardrails' judges alone: each becomes a node that runs only\n"+
|
||||
" the judges, so a busy control plane cannot leave a request waiting for a judgment.") + "\n\n")
|
||||
}
|
||||
// Padded before it is coloured: the colour's escape codes count
|
||||
// toward a width and threw the focused row out of line.
|
||||
label := func(f int, s string) string {
|
||||
@@ -731,7 +804,7 @@ func (m setupModel) machinesView() string {
|
||||
}
|
||||
return " " + s
|
||||
}
|
||||
fmt.Fprintf(&b, " %s\n", lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1)))
|
||||
fmt.Fprintf(&b, " %s\n", lipgloss.NewStyle().Bold(true).Render(machineTitle(i)))
|
||||
if d.visible(mfAddr) {
|
||||
fmt.Fprintf(&b, " %s %s\n", label(mfAddr, "Address"), d.addr.View())
|
||||
fmt.Fprintf(&b, " %s %s\n", label(mfUser, "User"), d.user.View())
|
||||
@@ -756,7 +829,22 @@ func (m setupModel) machinesView() string {
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
b.WriteString(button(m.mfocus == 2*mfCount, "Check the three machines") + "\n\n")
|
||||
sel := m.shown() * mfCount
|
||||
var opts []string
|
||||
for n, name := range []string{"none", "one", "two"} {
|
||||
if n == m.judges {
|
||||
opts = append(opts, "("+sOK.Render("•")+") "+name)
|
||||
} else {
|
||||
opts = append(opts, "( ) "+sDim.Render(name))
|
||||
}
|
||||
}
|
||||
judgeLabel := fmt.Sprintf(" %-15s", "Judge machines")
|
||||
if m.mfocus == sel {
|
||||
judgeLabel = fSelected.Render(fmt.Sprintf("› %-15s", "Judge machines"))
|
||||
}
|
||||
fmt.Fprintf(&b, " %s %s\n", judgeLabel, strings.Join(opts, " "))
|
||||
b.WriteString(" " + sDim.Render("Without them the judges run on whichever node has room, added later.") + "\n\n")
|
||||
b.WriteString(button(m.mfocus == sel+1, fmt.Sprintf("Check the %d machines", m.shown()+1)) + "\n\n")
|
||||
b.WriteString(" " + sDim.Render("Passwords are kept in memory only, for this setup.") + "\n")
|
||||
return b.String()
|
||||
}
|
||||
@@ -764,11 +852,11 @@ func (m setupModel) machinesView() string {
|
||||
func (m setupModel) waitView() string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "\n On each machine below, as root. It dials this machine on port %d;\n nothing dials in. Each code is for one machine and good for an hour.\n\n", joinpkg.DefaultPort)
|
||||
for i, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
if d.auth != authJoin {
|
||||
continue
|
||||
}
|
||||
title := lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1))
|
||||
title := lipgloss.NewStyle().Bold(true).Render(machineTitle(i))
|
||||
if d.joined != nil {
|
||||
fmt.Fprintf(&b, " %s %s joined as %s (%s)\n\n", title, sOK.Render("✓"), d.joined.Hostname, d.joined.Addr)
|
||||
continue
|
||||
@@ -787,7 +875,7 @@ func (m setupModel) checkView() string {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n")
|
||||
if m.checking {
|
||||
b.WriteString(" " + m.spin.View() + " checking the three machines…\n")
|
||||
b.WriteString(" " + m.spin.View() + " checking the machines…\n")
|
||||
return b.String()
|
||||
}
|
||||
switch m.report.Worst() {
|
||||
@@ -796,7 +884,7 @@ func (m setupModel) checkView() string {
|
||||
case preflight.Warn:
|
||||
b.WriteString(" " + sWarn.Render("! It can go ahead; what is marked ! is worth a look.") + "\n")
|
||||
default:
|
||||
b.WriteString(" " + sOK.Render("✓ All three are ready.") + "\n")
|
||||
b.WriteString(" " + sOK.Render("✓ All are ready.") + "\n")
|
||||
}
|
||||
machine := ""
|
||||
for _, f := range m.report.Findings {
|
||||
@@ -822,7 +910,7 @@ func (m setupModel) checkView() string {
|
||||
func (m setupModel) planView() string {
|
||||
var b strings.Builder
|
||||
b.WriteString(m.choices.planView(m.width - navWidth - 6))
|
||||
b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render("The three machines") + "\n")
|
||||
b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render("The machines") + "\n")
|
||||
here := "control plane, leading at first; model mirror; etcd"
|
||||
there := "PostgreSQL under Patroni, etcd"
|
||||
if m.in.standbys {
|
||||
@@ -834,12 +922,16 @@ func (m setupModel) planView() string {
|
||||
}
|
||||
fmt.Fprintf(&b, " %s %s\n %s\n", lipgloss.NewStyle().Bold(true).Render(m.host), sDim.Render(m.thisAddr()), here)
|
||||
granted := false
|
||||
for _, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
addr := strings.TrimSpace(d.addr.Value())
|
||||
if d.joined != nil {
|
||||
addr = d.joined.Hostname + " " + sDim.Render(d.joined.Addr+", joined")
|
||||
}
|
||||
fmt.Fprintf(&b, " %s\n %s\n", lipgloss.NewStyle().Bold(true).Render(addr), there)
|
||||
role := there
|
||||
if i >= 2 {
|
||||
role = "a node for the guardrails' judges alone"
|
||||
}
|
||||
fmt.Fprintf(&b, " %s\n %s\n", lipgloss.NewStyle().Bold(true).Render(addr), role)
|
||||
granted = granted || d.sudo.Value() != ""
|
||||
}
|
||||
b.WriteString("\n The store is replicated from the first to the second; if the primary is lost\n" +
|
||||
@@ -851,7 +943,7 @@ func (m setupModel) planView() string {
|
||||
for _, l := range strings.Split(m.command(), "\n") {
|
||||
b.WriteString(" " + l + "\n")
|
||||
}
|
||||
for _, d := range m.db {
|
||||
for _, d := range m.db[:m.shown()] {
|
||||
if d.joined != nil {
|
||||
b.WriteString(" " + sDim.Render("(a machine that joined with a code needs ssh for that)") + "\n")
|
||||
break
|
||||
|
||||
@@ -0,0 +1,664 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"go.yaml.in/yaml/v3"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// How people sign in, as the console shows and sets it: single sign-on
|
||||
// through Entra ID or another OpenID provider, an Active Directory checked
|
||||
// over LDAPS, and provisioning by SCIM. The secrets these need are kept in
|
||||
// the store, so every control plane has them without a file put on each,
|
||||
// and a directory is tried before its settings are kept.
|
||||
|
||||
// inChainRule is AD's matching rule for nested group membership.
|
||||
const inChainRule = "1.2.840.113556.1.4.1941"
|
||||
|
||||
type signInState struct {
|
||||
view adminapi.AuthView
|
||||
err error
|
||||
loaded bool
|
||||
// ldap and oidc are the blocks the cluster's settings hold now, for the
|
||||
// forms to start from.
|
||||
ldap config.LDAP
|
||||
oidc config.OIDC
|
||||
}
|
||||
|
||||
type consoleSignInMsg struct {
|
||||
view adminapi.AuthView
|
||||
ldap config.LDAP
|
||||
oidc config.OIDC
|
||||
err error
|
||||
}
|
||||
|
||||
// signInRows are the panel's rows, one per way in.
|
||||
var signInRows = []string{"oidc", "ldap", "scim"}
|
||||
|
||||
func (m consoleModel) fetchSignIn() tea.Cmd {
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
var msg consoleSignInMsg
|
||||
if msg.err = c.get(ctx, "/api/v1/auth", &msg.view); msg.err != nil {
|
||||
return msg
|
||||
}
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err == nil {
|
||||
msg.ldap, msg.oidc = authBlocks(cur.Body)
|
||||
}
|
||||
return msg
|
||||
}
|
||||
}
|
||||
|
||||
// authBlocks are auth.ldap and auth.oidc as a settings document has them.
|
||||
func authBlocks(body string) (config.LDAP, config.OIDC) {
|
||||
var doc struct {
|
||||
Auth struct {
|
||||
LDAP yaml.Node `yaml:"ldap"`
|
||||
OIDC yaml.Node `yaml:"oidc"`
|
||||
} `yaml:"auth"`
|
||||
}
|
||||
var l config.LDAP
|
||||
var o config.OIDC
|
||||
if yaml.Unmarshal([]byte(body), &doc) != nil {
|
||||
return l, o
|
||||
}
|
||||
if doc.Auth.LDAP.Kind != 0 {
|
||||
if b, err := yaml.Marshal(&doc.Auth.LDAP); err == nil {
|
||||
// Read as kept, whether or not it would validate now.
|
||||
l, _ = config.ParseLDAP(b)
|
||||
}
|
||||
}
|
||||
if doc.Auth.OIDC.Kind != 0 {
|
||||
if m, err := yaml.Marshal(&doc.Auth.OIDC); err == nil {
|
||||
if cfg, err := config.FromMap(map[string]any{"auth": map[string]any{"oidc": yamlMap(m)}}); err == nil {
|
||||
o = cfg.Auth.OIDC
|
||||
}
|
||||
}
|
||||
}
|
||||
return l, o
|
||||
}
|
||||
|
||||
func yamlMap(b []byte) map[string]any {
|
||||
var m map[string]any
|
||||
_ = yaml.Unmarshal(b, &m)
|
||||
return m
|
||||
}
|
||||
|
||||
func (m consoleModel) signInPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ row · Enter actions · E Entra ID · D Active Directory · T try the directory · ← back · Q leave"
|
||||
}
|
||||
s := m.signIn
|
||||
var b strings.Builder
|
||||
b.WriteString("\n " + sDim.Render("How people sign in besides a password kept here. E for Entra ID, D for Active Directory.") + "\n\n")
|
||||
if s.err != nil {
|
||||
b.WriteString(" " + sBad.Render(s.err.Error()))
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
if !s.loaded {
|
||||
b.WriteString(" " + sDim.Render("asking the control plane…"))
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
v := s.view
|
||||
local := "passwords kept here; accounts made by an administrator"
|
||||
if v.SignupOpen {
|
||||
local = "passwords kept here; anyone may sign up, then wait for approval"
|
||||
}
|
||||
b.WriteString(" " + sTitle.Render("LOCAL") + " " + sDim.Render(local) + "\n\n")
|
||||
|
||||
var rows [][]string
|
||||
for _, r := range signInRows {
|
||||
switch r {
|
||||
case "oidc":
|
||||
if v.OIDC == nil {
|
||||
rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", "off", "E to set up"})
|
||||
continue
|
||||
}
|
||||
state := "on"
|
||||
if v.OIDC.NoPasswords {
|
||||
state = "on, the only way in"
|
||||
}
|
||||
rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", state, or(v.OIDC.Label, v.OIDC.Issuer)})
|
||||
case "ldap":
|
||||
if v.LDAP == nil {
|
||||
rows = append(rows, []string{"Active Directory (LDAP)", "off", "D to set up"})
|
||||
continue
|
||||
}
|
||||
state := "on"
|
||||
if v.LDAP.Off != "" {
|
||||
state = "broken here"
|
||||
}
|
||||
rows = append(rows, []string{"Active Directory (LDAP)", state, strings.Join(v.LDAP.URLs, " ")})
|
||||
case "scim":
|
||||
state, detail := "off", "Enter to make a token"
|
||||
if v.SCIM {
|
||||
state, detail = "on", "at /scim/v2"
|
||||
}
|
||||
rows = append(rows, []string{"Provisioning (SCIM)", state, detail})
|
||||
}
|
||||
}
|
||||
b.WriteString(table([]string{"WAY IN", "STATE", ""}, rows, m.row, pw, m.inPanel) + "\n")
|
||||
|
||||
if v.OIDC != nil {
|
||||
b.WriteString("\n " + sTitle.Render("SINGLE SIGN-ON") + "\n")
|
||||
b.WriteString(" issuer " + v.OIDC.Issuer + "\n")
|
||||
b.WriteString(" redirect " + redirectURL(v.BaseURL) + "\n")
|
||||
b.WriteString(" " + sDim.Render(pickText(v.OIDC.CreateAccounts, "accounts made at first sign-in", "only people with an account sign in")) + "\n")
|
||||
}
|
||||
if l := v.LDAP; l != nil {
|
||||
b.WriteString("\n " + sTitle.Render("ACTIVE DIRECTORY") + " " + sDim.Render(l.BaseDN) + "\n")
|
||||
if l.Off != "" {
|
||||
b.WriteString(" " + sBad.Render("off on this control plane: "+l.Off) + "\n")
|
||||
}
|
||||
switch ls := l.LastSync; {
|
||||
case ls == nil:
|
||||
b.WriteString(" " + sDim.Render("not read yet: the leader reads it every sync_interval") + "\n")
|
||||
case ls.Error != "":
|
||||
b.WriteString(" " + sBad.Render("last read "+agoText(ls.At)+": "+ls.Error) + "\n")
|
||||
default:
|
||||
b.WriteString(fmt.Sprintf(" last read %s: %d people there, %d accounts kept in step, %d suspended, %d reinstated\n",
|
||||
agoText(ls.At), ls.People, ls.Accounts, ls.Suspended, ls.Reinstated))
|
||||
}
|
||||
}
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
|
||||
func agoText(t time.Time) string { return time.Since(t).Round(time.Second).String() + " ago" }
|
||||
|
||||
func pickText(b bool, yes, no string) string {
|
||||
if b {
|
||||
return yes
|
||||
}
|
||||
return no
|
||||
}
|
||||
|
||||
// redirectURL is what an identity provider sends people back to.
|
||||
func redirectURL(base string) string {
|
||||
if base == "" {
|
||||
return "https://<this gateway>/login/oidc/callback"
|
||||
}
|
||||
return strings.TrimRight(base, "/") + "/login/oidc/callback"
|
||||
}
|
||||
|
||||
// signInMenu is what can be done to one way in.
|
||||
func (m consoleModel) signInMenu(row string) *menuDialog {
|
||||
v := m.signIn.view
|
||||
switch row {
|
||||
case "oidc":
|
||||
d := &menuDialog{title: "Single sign-on", items: []menuItem{
|
||||
{"Set up Entra ID…", func(m *consoleModel) tea.Cmd { m.entraForm(); return nil }},
|
||||
}}
|
||||
if v.OIDC != nil {
|
||||
d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("oidc", "Single sign-on", "People sign in with a password kept here again; accounts made through the provider have none.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
case "ldap":
|
||||
d := &menuDialog{title: "Active Directory", items: []menuItem{
|
||||
{"Set up…", func(m *consoleModel) tea.Cmd { m.directoryForm(); return nil }},
|
||||
}}
|
||||
if v.LDAP != nil {
|
||||
d.items = append(d.items,
|
||||
menuItem{"Try it", func(m *consoleModel) tea.Cmd { return m.tryDirectory() }},
|
||||
menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("ldap", "Active Directory", "Its people can no longer sign in; their accounts stay, and are no longer kept in step.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
default:
|
||||
d := &menuDialog{title: "Provisioning (SCIM)", items: []menuItem{
|
||||
{"Make a new token", func(m *consoleModel) tea.Cmd { return m.scimToken() }},
|
||||
}}
|
||||
if v.SCIM {
|
||||
d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("scim", "Provisioning", "The provider can no longer make, change or suspend accounts here.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
}
|
||||
}
|
||||
|
||||
// signInRowKey is the way in the cursor is on.
|
||||
func (m consoleModel) signInRowKey() string {
|
||||
if m.row >= 0 && m.row < len(signInRows) {
|
||||
return signInRows[m.row]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// directoryForm sets up auth.ldap, starting from what is kept.
|
||||
func (m *consoleModel) directoryForm() {
|
||||
l := m.signIn.ldap
|
||||
allowed := ""
|
||||
if f := l.UserFilter; strings.HasPrefix(f, "(memberOf:"+inChainRule+":=") {
|
||||
allowed = strings.TrimSuffix(strings.TrimPrefix(f, "(memberOf:"+inChainRule+":="), ")")
|
||||
} else {
|
||||
allowed = f
|
||||
}
|
||||
pwHint := "the service account's password"
|
||||
if l.BindPasswordFile != "" {
|
||||
pwHint = "empty keeps the one kept"
|
||||
}
|
||||
caHint := "a PEM file on this machine; empty uses the system's"
|
||||
if l.CA != "" || l.CAFile != "" {
|
||||
caHint = "empty keeps the one kept; - for the system's"
|
||||
}
|
||||
create := "yes"
|
||||
if l.Enabled() && !l.CreateAccounts {
|
||||
create = "no"
|
||||
}
|
||||
var teams []string
|
||||
for g, place := range l.Memberships {
|
||||
teams = append(teams, g+"="+place)
|
||||
}
|
||||
pw := textF("password", "", pwHint)
|
||||
pw.text.Mask = true
|
||||
m.openForm(&formDialog{title: "Active Directory",
|
||||
note: "People sign in with their AD name and password, checked over LDAPS. Tried before it is kept.",
|
||||
fields: []formField{
|
||||
textF("servers", strings.Join(l.URLs, " "), "ldaps://dc1.corp.example ldaps://dc2.corp.example"),
|
||||
textF("people under", l.BaseDN, "OU=Staff,DC=corp,DC=example"),
|
||||
textF("service account", l.BindDN, "CN=svc-llmgw,OU=Service,DC=corp,DC=example"),
|
||||
pw,
|
||||
textF("CA certificate", "", caHint),
|
||||
textF("who may sign in", allowed, "a group's DN, nested groups count; empty is everyone under it"),
|
||||
textF("administrators", strings.Join(l.Roles["admin"], ", "), "group names or DNs"),
|
||||
textF("operators", strings.Join(l.Roles["operator"], ", "), "group names or DNs"),
|
||||
textF("teams", strings.Join(teams, "; "), "ml-team=team:Acme/ml; sales=org:Sales"),
|
||||
textF("label", l.Label, "the form says: Sign in with your LABEL account"),
|
||||
choiceF("make accounts", []string{"yes", "no"}, create),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
next, err := directoryFromForm(l, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.saveDirectory(next, v[3]), nil
|
||||
}})
|
||||
}
|
||||
|
||||
// directoryFromForm is the auth.ldap block the form's values make, over
|
||||
// what was kept.
|
||||
func directoryFromForm(kept config.LDAP, v []string) (config.LDAP, error) {
|
||||
l := kept
|
||||
l.URLs = strings.Fields(strings.ReplaceAll(v[0], ",", " "))
|
||||
l.BaseDN, l.BindDN = v[1], v[2]
|
||||
if v[3] != "" || l.BindPasswordFile == "" {
|
||||
l.BindPasswordFile = adminapi.AuthSecretRef("ldap-bind")
|
||||
}
|
||||
switch ca := v[4]; ca {
|
||||
case "":
|
||||
case "-":
|
||||
l.CA, l.CAFile = "", ""
|
||||
default:
|
||||
b, err := os.ReadFile(ca) //nolint:gosec // a file the operator names
|
||||
if err != nil {
|
||||
return l, fmt.Errorf("CA certificate: %w", err)
|
||||
}
|
||||
l.CA, l.CAFile = string(b), ""
|
||||
}
|
||||
switch g := v[5]; {
|
||||
case g == "":
|
||||
l.UserFilter = ""
|
||||
case strings.HasPrefix(g, "("):
|
||||
l.UserFilter = g
|
||||
case strings.Contains(g, "="):
|
||||
l.UserFilter = "(memberOf:" + inChainRule + ":=" + g + ")"
|
||||
default:
|
||||
return l, errors.New("who may sign in: a group's DN (CN=…,OU=…,DC=…), or an LDAP filter in parentheses")
|
||||
}
|
||||
l.Roles = map[string][]string{}
|
||||
for role, i := range map[string]int{"admin": 6, "operator": 7} {
|
||||
for _, g := range strings.Split(v[i], ",") {
|
||||
if g = strings.TrimSpace(g); g != "" {
|
||||
l.Roles[role] = append(l.Roles[role], g)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(l.Roles) == 0 {
|
||||
l.Roles = nil
|
||||
}
|
||||
l.Memberships = nil
|
||||
for _, pair := range strings.Split(v[8], ";") {
|
||||
if pair = strings.TrimSpace(pair); pair == "" {
|
||||
continue
|
||||
}
|
||||
g, place, ok := strings.Cut(pair, "=")
|
||||
if !ok {
|
||||
return l, fmt.Errorf("teams: %q is not GROUP=team:ORG/TEAM", pair)
|
||||
}
|
||||
if l.Memberships == nil {
|
||||
l.Memberships = map[string]string{}
|
||||
}
|
||||
l.Memberships[strings.TrimSpace(g)] = strings.TrimSpace(place)
|
||||
}
|
||||
l.Label, l.CreateAccounts = v[9], v[10] == "yes"
|
||||
return l, l.Validate()
|
||||
}
|
||||
|
||||
// ldapYAML is a block as the cluster's settings write it.
|
||||
func ldapYAML(l config.LDAP) string {
|
||||
m := map[string]any{"urls": l.URLs, "bind_dn": l.BindDN, "bind_password_file": l.BindPasswordFile,
|
||||
"base_dn": l.BaseDN, "create_accounts": l.CreateAccounts}
|
||||
put := func(k, v string) {
|
||||
if v != "" {
|
||||
m[k] = v
|
||||
}
|
||||
}
|
||||
put("ca", l.CA)
|
||||
put("ca_file", l.CAFile)
|
||||
put("user_filter", l.UserFilter)
|
||||
put("group_base_dn", l.GroupBaseDN)
|
||||
put("label", l.Label)
|
||||
if l.StartTLS {
|
||||
m["start_tls"] = true
|
||||
}
|
||||
if len(l.Roles) > 0 {
|
||||
m["roles"] = l.Roles
|
||||
}
|
||||
if len(l.Memberships) > 0 {
|
||||
m["memberships"] = l.Memberships
|
||||
}
|
||||
if l.SyncInterval != 0 {
|
||||
m["sync_interval"] = l.SyncInterval.String()
|
||||
}
|
||||
if l.Timeout != 0 {
|
||||
m["timeout"] = l.Timeout.String()
|
||||
}
|
||||
b, _ := yaml.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// saveDirectory tries the block, and only if every step passes keeps the
|
||||
// password and puts the block in the cluster's settings.
|
||||
func (m *consoleModel) saveDirectory(l config.LDAP, password string) tea.Cmd {
|
||||
m.busy = "Active Directory"
|
||||
c, block := m.client, ldapYAML(l)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
var tv adminapi.LDAPTestView
|
||||
if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test",
|
||||
adminapi.LDAPTestRequest{Settings: block, Password: password}, &tv); err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
report := checksText(tv.Checks)
|
||||
if !tv.OK {
|
||||
return actionDoneMsg{title: "Active Directory", err: errors.New("not kept: the directory did not pass\n\n" + report)}
|
||||
}
|
||||
if password != "" {
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/ldap-bind", adminapi.SecretRequest{Value: password}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.ldap from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "ldap"}, block)
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
return actionDoneMsg{title: "Active Directory", text: report + "\n\n" + said}
|
||||
}
|
||||
}
|
||||
|
||||
// tryDirectory tries the block in force.
|
||||
func (m *consoleModel) tryDirectory() tea.Cmd {
|
||||
m.busy = "Trying the directory"
|
||||
c, block := m.client, ldapYAML(m.signIn.ldap)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
var tv adminapi.LDAPTestView
|
||||
if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test", adminapi.LDAPTestRequest{Settings: block}, &tv); err != nil {
|
||||
return actionDoneMsg{title: "The directory", err: err}
|
||||
}
|
||||
if !tv.OK {
|
||||
return actionDoneMsg{title: "The directory", err: errors.New("it did not pass\n\n" + checksText(tv.Checks))}
|
||||
}
|
||||
return actionDoneMsg{title: "The directory", text: checksText(tv.Checks)}
|
||||
}
|
||||
}
|
||||
|
||||
func checksText(checks []adminapi.CheckView) string {
|
||||
var b strings.Builder
|
||||
for i, c := range checks {
|
||||
if i > 0 {
|
||||
b.WriteString("\n")
|
||||
}
|
||||
mark := sOK.Render("✓")
|
||||
if !c.OK {
|
||||
mark = sBad.Render("✗")
|
||||
}
|
||||
fmt.Fprintf(&b, "%s %s: %s", mark, c.Step, c.Detail)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// entraForm sets up single sign-on through Entra ID, or another provider
|
||||
// when an issuer address is given in place of the tenant.
|
||||
func (m *consoleModel) entraForm() {
|
||||
o := m.signIn.oidc
|
||||
tenant := strings.TrimSuffix(strings.TrimPrefix(o.Issuer, "https://login.microsoftonline.com/"), "/v2.0")
|
||||
secretHint := "the app registration's client secret"
|
||||
if o.ClientSecretFile != "" {
|
||||
secretHint = "empty keeps the one kept"
|
||||
}
|
||||
create, only := "yes", "no"
|
||||
if o.Enabled() && !o.CreateAccounts {
|
||||
create = "no"
|
||||
}
|
||||
if o.NoPasswords {
|
||||
only = "yes"
|
||||
}
|
||||
secret := textF("client secret", "", secretHint)
|
||||
secret.text.Mask = true
|
||||
m.openForm(&formDialog{title: "Single sign-on: Entra ID",
|
||||
note: "Register an app in Entra with the redirect address " + redirectURL(m.signIn.view.BaseURL) +
|
||||
", and a groups claim of the groups assigned to it.",
|
||||
fields: []formField{
|
||||
textF("tenant", tenant, "the directory (tenant) ID; or another provider's issuer URL"),
|
||||
textF("client ID", o.ClientID, "the application (client) ID"),
|
||||
secret,
|
||||
textF("administrators", strings.Join(o.Roles["admin"], ", "), "group object IDs, as Entra sends them"),
|
||||
textF("operators", strings.Join(o.Roles["operator"], ", "), "group object IDs"),
|
||||
textF("label", or(o.Label, "Microsoft"), "the button says: Sign in with LABEL"),
|
||||
choiceF("make accounts", []string{"yes", "no"}, create),
|
||||
choiceF("only this way", []string{"no", "yes"}, only),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
next, err := oidcFromForm(o, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.saveOIDC(next, v[2]), nil
|
||||
}})
|
||||
}
|
||||
|
||||
func oidcFromForm(kept config.OIDC, v []string) (config.OIDC, error) {
|
||||
o := kept
|
||||
switch t := strings.TrimSpace(v[0]); {
|
||||
case t == "":
|
||||
return o, errors.New("tenant: the directory (tenant) ID, or an issuer URL")
|
||||
case strings.Contains(t, "://"):
|
||||
o.Issuer = t
|
||||
default:
|
||||
o.Issuer = "https://login.microsoftonline.com/" + t + "/v2.0"
|
||||
}
|
||||
if o.ClientID = v[1]; o.ClientID == "" {
|
||||
return o, errors.New("client ID: the application (client) ID")
|
||||
}
|
||||
if v[2] != "" || o.ClientSecretFile == "" {
|
||||
if v[2] == "" {
|
||||
return o, errors.New("client secret: the app registration's secret")
|
||||
}
|
||||
o.ClientSecretFile = adminapi.AuthSecretRef("oidc-client-secret")
|
||||
}
|
||||
o.Roles = map[string][]string{}
|
||||
for role, i := range map[string]int{"admin": 3, "operator": 4} {
|
||||
for _, g := range strings.Split(v[i], ",") {
|
||||
if g = strings.TrimSpace(g); g != "" {
|
||||
o.Roles[role] = append(o.Roles[role], g)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(o.Roles) == 0 {
|
||||
o.Roles = nil
|
||||
}
|
||||
o.Label, o.CreateAccounts, o.NoPasswords = v[5], v[6] == "yes", v[7] == "yes"
|
||||
return o, o.Validate()
|
||||
}
|
||||
|
||||
func oidcYAML(o config.OIDC) string {
|
||||
m := map[string]any{"issuer": o.Issuer, "client_id": o.ClientID, "client_secret_file": o.ClientSecretFile,
|
||||
"create_accounts": o.CreateAccounts}
|
||||
if o.Label != "" {
|
||||
m["label"] = o.Label
|
||||
}
|
||||
if o.NoPasswords {
|
||||
m["no_passwords"] = true
|
||||
}
|
||||
if o.RedirectURL != "" {
|
||||
m["redirect_url"] = o.RedirectURL
|
||||
}
|
||||
if len(o.Scopes) > 0 {
|
||||
m["scopes"] = o.Scopes
|
||||
}
|
||||
if o.GroupsClaim != "" {
|
||||
m["groups_claim"] = o.GroupsClaim
|
||||
}
|
||||
if len(o.Roles) > 0 {
|
||||
m["roles"] = o.Roles
|
||||
}
|
||||
if len(o.Memberships) > 0 {
|
||||
m["memberships"] = o.Memberships
|
||||
}
|
||||
b, _ := yaml.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func (m *consoleModel) saveOIDC(o config.OIDC, secret string) tea.Cmd {
|
||||
m.busy = "Single sign-on"
|
||||
c, block, redirect := m.client, oidcYAML(o), redirectURL(m.signIn.view.BaseURL)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if secret != "" {
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/oidc-client-secret", adminapi.SecretRequest{Value: secret}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Single sign-on", err: err}
|
||||
}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.oidc from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "oidc"}, block)
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Single sign-on", err: err}
|
||||
}
|
||||
return actionDoneMsg{title: "Single sign-on", text: said + "\n\nIn the app registration the redirect address is " + redirect +
|
||||
". For people in more than 200 groups, emit the groups assigned to the application."}
|
||||
}
|
||||
}
|
||||
|
||||
// scimToken makes a new provisioning token, keeps it, and shows it once.
|
||||
func (m *consoleModel) scimToken() tea.Cmd {
|
||||
m.busy = "Provisioning"
|
||||
c, base := m.client, m.signIn.view.BaseURL
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
token := hex.EncodeToString(b)
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/scim-token", adminapi.SecretRequest{Value: token}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.scim from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "scim", "token_file"}, adminapi.AuthSecretRef("scim-token"))
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
url := "https://<this gateway>/scim/v2"
|
||||
if base != "" {
|
||||
url = strings.TrimRight(base, "/") + "/scim/v2"
|
||||
}
|
||||
return actionDoneMsg{title: "Provisioning", text: "Give the provider the tenant URL " + url + " and this token, shown only now:\n\n " +
|
||||
token + "\n\n" + said}
|
||||
}
|
||||
}
|
||||
|
||||
// signInOffConfirm turns one way in off, after asking.
|
||||
func (m *consoleModel) signInOffConfirm(key, title, text string) {
|
||||
m.confirmD = &confirmDialog{title: "Turn off " + strings.ToLower(title), text: text,
|
||||
yes: func(m *consoleModel) tea.Cmd {
|
||||
m.busy = title
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
said, err := changeClusterSettings(ctx, c, "auth."+key+" off from the console", func(doc *settings.Doc) error {
|
||||
doc.Unset([]string{"auth", key})
|
||||
return nil
|
||||
})
|
||||
return actionDoneMsg{title: title, text: said, err: err}
|
||||
}
|
||||
}}
|
||||
}
|
||||
|
||||
// changeClusterSettings is changeSettings for the console: the newest
|
||||
// settings read, changed, and put in force, said in a sentence.
|
||||
func changeClusterSettings(ctx context.Context, c *ctlClient, note string, change func(doc *settings.Doc) error) (string, error) {
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return "", err
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := change(doc); err != nil {
|
||||
return "", err
|
||||
}
|
||||
body, err := doc.Bytes()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
before, _ := config.Parse([]byte(cur.Body))
|
||||
after, err := config.Parse(body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
changed := config.Changed(before, after)
|
||||
if len(changed) == 0 {
|
||||
return "Nothing changed.", nil
|
||||
}
|
||||
var v adminapi.SettingsView
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/settings",
|
||||
adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note}, &v); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprintf("Settings version %d. Every control plane starts again on it within seconds, standbys first; "+
|
||||
"engines keep serving.", v.Version), nil
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// adForm is the directory form's values, in its order.
|
||||
func adForm(password, ca, allowed, teams string) []string {
|
||||
return []string{"ldaps://dc1.corp.example, ldaps://dc2.corp.example", "OU=Staff,DC=corp,DC=example",
|
||||
"CN=svc,OU=Service,DC=corp,DC=example", password, ca, allowed, "llmgw-admins", "", teams, "Acme", "yes"}
|
||||
}
|
||||
|
||||
func TestTheDirectoryFormMakesTheBlockTheSettingsKeep(t *testing.T) {
|
||||
pem := "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n"
|
||||
caFile := filepath.Join(t.TempDir(), "ca.pem")
|
||||
if err := os.WriteFile(caFile, []byte(pem), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, err := directoryFromForm(config.LDAP{}, adForm("pw", caFile, "CN=llmgw-users,OU=Groups,DC=corp,DC=example",
|
||||
"ml-team=team:Acme/ml; sales=org:Sales"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The group, nested ones counting; the CA written into the settings;
|
||||
// the password named in the store, not a file on one machine.
|
||||
if l.UserFilter != "(memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)" ||
|
||||
l.CA != pem || l.BindPasswordFile != "store:auth/ldap-bind" || len(l.URLs) != 2 ||
|
||||
l.Memberships["sales"] != "org:Sales" || l.Roles["admin"][0] != "llmgw-admins" || l.Roles["operator"] != nil {
|
||||
t.Errorf("%+v", l)
|
||||
}
|
||||
// What the settings keep reads back the same.
|
||||
back, err := config.ParseLDAP([]byte(ldapYAML(l)))
|
||||
if err != nil || back.UserFilter != l.UserFilter || back.CA != pem || back.Memberships["ml-team"] != "team:Acme/ml" {
|
||||
t.Errorf("read back %+v %v", back, err)
|
||||
}
|
||||
|
||||
// Changed again with nothing typed for the password or the CA: both kept.
|
||||
again, err := directoryFromForm(l, adForm("", "", "(objectClass=user)", ""))
|
||||
if err != nil || again.CA != pem || again.BindPasswordFile != l.BindPasswordFile || again.UserFilter != "(objectClass=user)" ||
|
||||
again.Memberships != nil {
|
||||
t.Errorf("kept: %+v %v", again, err)
|
||||
}
|
||||
// A group by its bare name cannot be a filter, and a plaintext server is refused.
|
||||
if _, err := directoryFromForm(config.LDAP{}, adForm("pw", "", "llmgw-users", "")); err == nil {
|
||||
t.Error("a bare group name was taken")
|
||||
}
|
||||
plain := adForm("pw", "", "", "")
|
||||
plain[0] = "ldap://dc1"
|
||||
if _, err := directoryFromForm(config.LDAP{}, plain); err == nil || !strings.Contains(err.Error(), "plaintext") {
|
||||
t.Errorf("plaintext: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheEntraFormMakesTheIssuerFromTheTenant(t *testing.T) {
|
||||
o, err := oidcFromForm(config.OIDC{}, []string{"0f2c-tenant", "app-id", "s3cret", "g-admins", "", "Microsoft", "yes", "no"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if o.Issuer != "https://login.microsoftonline.com/0f2c-tenant/v2.0" || o.ClientSecretFile != "store:auth/oidc-client-secret" ||
|
||||
o.Roles["admin"][0] != "g-admins" || !o.CreateAccounts || o.NoPasswords {
|
||||
t.Errorf("%+v", o)
|
||||
}
|
||||
// Another provider by its issuer; the secret kept when none is typed.
|
||||
k, err := oidcFromForm(o, []string{"https://sso.example/realms/acme", "llmgw", "", "", "", "Acme SSO", "no", "yes"})
|
||||
if err != nil || k.Issuer != "https://sso.example/realms/acme" || k.ClientSecretFile != o.ClientSecretFile || !k.NoPasswords {
|
||||
t.Errorf("%+v %v", k, err)
|
||||
}
|
||||
if _, err := oidcFromForm(config.OIDC{}, []string{"t", "app", "", "", "", "", "yes", "no"}); err == nil {
|
||||
t.Error("a first setup without a secret was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSignInPanelSaysWhatIsOnAndHowTheDirectoryLastWent(t *testing.T) {
|
||||
m := consoleModel{menu: cSignIn, loaded: true, width: 120, height: 40}
|
||||
m.signIn = signInState{loaded: true, view: adminapi.AuthView{
|
||||
BaseURL: "https://llm.example",
|
||||
OIDC: &adminapi.OIDCView{Issuer: "https://login.microsoftonline.com/t/v2.0", Label: "Microsoft", CreateAccounts: true},
|
||||
LDAP: &adminapi.LDAPView{URLs: []string{"ldaps://dc1"}, BaseDN: "OU=Staff,DC=corp",
|
||||
LastSync: &adminapi.DirectorySyncView{At: time.Now().Add(-time.Minute), People: 40, Accounts: 12, Suspended: 1}},
|
||||
}}
|
||||
_, body, _ := m.signInPanel(110)
|
||||
for _, want := range []string{"Single sign-on (Entra ID, OIDC)", "Microsoft", "https://llm.example/login/oidc/callback",
|
||||
"Active Directory (LDAP)", "ldaps://dc1", "40 people there, 12 accounts kept in step, 1 suspended", "Provisioning (SCIM)"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the panel lacks %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
m.signIn.view.LDAP.Off = "auth.ldap.ca: holds no certificate"
|
||||
if _, body, _ := m.signInPanel(110); !strings.Contains(body, "broken here") || !strings.Contains(body, "holds no certificate") {
|
||||
t.Errorf("a directory off here is not said:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuthBlocksAreReadFromTheSettingsDocument(t *testing.T) {
|
||||
body := "auth:\n ldap:\n urls: [ldaps://dc1]\n base_dn: DC=corp\n bind_dn: CN=svc\n bind_password_file: store:auth/ldap-bind\n" +
|
||||
" sync_interval: 1m\n oidc:\n issuer: https://sso.example\n client_id: llmgw\n"
|
||||
l, o := authBlocks(body)
|
||||
if l.BaseDN != "DC=corp" || l.SyncInterval != time.Minute || o.ClientID != "llmgw" {
|
||||
t.Errorf("%+v %+v", l, o)
|
||||
}
|
||||
}
|
||||
@@ -503,6 +503,10 @@ type Personal struct {
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// CachedPer1k is the cost of a prompt token the engine or provider read
|
||||
// from its cache (the answer's cached_tokens, counted within the
|
||||
// prompt's). Zero charges them at prompt_per_1k.
|
||||
CachedPer1k int64 `koanf:"cached_per_1k"`
|
||||
// PerImage is the cost of one picture, for a model that makes them
|
||||
// (/v1/images/generations), on top of its prompt's tokens.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
@@ -794,6 +798,11 @@ type Cache struct {
|
||||
// MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
// unless set.
|
||||
MaxEntryBytes int64 `koanf:"max_entry_bytes"`
|
||||
// EmbeddingModel embeds the questions of projects that reuse answers to
|
||||
// similar ones (llmgw keys cache --similar): an embedding model in the
|
||||
// catalog. Unset, those projects reuse only the same request's answer.
|
||||
// The gateway asks it on its own account, as it asks a judge.
|
||||
EmbeddingModel string `koanf:"embedding_model"`
|
||||
}
|
||||
|
||||
// Default returns the configuration used when nothing is specified.
|
||||
|
||||
@@ -6,6 +6,9 @@ import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/knadh/koanf/parsers/yaml"
|
||||
"github.com/knadh/koanf/v2"
|
||||
)
|
||||
|
||||
// LDAP is sign-in against a directory — Active Directory above all — with
|
||||
@@ -38,6 +41,10 @@ type LDAP struct {
|
||||
// CAFile is the authority the servers' certificates are checked against,
|
||||
// when it is not the system's: an AD Certificate Services root.
|
||||
CAFile string `koanf:"ca_file"`
|
||||
// CA is the same authority written out, PEM: kept in the cluster's
|
||||
// settings, so no file has to be put on every control plane. Used in
|
||||
// place of CAFile when both are set.
|
||||
CA string `koanf:"ca"`
|
||||
// BindDN and the file holding its password: a service account that may
|
||||
// read users and groups. Nothing is written to the directory.
|
||||
BindDN string `koanf:"bind_dn"`
|
||||
@@ -99,6 +106,8 @@ func (l LDAP) Validate() error {
|
||||
switch {
|
||||
case strings.TrimSpace(l.BindDN) == "" || l.BindPasswordFile == "":
|
||||
return errors.New("auth.ldap: bind_dn and bind_password_file are needed, for the account that finds people")
|
||||
case l.CA != "" && !strings.Contains(l.CA, "-----BEGIN CERTIFICATE-----"):
|
||||
return errors.New("auth.ldap.ca: not a PEM certificate")
|
||||
case strings.TrimSpace(l.BaseDN) == "":
|
||||
return errors.New("auth.ldap: base_dn is needed, where people are searched for")
|
||||
case l.UserFilter != "" && !(strings.HasPrefix(l.UserFilter, "(") && strings.HasSuffix(l.UserFilter, ")")):
|
||||
@@ -135,3 +144,17 @@ func (l LDAP) RoundTrip() time.Duration {
|
||||
}
|
||||
return l.Timeout
|
||||
}
|
||||
|
||||
// ParseLDAP is an auth.ldap block, as YAML, read on its own as the cluster's
|
||||
// settings would read it, and checked.
|
||||
func ParseLDAP(block []byte) (LDAP, error) {
|
||||
var l LDAP
|
||||
k := koanf.New(".")
|
||||
if err := k.Load(bytesProvider(block), yaml.Parser()); err != nil {
|
||||
return l, fmt.Errorf("auth.ldap: %w", err)
|
||||
}
|
||||
if err := k.UnmarshalWithConf("", &l, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
|
||||
return l, fmt.Errorf("auth.ldap: %w", err)
|
||||
}
|
||||
return l, l.Validate()
|
||||
}
|
||||
@@ -76,6 +76,8 @@ type API struct {
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
settings *Settings
|
||||
auth *AuthAdmin
|
||||
mcp MCPStore
|
||||
// restartAgents tells every connected agent to start again.
|
||||
restartAgents func(reason string) []string
|
||||
// revoke records a removed node, so its certificate is refused.
|
||||
@@ -129,6 +131,8 @@ func (a *API) Register(mux Mux) {
|
||||
a.registerTrain(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
a.registerAuth(mux)
|
||||
a.registerMCP(mux)
|
||||
mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
|
||||
mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
|
||||
mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AuthAdmin is what the sign-in endpoints need: how people sign in now, a
|
||||
// directory's settings tried before they are kept, and where the secrets
|
||||
// those settings name are kept for every control plane.
|
||||
type AuthAdmin struct {
|
||||
Status func() AuthView
|
||||
// TestLDAP tries an auth.ldap block, as YAML, binding with password
|
||||
// when one is given rather than the one the block names.
|
||||
TestLDAP func(ctx context.Context, settings, password string) ([]CheckView, error)
|
||||
Secrets interface {
|
||||
SetClusterSecret(ctx context.Context, name, value string) error
|
||||
}
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// SetAuth enables the sign-in endpoints.
|
||||
func (a *API) SetAuth(x *AuthAdmin) { a.auth = x }
|
||||
|
||||
// AuthView is how people sign in on this gateway.
|
||||
type AuthView struct {
|
||||
SignupOpen bool `json:"signup_open"`
|
||||
// OIDC is single sign-on through an identity provider: Entra ID,
|
||||
// Keycloak, Okta.
|
||||
OIDC *OIDCView `json:"oidc,omitempty"`
|
||||
// LDAP is sign-in checked by a directory: Active Directory.
|
||||
LDAP *LDAPView `json:"ldap,omitempty"`
|
||||
// SCIM is whether a provider may keep accounts in step.
|
||||
SCIM bool `json:"scim"`
|
||||
// BaseURL is the portal's address, when the settings say it: what a
|
||||
// provider's redirect address is made from.
|
||||
BaseURL string `json:"base_url,omitempty"`
|
||||
}
|
||||
|
||||
// OIDCView is auth.oidc as it is in force.
|
||||
type OIDCView struct {
|
||||
Issuer string `json:"issuer"`
|
||||
ClientID string `json:"client_id"`
|
||||
Label string `json:"label,omitempty"`
|
||||
CreateAccounts bool `json:"create_accounts"`
|
||||
NoPasswords bool `json:"no_passwords"`
|
||||
}
|
||||
|
||||
// LDAPView is auth.ldap as it is in force, and how its last sync went.
|
||||
type LDAPView struct {
|
||||
URLs []string `json:"urls"`
|
||||
BaseDN string `json:"base_dn"`
|
||||
BindDN string `json:"bind_dn"`
|
||||
Label string `json:"label,omitempty"`
|
||||
// Off is why this control plane could not use the settings, when it
|
||||
// could not; local accounts still sign in.
|
||||
Off string `json:"off,omitempty"`
|
||||
LastSync *DirectorySyncView `json:"last_sync,omitempty"`
|
||||
}
|
||||
|
||||
// DirectorySyncView is the last time the leader read the directory.
|
||||
type DirectorySyncView struct {
|
||||
At time.Time `json:"at"`
|
||||
People int `json:"people"`
|
||||
Accounts int `json:"accounts"`
|
||||
Suspended int `json:"suspended"`
|
||||
Reinstated int `json:"reinstated"`
|
||||
RolesChanged int `json:"roles_changed"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// CheckView is one step of trying a directory.
|
||||
type CheckView struct {
|
||||
Step string `json:"step"`
|
||||
OK bool `json:"ok"`
|
||||
Detail string `json:"detail"`
|
||||
}
|
||||
|
||||
// LDAPTestRequest is an auth.ldap block to try, as YAML, and the service
|
||||
// account's password when it is not kept yet.
|
||||
type LDAPTestRequest struct {
|
||||
Settings string `json:"settings"`
|
||||
Password string `json:"password,omitempty"`
|
||||
}
|
||||
|
||||
// LDAPTestView is how trying the directory went.
|
||||
type LDAPTestView struct {
|
||||
OK bool `json:"ok"`
|
||||
Checks []CheckView `json:"checks"`
|
||||
}
|
||||
|
||||
// SecretRequest is a secret for the store to keep.
|
||||
type SecretRequest struct {
|
||||
Value string `json:"value"`
|
||||
}
|
||||
|
||||
// authSecrets are the secrets an operator may give through the API, and
|
||||
// the reference the settings then name each by.
|
||||
var authSecrets = map[string]bool{"ldap-bind": true, "oidc-client-secret": true, "scim-token": true}
|
||||
|
||||
// AuthSecretRef is the reference auth's settings name a kept secret by.
|
||||
func AuthSecretRef(name string) string { return "store:auth/" + name }
|
||||
|
||||
func (a *API) registerAuth(mux Mux) {
|
||||
need := func(w http.ResponseWriter) bool {
|
||||
if a.auth == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no accounts")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
mux.HandleFunc("GET /api/v1/auth", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.auth.Status())
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/auth/ldap/test", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
var req LDAPTestRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
checks, err := a.auth.TestLDAP(r.Context(), req.Settings, req.Password)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
v := LDAPTestView{OK: len(checks) > 0, Checks: checks}
|
||||
for _, c := range checks {
|
||||
v.OK = v.OK && c.OK
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/auth/secrets/{name}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
name := r.PathValue("name")
|
||||
if !authSecrets[name] {
|
||||
writeError(w, http.StatusNotFound, "no such secret: ldap-bind, oidc-client-secret or scim-token")
|
||||
return
|
||||
}
|
||||
var req SecretRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.Value) == "" {
|
||||
writeError(w, http.StatusBadRequest, "an empty secret")
|
||||
return
|
||||
}
|
||||
if err := a.auth.Secrets.SetClusterSecret(r.Context(), "auth/"+name, req.Value); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
if a.auth.Log != nil {
|
||||
// What changed and who changed it; never the value.
|
||||
a.auth.Log.Info("a sign-in secret was set", "secret", name, "by", actorOf(r))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"ref": AuthSecretRef(name)})
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type memSecrets map[string]string
|
||||
|
||||
func (m memSecrets) SetClusterSecret(_ context.Context, name, value string) error {
|
||||
m[name] = value
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestSignInIsShownTriedAndItsSecretsKept(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
kept := memSecrets{}
|
||||
var tried string
|
||||
a.SetAuth(&AuthAdmin{
|
||||
Status: func() AuthView { return AuthView{LDAP: &LDAPView{URLs: []string{"ldaps://dc1"}}} },
|
||||
TestLDAP: func(_ context.Context, settings, password string) ([]CheckView, error) {
|
||||
tried = settings + "|" + password
|
||||
return []CheckView{{Step: "ldaps://dc1", OK: true}, {Step: "people", OK: false, Detail: "no one"}}, nil
|
||||
},
|
||||
Secrets: kept,
|
||||
})
|
||||
var v AuthView
|
||||
if code := call(t, a, "GET", "/api/v1/auth", "", &v); code != 200 || v.LDAP == nil {
|
||||
t.Fatalf("status: %d %+v", code, v)
|
||||
}
|
||||
// One step failing fails the whole.
|
||||
var tv LDAPTestView
|
||||
if code := call(t, a, "POST", "/api/v1/auth/ldap/test", `{"settings":"urls: [ldaps://dc1]","password":"pw"}`, &tv); code != 200 ||
|
||||
tv.OK || len(tv.Checks) != 2 || tried != "urls: [ldaps://dc1]|pw" {
|
||||
t.Errorf("test: %d %+v %q", code, tv, tried)
|
||||
}
|
||||
var ref map[string]string
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":"s3cret"}`, &ref); code != 200 ||
|
||||
kept["auth/ldap-bind"] != "s3cret" || ref["ref"] != "store:auth/ldap-bind" {
|
||||
t.Errorf("secret: %d %v %v", code, kept, ref)
|
||||
}
|
||||
// Only the sign-in secrets, and never an empty one.
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/engine-key", `{"value":"x"}`, nil); code != 404 {
|
||||
t.Errorf("another secret: %d", code)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":" "}`, nil); code != 400 {
|
||||
t.Errorf("an empty secret: %d", code)
|
||||
}
|
||||
if strings.Contains(kept["auth/ldap-bind"], " ") {
|
||||
t.Error("an empty secret replaced the kept one")
|
||||
}
|
||||
}
|
||||
@@ -209,6 +209,7 @@ type GuardAttachmentRequest struct {
|
||||
func (a *API) registerGuard(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
|
||||
mux.HandleFunc("POST /api/v1/guard/judges/ask", a.askJudge)
|
||||
mux.HandleFunc("POST /api/v1/guard/judges/check", a.checkJudge)
|
||||
mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.guard == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
@@ -30,6 +31,9 @@ type JudgeView struct {
|
||||
Judge bool `json:"judge"`
|
||||
Ready int `json:"ready_replicas"`
|
||||
Wanted int `json:"wanted_replicas"`
|
||||
// BudgetMs is how long a request waits for it: the shortest of its
|
||||
// policies' latency budgets.
|
||||
BudgetMs int64 `json:"budget_ms,omitempty"`
|
||||
// LastError is its last failure to give a verdict, if any.
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
LastErrorAt *time.Time `json:"last_error_at,omitempty"`
|
||||
@@ -65,6 +69,9 @@ func (a *API) Judges() []JudgeView {
|
||||
if !slices.Contains(v.Policies, name) {
|
||||
v.Policies = append(v.Policies, name)
|
||||
}
|
||||
if b := p.Judge.Budget().Milliseconds(); v.BudgetMs == 0 || b < v.BudgetMs {
|
||||
v.BudgetMs = b
|
||||
}
|
||||
if p.Fail == "closed" && p.Mode != "shadow" && !slices.Contains(v.Closed, name) {
|
||||
v.Closed = append(v.Closed, name)
|
||||
}
|
||||
@@ -198,3 +205,148 @@ func (a *API) askJudge(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// JudgeCheckRequest asks how a judge keeps up: Burst judgments at once after
|
||||
// three in turn; zero is two for each ready replica.
|
||||
type JudgeCheckRequest struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Burst int `json:"burst,omitempty"`
|
||||
}
|
||||
|
||||
// JudgeCheckView is how long a judgment took, alone and among others, set
|
||||
// against the time a request waits for one.
|
||||
type JudgeCheckView struct {
|
||||
Model string `json:"model"`
|
||||
BudgetMs int64 `json:"budget_ms"`
|
||||
Ready int `json:"ready_replicas"`
|
||||
// TookMs are three judgments of the same text, one after another.
|
||||
TookMs []int64 `json:"took_ms"`
|
||||
SlowestMs int64 `json:"slowest_ms"`
|
||||
// Burst judgments at once: the slowest of them, how many gave no
|
||||
// verdict, and how many a second the judges read.
|
||||
Burst int `json:"burst"`
|
||||
BurstSlowestMs int64 `json:"burst_slowest_ms"`
|
||||
BurstFailed int `json:"burst_failed"`
|
||||
PerSecond float64 `json:"judgments_per_second"`
|
||||
// Verdict is ok, tight (past half the budget alone, or past it among
|
||||
// others) or slow (past it alone); Advice what to do when not ok.
|
||||
Verdict string `json:"verdict"`
|
||||
Advice []string `json:"advice,omitempty"`
|
||||
}
|
||||
|
||||
// judgeCheckText is a prompt of about 280 tokens, the length the judges
|
||||
// were measured at: ordinary work, so the verdict is safe and the time is
|
||||
// what a request pays.
|
||||
const judgeCheckText = "Our support team has been asked to write a short guide for new customers who are setting " +
|
||||
"up the gateway for the first time. The guide should explain how to create an API key in the portal, how to " +
|
||||
"choose a model from the list the portal shows, and how to send a first request with curl or with the OpenAI " +
|
||||
"Python client by changing only the base URL and the key. It should also say what the usage page shows, how " +
|
||||
"budgets and rate limits are set by an administrator for each team, and what a person should do when a " +
|
||||
"request is refused because a limit was reached. Please keep the tone friendly and plain, use short " +
|
||||
"paragraphs, and add a small table that lists the three most common error codes with what each means and " +
|
||||
"what to try next. The audience is technical but new to the product, so avoid internal names, and end with " +
|
||||
"a short checklist the reader can follow on their first day. Write it in English, about four hundred words."
|
||||
|
||||
func (a *API) checkJudge(w http.ResponseWriter, r *http.Request) {
|
||||
if a.guard == nil || a.guard.Ask == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
|
||||
return
|
||||
}
|
||||
var req JudgeCheckRequest
|
||||
_ = json.NewDecoder(r.Body).Decode(&req)
|
||||
var jv *JudgeView
|
||||
for _, j := range a.Judges() {
|
||||
if req.Model == "" || j.Model == req.Model {
|
||||
j := j
|
||||
jv = &j
|
||||
break
|
||||
}
|
||||
}
|
||||
if jv == nil {
|
||||
writeError(w, http.StatusBadRequest, "no policy asks a judge of that name")
|
||||
return
|
||||
}
|
||||
if jv.Ready == 0 {
|
||||
writeError(w, http.StatusConflict, "the judge "+jv.Model+" has no replica ready: "+jv.Problem)
|
||||
return
|
||||
}
|
||||
v := JudgeCheckView{Model: jv.Model, BudgetMs: jv.BudgetMs, Ready: jv.Ready, Burst: req.Burst}
|
||||
if v.Burst <= 0 {
|
||||
v.Burst = 2 * jv.Ready
|
||||
}
|
||||
v.Burst = min(v.Burst, 32)
|
||||
var mu sync.Mutex
|
||||
start := time.Now()
|
||||
// Each text begins differently: the engine keeps what it read before,
|
||||
// and the same text again is answered from that in a tenth of the time
|
||||
// a request's own prompt takes.
|
||||
n := 0
|
||||
text := func() string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
n++
|
||||
return fmt.Sprintf("Request %d of a check begun at %s.\n", n, start.Format(time.RFC3339Nano)) + judgeCheckText
|
||||
}
|
||||
for range 3 {
|
||||
ans, err := a.guard.Ask(r.Context(), jv.Model, text(), "")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, fmt.Sprintf("the judge %s gave no verdict: %v", jv.Model, err))
|
||||
return
|
||||
}
|
||||
v.TookMs = append(v.TookMs, ans.Took.Milliseconds())
|
||||
v.SlowestMs = max(v.SlowestMs, ans.Took.Milliseconds())
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
start = time.Now()
|
||||
for range v.Burst {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
ans, err := a.guard.Ask(r.Context(), jv.Model, text(), "")
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if err != nil {
|
||||
v.BurstFailed++
|
||||
return
|
||||
}
|
||||
v.BurstSlowestMs = max(v.BurstSlowestMs, ans.Took.Milliseconds())
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if wall := time.Since(start).Seconds(); wall > 0 {
|
||||
v.PerSecond = float64(v.Burst-v.BurstFailed) / wall
|
||||
}
|
||||
v.Verdict, v.Advice = judgeVerdict(v, *jv)
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// judgeVerdict reads a check: a judgment alone past the budget refuses every
|
||||
// request under fail: closed; past half of it, or past it among others, a
|
||||
// busy moment does.
|
||||
func judgeVerdict(v JudgeCheckView, jv JudgeView) (string, []string) {
|
||||
verdict := "ok"
|
||||
switch {
|
||||
case v.BudgetMs > 0 && v.SlowestMs > v.BudgetMs:
|
||||
verdict = "slow"
|
||||
case v.BudgetMs > 0 && (v.SlowestMs*2 > v.BudgetMs || v.BurstSlowestMs > v.BudgetMs || v.BurstFailed > 0):
|
||||
verdict = "tight"
|
||||
}
|
||||
if verdict == "ok" {
|
||||
return verdict, nil
|
||||
}
|
||||
refused := "served unjudged"
|
||||
if len(jv.Closed) > 0 {
|
||||
refused = "refused"
|
||||
}
|
||||
advice := []string{
|
||||
fmt.Sprintf("a judgment past %d ms is %s: each request is judged once for its prompt and again every 600 characters of its answer", v.BudgetMs, refused),
|
||||
"more cores for each judge: llmgw ctl model resources " + v.Model + " --cores 4, then llmgw ctl restart --model " + v.Model,
|
||||
"more judges, on machines of their own: llmgw ctl scale " + v.Model + " N, or judge machines at install",
|
||||
"a judge on a GPU: llmgw ctl guard judge use qwen3guard-4b " + v.Model,
|
||||
"a longer wait: guard.policies.<policy>.judge.latency_budget in llmgw ctl config edit",
|
||||
}
|
||||
if len(jv.Closed) > 0 {
|
||||
advice = append(advice, "or fail: open, which serves what the judge did not read, flagged")
|
||||
}
|
||||
return verdict, advice
|
||||
}
|
||||
@@ -123,3 +123,44 @@ func TestANodeSaysWhetherItsFabricsAreUp(t *testing.T) {
|
||||
t.Errorf("no fabric: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A judge alone past its budget is slow; past half of it, or past it among
|
||||
// others, tight; and what to do is said, fail: open only where it is closed.
|
||||
func TestAJudgeCheckIsReadAgainstTheBudget(t *testing.T) {
|
||||
closed := JudgeView{Model: "qwen3guard-0.6b", Closed: []string{"baseline"}}
|
||||
for _, c := range []struct {
|
||||
alone, burst int64
|
||||
failed int
|
||||
want string
|
||||
}{{1200, 2400, 0, "ok"}, {2600, 3000, 0, "tight"}, {1200, 6000, 0, "tight"}, {1200, 2000, 1, "tight"}, {5200, 9000, 0, "slow"}} {
|
||||
v := JudgeCheckView{Model: closed.Model, BudgetMs: 5000, SlowestMs: c.alone, BurstSlowestMs: c.burst, BurstFailed: c.failed}
|
||||
got, advice := judgeVerdict(v, closed)
|
||||
if got != c.want || (got == "ok") != (advice == nil) {
|
||||
t.Errorf("%+v: %s %v", c, got, advice)
|
||||
}
|
||||
if got != "ok" && !strings.Contains(strings.Join(advice, " "), "fail: open") {
|
||||
t.Errorf("a closed policy is not offered fail: open: %v", advice)
|
||||
}
|
||||
}
|
||||
if _, advice := judgeVerdict(JudgeCheckView{BudgetMs: 5000, SlowestMs: 6000}, JudgeView{}); strings.Contains(strings.Join(advice, " "), "fail: open") {
|
||||
t.Error("fail: open offered to a policy already open")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJudgeWithNoReplicaIsNotTimed(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
lib := guard.Config{Policies: map[string]guard.Policy{
|
||||
"baseline": {Judge: &guard.JudgePolicy{Model: "nowhere-judge", Input: "block", LatencyBudget: 5 * time.Second}}}}
|
||||
a.SetGuard(&GuardAdmin{Library: func() guard.Config { return lib },
|
||||
Ask: func(context.Context, string, string, string) (guard.JudgeAnswer, error) {
|
||||
return guard.JudgeAnswer{}, nil
|
||||
}})
|
||||
if code := call(t, a, "POST", "/api/v1/guard/judges/check", `{}`, nil); code != 409 {
|
||||
t.Errorf("a judge with no replica: %d", code)
|
||||
}
|
||||
var judges []JudgeView
|
||||
call(t, a, "GET", "/api/v1/guard/judges", "", &judges)
|
||||
if len(judges) != 1 || judges[0].BudgetMs != 5000 {
|
||||
t.Errorf("judges %+v", judges)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mcpclient"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// MCPStore keeps the company's MCP servers and the secrets their headers
|
||||
// carry.
|
||||
type MCPStore interface {
|
||||
MCPServers(ctx context.Context) ([]store.MCPServer, error)
|
||||
MCPServer(ctx context.Context, name string) (store.MCPServer, error)
|
||||
PutMCPServer(ctx context.Context, m store.MCPServer) error
|
||||
DeleteMCPServer(ctx context.Context, name string) error
|
||||
SetClusterSecret(ctx context.Context, name, value string) error
|
||||
}
|
||||
|
||||
// SetMCP enables the endpoints for the company's MCP servers.
|
||||
func (a *API) SetMCP(st MCPStore) { a.mcp = st }
|
||||
|
||||
// MCPServerView is a server as the API shows it: header values only for the
|
||||
// headers sent as written, and for the others the reference read.
|
||||
type MCPServerView struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Headers map[string]string `json:"headers,omitempty"`
|
||||
HeaderRefs map[string]string `json:"header_refs,omitempty"`
|
||||
Enabled bool `json:"enabled"`
|
||||
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
|
||||
RatePerMinute int `json:"rate_per_minute,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
CreatedBy string `json:"created_by,omitempty"`
|
||||
}
|
||||
|
||||
// MCPServerRequest adds a server or changes one. Secrets are header values
|
||||
// to keep in the store, sealed, each then read by reference; HeaderRefs name
|
||||
// secrets kept elsewhere (a file, env:, vault:).
|
||||
type MCPServerRequest struct {
|
||||
URL string `json:"url"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Headers map[string]string `json:"headers,omitempty"`
|
||||
HeaderRefs map[string]string `json:"header_refs,omitempty"`
|
||||
Secrets map[string]string `json:"secrets,omitempty"`
|
||||
Enabled *bool `json:"enabled,omitempty"`
|
||||
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
|
||||
// RatePerMinute bounds each caller's calls to its tools; a negative
|
||||
// number takes the bound away.
|
||||
RatePerMinute int `json:"rate_per_minute,omitempty"`
|
||||
}
|
||||
|
||||
// MCPTestView is what the server said when asked: its tools, or why not.
|
||||
type MCPTestView struct {
|
||||
OK bool `json:"ok"`
|
||||
Tools []string `json:"tools,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
func mcpServerView(m store.MCPServer) MCPServerView {
|
||||
return MCPServerView{Name: m.Name, URL: m.URL, Description: m.Description, Headers: m.Headers,
|
||||
HeaderRefs: m.HeaderRefs, Enabled: m.Enabled, TimeoutSeconds: int(m.Timeout / time.Second), RatePerMinute: m.RatePerMinute,
|
||||
CreatedAt: m.CreatedAt, CreatedBy: m.CreatedBy}
|
||||
}
|
||||
|
||||
func (a *API) registerMCP(mux Mux) {
|
||||
need := func(w http.ResponseWriter) bool {
|
||||
if a.mcp == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep MCP servers in")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
mux.HandleFunc("POST /api/v1/mcp", a.MCP)
|
||||
mux.HandleFunc("GET /api/v1/mcp/servers", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
list, err := a.mcp.MCPServers(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]MCPServerView, len(list))
|
||||
for i, m := range list {
|
||||
out[i] = mcpServerView(m)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/mcp/servers/{name}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
name := r.PathValue("name")
|
||||
var req MCPServerRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
m, err := a.mcpServerFrom(r.Context(), name, req, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if err := a.mcp.PutMCPServer(r.Context(), m); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, mcpServerView(m))
|
||||
})
|
||||
mux.HandleFunc("DELETE /api/v1/mcp/servers/{name}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
switch err := a.mcp.DeleteMCPServer(r.Context(), r.PathValue("name")); {
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
writeError(w, http.StatusNotFound, "no MCP server "+r.PathValue("name"))
|
||||
case err != nil:
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
default:
|
||||
writeJSON(w, http.StatusOK, map[string]string{"removed": r.PathValue("name")})
|
||||
}
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/mcp/servers/{name}/test", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
m, err := a.mcp.MCPServer(r.Context(), r.PathValue("name"))
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no MCP server "+r.PathValue("name"))
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.testMCPServer(r.Context(), m))
|
||||
})
|
||||
}
|
||||
|
||||
// mcpServerFrom is the server a request makes, over the one of that name if
|
||||
// there is one: its secrets kept in the store, and named by reference.
|
||||
func (a *API) mcpServerFrom(ctx context.Context, name string, req MCPServerRequest, by string) (store.MCPServer, error) {
|
||||
if !store.ValidMCPServerName(name) {
|
||||
return store.MCPServer{}, errors.New("a server's name is lowercase letters, digits, - and _, starting with a letter: " +
|
||||
"its tools are NAME.tool")
|
||||
}
|
||||
m, err := a.mcp.MCPServer(ctx, name)
|
||||
switch {
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
m = store.MCPServer{Name: name, Enabled: true, CreatedBy: by}
|
||||
case err != nil:
|
||||
return m, err
|
||||
}
|
||||
if req.URL != "" {
|
||||
m.URL = req.URL
|
||||
}
|
||||
if u, err := url.Parse(m.URL); m.URL != AdminMCPAddress && (err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "") {
|
||||
return m, errors.New("the server's url is an http:// or https:// address, or " + AdminMCPAddress + " for this cluster's own")
|
||||
}
|
||||
if req.Description != "" {
|
||||
m.Description = req.Description
|
||||
}
|
||||
if req.Headers != nil {
|
||||
m.Headers = req.Headers
|
||||
}
|
||||
if req.HeaderRefs != nil {
|
||||
m.HeaderRefs = req.HeaderRefs
|
||||
}
|
||||
if m.HeaderRefs == nil {
|
||||
m.HeaderRefs = map[string]string{}
|
||||
}
|
||||
for header, value := range req.Secrets {
|
||||
if strings.TrimSpace(header) == "" || value == "" {
|
||||
return m, errors.New("a secret header needs a name and a value")
|
||||
}
|
||||
key := "mcp/" + name + "/" + header
|
||||
if err := a.mcp.SetClusterSecret(ctx, key, value); err != nil {
|
||||
return m, err
|
||||
}
|
||||
m.HeaderRefs[header] = "store:" + key
|
||||
delete(m.Headers, header)
|
||||
}
|
||||
if req.Enabled != nil {
|
||||
m.Enabled = *req.Enabled
|
||||
}
|
||||
if req.TimeoutSeconds > 0 {
|
||||
m.Timeout = time.Duration(req.TimeoutSeconds) * time.Second
|
||||
}
|
||||
switch {
|
||||
case req.RatePerMinute > 0:
|
||||
m.RatePerMinute = req.RatePerMinute
|
||||
case req.RatePerMinute < 0:
|
||||
m.RatePerMinute = 0
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// testMCPServer asks a server for its tools, with its credentials, as /mcp
|
||||
// would.
|
||||
func (a *API) testMCPServer(ctx context.Context, m store.MCPServer) MCPTestView {
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
c := &mcpclient.Client{URL: m.URL, Header: func(ctx context.Context) (http.Header, error) {
|
||||
h := http.Header{}
|
||||
for k, v := range m.Headers {
|
||||
h.Set(k, v)
|
||||
}
|
||||
for k, ref := range m.HeaderRefs {
|
||||
v, err := secrets.ReadString(ctx, ref)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h.Set(k, v)
|
||||
}
|
||||
return h, nil
|
||||
}}
|
||||
if m.URL == AdminMCPAddress {
|
||||
c.URL, c.HTTP = InProcessURL, &http.Client{Transport: InProcess(http.HandlerFunc(a.MCP))}
|
||||
}
|
||||
tools, err := c.Tools(ctx)
|
||||
if err != nil {
|
||||
return MCPTestView{Error: err.Error()}
|
||||
}
|
||||
v := MCPTestView{OK: true}
|
||||
for _, t := range tools {
|
||||
v.Tools = append(v.Tools, m.Name+"."+t.Name)
|
||||
}
|
||||
sort.Strings(v.Tools)
|
||||
return v
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// The cluster as an MCP server, read only: what `llmgw ctl status`, `nodes`,
|
||||
// `models`, `replicas`, `explain-placement` and `logs` show, as tools an
|
||||
// agent can be given. It answers at POST /api/v1/mcp, behind the admin
|
||||
// token, and in-process to the gateway as the MCP server llmgw:admin, which
|
||||
// an administrator registers like any other and grants like any other.
|
||||
// Nothing here changes the cluster.
|
||||
|
||||
// AdminMCPAddress is the address that registers this server with the
|
||||
// gateway: called in-process, with no token and no network.
|
||||
const AdminMCPAddress = "llmgw:admin"
|
||||
|
||||
// adminMCPAnswerBytes bounds one tool's answer: it becomes a prompt.
|
||||
const adminMCPAnswerBytes = 256 << 10
|
||||
|
||||
var adminMCPTools = []map[string]any{
|
||||
adminTool("cluster_status", "The whole cluster in one answer: control planes, store, nodes ready, models with the replicas they "+
|
||||
"asked for, GPUs in use, and what is unhealthy or idle.", nil, nil),
|
||||
adminTool("list_nodes", "Every node: its state, GPUs and their use, CPU and memory load, fabrics, agent version and engines.", nil, nil),
|
||||
adminTool("list_models", "Every model in the catalog: its replicas ready against wanted, state, profiles and why one is parked.", nil, nil),
|
||||
adminTool("list_replicas", "Every replica, with its shards: the node and GPUs each is on, its state and how long it has run.",
|
||||
map[string]any{"model": map[string]any{"type": "string", "description": "Only this model's replicas."}}, nil),
|
||||
adminTool("explain_placement", "Why a model is not placed: each node, and what kept the model off it.",
|
||||
map[string]any{"model": map[string]any{"type": "string", "description": "The model's name or an alias."}}, []string{"model"}),
|
||||
adminTool("node_logs", "The last lines of a node's agent log, or of one of its engines.",
|
||||
map[string]any{
|
||||
"node": map[string]any{"type": "string", "description": "The node's id."},
|
||||
"lines": map[string]any{"type": "integer", "description": "How many lines, up to 300; 100 when absent."},
|
||||
"source": map[string]any{"type": "string", "description": "agent when absent, or a shard id for that engine's output."},
|
||||
}, []string{"node"}),
|
||||
}
|
||||
|
||||
func adminTool(name, desc string, props map[string]any, required []string) map[string]any {
|
||||
schema := map[string]any{"type": "object", "properties": map[string]any{}}
|
||||
if props != nil {
|
||||
schema["properties"] = props
|
||||
}
|
||||
if len(required) > 0 {
|
||||
schema["required"] = required
|
||||
}
|
||||
return map[string]any{"name": name, "description": desc, "inputSchema": schema,
|
||||
"annotations": map[string]any{"readOnlyHint": true, "openWorldHint": false}}
|
||||
}
|
||||
|
||||
// MCP serves the cluster as a read-only MCP server: JSON-RPC over POST, the
|
||||
// streamable HTTP transport answered as plain JSON, no session.
|
||||
func (a *API) MCP(w http.ResponseWriter, r *http.Request) {
|
||||
var msg struct {
|
||||
ID json.RawMessage `json:"id"`
|
||||
Method string `json:"method"`
|
||||
Params json.RawMessage `json:"params"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&msg); err != nil || msg.Method == "" {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]any{"jsonrpc": "2.0", "id": nil,
|
||||
"error": map[string]any{"code": -32600, "message": "expected one JSON-RPC request object"}})
|
||||
return
|
||||
}
|
||||
if len(msg.ID) == 0 {
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
return
|
||||
}
|
||||
answer := func(result any) {
|
||||
writeJSON(w, http.StatusOK, map[string]any{"jsonrpc": "2.0", "id": msg.ID, "result": result})
|
||||
}
|
||||
switch msg.Method {
|
||||
case "initialize":
|
||||
var p struct {
|
||||
ProtocolVersion string `json:"protocolVersion"`
|
||||
}
|
||||
_ = json.Unmarshal(msg.Params, &p)
|
||||
version := p.ProtocolVersion
|
||||
if version != "2025-03-26" && version != "2025-06-18" {
|
||||
version = "2025-06-18"
|
||||
}
|
||||
answer(map[string]any{"protocolVersion": version, "capabilities": map[string]any{"tools": map[string]any{}},
|
||||
"serverInfo": map[string]any{"name": "llmgw-admin", "version": "1"},
|
||||
"instructions": "This llmgw cluster, read only: its control planes, nodes, models and replicas, why a model is " +
|
||||
"not placed, and a node's logs. Start with cluster_status."})
|
||||
case "ping":
|
||||
answer(map[string]any{})
|
||||
case "tools/list":
|
||||
answer(map[string]any{"tools": adminMCPTools})
|
||||
case "tools/call":
|
||||
var call struct {
|
||||
Name string `json:"name"`
|
||||
Arguments struct {
|
||||
Model string `json:"model"`
|
||||
Node string `json:"node"`
|
||||
Source string `json:"source"`
|
||||
Lines int `json:"lines"`
|
||||
} `json:"arguments"`
|
||||
}
|
||||
_ = json.Unmarshal(msg.Params, &call)
|
||||
answer(a.callAdminTool(r, call.Name, call.Arguments.Model, call.Arguments.Node, call.Arguments.Source, call.Arguments.Lines))
|
||||
default:
|
||||
writeJSON(w, http.StatusOK, map[string]any{"jsonrpc": "2.0", "id": msg.ID,
|
||||
"error": map[string]any{"code": -32601, "message": "method not found: " + msg.Method}})
|
||||
}
|
||||
}
|
||||
|
||||
// callAdminTool answers a tool from the API's own handler for it, so the
|
||||
// tool says what `llmgw ctl` would.
|
||||
func (a *API) callAdminTool(r *http.Request, name, model, node, source string, lines int) map[string]any {
|
||||
var h http.HandlerFunc
|
||||
q := url.Values{}
|
||||
path := ""
|
||||
switch name {
|
||||
case "cluster_status":
|
||||
h, path = a.summary, "/api/v1/summary"
|
||||
case "list_nodes":
|
||||
h, path = a.listNodes, "/api/v1/nodes"
|
||||
case "list_models":
|
||||
h, path = a.listModels, "/api/v1/models"
|
||||
case "list_replicas":
|
||||
h, path = a.listReplicas, "/api/v1/replicas"
|
||||
if model != "" {
|
||||
q.Set("model", model)
|
||||
}
|
||||
case "explain_placement":
|
||||
if model == "" {
|
||||
return adminToolText("explain_placement needs the model", true)
|
||||
}
|
||||
h, path = a.explainPlacement, "/api/v1/explain-placement"
|
||||
q.Set("model", model)
|
||||
case "node_logs":
|
||||
if node == "" {
|
||||
return adminToolText("node_logs needs the node", true)
|
||||
}
|
||||
if lines <= 0 {
|
||||
lines = 100
|
||||
}
|
||||
h, path = a.nodeLogs, "/api/v1/nodes/"+url.PathEscape(node)+"/logs"
|
||||
q.Set("lines", strconv.Itoa(min(lines, 300)))
|
||||
if source != "" {
|
||||
q.Set("source", source)
|
||||
}
|
||||
default:
|
||||
return adminToolText("no tool "+name, true)
|
||||
}
|
||||
req := httptest.NewRequestWithContext(r.Context(), http.MethodGet, path+"?"+q.Encode(), nil)
|
||||
if name == "node_logs" {
|
||||
req.SetPathValue("id", node)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
h(rec, req)
|
||||
body := bytes.TrimSpace(rec.Body.Bytes())
|
||||
if rec.Code >= 300 {
|
||||
return adminToolText(fmt.Sprintf("%s: %s", name, body), true)
|
||||
}
|
||||
if len(body) > adminMCPAnswerBytes {
|
||||
body = append(body[:adminMCPAnswerBytes], []byte("… (cut: the answer is larger than an agent should read)")...)
|
||||
}
|
||||
return adminToolText(string(body), false)
|
||||
}
|
||||
|
||||
func adminToolText(text string, isError bool) map[string]any {
|
||||
return map[string]any{"content": []any{map[string]any{"type": "text", "text": text}}, "isError": isError}
|
||||
}
|
||||
|
||||
// InProcessURL is the address an in-process client posts to; the transport
|
||||
// ignores it.
|
||||
const InProcessURL = "http://llmgw-admin/api/v1/mcp"
|
||||
|
||||
// InProcess is a transport that answers every request from h, in this
|
||||
// process: the gateway's way to the cluster's own MCP server.
|
||||
func InProcess(h http.Handler) http.RoundTripper { return inProcess{h} }
|
||||
|
||||
type inProcess struct{ h http.Handler }
|
||||
|
||||
func (t inProcess) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
rec := httptest.NewRecorder()
|
||||
t.h.ServeHTTP(rec, req)
|
||||
return rec.Result(), nil
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func adminMCP(t *testing.T, a *API, method string, params any) map[string]any {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": method, "params": params})
|
||||
rec := httptest.NewRecorder()
|
||||
a.MCP(rec, httptest.NewRequest(http.MethodPost, "/api/v1/mcp", bytes.NewReader(b)))
|
||||
var resp struct {
|
||||
Result map[string]any `json:"result"`
|
||||
}
|
||||
if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &resp) != nil {
|
||||
t.Fatalf("%s: %d %s", method, rec.Code, rec.Body.String())
|
||||
}
|
||||
return resp.Result
|
||||
}
|
||||
|
||||
func TestTheClusterIsAReadOnlyMCPServer(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
if init := adminMCP(t, a, "initialize", map[string]any{"protocolVersion": "2025-06-18"}); init["protocolVersion"] != "2025-06-18" {
|
||||
t.Errorf("initialize %v", init)
|
||||
}
|
||||
tools := adminMCP(t, a, "tools/list", map[string]any{})["tools"].([]any)
|
||||
var names []string
|
||||
for _, tl := range tools {
|
||||
m := tl.(map[string]any)
|
||||
names = append(names, m["name"].(string))
|
||||
if m["annotations"].(map[string]any)["readOnlyHint"] != true {
|
||||
t.Errorf("%s is not marked read only", m["name"])
|
||||
}
|
||||
}
|
||||
if got := strings.Join(names, ","); got != "cluster_status,list_nodes,list_models,list_replicas,explain_placement,node_logs" {
|
||||
t.Errorf("tools %s", got)
|
||||
}
|
||||
// Each answers what the API would.
|
||||
for _, name := range []string{"cluster_status", "list_nodes", "list_models", "list_replicas"} {
|
||||
res := adminMCP(t, a, "tools/call", map[string]any{"name": name, "arguments": map[string]any{}})
|
||||
text := res["content"].([]any)[0].(map[string]any)["text"].(string)
|
||||
if res["isError"] == true || !json.Valid([]byte(text)) {
|
||||
t.Errorf("%s: %v", name, res)
|
||||
}
|
||||
}
|
||||
if res := adminMCP(t, a, "tools/call", map[string]any{"name": "explain_placement", "arguments": map[string]any{}}); res["isError"] != true {
|
||||
t.Errorf("explain_placement without a model: %v", res)
|
||||
}
|
||||
if res := adminMCP(t, a, "tools/call", map[string]any{"name": "remove_model", "arguments": map[string]any{}}); res["isError"] != true {
|
||||
t.Errorf("a tool that would change something: %v", res)
|
||||
}
|
||||
}
|
||||
|
||||
// The gateway reaches it in this process, as llmgw:admin.
|
||||
func TestTheClustersOwnMCPServerIsReachedInProcess(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
c := &http.Client{Transport: InProcess(http.HandlerFunc(a.MCP))}
|
||||
b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/list"})
|
||||
resp, err := c.Post(InProcessURL, "application/json", bytes.NewReader(b))
|
||||
if err != nil || resp.StatusCode != 200 {
|
||||
t.Fatalf("%v %v", resp, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var out struct {
|
||||
Result struct {
|
||||
Tools []any `json:"tools"`
|
||||
} `json:"result"`
|
||||
}
|
||||
if json.NewDecoder(resp.Body).Decode(&out) != nil || len(out.Result.Tools) != 6 {
|
||||
t.Errorf("tools %+v", out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mcpclient/mcptest"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type memMCP struct {
|
||||
mu sync.Mutex
|
||||
servers map[string]store.MCPServer
|
||||
secrets map[string]string
|
||||
}
|
||||
|
||||
func (m *memMCP) MCPServers(context.Context) ([]store.MCPServer, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
var out []store.MCPServer
|
||||
for _, s := range m.servers {
|
||||
out = append(out, s)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (m *memMCP) MCPServer(_ context.Context, name string) (store.MCPServer, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s, ok := m.servers[name]
|
||||
if !ok {
|
||||
return s, store.ErrNotFound
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func (m *memMCP) PutMCPServer(_ context.Context, s store.MCPServer) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.servers[s.Name] = s
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *memMCP) DeleteMCPServer(_ context.Context, name string) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if _, ok := m.servers[name]; !ok {
|
||||
return store.ErrNotFound
|
||||
}
|
||||
delete(m.servers, name)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *memMCP) SetClusterSecret(_ context.Context, name, value string) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.secrets[name] = value
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestAnMCPServerIsRegisteredWithItsTokenKeptAsASecretAndTried(t *testing.T) {
|
||||
srv := mcptest.New()
|
||||
defer srv.Close()
|
||||
a, _ := setup(t, testModels())
|
||||
st := &memMCP{servers: map[string]store.MCPServer{}, secrets: map[string]string{}}
|
||||
a.SetMCP(st)
|
||||
secrets.SetStore(func(_ context.Context, name string) (string, error) {
|
||||
st.mu.Lock()
|
||||
defer st.mu.Unlock()
|
||||
return st.secrets[name], nil
|
||||
})
|
||||
t.Cleanup(func() { secrets.SetStore(nil) })
|
||||
|
||||
var v MCPServerView
|
||||
body := `{"url":"` + srv.URL + `","description":"tickets","headers":{"X-Org":"acme"},"secrets":{"Authorization":"Bearer t0ken"}}`
|
||||
if code := call(t, a, "PUT", "/api/v1/mcp/servers/jira", body, &v); code != 200 {
|
||||
t.Fatalf("add: %d", code)
|
||||
}
|
||||
// The token is in the store, named by reference; the row has none.
|
||||
if st.secrets["mcp/jira/Authorization"] != "Bearer t0ken" || v.HeaderRefs["Authorization"] != "store:mcp/jira/Authorization" ||
|
||||
strings.Contains(v.Headers["Authorization"], "t0ken") || !v.Enabled {
|
||||
t.Errorf("view %+v, secrets %v", v, st.secrets)
|
||||
}
|
||||
var tv MCPTestView
|
||||
if code := call(t, a, "POST", "/api/v1/mcp/servers/jira/test", "", &tv); code != 200 || !tv.OK || len(tv.Tools) != 3 || tv.Tools[1] != "jira.echo" {
|
||||
t.Errorf("test: %d %+v", code, tv)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/mcp/servers/Bad.Name", body, nil); code != 400 {
|
||||
t.Errorf("a name with a dot: %d", code)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/mcp/servers/x", `{"url":"ftp://x"}`, nil); code != 400 {
|
||||
t.Errorf("not an http address: %d", code)
|
||||
}
|
||||
// Switched off, kept; then removed.
|
||||
if code := call(t, a, "PUT", "/api/v1/mcp/servers/jira", `{"enabled":false}`, &v); code != 200 || v.Enabled || v.URL != srv.URL {
|
||||
t.Errorf("off: %d %+v", code, v)
|
||||
}
|
||||
if code := call(t, a, "DELETE", "/api/v1/mcp/servers/jira", "", nil); code != 200 {
|
||||
t.Errorf("remove: %d", code)
|
||||
}
|
||||
if code := call(t, a, "DELETE", "/api/v1/mcp/servers/jira", "", nil); code != 404 {
|
||||
t.Errorf("remove again: %d", code)
|
||||
}
|
||||
}
|
||||
@@ -67,6 +67,8 @@ var PathParams = map[string]string{
|
||||
"models/{name}": "The model's name in the catalog.",
|
||||
"presets/{name}": "The preset's name, as `llmgw ctl model presets` lists it.",
|
||||
"nodes/{id}": "The node's ID, as `llmgw ctl nodes` lists it.",
|
||||
"secrets/{name}": "ldap-bind, oidc-client-secret or scim-token.",
|
||||
"servers/{name}": "The MCP server's name: its tools are NAME.tool in /mcp.",
|
||||
"jobs/{id}": "The job's ID, as starting it answered.",
|
||||
"runs/{id}": "The quality run's ID, as starting it answered.",
|
||||
"requests/{id}": "The request's ID: the x-request-id a client was answered with.",
|
||||
@@ -280,6 +282,8 @@ var Routes = []Route{
|
||||
Response: []JudgeView{}},
|
||||
{Method: "POST", Path: "/api/v1/guard/judges/ask", Summary: "Have a judge read a text and say what it answered; nothing is kept",
|
||||
Request: JudgeAskRequest{}, Response: JudgeAskView{}},
|
||||
{Method: "POST", Path: "/api/v1/guard/judges/check", Summary: "How a judge keeps up: three judgments in turn, then a burst at once, against the budget",
|
||||
Request: JudgeCheckRequest{}, Response: JudgeCheckView{}},
|
||||
{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
|
||||
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
|
||||
{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
|
||||
@@ -396,6 +400,28 @@ var Routes = []Route{
|
||||
{Method: "POST", Path: "/api/v1/settings/rollback", Summary: "Put an earlier version of the cluster's settings back in force",
|
||||
Request: SettingsRollbackRequest{}, Response: SettingsView{}},
|
||||
|
||||
// How people sign in: single sign-on, a directory, and the secrets
|
||||
// they need kept for every control plane.
|
||||
{Method: "GET", Path: "/api/v1/auth", Summary: "How people sign in: OIDC, a directory, SCIM, and the directory's last sync",
|
||||
Response: AuthView{}},
|
||||
{Method: "POST", Path: "/api/v1/auth/ldap/test", Summary: "Try an auth.ldap block before it is kept: servers, the service account, people and groups",
|
||||
Request: LDAPTestRequest{}, Response: LDAPTestView{}},
|
||||
{Method: "PUT", Path: "/api/v1/auth/secrets/{name}", Summary: "Keep ldap-bind, oidc-client-secret or scim-token in the store, for settings to name as store:auth/NAME",
|
||||
Request: SecretRequest{}, Response: map[string]string{}},
|
||||
|
||||
// The company's own MCP servers, served through /mcp.
|
||||
{Method: "POST", Path: "/api/v1/mcp", Summary: "This cluster as a read-only MCP server: status, nodes, models, replicas, placement, logs",
|
||||
Description: "JSON-RPC: initialize, tools/list, tools/call. Registered with the gateway as llmgw:admin.",
|
||||
Request: map[string]any{}, Response: map[string]any{}},
|
||||
{Method: "GET", Path: "/api/v1/mcp/servers", Summary: "The company's MCP servers /mcp serves, and how each is reached",
|
||||
Response: []MCPServerView{}},
|
||||
{Method: "PUT", Path: "/api/v1/mcp/servers/{name}", Summary: "Add an MCP server, or change one: its address, headers and the secrets they carry",
|
||||
Request: MCPServerRequest{}, Response: MCPServerView{}},
|
||||
{Method: "DELETE", Path: "/api/v1/mcp/servers/{name}", Summary: "Remove an MCP server: its tools leave /mcp",
|
||||
Response: map[string]string{}},
|
||||
{Method: "POST", Path: "/api/v1/mcp/servers/{name}/test", Summary: "Ask an MCP server for its tools, with its credentials, as /mcp would",
|
||||
Response: MCPTestView{}},
|
||||
|
||||
// Usage, requests and the record of changes.
|
||||
{Method: "GET", Path: "/api/v1/usage", Summary: "What each group of the tenancy tree spent",
|
||||
Query: []Param{
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net/url"
|
||||
@@ -23,6 +24,9 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
)
|
||||
|
||||
// ErrExists is a model added only if absent that was there already.
|
||||
var ErrExists = errors.New("already in the catalog")
|
||||
|
||||
// Scaling is how many replicas a model should have.
|
||||
type Scaling struct {
|
||||
Min int `json:"min,omitempty" koanf:"min"`
|
||||
|
||||
@@ -163,6 +163,18 @@ type Request struct {
|
||||
// CPUCores is what a replica computes on; zero takes half the smallest
|
||||
// node it fits on.
|
||||
CPUCores int `json:"cpu_cores,omitempty"`
|
||||
// Env is set in the engine's environment, on every node of a replica:
|
||||
// NCCL_DEBUG=INFO to see how its cards talk, NCCL_IB_* to steer them.
|
||||
// Only the variables engines are tuned with; the catalog refuses others.
|
||||
Env map[string]string `json:"env,omitempty"`
|
||||
// IfAbsent adds the model only if nobody has meanwhile: the leader's own
|
||||
// serving of a judge a policy names gives way to the one init or an
|
||||
// operator added while it fetched, rather than replacing it.
|
||||
IfAbsent bool `json:"-"`
|
||||
// MemoryGB is what a replica on CPUs reserves, in place of the figure
|
||||
// worked out from its weights and context: a judge's, whose own sizing
|
||||
// is known.
|
||||
MemoryGB int `json:"memory_gb,omitempty"`
|
||||
// By is who asked, for the catalog's record.
|
||||
By string `json:"-"`
|
||||
}
|
||||
@@ -453,8 +465,17 @@ func (s *Server) run(j *Job) {
|
||||
for _, o := range layout {
|
||||
where = append(where, o.Label())
|
||||
}
|
||||
s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s; runs on %s",
|
||||
set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision(), strings.Join(where, ", else ")))
|
||||
// What the first way needs of each card, for the context asked for.
|
||||
need := ""
|
||||
if first := layout[0]; first.NeedGB > 0 {
|
||||
tokens := req.ContextLength
|
||||
if req.Format != "safetensors" && req.CtxPool > 0 {
|
||||
tokens = req.CtxPool
|
||||
}
|
||||
need = "; needs " + first.WithContext(plan.KVBytesPerToken, tokens).MemoryText()
|
||||
}
|
||||
s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s; runs on %s%s",
|
||||
set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision(), strings.Join(where, ", else "), need))
|
||||
|
||||
// 2. The mirror: fetch, and check what arrived against what was published.
|
||||
source, err := s.mirror(ctx, j, repo, set, name)
|
||||
@@ -518,6 +539,9 @@ func (s *Server) run(j *Job) {
|
||||
memGB = int(gb(set.Bytes)*1.5) + 2
|
||||
basis = "weights plus half again plus two; the header gave no shape"
|
||||
}
|
||||
if req.MemoryGB > 0 {
|
||||
memGB, basis = req.MemoryGB, "as asked, in place of "+basis
|
||||
}
|
||||
s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
|
||||
|
||||
cores := req.CPUCores
|
||||
@@ -560,6 +584,8 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
|
||||
o = p
|
||||
}
|
||||
}
|
||||
// A way the plan did not offer is still a model of experts.
|
||||
o.ExpertParallel = engine == "vllm" && plan.Experts > 1 && o.GPUs() > 1
|
||||
out = append(out, o)
|
||||
case len(opts) > 0:
|
||||
out = append(out, opts[0])
|
||||
@@ -594,7 +620,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
|
||||
func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []catalog.Profile {
|
||||
var out []catalog.Profile
|
||||
for i, o := range layout {
|
||||
p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i}
|
||||
p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env}
|
||||
switch {
|
||||
case o.CPU:
|
||||
p.Name = "cpu"
|
||||
@@ -611,6 +637,11 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
||||
p.PP, p.Nodes, p.GPUsPerNode = o.Nodes, o.Nodes, o.GPUsPerNode
|
||||
}
|
||||
p.Args = []string{"--max-model-len", strconv.Itoa(req.ContextLength)}
|
||||
if o.ExpertParallel {
|
||||
// The experts whole on each card rather than each sliced
|
||||
// across them: less traffic between the cards for every token.
|
||||
p.Args = append(p.Args, "--enable-expert-parallel")
|
||||
}
|
||||
}
|
||||
out = append(out, p)
|
||||
}
|
||||
@@ -631,6 +662,12 @@ func llamaArgs(req Request) []string {
|
||||
n := strconv.Itoa(req.ContextLength)
|
||||
args = append([]string{"--embedding"}, append(args, "--batch-size", n, "--ubatch-size", n)...)
|
||||
}
|
||||
if req.GuardFormat != "" {
|
||||
// No prompt cache: a judge's prompts, each a different text, are
|
||||
// not asked again, and llama.cpp's default of up to 8 GB grew one
|
||||
// judge to 3.2 GB over a day.
|
||||
args = append(args, "--cache-ram", "0")
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
@@ -678,7 +715,18 @@ func (s *Server) addAndStart(ctx context.Context, j *Job, repo hf.Repo, set hf.F
|
||||
}
|
||||
m.Class = req.Class
|
||||
s.step(j, "catalog", Running, "adding "+name)
|
||||
if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil {
|
||||
add := s.deps.Catalog.Add
|
||||
if adder, ok := s.deps.Catalog.(interface {
|
||||
AddNew(ctx context.Context, m catalog.Model, by string) error
|
||||
}); ok && req.IfAbsent {
|
||||
add = adder.AddNew
|
||||
}
|
||||
if err := add(ctx, m, req.By); err != nil {
|
||||
if req.IfAbsent && errors.Is(err, catalog.ErrExists) {
|
||||
s.step(j, "catalog", Skipped, name+" was added meanwhile; left as it is")
|
||||
s.finish(j)
|
||||
return
|
||||
}
|
||||
s.fail(j, "catalog", err)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -428,3 +428,73 @@ func TestAPoolTheCardCannotHoldIsShrunkNotStarted(t *testing.T) {
|
||||
t.Errorf("a pool that fits was changed: %+v %q", got, why)
|
||||
}
|
||||
}
|
||||
|
||||
// A judge is sized as asked, not from its weights, and runs with no prompt
|
||||
// cache: what `server init` gives its own.
|
||||
func TestAJudgeIsServedWithTheMemoryAskedAndNoPromptCache(t *testing.T) {
|
||||
files := []hf.File{{Path: "Qwen3Guard-Gen-0.6B-Q8_0.gguf", Bytes: 600 << 20, SHA256: "aa"}}
|
||||
s, _, c, _ := setup(t, files, nil)
|
||||
job, err := s.Start(Request{Repo: "example/Qwen3Guard-Gen-0.6B-GGUF", File: "Qwen3Guard-Gen-0.6B-Q8_0.gguf",
|
||||
GuardFormat: "qwen3guard", CPU: true, CPUCores: 2, MemoryGB: 2, Replicas: 2, CtxPool: 8192, ContextLength: 4096})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wait(t, s, job.ID)
|
||||
if len(c.added) != 1 {
|
||||
t.Fatalf("added %+v", c.added)
|
||||
}
|
||||
p := c.added[0].Profiles[0]
|
||||
if p.MinMemoryGB != 2 || p.CPUCores != 2 || !strings.Contains(strings.Join(p.Args, " "), "--cache-ram 0") {
|
||||
t.Errorf("profile %+v", p)
|
||||
}
|
||||
// Any other model keeps llama.cpp's cache.
|
||||
if strings.Contains(strings.Join(llamaArgs(Request{CtxPool: 8192, ContextLength: 4096}), " "), "cache-ram") {
|
||||
t.Error("a model that is not a judge lost its prompt cache")
|
||||
}
|
||||
}
|
||||
|
||||
// takenCatalog has the model by the time a job comes to add it.
|
||||
type takenCatalog struct{ fakeCatalog }
|
||||
|
||||
func (c *takenCatalog) AddNew(context.Context, catalog.Model, string) error {
|
||||
return fmt.Errorf("model %q: %w", "x", catalog.ErrExists)
|
||||
}
|
||||
|
||||
func TestAJobAddingOnlyIfAbsentGivesWayToAModelAddedMeanwhile(t *testing.T) {
|
||||
files := []hf.File{{Path: "Qwen3Guard-Gen-0.6B-Q8_0.gguf", Bytes: 600 << 20, SHA256: "aa"}}
|
||||
s, _, _, _ := setup(t, files, nil)
|
||||
taken := &takenCatalog{}
|
||||
s.deps.Catalog = taken
|
||||
job, err := s.Start(Request{Repo: "example/Qwen3Guard-Gen-0.6B-GGUF", File: "Qwen3Guard-Gen-0.6B-Q8_0.gguf",
|
||||
GuardFormat: "qwen3guard", CPU: true, CPUCores: 2, MemoryGB: 2, Replicas: 2, IfAbsent: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
j := wait(t, s, job.ID)
|
||||
if j.State != Done || len(taken.added) != 0 {
|
||||
t.Errorf("job %s, added %d: %+v", j.State, len(taken.added), j.Steps)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of experts given its cards by hand spreads them still, and the
|
||||
// environment asked for reaches every profile.
|
||||
func TestAHandGivenLayoutOfExpertsKeepsExpertParallelAndTheEnvironment(t *testing.T) {
|
||||
plan := Plan{Repo: "Qwen/Qwen3-235B-A22B", Experts: 128}
|
||||
req := Request{Format: "safetensors", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2, ContextLength: 8192,
|
||||
Env: map[string]string{"NCCL_DEBUG": "INFO"}}
|
||||
layout, err := chooseLayout(req, plan, true, hf.FileSet{Name: "original"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profs := profilesFor(layout, req, 0, 0)
|
||||
p := profs[0]
|
||||
if p.TP != 8 || p.PP != 2 || p.Nodes != 2 || !slices.Contains(p.Args, "--enable-expert-parallel") || p.Env["NCCL_DEBUG"] != "INFO" {
|
||||
t.Errorf("profile %+v", p)
|
||||
}
|
||||
// A dense model given the same cards does not.
|
||||
plan.Experts = 0
|
||||
layout, _ = chooseLayout(req, plan, true, hf.FileSet{Name: "original"})
|
||||
if slices.Contains(profilesFor(layout, req, 0, 0)[0].Args, "--enable-expert-parallel") {
|
||||
t.Error("a dense model was given expert parallelism")
|
||||
}
|
||||
}
|
||||
@@ -42,7 +42,12 @@ type Plan struct {
|
||||
License string `json:"license,omitempty"`
|
||||
Params int64 `json:"params,omitempty"`
|
||||
Context int `json:"context"`
|
||||
Gated bool `json:"gated,omitempty"`
|
||||
// KVBytesPerToken is one token of context, the whole model: what an
|
||||
// option needs for another context is worked out from it. 0 is unknown.
|
||||
KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
|
||||
// Experts is a mixture-of-experts model's routed experts; 0 for dense.
|
||||
Experts int `json:"experts,omitempty"`
|
||||
Gated bool `json:"gated,omitempty"`
|
||||
// Servable is false when the nodes' engine cannot run the model at all,
|
||||
// and Reason says why.
|
||||
Servable bool `json:"servable"`
|
||||
@@ -144,6 +149,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
if err == nil {
|
||||
shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
|
||||
shape.KVHeads = m.KVHeads
|
||||
shape.Experts = m.Experts
|
||||
if p.Params == 0 {
|
||||
p.Params = m.Params
|
||||
}
|
||||
@@ -157,6 +163,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
p.Note = "the model's shape was not readable, so the context's memory is not counted"
|
||||
}
|
||||
shape.Params = p.Params
|
||||
p.KVBytesPerToken, p.Experts = shape.KVBytesPerToken, shape.Experts
|
||||
p.Options = modelplan.Options(arts, shape, s.deps.Pool(), p.Context)
|
||||
if !p.Servable {
|
||||
p.Options = nil
|
||||
|
||||
@@ -106,6 +106,17 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
|
||||
// than persisted and then skipped on every future start — which would be a
|
||||
// model that looks added and never appears.
|
||||
func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
|
||||
return s.add(ctx, m, by, false)
|
||||
}
|
||||
|
||||
// AddNew is Add for a model nobody has added meanwhile: one that is in the
|
||||
// catalog by now is left as it is, and catalog.ErrExists said. The leader serving a
|
||||
// judge a policy names gives way so to the judge init added while it fetched.
|
||||
func (s *Service) AddNew(ctx context.Context, m catalog.Model, by string) error {
|
||||
return s.add(ctx, m, by, true)
|
||||
}
|
||||
|
||||
func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew bool) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
@@ -135,6 +146,9 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
|
||||
next := s.base(stored)
|
||||
for _, e := range stored {
|
||||
if e.Name == m.Name {
|
||||
if onlyNew {
|
||||
return fmt.Errorf("model %q: %w", m.Name, catalog.ErrExists)
|
||||
}
|
||||
continue // this call replaces it
|
||||
}
|
||||
next = append(next, e)
|
||||
|
||||
@@ -3,6 +3,7 @@ package modelsvc
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -377,3 +378,25 @@ func TestResourcesAreSetOnAStoredModelAndSurviveARestart(t *testing.T) {
|
||||
t.Error("the cores were lost on reload")
|
||||
}
|
||||
}
|
||||
|
||||
// The leader serving a judge gives way to the one init added while it
|
||||
// fetched: AddNew leaves a model already there as it is.
|
||||
func TestAddNewLeavesAModelAddedMeanwhile(t *testing.T) {
|
||||
s, _, rec := newService(t)
|
||||
first := model("judge")
|
||||
first.Description = "init's"
|
||||
if err := s.Add(t.Context(), first, "init"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second := model("judge")
|
||||
second.Description = "the leader's"
|
||||
if err := s.AddNew(t.Context(), second, "llmgw"); !errors.Is(err, catalog.ErrExists) {
|
||||
t.Fatalf("err %v", err)
|
||||
}
|
||||
if m, _ := rec.last.Resolve("judge"); m.Description != "init's" {
|
||||
t.Errorf("replaced: %q", m.Description)
|
||||
}
|
||||
if err := s.AddNew(t.Context(), model("fresh"), "llmgw"); err != nil {
|
||||
t.Errorf("a new model: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -548,6 +548,10 @@ func hasWeights(source string, n inventory.Node) (present, known bool) {
|
||||
// It returns the label rather than a bare false so the rejection can name it:
|
||||
// "does not carry ram=32g" is a placement failure someone can act on, and
|
||||
// "no candidates were considered" is not.
|
||||
// RoleLabel gives a node a role, such as judge: it then takes only profiles
|
||||
// whose node selector asks for that role.
|
||||
const RoleLabel = "llmgw.role"
|
||||
|
||||
func unmatchedLabel(selector, labels map[string]string) (string, bool) {
|
||||
for k, want := range selector {
|
||||
if got, ok := labels[k]; !ok || got != want {
|
||||
@@ -602,6 +606,13 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
reject("does not carry " + miss)
|
||||
continue
|
||||
}
|
||||
// A node given a role is kept for what asks for it: the judges'
|
||||
// machine runs the judges alone, so nothing beside them takes the
|
||||
// cores a judgment waits on.
|
||||
if role, ok := n.Labels[RoleLabel]; ok && p.NodeSelector[RoleLabel] != role {
|
||||
reject("kept for " + RoleLabel + "=" + role)
|
||||
continue
|
||||
}
|
||||
// The engine has to be installed here. The field has been on this
|
||||
// struct since placement was written and was never read, so a node
|
||||
// with no vLLM was as good a candidate as one with it: the replica was
|
||||
|
||||
@@ -1099,3 +1099,19 @@ func TestAProfileCanAskForAnEngineVersion(t *testing.T) {
|
||||
t.Errorf("the rejection should say what the node has:\n%s", res.Explain())
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeGivenARoleTakesOnlyWhatAsksForIt(t *testing.T) {
|
||||
judge := node("judge", "", 0)
|
||||
judge.Labels = map[string]string{RoleLabel: "judge"}
|
||||
s := New(store(judge))
|
||||
// A model that does not ask for the judges' machine is not put there.
|
||||
res := s.Place([]Profile{{Name: "cpu", Engine: "llamacpp"}}, nil)
|
||||
if res.Placement != nil || !strings.Contains(res.Explain(), "kept for llmgw.role=judge") {
|
||||
t.Fatalf("an ordinary model went on the judges' machine:\n%s", res.Explain())
|
||||
}
|
||||
// The judge, which asks for it, is.
|
||||
res = s.Place([]Profile{{Name: "cpu", Engine: "llamacpp", NodeSelector: map[string]string{RoleLabel: "judge"}}}, nil)
|
||||
if res.Placement == nil || res.Placement.Nodes[0].NodeID != "judge" {
|
||||
t.Fatalf("the judge was not placed on its machine:\n%s", res.Explain())
|
||||
}
|
||||
}
|
||||
@@ -582,7 +582,11 @@ func (r *Reconciler) expireStuckReplicas() {
|
||||
if rep.State != Starting && rep.State != Placing {
|
||||
continue
|
||||
}
|
||||
if now.Sub(rep.CreatedAt) < r.startupTimeout(rep) {
|
||||
since := rep.CreatedAt
|
||||
if rep.fetchedAt.After(since) {
|
||||
since = rep.fetchedAt
|
||||
}
|
||||
if now.Sub(since) < r.startupTimeout(rep) {
|
||||
continue
|
||||
}
|
||||
rep.State = Failed
|
||||
@@ -1328,6 +1332,9 @@ func (r *Reconciler) OnFetchProgress(replicaID, shardID, source string, done, to
|
||||
if rep.Shards[i].ID != shardID {
|
||||
continue
|
||||
}
|
||||
if prev := rep.Shards[i].Fetch; prev == nil || done > prev.BytesDone {
|
||||
rep.fetchedAt = r.now()
|
||||
}
|
||||
rep.Shards[i].Fetch = &FetchProgress{
|
||||
Source: source, BytesDone: done, BytesTotal: total,
|
||||
}
|
||||
|
||||
@@ -160,6 +160,11 @@ type Replica struct {
|
||||
AdoptedAt time.Time
|
||||
|
||||
CreatedAt time.Time
|
||||
// fetchedAt is when a shard's download last moved: the startup timeout
|
||||
// runs from then, so a model of hundreds of gigabytes on a slow mirror
|
||||
// is not failed while it is still arriving, and one whose download
|
||||
// stalls is failed a timeout after it stopped.
|
||||
fetchedAt time.Time
|
||||
ReadyAt time.Time
|
||||
// LastUsedAt is when a request was last routed to this replica, zero if
|
||||
// none has been since the router started.
|
||||
|
||||
@@ -406,3 +406,34 @@ func TestAMultiNodeReplicaKnowsHowManyShardsItWants(t *testing.T) {
|
||||
t.Errorf("WantShards = %d, want 2", reps[0].WantShards)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of hundreds of gigabytes on a slow mirror takes longer to arrive
|
||||
// than to load: the startup timeout runs from when its download last moved,
|
||||
// and a download that stops is failed a timeout after it stopped.
|
||||
func TestAReplicaStillDownloadingIsNotFailed(t *testing.T) {
|
||||
m := testModel(1, catalog.Profile{
|
||||
Name: "tp4", GPUClass: "h100-80g", TP: 4, StartupTimeout: 20 * time.Minute,
|
||||
})
|
||||
h := newHarness(t, m, gpuNode("1", 8))
|
||||
h.r.Reconcile(context.Background())
|
||||
rep := h.r.Replicas()[0]
|
||||
shard := rep.Shards[0].ID
|
||||
|
||||
// Forty minutes of download, moving all the while.
|
||||
for i := int64(1); i <= 8; i++ {
|
||||
h.advance(5 * time.Minute)
|
||||
h.r.OnFetchProgress(rep.ID, shard, "mirror", i<<35, 400<<30)
|
||||
h.r.expireStuckReplicas()
|
||||
}
|
||||
if got := h.r.Replicas()[0].State; got == Failed {
|
||||
t.Fatal("a replica whose download kept moving was failed")
|
||||
}
|
||||
// The same figure again is not progress: stalled, it fails a timeout on.
|
||||
h.advance(10 * time.Minute)
|
||||
h.r.OnFetchProgress(rep.ID, shard, "mirror", 8<<35, 400<<30)
|
||||
h.advance(11 * time.Minute)
|
||||
h.r.expireStuckReplicas()
|
||||
if got := h.r.Replicas()[0].State; got != Failed {
|
||||
t.Errorf("a stalled download was never failed: %s", got)
|
||||
}
|
||||
}
|
||||
+138
-11
@@ -5,6 +5,7 @@ import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/binary"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -24,6 +25,26 @@ type LDAP struct {
|
||||
cfg config.LDAP
|
||||
tls *tls.Config
|
||||
dial func(url string) (ldapConn, error)
|
||||
// password, when set, is the service account's in place of the one
|
||||
// bind_password_file names: settings tried before they are kept.
|
||||
password string
|
||||
}
|
||||
|
||||
// WithPassword has the service account bind with pw rather than read it.
|
||||
func (d *LDAP) WithPassword(pw string) *LDAP {
|
||||
d.password = pw
|
||||
return d
|
||||
}
|
||||
|
||||
func (d *LDAP) servicePassword(ctx context.Context) (string, error) {
|
||||
if d.password != "" {
|
||||
return d.password, nil
|
||||
}
|
||||
pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("auth.ldap.bind_password_file: %w", err)
|
||||
}
|
||||
return strings.TrimSpace(string(pw)), nil
|
||||
}
|
||||
|
||||
// ldapConn is the part of a connection used here, so tests need no server.
|
||||
@@ -49,14 +70,18 @@ var ErrBadCredentials = errors.New("the directory does not take that name and pa
|
||||
// NewLDAP prepares a directory from its configuration.
|
||||
func NewLDAP(cfg config.LDAP) (*LDAP, error) {
|
||||
tc := &tls.Config{MinVersion: tls.VersionTLS12}
|
||||
if cfg.CAFile != "" {
|
||||
pem, err := os.ReadFile(cfg.CAFile)
|
||||
if err != nil {
|
||||
ca, where := []byte(cfg.CA), "auth.ldap.ca"
|
||||
if cfg.CA == "" && cfg.CAFile != "" {
|
||||
var err error
|
||||
if ca, err = os.ReadFile(cfg.CAFile); err != nil {
|
||||
return nil, fmt.Errorf("auth.ldap.ca_file: %w", err)
|
||||
}
|
||||
pool := x509.NewCertPool()
|
||||
if !pool.AppendCertsFromPEM(pem) {
|
||||
return nil, fmt.Errorf("auth.ldap.ca_file %s holds no certificate", cfg.CAFile)
|
||||
where = "auth.ldap.ca_file " + cfg.CAFile
|
||||
}
|
||||
if len(ca) > 0 {
|
||||
pool, err := certPool(ca)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", where, err)
|
||||
}
|
||||
tc.RootCAs = pool
|
||||
}
|
||||
@@ -65,6 +90,33 @@ func NewLDAP(cfg config.LDAP) (*LDAP, error) {
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// certPool is the certificates in pem, each of which must parse: a
|
||||
// certificate Go refuses (a negative serial, as Samba's own has) is said,
|
||||
// not skipped into an empty pool.
|
||||
func certPool(data []byte) (*x509.CertPool, error) {
|
||||
pool := x509.NewCertPool()
|
||||
n := 0
|
||||
for rest := data; ; {
|
||||
var b *pem.Block
|
||||
if b, rest = pem.Decode(rest); b == nil {
|
||||
break
|
||||
}
|
||||
if b.Type != "CERTIFICATE" {
|
||||
continue
|
||||
}
|
||||
c, err := x509.ParseCertificate(b.Bytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pool.AddCert(c)
|
||||
n++
|
||||
}
|
||||
if n == 0 {
|
||||
return nil, errors.New("holds no certificate")
|
||||
}
|
||||
return pool, nil
|
||||
}
|
||||
|
||||
func (d *LDAP) dialURL(u string) (ldapConn, error) {
|
||||
c, err := ldap.DialURL(u, ldap.DialWithTLSConfig(d.tls))
|
||||
if err != nil {
|
||||
@@ -84,9 +136,9 @@ func (d *LDAP) dialURL(u string) (ldapConn, error) {
|
||||
// service is a connection bound as the account that finds people, to the
|
||||
// first server that answers.
|
||||
func (d *LDAP) service(ctx context.Context) (ldapConn, error) {
|
||||
pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
|
||||
pw, err := d.servicePassword(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("auth.ldap.bind_password_file: %w", err)
|
||||
return nil, err
|
||||
}
|
||||
var errs []error
|
||||
for _, u := range d.cfg.URLs {
|
||||
@@ -95,7 +147,7 @@ func (d *LDAP) service(ctx context.Context) (ldapConn, error) {
|
||||
errs = append(errs, fmt.Errorf("%s: %w", u, err))
|
||||
continue
|
||||
}
|
||||
if err := c.Bind(d.cfg.BindDN, strings.TrimSpace(string(pw))); err != nil {
|
||||
if err := c.Bind(d.cfg.BindDN, pw); err != nil {
|
||||
c.Close()
|
||||
// A server that refuses the service account refuses it
|
||||
// everywhere: not tried on the next.
|
||||
@@ -144,11 +196,11 @@ func (d *LDAP) Authenticate(ctx context.Context, login, password string) (Identi
|
||||
return Identity{}, fmt.Errorf("checking %s's password: %w", login, err)
|
||||
}
|
||||
// Back to the service account to read groups, which a person may not.
|
||||
pw, err := secrets.Read(ctx, d.cfg.BindPasswordFile)
|
||||
pw, err := d.servicePassword(ctx)
|
||||
if err != nil {
|
||||
return Identity{}, err
|
||||
}
|
||||
if err := c.Bind(d.cfg.BindDN, strings.TrimSpace(string(pw))); err != nil {
|
||||
if err := c.Bind(d.cfg.BindDN, pw); err != nil {
|
||||
return Identity{}, err
|
||||
}
|
||||
if id.Groups, err = d.groupsOf(c, id.DN); err != nil {
|
||||
@@ -216,6 +268,81 @@ func (d *LDAP) People(ctx context.Context, m Mapping) ([]Identity, error) {
|
||||
return people, nil
|
||||
}
|
||||
|
||||
// Check is one step of trying a directory.
|
||||
type Check struct {
|
||||
Step string `json:"step"`
|
||||
OK bool `json:"ok"`
|
||||
Detail string `json:"detail"`
|
||||
}
|
||||
|
||||
// Check tries the directory as configured, before its settings are kept:
|
||||
// each server reached and the service account bound on it, how many people
|
||||
// may sign in, and how many are in each group the mapping names.
|
||||
func (d *LDAP) Check(ctx context.Context, m Mapping) []Check {
|
||||
pw, err := d.servicePassword(ctx)
|
||||
if err != nil {
|
||||
return []Check{{Step: "password", Detail: err.Error()}}
|
||||
}
|
||||
var out []Check
|
||||
var c ldapConn
|
||||
for _, u := range d.cfg.URLs {
|
||||
conn, err := d.dial(u)
|
||||
if err == nil {
|
||||
if err = conn.Bind(d.cfg.BindDN, pw); err != nil {
|
||||
conn.Close()
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
out = append(out, Check{Step: u, Detail: err.Error()})
|
||||
continue
|
||||
}
|
||||
out = append(out, Check{Step: u, OK: true, Detail: "reached, bound as " + d.cfg.BindDN})
|
||||
if c == nil {
|
||||
c = conn
|
||||
} else {
|
||||
conn.Close()
|
||||
}
|
||||
}
|
||||
if c == nil {
|
||||
return out
|
||||
}
|
||||
defer c.Close()
|
||||
res, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
|
||||
0, 0, false, d.people(false), []string{"objectGUID"}, nil), 500)
|
||||
switch {
|
||||
case err != nil:
|
||||
out = append(out, Check{Step: "people", Detail: err.Error()})
|
||||
case len(res.Entries) == 0:
|
||||
out = append(out, Check{Step: "people", Detail: "no enabled user under base_dn matches user_filter: no one could sign in"})
|
||||
default:
|
||||
out = append(out, Check{Step: "people", OK: true,
|
||||
Detail: fmt.Sprintf("%d enabled people under base_dn may sign in", len(res.Entries))})
|
||||
}
|
||||
for _, g := range m.Groups() {
|
||||
dns, err := d.groupDNs(c, g)
|
||||
if err != nil || len(dns) == 0 {
|
||||
detail := "no group of that name under " + d.groupBase()
|
||||
if err != nil {
|
||||
detail = err.Error()
|
||||
}
|
||||
out = append(out, Check{Step: "group " + g, Detail: detail})
|
||||
continue
|
||||
}
|
||||
n := 0
|
||||
for _, gdn := range dns {
|
||||
filter := "(&" + d.people(false) + "(memberOf:" + inChain + ":=" + ldap.EscapeFilter(gdn) + "))"
|
||||
r, err := c.SearchWithPaging(ldap.NewSearchRequest(d.cfg.BaseDN, ldap.ScopeWholeSubtree,
|
||||
ldap.NeverDerefAliases, 0, 0, false, filter, []string{"objectGUID"}, nil), 500)
|
||||
if err == nil {
|
||||
n += len(r.Entries)
|
||||
}
|
||||
}
|
||||
out = append(out, Check{Step: "group " + g, OK: true,
|
||||
Detail: fmt.Sprintf("%d of the people who may sign in are in it, nested groups included", n)})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// groupDNs are the groups a mapping's name means: a DN as it is, a CN
|
||||
// looked up.
|
||||
func (d *LDAP) groupDNs(c ldapConn, g string) ([]string, error) {
|
||||
|
||||
@@ -18,12 +18,16 @@ type fakeConn struct {
|
||||
filters []string
|
||||
password string // the person's
|
||||
people int // how many the person search finds
|
||||
refused bool // the service account's password is wrong
|
||||
}
|
||||
|
||||
func (f *fakeConn) Bind(user, password string) error {
|
||||
f.binds = append(f.binds, user)
|
||||
if f.refused && strings.HasPrefix(user, "CN=llmgw") {
|
||||
return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
|
||||
}
|
||||
if strings.HasPrefix(user, "CN=Ada") && password != f.password {
|
||||
return &ldap.Error{ResultCode: ldap.LDAPResultInvalidCredentials}
|
||||
return ldap.NewError(ldap.LDAPResultInvalidCredentials, errors.New("invalid credentials"))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -128,3 +132,31 @@ func slicesHas(xs []string, x string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestADirectoryIsTriedStepByStepBeforeItsSettingsAreKept(t *testing.T) {
|
||||
m := Mapping{Roles: map[string][]string{"admin": {"llmgw-admins"}}}
|
||||
checks := fakeLDAP(t, &fakeConn{people: 1}).WithPassword("typed-in").Check(t.Context(), m)
|
||||
if len(checks) != 3 || !checks[0].OK || checks[0].Step != "ldaps://dc1" || !checks[1].OK ||
|
||||
checks[2].Step != "group llmgw-admins" || !checks[2].OK {
|
||||
t.Errorf("checks %+v", checks)
|
||||
}
|
||||
// A refused service account stops at the server, and says why.
|
||||
checks = fakeLDAP(t, &fakeConn{refused: true}).Check(t.Context(), m)
|
||||
if len(checks) != 1 || checks[0].OK || !strings.Contains(checks[0].Detail, "Invalid Credentials") {
|
||||
t.Errorf("refused: %+v", checks)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACertificateGoCannotReadIsSaidNotSkipped(t *testing.T) {
|
||||
if _, err := certPool([]byte("not a certificate")); err == nil {
|
||||
t.Error("an empty pool was taken")
|
||||
}
|
||||
// A CERTIFICATE block Go cannot parse, as it cannot Samba's own (its
|
||||
// serial is negative): an error, not an empty pool.
|
||||
neg := "-----BEGIN CERTIFICATE-----\nMIIBNTCB3KADAgECAgT/////MAoGCCqGSM49BAMCMA8xDTALBgNVBAMTBHRlc3Qw\n" +
|
||||
"HhcNMjYwMTAxMDAwMDAwWhcNMjgwMTAxMDAwMDAwWjAPMQ0wCwYDVQQDEwR0ZXN0\n" +
|
||||
"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE\n-----END CERTIFICATE-----\n"
|
||||
if _, err := certPool([]byte(neg)); err == nil {
|
||||
t.Error("a broken certificate made a pool")
|
||||
}
|
||||
}
|
||||
@@ -53,6 +53,7 @@ var Reference = []Block{
|
||||
{Package: "config", Name: "Cache", Title: "Cache", Doc: "", Fields: []Field{
|
||||
{Key: "max_bytes", Type: "int64", Doc: "MaxBytes bounds every project's answers together; 128 MB unless set.\nPast it, the answers used longest ago go first.", Nested: ""},
|
||||
{Key: "max_entry_bytes", Type: "int64", Doc: "MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB\nunless set.", Nested: ""},
|
||||
{Key: "embedding_model", Type: "string", Doc: "EmbeddingModel embeds the questions of projects that reuse answers to\nsimilar ones (llmgw keys cache --similar): an embedding model in the\ncatalog. Unset, those projects reuse only the same request's answer.\nThe gateway asks it on its own account, as it asks a judge.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "Canary", Title: "Canary", Doc: "", Fields: []Field{
|
||||
{Key: "model", Type: "string", Doc: "Model is the candidate: a catalog model, usually this one at newer\nweights, which clients cannot name themselves.", Nested: ""},
|
||||
@@ -193,6 +194,7 @@ var Reference = []Block{
|
||||
{Key: "urls", Type: "[]string", Doc: "URLs are the directory's servers, tried in turn: ldaps:// (636), or\nldap:// (389) with StartTLS.", Nested: ""},
|
||||
{Key: "start_tls", Type: "bool", Doc: "StartTLS upgrades an ldap:// connection before anything is sent. A\npassword never crosses a connection without TLS: ldap:// without it\nis refused.", Nested: ""},
|
||||
{Key: "ca_file", Type: "string", Doc: "CAFile is the authority the servers' certificates are checked against,\nwhen it is not the system's: an AD Certificate Services root.", Nested: ""},
|
||||
{Key: "ca", Type: "string", Doc: "CA is the same authority written out, PEM: kept in the cluster's\nsettings, so no file has to be put on every control plane. Used in\nplace of CAFile when both are set.", Nested: ""},
|
||||
{Key: "bind_dn", Type: "string", Doc: "BindDN and the file holding its password: a service account that may\nread users and groups. Nothing is written to the directory.", Nested: ""},
|
||||
{Key: "bind_password_file", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "base_dn", Type: "string", Doc: "BaseDN is where people are searched for, subtree included.", Nested: ""},
|
||||
@@ -328,6 +330,7 @@ var Reference = []Block{
|
||||
{Key: "require_capabilities", Type: "bool", Doc: "RequireCapabilities refuses a request offering tools, or sending\nimages, to a model whose catalog entry does not say it takes them.", Nested: ""},
|
||||
{Key: "validate_structured", Type: "bool", Doc: "ValidateStructured checks an answer against the JSON schema the\nrequest asked for in response_format: one that does not fit is\nstopped — or, streamed a sentence at a time, flagged, having been\nsent already.", Nested: ""},
|
||||
{Key: "judge", Type: "*JudgePolicy", Doc: "Judge has a judge model on the cluster read the text too.", Nested: "guard.JudgePolicy"},
|
||||
{Key: "services", Type: "[]Service", Doc: "Services are outside guardrails asked over HTTP, beside the judge.", Nested: ""},
|
||||
{Key: "stream", Type: "string", Doc: "Stream: sentence scans a streamed answer a sentence at a time,\nreleasing each once it passes; buffer_all holds the whole answer\nuntil it is judged. Empty is sentence.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Prefix", Title: "Prefix", Doc: "", Fields: []Field{
|
||||
@@ -405,6 +408,7 @@ var Reference = []Block{
|
||||
{Key: "injection", Type: "string", Doc: "Injection is what is done on signs of prompt injection: flag or block.", Nested: ""},
|
||||
{Key: "deny", Type: "[]Pattern", Doc: "Deny are patterns, matched against the text as a model reads it.", Nested: ""},
|
||||
{Key: "max_chars", Type: "int", Doc: "MaxChars refuses text longer than this; zero is no limit.", Nested: ""},
|
||||
{Key: "restore", Type: "bool", Doc: "Restore, on input, puts what redact rewrote as placeholders back into\nthe answer: the model reads \"[EMAIL_1]\", the client reads its own\naddress. Only the values that request sent, only into its answer.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "SCIM", Title: "SCIM", Doc: "", Fields: []Field{
|
||||
{Key: "token_file", Type: "string", Doc: "TokenFile holds the bearer token the provider sends. Empty turns\nprovisioning off.", Nested: ""},
|
||||
@@ -482,6 +486,17 @@ var Reference = []Block{
|
||||
{Key: "federation", Type: "Federation", Doc: "Federation is how this cluster takes part in federation with other\nllmgw clusters. The peers themselves are added at runtime, with\n`llmgw ctl federation add`, and kept in the store.", Nested: "config.Federation"},
|
||||
{Key: "endpoints", Type: "[]Endpoint", Doc: "Endpoints is the phase-1 static replica list. It is only consulted when\nno model declares a profile, which is what makes a router-only smoke test\nagainst sim-engines possible without a control plane.", Nested: ""},
|
||||
}},
|
||||
{Package: "guard", Name: "Service", Title: "Service", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "Name says which service decided, in the verdict's rule.", Nested: ""},
|
||||
{Key: "url", Type: "string", Doc: "URL is POSTed the body.", Nested: ""},
|
||||
{Key: "header_files", Type: "map[string]string", Doc: "HeaderFiles are headers whose values are read from files each time,\nso a rotated key needs no restart: {Authorization: /etc/…/lakera}.", Nested: ""},
|
||||
{Key: "body", Type: "string", Doc: "Body is the request, a template given .Text; {{json .Text}} writes it\nas a JSON string. Empty is {\"input\": <text>}.", Nested: ""},
|
||||
{Key: "blocked", Type: "string", Doc: "Blocked is where in the answer the service says the text is unsafe:\na dotted path, numbers for list items (results.0.flagged). True, a\nnumber above zero, a non-empty list or string other than \"false\" is\nunsafe.", Nested: ""},
|
||||
{Key: "reason", Type: "string", Doc: "Reason is where it says why, for the verdict; optional.", Nested: ""},
|
||||
{Key: "on", Type: "string", Doc: "On: input, output or both. Empty is input.", Nested: ""},
|
||||
{Key: "action", Type: "string", Doc: "Action on unsafe: block or flag. Empty is block.", Nested: ""},
|
||||
{Key: "timeout", Type: "time.Duration", Doc: "Timeout bounds one call. Empty is 3s; past it the policy's fail says.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "ServingSettings", Title: "ServingSettings", Doc: "", Fields: []Field{
|
||||
{Key: "stopped", Type: "bool", Doc: "Stopped takes every model down and answers every request for one with\n503 (system_stopped), leaving the control planes, agents and mirror\nrunning and every model's definition as it is: llmgw stop sets it,\nllmgw start clears it, and the models come back as their catalog\nentries ask. In force on every control plane within seconds.", Nested: ""},
|
||||
}},
|
||||
@@ -536,6 +551,7 @@ var Reference = []Block{
|
||||
{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "cached_per_1k", Type: "int64", Doc: "CachedPer1k is what a prompt token costs that the engine or provider\nread from its cache instead of computing: the part of prompt_tokens\nthe answer counts as cached_tokens. Zero charges those at\nPromptPer1k, as before there was a rate for them.", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is what one picture costs, for a model that makes them: a\npicture's cost is the card's time, which its prompt's few tokens do\nnot measure.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is what one 3D model costs: minutes of a card, which no\ntoken count measures.", Nested: ""},
|
||||
}},
|
||||
@@ -584,6 +600,7 @@ var Reference = []Block{
|
||||
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "cached_per_1k", Type: "int64", Doc: "CachedPer1k is the cost of a prompt token the engine or provider read\nfrom its cache (the answer's cached_tokens, counted within the\nprompt's). Zero charges them at prompt_per_1k.", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is the cost of one picture, for a model that makes them\n(/v1/images/generations), on top of its prompt's tokens.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is the cost of one 3D model (/v1/3d/generations).", Nested: ""},
|
||||
}},
|
||||
|
||||
@@ -31,6 +31,7 @@ var Sources = []string{
|
||||
"internal/router/admission/controller.go",
|
||||
"internal/accounting/cost.go",
|
||||
"internal/guard/policy.go",
|
||||
"internal/guard/service.go",
|
||||
"internal/guard/judge.go",
|
||||
"internal/config/oidc.go",
|
||||
"internal/config/ldap.go",
|
||||
|
||||
@@ -307,6 +307,15 @@ func (p *placeholders) rewrite(text string, spans []detect.Span) string {
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// inverse is each placeholder and the value it replaced.
|
||||
func (p *placeholders) inverse() map[string]string {
|
||||
out := make(map[string]string, len(p.byValue))
|
||||
for v, ph := range p.byValue {
|
||||
out[ph] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var _ Guard = (*Gate)(nil)
|
||||
|
||||
// Directives are the policy in force's demands of the gateway. Not in
|
||||
@@ -406,6 +415,19 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
|
||||
if found && best.action == Block {
|
||||
return done(best), nil
|
||||
}
|
||||
// Outside services next, then the judge: what a pattern already stopped
|
||||
// need not wait for either.
|
||||
if user != "" {
|
||||
sf, serr := g.askServices(ctx, p, true, user)
|
||||
if serr != nil && !v.Shadow {
|
||||
v.Took = g.now().Sub(start)
|
||||
return v, serr
|
||||
}
|
||||
fs = append(fs, sf...)
|
||||
if best, found = strictest(fs); found && best.action == Block {
|
||||
return done(best), nil
|
||||
}
|
||||
}
|
||||
// The judge last: what a pattern already stopped need not wait for it.
|
||||
if jp := p.Judge; jp != nil && jp.Input != "" && user != "" {
|
||||
res, err := g.judger.ask(ctx, jp, user, "")
|
||||
@@ -432,6 +454,9 @@ func (g *Gate) Input(ctx context.Context, s Subject, body []byte) (Verdict, erro
|
||||
rw.seg.set(rw.text)
|
||||
changed = true
|
||||
}
|
||||
if p.Input.Restore && len(rewrites) > 0 {
|
||||
v.Restore = ph.inverse()
|
||||
}
|
||||
if d.pin(p.SystemPrompt, p.DropClientSystem) {
|
||||
changed = true
|
||||
}
|
||||
|
||||
@@ -309,3 +309,27 @@ func TestAJudgesRefusalNamesTheHazard(t *testing.T) {
|
||||
t.Errorf("%q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreSaysWhichPlaceholderWasWhichOnlyWhenAsked(t *testing.T) {
|
||||
body := []byte(`{"messages":[{"role":"user","content":"mail ayse@example.com and ayse@example.com"}]}`)
|
||||
for _, c := range []struct {
|
||||
mode string
|
||||
restore bool
|
||||
want int
|
||||
}{{"enforce", true, 1}, {"enforce", false, 0}, {"shadow", true, 0}} {
|
||||
p := Policy{Mode: c.mode, Input: Rules{PII: map[string]string{"email": "redact"}, Restore: c.restore}}
|
||||
g := gate(t, lib(map[string]Policy{"pii": p}, "pii"))
|
||||
v, err := g.Input(context.Background(), Subject{}, body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(v.Restore) != c.want {
|
||||
t.Errorf("%s restore=%v: %v", c.mode, c.restore, v.Restore)
|
||||
}
|
||||
for ph, val := range v.Restore {
|
||||
if val != "ayse@example.com" || !strings.HasPrefix(ph, "[EMAIL_") {
|
||||
t.Errorf("%s -> %s", ph, val)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -75,6 +75,10 @@ type Verdict struct {
|
||||
// Shadow says the policy only records: the action is what it would have
|
||||
// done, and nothing is done.
|
||||
Shadow bool
|
||||
// Restore maps each placeholder the input was given to the value it
|
||||
// stands for, when the policy puts them back in the answer. Held for
|
||||
// the request's life only: never stored, logged or traced.
|
||||
Restore map[string]string
|
||||
}
|
||||
|
||||
// Stricter reports whether v outranks w: block over redact over flag over
|
||||
|
||||
@@ -61,6 +61,9 @@ func (j *JudgePolicy) Name() string { return strings.Join(j.Judges(), ", else ")
|
||||
// not say.
|
||||
const DefaultJudgeBudget = 250 * time.Millisecond
|
||||
|
||||
// Budget is how long each judge is waited for.
|
||||
func (j *JudgePolicy) Budget() time.Duration { return j.budget() }
|
||||
|
||||
func (j *JudgePolicy) budget() time.Duration {
|
||||
if j.LatencyBudget > 0 {
|
||||
return j.LatencyBudget
|
||||
|
||||
@@ -3,6 +3,7 @@ package guard
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -55,6 +56,7 @@ func outputWatched(p Policy) bool {
|
||||
o, t := p.Output, p.Tools
|
||||
return len(o.PII) > 0 || o.Injection != "" || len(o.Deny) > 0 || o.MaxChars > 0 || p.ValidateStructured ||
|
||||
p.Judge != nil && p.Judge.Output != "" ||
|
||||
slices.ContainsFunc(p.Services, Service.onOutput) ||
|
||||
len(t.Allow) > 0 || len(t.Deny) > 0 || t.MaxCallsPerTurn > 0 || t.ValidateArguments
|
||||
}
|
||||
|
||||
@@ -82,6 +84,7 @@ type outFilter struct {
|
||||
full strings.Builder // the whole answer, when it must fit a schema
|
||||
user string // the request's last user turn, for the judge
|
||||
sofar strings.Builder // the answer so far, for the judge
|
||||
served strings.Builder // the answer so far, for outside services, asked once at its end
|
||||
judged int // how much of it the judge has read
|
||||
v Verdict
|
||||
took time.Duration
|
||||
@@ -203,6 +206,21 @@ func (f *outFilter) askJudge(text string, final bool) finding {
|
||||
return f.record([]finding{jf})
|
||||
}
|
||||
|
||||
// askServices has the outside services read the whole answer, once, at its
|
||||
// end. One that cannot be asked under fail: closed stops it.
|
||||
func (f *outFilter) askServices(text string) finding {
|
||||
if !slices.ContainsFunc(f.p.Services, Service.onOutput) || text == "" {
|
||||
return finding{}
|
||||
}
|
||||
start := time.Now()
|
||||
fs, err := f.g.askServices(f.ctx, f.p, false, text)
|
||||
f.took += time.Since(start)
|
||||
if err != nil {
|
||||
fs = append(fs, finding{action: Block, category: "service", rule: "service:unavailable"})
|
||||
}
|
||||
return f.record(fs)
|
||||
}
|
||||
|
||||
// judgeStructured checks a whole answer against the schema the request
|
||||
// asked it to fit.
|
||||
func (f *outFilter) judgeStructured(text string, enforce bool) finding {
|
||||
@@ -259,6 +277,9 @@ func (f *outFilter) Buffered(body []byte) []byte {
|
||||
if f.askJudge(t, true).action == Block {
|
||||
blocked = true
|
||||
}
|
||||
if f.askServices(t).action == Block {
|
||||
blocked = true
|
||||
}
|
||||
if out != t {
|
||||
msg["content"] = rawString(out)
|
||||
ch["message"], _ = json.Marshal(msg)
|
||||
@@ -422,6 +443,12 @@ func (f *outFilter) flush(final bool) (bool, map[int]string, bool) {
|
||||
if i == 0 && f.askJudge(head, final).action == Block {
|
||||
blocked = true
|
||||
}
|
||||
if i == 0 {
|
||||
f.served.WriteString(head)
|
||||
if final && f.askServices(f.served.String()).action == Block {
|
||||
blocked = true
|
||||
}
|
||||
}
|
||||
if head != "" {
|
||||
fd, r := f.judgeText(head)
|
||||
blocked = blocked || fd.action == Block
|
||||
|
||||
@@ -69,6 +69,8 @@ type Policy struct {
|
||||
ValidateStructured bool `koanf:"validate_structured"`
|
||||
// Judge has a judge model on the cluster read the text too.
|
||||
Judge *JudgePolicy `koanf:"judge"`
|
||||
// Services are outside guardrails asked over HTTP, beside the judge.
|
||||
Services []Service `koanf:"services"`
|
||||
// Stream: sentence scans a streamed answer a sentence at a time,
|
||||
// releasing each once it passes; buffer_all holds the whole answer
|
||||
// until it is judged. Empty is sentence.
|
||||
@@ -86,6 +88,10 @@ type Rules struct {
|
||||
Deny []Pattern `koanf:"deny"`
|
||||
// MaxChars refuses text longer than this; zero is no limit.
|
||||
MaxChars int `koanf:"max_chars"`
|
||||
// Restore, on input, puts what redact rewrote as placeholders back into
|
||||
// the answer: the model reads "[EMAIL_1]", the client reads its own
|
||||
// address. Only the values that request sent, only into its answer.
|
||||
Restore bool `koanf:"restore"`
|
||||
}
|
||||
|
||||
// Pattern is a named regular expression and what its match does.
|
||||
@@ -150,6 +156,11 @@ func (p Policy) validate() error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, sv := range p.Services {
|
||||
if err := sv.validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
switch p.Tools.OnDenied {
|
||||
case "", "strip", "block":
|
||||
default:
|
||||
@@ -209,6 +220,12 @@ func Merge(names []string, ps []Policy) Policy {
|
||||
out.RequireCapabilities = out.RequireCapabilities || p.RequireCapabilities
|
||||
out.ValidateStructured = out.ValidateStructured || p.ValidateStructured
|
||||
out.Judge = mergeJudge(out.Judge, p.Judge)
|
||||
// Every service any policy asks, each once.
|
||||
for _, sv := range p.Services {
|
||||
if !slices.ContainsFunc(out.Services, func(e Service) bool { return e.Name == sv.Name }) {
|
||||
out.Services = append(out.Services, sv)
|
||||
}
|
||||
}
|
||||
out.MaxOutputTokens = minSet(out.MaxOutputTokens, p.MaxOutputTokens)
|
||||
if p.SystemPrompt != "" {
|
||||
prompts = append(prompts, p.SystemPrompt)
|
||||
@@ -253,6 +270,9 @@ func mergeRules(dst *Rules, r Rules) {
|
||||
}
|
||||
}
|
||||
dst.MaxChars = minSet(dst.MaxChars, r.MaxChars)
|
||||
// Any policy may ask: what is put back is the caller's own, to the
|
||||
// caller, and never reaches the model either way.
|
||||
dst.Restore = dst.Restore || r.Restore
|
||||
}
|
||||
|
||||
func stricterOf(a, b string, order []string) string {
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
package guard
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"text/template"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A guardrail that is someone else's service: Azure Content Safety, Lakera,
|
||||
// NeMo Guardrails, a company's own classifier. A site may be required to use
|
||||
// one; this asks it beside the patterns and the judge, and its answer counts
|
||||
// like theirs. Told only the text, never who sent it, and nothing of the
|
||||
// call is kept but its verdict.
|
||||
|
||||
// Service is one outside guardrail, asked over HTTP.
|
||||
type Service struct {
|
||||
// Name says which service decided, in the verdict's rule.
|
||||
Name string `koanf:"name"`
|
||||
// URL is POSTed the body.
|
||||
URL string `koanf:"url"`
|
||||
// HeaderFiles are headers whose values are read from files each time,
|
||||
// so a rotated key needs no restart: {Authorization: /etc/…/lakera}.
|
||||
HeaderFiles map[string]string `koanf:"header_files"`
|
||||
// Body is the request, a template given .Text; {{json .Text}} writes it
|
||||
// as a JSON string. Empty is {"input": <text>}.
|
||||
Body string `koanf:"body"`
|
||||
// Blocked is where in the answer the service says the text is unsafe:
|
||||
// a dotted path, numbers for list items (results.0.flagged). True, a
|
||||
// number above zero, a non-empty list or string other than "false" is
|
||||
// unsafe.
|
||||
Blocked string `koanf:"blocked"`
|
||||
// Reason is where it says why, for the verdict; optional.
|
||||
Reason string `koanf:"reason"`
|
||||
// On: input, output or both. Empty is input.
|
||||
On string `koanf:"on"`
|
||||
// Action on unsafe: block or flag. Empty is block.
|
||||
Action string `koanf:"action"`
|
||||
// Timeout bounds one call. Empty is 3s; past it the policy's fail says.
|
||||
Timeout time.Duration `koanf:"timeout"`
|
||||
}
|
||||
|
||||
func (s Service) validate() error {
|
||||
if s.Name == "" {
|
||||
return fmt.Errorf("services: one has no name")
|
||||
}
|
||||
u, err := url.Parse(s.URL)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return fmt.Errorf("services.%s: url %q is not an http(s) address", s.Name, s.URL)
|
||||
}
|
||||
if s.Blocked == "" {
|
||||
return fmt.Errorf("services.%s: blocked says where the answer says unsafe, and is empty", s.Name)
|
||||
}
|
||||
switch s.On {
|
||||
case "", "input", "output", "both":
|
||||
default:
|
||||
return fmt.Errorf("services.%s: on is %q: input, output or both", s.Name, s.On)
|
||||
}
|
||||
switch s.Action {
|
||||
case "", "block", "flag":
|
||||
default:
|
||||
return fmt.Errorf("services.%s: action is %q: block or flag", s.Name, s.Action)
|
||||
}
|
||||
if _, err := s.template(); err != nil {
|
||||
return fmt.Errorf("services.%s: body: %w", s.Name, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s Service) onInput() bool { return s.On == "" || s.On == "input" || s.On == "both" }
|
||||
func (s Service) onOutput() bool { return s.On == "output" || s.On == "both" }
|
||||
|
||||
func (s Service) action() Action {
|
||||
if s.Action == "flag" {
|
||||
return Flag
|
||||
}
|
||||
return Block
|
||||
}
|
||||
|
||||
func (s Service) template() (*template.Template, error) {
|
||||
body := s.Body
|
||||
if body == "" {
|
||||
body = `{"input": {{json .Text}}}`
|
||||
}
|
||||
return template.New(s.Name).Funcs(template.FuncMap{"json": func(v any) (string, error) {
|
||||
b, err := json.Marshal(v)
|
||||
return string(b), err
|
||||
}}).Parse(body)
|
||||
}
|
||||
|
||||
// serviceClient asks every service; each call is bounded by its own timeout.
|
||||
var serviceClient = &http.Client{}
|
||||
|
||||
// ask sends the service the text and says whether it found it unsafe, and why.
|
||||
func (s Service) ask(ctx context.Context, text string) (unsafe bool, reason string, err error) {
|
||||
t, err := s.template()
|
||||
if err != nil {
|
||||
return false, "", err
|
||||
}
|
||||
var body bytes.Buffer
|
||||
if err := t.Execute(&body, struct{ Text string }{text}); err != nil {
|
||||
return false, "", err
|
||||
}
|
||||
timeout := s.Timeout
|
||||
if timeout <= 0 {
|
||||
timeout = 3 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.URL, &body)
|
||||
if err != nil {
|
||||
return false, "", err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
for h, file := range s.HeaderFiles {
|
||||
v, err := os.ReadFile(file)
|
||||
if err != nil {
|
||||
return false, "", fmt.Errorf("header %s: %w", h, err)
|
||||
}
|
||||
req.Header.Set(h, strings.TrimSpace(string(v)))
|
||||
}
|
||||
resp, err := serviceClient.Do(req)
|
||||
if err != nil {
|
||||
return false, "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if err != nil {
|
||||
return false, "", err
|
||||
}
|
||||
if resp.StatusCode/100 != 2 {
|
||||
return false, "", fmt.Errorf("answered %d", resp.StatusCode)
|
||||
}
|
||||
var doc any
|
||||
if err := json.Unmarshal(b, &doc); err != nil {
|
||||
return false, "", fmt.Errorf("answer is not JSON: %w", err)
|
||||
}
|
||||
v, ok := at(doc, s.Blocked)
|
||||
if !ok {
|
||||
return false, "", fmt.Errorf("the answer has no %s", s.Blocked)
|
||||
}
|
||||
if s.Reason != "" {
|
||||
if r, ok := at(doc, s.Reason); ok {
|
||||
reason = fmt.Sprint(r)
|
||||
}
|
||||
}
|
||||
return truthy(v), reason, nil
|
||||
}
|
||||
|
||||
// at is the value at a dotted path in decoded JSON.
|
||||
func at(doc any, path string) (any, bool) {
|
||||
cur := doc
|
||||
for _, k := range strings.Split(path, ".") {
|
||||
switch c := cur.(type) {
|
||||
case map[string]any:
|
||||
v, ok := c[k]
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
cur = v
|
||||
case []any:
|
||||
i, err := strconv.Atoi(k)
|
||||
if err != nil || i < 0 || i >= len(c) {
|
||||
return nil, false
|
||||
}
|
||||
cur = c[i]
|
||||
default:
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
return cur, true
|
||||
}
|
||||
|
||||
func truthy(v any) bool {
|
||||
switch x := v.(type) {
|
||||
case bool:
|
||||
return x
|
||||
case float64:
|
||||
return x > 0
|
||||
case string:
|
||||
return x != "" && !strings.EqualFold(x, "false") && x != "0"
|
||||
case []any:
|
||||
return len(x) > 0
|
||||
case map[string]any:
|
||||
return len(x) > 0
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// askServices asks each service of a direction, and says what the policy
|
||||
// makes of their answers: a finding per service that found the text unsafe,
|
||||
// or could not be asked under fail: closed. failed is true when a service
|
||||
// gave no answer, for the caller to refuse under fail: closed.
|
||||
func (g *Gate) askServices(ctx context.Context, p Policy, input bool, text string) (fs []finding, failed error) {
|
||||
for _, s := range p.Services {
|
||||
if input && !s.onInput() || !input && !s.onOutput() || text == "" {
|
||||
continue
|
||||
}
|
||||
unsafe, reason, err := s.ask(ctx, text)
|
||||
if err != nil {
|
||||
g.log.Warn("a guardrail service gave no verdict", "service", s.Name, "err", err)
|
||||
if p.Fail == "closed" {
|
||||
failed = fmt.Errorf("the guardrail service %s could not be asked: %w", s.Name, err)
|
||||
continue
|
||||
}
|
||||
g.m.failOpen.Inc()
|
||||
continue
|
||||
}
|
||||
if unsafe {
|
||||
rule := "service:" + s.Name
|
||||
if reason != "" {
|
||||
rule += ":" + reason
|
||||
}
|
||||
fs = append(fs, finding{action: s.action(), category: "service", rule: rule})
|
||||
}
|
||||
}
|
||||
return fs, failed
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package guard
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// classifier is a stand-in for Lakera or Azure Content Safety: it finds
|
||||
// "evil" unsafe, and refuses a call without its key.
|
||||
func classifier(t *testing.T) (*httptest.Server, string) {
|
||||
key := filepath.Join(t.TempDir(), "key")
|
||||
if err := os.WriteFile(key, []byte("Bearer k1\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get("Authorization") != "Bearer k1" {
|
||||
http.Error(w, "no", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
var in struct{ Input string }
|
||||
_ = json.NewDecoder(r.Body).Decode(&in)
|
||||
flagged := strings.Contains(in.Input, "evil")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"results": []any{map[string]any{"flagged": flagged, "category": "injection"}}})
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv, key
|
||||
}
|
||||
|
||||
func svcPolicy(url, key, on, action, fail string) Config {
|
||||
return lib(map[string]Policy{"svc": {Fail: fail, Services: []Service{{
|
||||
Name: "cls", URL: url, HeaderFiles: map[string]string{"Authorization": key},
|
||||
Blocked: "results.0.flagged", Reason: "results.0.category", On: on, Action: action,
|
||||
}}}}, "svc")
|
||||
}
|
||||
|
||||
func TestAnOutsideServiceBlocksOrFlagsWhatItFindsUnsafe(t *testing.T) {
|
||||
srv, key := classifier(t)
|
||||
ctx := context.Background()
|
||||
g := gate(t, svcPolicy(srv.URL, key, "", "", ""))
|
||||
v, err := g.Input(ctx, Subject{}, chat("user", "something evil"))
|
||||
if err != nil || v.Action != Block || v.Rule != "service:cls:injection" {
|
||||
t.Errorf("unsafe: %+v %v", v, err)
|
||||
}
|
||||
if v, _ := g.Input(ctx, Subject{}, chat("user", "hello")); v.Action != Allow {
|
||||
t.Errorf("safe: %+v", v)
|
||||
}
|
||||
g = gate(t, svcPolicy(srv.URL, key, "input", "flag", ""))
|
||||
if v, _ := g.Input(ctx, Subject{}, chat("user", "evil")); v.Action != Flag {
|
||||
t.Errorf("flag: %+v", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAServiceThatCannotBeAskedIsWhatFailSays(t *testing.T) {
|
||||
srv, _ := classifier(t)
|
||||
ctx := context.Background()
|
||||
// The wrong key: the service answers 401.
|
||||
wrong := filepath.Join(t.TempDir(), "wrong")
|
||||
_ = os.WriteFile(wrong, []byte("Bearer nope"), 0o600)
|
||||
if _, err := gate(t, svcPolicy(srv.URL, wrong, "", "", "closed")).Input(ctx, Subject{}, chat("user", "hi")); err == nil {
|
||||
t.Error("fail: closed served a request no service could check")
|
||||
}
|
||||
if v, err := gate(t, svcPolicy(srv.URL, wrong, "", "", "open")).Input(ctx, Subject{}, chat("user", "hi")); err != nil || v.Action != Allow {
|
||||
t.Errorf("fail: open: %+v %v", v, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOutsideServiceReadsTheWholeAnswer(t *testing.T) {
|
||||
srv, key := classifier(t)
|
||||
g := gate(t, svcPolicy(srv.URL, key, "output", "", ""))
|
||||
f := g.Output(context.Background(), Subject{})
|
||||
if f == nil {
|
||||
t.Fatal("no output filter for an output service")
|
||||
}
|
||||
out := f.Buffered([]byte(`{"choices":[{"index":0,"message":{"role":"assistant","content":"an evil plan"},"finish_reason":"stop"}]}`))
|
||||
if !strings.Contains(string(out), "content_filter") || strings.Contains(string(out), "evil") || f.Verdict().Action != Block {
|
||||
t.Errorf("%s %+v", out, f.Verdict())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAServiceIsCheckedWhenThePolicyIsRead(t *testing.T) {
|
||||
for _, s := range []Service{
|
||||
{Name: "a", URL: "ftp://x", Blocked: "x"},
|
||||
{Name: "b", URL: "https://x"},
|
||||
{Name: "c", URL: "https://x", Blocked: "x", On: "sideways"},
|
||||
{Name: "d", URL: "https://x", Blocked: "x", Body: "{{"},
|
||||
} {
|
||||
if err := (Policy{Services: []Service{s}}).validate(); err == nil {
|
||||
t.Errorf("%s passed", s.Name)
|
||||
}
|
||||
}
|
||||
if v, ok := at(map[string]any{"a": []any{map[string]any{"b": 2.0}}}, "a.0.b"); !ok || !truthy(v) {
|
||||
t.Errorf("at: %v %v", v, ok)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,346 @@
|
||||
// Package mcpclient speaks to an MCP server over the streamable HTTP
|
||||
// transport: the company's own tool servers, which llmgw's /mcp serves on to
|
||||
// its callers. It initializes once and keeps the session the server gives,
|
||||
// lists the server's tools, and calls them, taking an answer as JSON or as a
|
||||
// stream of events, whichever the server sends.
|
||||
package mcpclient
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// ProtocolVersion is the version asked for; a server may answer an older one.
|
||||
const ProtocolVersion = "2025-06-18"
|
||||
|
||||
// Client is one MCP server.
|
||||
type Client struct {
|
||||
URL string
|
||||
// Header gives what each request carries beyond the protocol's own: the
|
||||
// server's credentials. Nil sends none.
|
||||
Header func(ctx context.Context) (http.Header, error)
|
||||
HTTP *http.Client
|
||||
// MaxBytes bounds one answer; zero is 4 MiB.
|
||||
MaxBytes int64
|
||||
|
||||
mu sync.Mutex
|
||||
session string
|
||||
version string
|
||||
ready bool
|
||||
nextID atomic.Int64
|
||||
}
|
||||
|
||||
// Tool is one tool as the server describes it.
|
||||
type Tool struct {
|
||||
Name string `json:"name"`
|
||||
Title string `json:"title,omitempty"`
|
||||
Description string `json:"description,omitempty"`
|
||||
InputSchema json.RawMessage `json:"inputSchema,omitempty"`
|
||||
Annotations json.RawMessage `json:"annotations,omitempty"`
|
||||
}
|
||||
|
||||
// Error is the server's own JSON-RPC error.
|
||||
type Error struct {
|
||||
Code int `json:"code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func (e *Error) Error() string {
|
||||
return fmt.Sprintf("the MCP server answered %d: %s", e.Code, e.Message)
|
||||
}
|
||||
|
||||
// errSessionGone is the server no longer knowing the session: it is
|
||||
// initialized again, once.
|
||||
var errSessionGone = errors.New("the MCP server no longer knows the session")
|
||||
|
||||
type rpcMessage struct {
|
||||
JSONRPC string `json:"jsonrpc"`
|
||||
ID json.RawMessage `json:"id,omitempty"`
|
||||
Method string `json:"method,omitempty"`
|
||||
Params json.RawMessage `json:"params,omitempty"`
|
||||
Result json.RawMessage `json:"result,omitempty"`
|
||||
Error *Error `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// Tools lists every tool, page by page.
|
||||
func (c *Client) Tools(ctx context.Context) ([]Tool, error) {
|
||||
var out []Tool
|
||||
cursor := ""
|
||||
for range 100 {
|
||||
params := map[string]any{}
|
||||
if cursor != "" {
|
||||
params["cursor"] = cursor
|
||||
}
|
||||
raw, err := c.request(ctx, "tools/list", params, nil, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var page struct {
|
||||
Tools []Tool `json:"tools"`
|
||||
NextCursor string `json:"nextCursor"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &page); err != nil {
|
||||
return nil, fmt.Errorf("the MCP server's tool list did not parse: %w", err)
|
||||
}
|
||||
out = append(out, page.Tools...)
|
||||
if page.NextCursor == "" {
|
||||
return out, nil
|
||||
}
|
||||
cursor = page.NextCursor
|
||||
}
|
||||
return out, errors.New("the MCP server's tool list never ended")
|
||||
}
|
||||
|
||||
// Call calls a tool and gives its result as the server sent it. extra is
|
||||
// sent with this request alone (who is calling, on whose behalf); progress
|
||||
// hears the server's progress notifications while it works, when it sends
|
||||
// them as a stream.
|
||||
func (c *Client) Call(ctx context.Context, name string, args json.RawMessage, extra http.Header,
|
||||
progress func(params json.RawMessage),
|
||||
) (json.RawMessage, error) {
|
||||
if len(args) == 0 {
|
||||
args = json.RawMessage("{}")
|
||||
}
|
||||
params := map[string]any{"name": name, "arguments": args}
|
||||
if progress != nil {
|
||||
params["_meta"] = map[string]any{"progressToken": c.nextID.Add(1)}
|
||||
}
|
||||
return c.request(ctx, "tools/call", params, extra, progress)
|
||||
}
|
||||
|
||||
// request sends one request, initializing first when there is no session,
|
||||
// and again once when the server has dropped it.
|
||||
func (c *Client) request(ctx context.Context, method string, params any, extra http.Header,
|
||||
progress func(json.RawMessage),
|
||||
) (json.RawMessage, error) {
|
||||
for attempt := 0; ; attempt++ {
|
||||
if err := c.ensureReady(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
raw, err := c.send(ctx, method, params, extra, progress)
|
||||
if errors.Is(err, errSessionGone) && attempt == 0 {
|
||||
c.mu.Lock()
|
||||
c.ready, c.session = false, ""
|
||||
c.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
return raw, err
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) ensureReady(ctx context.Context) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.ready {
|
||||
return nil
|
||||
}
|
||||
c.session, c.version = "", ""
|
||||
raw, err := c.sendLocked(ctx, "initialize", map[string]any{
|
||||
"protocolVersion": ProtocolVersion,
|
||||
"capabilities": map[string]any{},
|
||||
"clientInfo": map[string]any{"name": "llmgw", "version": "1"},
|
||||
}, nil, nil, true)
|
||||
if err != nil {
|
||||
return fmt.Errorf("initializing: %w", err)
|
||||
}
|
||||
var init struct {
|
||||
ProtocolVersion string `json:"protocolVersion"`
|
||||
}
|
||||
_ = json.Unmarshal(raw, &init)
|
||||
c.version = init.ProtocolVersion
|
||||
if err := c.notifyLocked(ctx, "notifications/initialized"); err != nil {
|
||||
return fmt.Errorf("initializing: %w", err)
|
||||
}
|
||||
c.ready = true
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Client) send(ctx context.Context, method string, params any, extra http.Header,
|
||||
progress func(json.RawMessage),
|
||||
) (json.RawMessage, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.sendLocked(ctx, method, params, extra, progress, false)
|
||||
}
|
||||
|
||||
// sendLocked posts one request and reads its answer. Held under the lock
|
||||
// only for what the session needs: the request itself runs unlocked for a
|
||||
// tool call, so slow calls do not queue behind each other.
|
||||
func (c *Client) sendLocked(ctx context.Context, method string, params any, extra http.Header,
|
||||
progress func(json.RawMessage), initializing bool,
|
||||
) (json.RawMessage, error) {
|
||||
id := c.nextID.Add(1)
|
||||
p, err := json.Marshal(params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
body, _ := json.Marshal(rpcMessage{JSONRPC: "2.0", ID: json.RawMessage(fmt.Sprint(id)), Method: method, Params: p})
|
||||
req, err := c.newRequest(ctx, body, extra)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !initializing {
|
||||
// The request may be long (a tool that searches); the session is
|
||||
// read, the lock let go for the wait, and taken again after.
|
||||
c.mu.Unlock()
|
||||
defer c.mu.Lock()
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if s := resp.Header.Get("Mcp-Session-Id"); s != "" && initializing {
|
||||
c.session = s
|
||||
}
|
||||
switch {
|
||||
case resp.StatusCode == http.StatusNotFound && req.Header.Get("Mcp-Session-Id") != "":
|
||||
return nil, errSessionGone
|
||||
case resp.StatusCode >= 300:
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<10))
|
||||
return nil, fmt.Errorf("the MCP server answered HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(b)))
|
||||
}
|
||||
limited := io.LimitReader(resp.Body, c.maxBytes()+1)
|
||||
var msg rpcMessage
|
||||
if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/event-stream") {
|
||||
msg, err = readStream(limited, id, progress, c.maxBytes())
|
||||
} else {
|
||||
var b []byte
|
||||
if b, err = io.ReadAll(limited); err == nil {
|
||||
if int64(len(b)) > c.maxBytes() {
|
||||
err = fmt.Errorf("the MCP server's answer is larger than %d bytes", c.maxBytes())
|
||||
} else {
|
||||
err = json.Unmarshal(b, &msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if msg.Error != nil {
|
||||
return nil, msg.Error
|
||||
}
|
||||
return msg.Result, nil
|
||||
}
|
||||
|
||||
// notifyLocked sends a notification, which has no answer.
|
||||
func (c *Client) notifyLocked(ctx context.Context, method string) error {
|
||||
body, _ := json.Marshal(rpcMessage{JSONRPC: "2.0", Method: method})
|
||||
req, err := c.newRequest(ctx, body, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<16))
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode >= 300 {
|
||||
return fmt.Errorf("the MCP server answered HTTP %d to %s", resp.StatusCode, method)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Client) newRequest(ctx context.Context, body []byte, extra http.Header) (*http.Request, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.URL, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if c.Header != nil {
|
||||
h, err := c.Header(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for k, vs := range h {
|
||||
for _, v := range vs {
|
||||
req.Header.Add(k, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
for k, vs := range extra {
|
||||
for _, v := range vs {
|
||||
req.Header.Add(k, v)
|
||||
}
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Accept", "application/json, text/event-stream")
|
||||
if c.session != "" {
|
||||
req.Header.Set("Mcp-Session-Id", c.session)
|
||||
}
|
||||
if c.version != "" {
|
||||
req.Header.Set("MCP-Protocol-Version", c.version)
|
||||
}
|
||||
return req, nil
|
||||
}
|
||||
|
||||
// readStream reads server-sent events until the answer to id, handing
|
||||
// progress notifications on.
|
||||
func readStream(r io.Reader, id int64, progress func(json.RawMessage), limit int64) (rpcMessage, error) {
|
||||
sc := bufio.NewScanner(r)
|
||||
sc.Buffer(make([]byte, 64<<10), int(limit))
|
||||
want := fmt.Sprint(id)
|
||||
var data strings.Builder
|
||||
flush := func() (rpcMessage, bool) {
|
||||
defer data.Reset()
|
||||
if data.Len() == 0 {
|
||||
return rpcMessage{}, false
|
||||
}
|
||||
var m rpcMessage
|
||||
if json.Unmarshal([]byte(data.String()), &m) != nil {
|
||||
return rpcMessage{}, false
|
||||
}
|
||||
if m.Method == "notifications/progress" && progress != nil {
|
||||
progress(m.Params)
|
||||
return rpcMessage{}, false
|
||||
}
|
||||
if string(m.ID) == want && m.Method == "" {
|
||||
return m, true
|
||||
}
|
||||
return rpcMessage{}, false
|
||||
}
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
switch {
|
||||
case line == "":
|
||||
if m, ok := flush(); ok {
|
||||
return m, nil
|
||||
}
|
||||
case strings.HasPrefix(line, "data:"):
|
||||
if data.Len() > 0 {
|
||||
data.WriteByte('\n')
|
||||
}
|
||||
data.WriteString(strings.TrimPrefix(strings.TrimPrefix(line, "data:"), " "))
|
||||
}
|
||||
}
|
||||
if m, ok := flush(); ok {
|
||||
return m, nil
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return rpcMessage{}, fmt.Errorf("reading the MCP server's stream: %w", err)
|
||||
}
|
||||
return rpcMessage{}, errors.New("the MCP server's stream ended without an answer")
|
||||
}
|
||||
|
||||
func (c *Client) client() *http.Client {
|
||||
if c.HTTP != nil {
|
||||
return c.HTTP
|
||||
}
|
||||
return http.DefaultClient
|
||||
}
|
||||
|
||||
func (c *Client) maxBytes() int64 {
|
||||
if c.MaxBytes > 0 {
|
||||
return c.MaxBytes
|
||||
}
|
||||
return 4 << 20
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package mcpclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mcpclient/mcptest"
|
||||
)
|
||||
|
||||
func TestToolsAreListedPageByPageAndCalledInOneSession(t *testing.T) {
|
||||
srv := mcptest.New()
|
||||
defer srv.Close()
|
||||
srv.PageSize = 2
|
||||
c := &Client{URL: srv.URL, Header: func(context.Context) (http.Header, error) {
|
||||
return http.Header{"Authorization": {"Bearer service"}}, nil
|
||||
}}
|
||||
tools, err := c.Tools(t.Context())
|
||||
if err != nil || len(tools) != 3 || tools[2].Name != "big" {
|
||||
t.Fatalf("tools %+v %v", tools, err)
|
||||
}
|
||||
raw, err := c.Call(t.Context(), "echo", json.RawMessage(`{"text":"hello"}`), http.Header{"X-Llmgw-User": {"ada"}}, nil)
|
||||
if err != nil || !strings.Contains(string(raw), "hello") {
|
||||
t.Fatalf("call %s %v", raw, err)
|
||||
}
|
||||
if srv.Sessions() != 1 {
|
||||
t.Errorf("initialized %d times", srv.Sessions())
|
||||
}
|
||||
h := srv.Headers[0]
|
||||
if h.Get("Authorization") != "Bearer service" || h.Get("X-Llmgw-User") != "ada" || h.Get("MCP-Protocol-Version") != "2025-06-18" {
|
||||
t.Errorf("headers %v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStreamedAnswerIsReadAndItsProgressHeard(t *testing.T) {
|
||||
srv := mcptest.New()
|
||||
defer srv.Close()
|
||||
srv.Stream = true
|
||||
c := &Client{URL: srv.URL}
|
||||
var heard []string
|
||||
raw, err := c.Call(t.Context(), "echo", json.RawMessage(`{"text":"streamed"}`), nil, func(p json.RawMessage) {
|
||||
heard = append(heard, string(p))
|
||||
})
|
||||
if err != nil || !strings.Contains(string(raw), "streamed") {
|
||||
t.Fatalf("call %s %v", raw, err)
|
||||
}
|
||||
if len(heard) != 1 || !strings.Contains(heard[0], "halfway") {
|
||||
t.Errorf("progress %v", heard)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASessionTheServerForgotIsBegunAgainOnce(t *testing.T) {
|
||||
srv := mcptest.New()
|
||||
defer srv.Close()
|
||||
c := &Client{URL: srv.URL}
|
||||
if _, err := c.Tools(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.Forget()
|
||||
if _, err := c.Call(t.Context(), "echo", json.RawMessage(`{"text":"again"}`), nil, nil); err != nil {
|
||||
t.Fatalf("after the server restarted: %v", err)
|
||||
}
|
||||
if srv.Sessions() != 2 {
|
||||
t.Errorf("sessions %d", srv.Sessions())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheServersErrorsAndAnAnswerTooLargeAreSaid(t *testing.T) {
|
||||
srv := mcptest.New()
|
||||
defer srv.Close()
|
||||
c := &Client{URL: srv.URL, MaxBytes: 64 << 10}
|
||||
var rpcErr *Error
|
||||
if _, err := c.Call(t.Context(), "nosuch", nil, nil, nil); !errors.As(err, &rpcErr) || rpcErr.Code != -32602 {
|
||||
t.Errorf("unknown tool: %v", err)
|
||||
}
|
||||
if _, err := c.Call(t.Context(), "big", nil, nil, nil); err == nil || !strings.Contains(err.Error(), "larger than") {
|
||||
t.Errorf("a megabyte through a 64 KiB limit: %v", err)
|
||||
}
|
||||
// A tool's own failure is its result, not an error of the protocol.
|
||||
raw, err := c.Call(t.Context(), "fail", nil, nil, nil)
|
||||
if err != nil || !strings.Contains(string(raw), `"isError":true`) {
|
||||
t.Errorf("fail: %s %v", raw, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
// Package mcptest is an MCP server for tests: a few tools, a session it
|
||||
// gives at initialize and can forget, its tool list in pages, and answers
|
||||
// as JSON or, when asked, as a stream with progress before the answer.
|
||||
package mcptest
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Server is the fake server and what it saw.
|
||||
type Server struct {
|
||||
*httptest.Server
|
||||
|
||||
// Stream answers tools/call as server-sent events, with a progress
|
||||
// notification first.
|
||||
Stream bool
|
||||
// PageSize splits the tool list into pages; zero sends it whole.
|
||||
PageSize int
|
||||
|
||||
mu sync.Mutex
|
||||
session string
|
||||
sessions int
|
||||
// Calls are the tools called, and Headers the headers of each call.
|
||||
Calls []string
|
||||
Args []json.RawMessage
|
||||
Headers []http.Header
|
||||
}
|
||||
|
||||
// Tools are what it offers: echo returns its text, fail answers with a
|
||||
// tool error, big returns more than most limits allow.
|
||||
var Tools = []map[string]any{
|
||||
{"name": "echo", "description": "Says the text back.", "inputSchema": map[string]any{
|
||||
"type": "object", "properties": map[string]any{"text": map[string]any{"type": "string"}}}},
|
||||
{"name": "fail", "description": "Always fails.", "inputSchema": map[string]any{"type": "object"}},
|
||||
{"name": "big", "description": "Answers with a megabyte.", "inputSchema": map[string]any{"type": "object"}},
|
||||
}
|
||||
|
||||
// New starts one.
|
||||
func New() *Server {
|
||||
s := &Server{}
|
||||
s.Server = httptest.NewServer(http.HandlerFunc(s.serve))
|
||||
return s
|
||||
}
|
||||
|
||||
// Forget drops the session, as a restarted server would.
|
||||
func (s *Server) Forget() {
|
||||
s.mu.Lock()
|
||||
s.session = ""
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// Sessions is how many times it was initialized.
|
||||
func (s *Server) Sessions() int {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.sessions
|
||||
}
|
||||
|
||||
func (s *Server) serve(w http.ResponseWriter, r *http.Request) {
|
||||
var msg struct {
|
||||
ID json.RawMessage `json:"id"`
|
||||
Method string `json:"method"`
|
||||
Params json.RawMessage `json:"params"`
|
||||
}
|
||||
if json.NewDecoder(r.Body).Decode(&msg) != nil {
|
||||
http.Error(w, "bad", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
if msg.Method == "initialize" {
|
||||
s.sessions++
|
||||
s.session = fmt.Sprintf("s%d", s.sessions)
|
||||
w.Header().Set("Mcp-Session-Id", s.session)
|
||||
} else if r.Header.Get("Mcp-Session-Id") != s.session || s.session == "" {
|
||||
s.mu.Unlock()
|
||||
http.Error(w, "no such session", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
s.mu.Unlock()
|
||||
if len(msg.ID) == 0 {
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
return
|
||||
}
|
||||
answer := func(result any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": msg.ID, "result": result})
|
||||
}
|
||||
switch msg.Method {
|
||||
case "initialize":
|
||||
answer(map[string]any{"protocolVersion": "2025-06-18", "capabilities": map[string]any{"tools": map[string]any{}},
|
||||
"serverInfo": map[string]any{"name": "test", "version": "1"}})
|
||||
case "tools/list":
|
||||
var p struct {
|
||||
Cursor string `json:"cursor"`
|
||||
}
|
||||
_ = json.Unmarshal(msg.Params, &p)
|
||||
start := 0
|
||||
_, _ = fmt.Sscan(p.Cursor, &start)
|
||||
end := len(Tools)
|
||||
if s.PageSize > 0 && start+s.PageSize < end {
|
||||
end = start + s.PageSize
|
||||
}
|
||||
res := map[string]any{"tools": Tools[start:end]}
|
||||
if end < len(Tools) {
|
||||
res["nextCursor"] = fmt.Sprint(end)
|
||||
}
|
||||
answer(res)
|
||||
case "tools/call":
|
||||
var p struct {
|
||||
Name string `json:"name"`
|
||||
Arguments json.RawMessage `json:"arguments"`
|
||||
}
|
||||
_ = json.Unmarshal(msg.Params, &p)
|
||||
s.mu.Lock()
|
||||
s.Calls, s.Args, s.Headers = append(s.Calls, p.Name), append(s.Args, p.Arguments), append(s.Headers, r.Header.Clone())
|
||||
s.mu.Unlock()
|
||||
var result map[string]any
|
||||
switch p.Name {
|
||||
case "echo":
|
||||
var a struct {
|
||||
Text string `json:"text"`
|
||||
}
|
||||
_ = json.Unmarshal(p.Arguments, &a)
|
||||
result = map[string]any{"content": []any{map[string]any{"type": "text", "text": a.Text}}}
|
||||
case "fail":
|
||||
result = map[string]any{"content": []any{map[string]any{"type": "text", "text": "it failed"}}, "isError": true}
|
||||
case "big":
|
||||
result = map[string]any{"content": []any{map[string]any{"type": "text", "text": strings.Repeat("x", 1<<20)}}}
|
||||
default:
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": msg.ID,
|
||||
"error": map[string]any{"code": -32602, "message": "unknown tool " + p.Name}})
|
||||
return
|
||||
}
|
||||
if s.Stream {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
fmt.Fprintf(w, "data: %s\n\n", mustJSON(map[string]any{"jsonrpc": "2.0", "method": "notifications/progress",
|
||||
"params": map[string]any{"progress": 1, "total": 2, "message": "halfway"}}))
|
||||
fmt.Fprintf(w, "event: message\ndata: %s\n\n", mustJSON(map[string]any{"jsonrpc": "2.0", "id": msg.ID, "result": result}))
|
||||
return
|
||||
}
|
||||
answer(result)
|
||||
default:
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"jsonrpc": "2.0", "id": msg.ID,
|
||||
"error": map[string]any{"code": -32601, "message": "method not found"}})
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(v any) string {
|
||||
b, _ := json.Marshal(v)
|
||||
return string(b)
|
||||
}
|
||||
@@ -155,11 +155,11 @@ func fitsOn(m Model, h Hardware, d Demand, p Precision, context int) []Fit {
|
||||
if nodes > 1 && h.Fabric == "" {
|
||||
continue
|
||||
}
|
||||
// The KV heads have to divide too, and they are far fewer than the
|
||||
// attention heads under grouped-query attention: a model with 8 KV
|
||||
// heads cannot be split sixteen ways.
|
||||
if m.KVHeads > 0 && width > m.KVHeads {
|
||||
break
|
||||
// The KV heads are split among the cards, or past their number each
|
||||
// card holds a copy of one: the width divides them or they divide
|
||||
// it. A model with 8 KV heads splits 2, 4, 8 or 16 ways, not 12.
|
||||
if m.KVHeads > 0 && m.KVHeads%width != 0 && width%m.KVHeads != 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
per := GB(d.PerGPU(width))
|
||||
|
||||
@@ -89,13 +89,14 @@ func TestATensorGroupNeverSpansFabrics(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTensorWidthCannotExceedTheKVHeads(t *testing.T) {
|
||||
// Grouped-query attention leaves far fewer KV heads than attention heads,
|
||||
// and an engine refuses a width that cannot divide them. Offering tp=16 for
|
||||
// a model with 8 KV heads is a plan that fails at launch.
|
||||
func TestTensorWidthFitsTheKVHeads(t *testing.T) {
|
||||
// Grouped-query attention leaves far fewer KV heads than attention heads.
|
||||
// An engine splits them among the cards, or past their number gives each
|
||||
// card a copy of one: so a width that neither divides them nor is a
|
||||
// multiple of them fails at launch, and is not offered.
|
||||
for _, f := range Fits(llama70B, []Hardware{h100Node(8, 8, "ib-a")}, 8192, []Precision{Q4GGUF}) {
|
||||
if f.TP > llama70B.KVHeads {
|
||||
t.Errorf("tp=%d exceeds the model's %d KV heads", f.TP, llama70B.KVHeads)
|
||||
if llama70B.KVHeads%f.TP != 0 && f.TP%llama70B.KVHeads != 0 {
|
||||
t.Errorf("tp=%d does not fit the model's %d KV heads", f.TP, llama70B.KVHeads)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,9 +47,20 @@ type hfConfig struct {
|
||||
HeadDim int `json:"head_dim"`
|
||||
MaxPositions int `json:"max_position_embeddings"`
|
||||
ModelType string `json:"model_type"`
|
||||
// The routed experts of a mixture-of-experts model, by whichever name
|
||||
// its family gives them: Qwen's and Mixtral's, DeepSeek's, Llama 4's.
|
||||
NumExperts int `json:"num_experts"`
|
||||
NRoutedExperts int `json:"n_routed_experts"`
|
||||
NumLocalExperts int `json:"num_local_experts"`
|
||||
// TextConfig is where a model that also reads pictures keeps its
|
||||
// language model's figures (Llama 4, Qwen3-VL, Gemma 3).
|
||||
TextConfig *hfConfig `json:"text_config"`
|
||||
}
|
||||
|
||||
func (c hfConfig) toModel(repo string) (Model, error) {
|
||||
if c.NumHiddenLayers == 0 && c.TextConfig != nil {
|
||||
c = *c.TextConfig
|
||||
}
|
||||
if c.NumHiddenLayers == 0 {
|
||||
return Model{}, fmt.Errorf("%s: config.json has no num_hidden_layers; is it a text model?", repo)
|
||||
}
|
||||
@@ -57,6 +68,7 @@ func (c hfConfig) toModel(repo string) (Model, error) {
|
||||
Name: repo[strings.LastIndex(repo, "/")+1:],
|
||||
Layers: c.NumHiddenLayers,
|
||||
MaxContext: c.MaxPositions,
|
||||
Experts: max(c.NumExperts, c.NRoutedExperts, c.NumLocalExperts),
|
||||
}
|
||||
|
||||
// Grouped-query attention: when the KV head count is absent the model has
|
||||
|
||||
@@ -75,3 +75,15 @@ func TestTheModelNameIsTheRepositoryLeaf(t *testing.T) {
|
||||
t.Errorf("Name = %q", m.Name)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelThatReadsPicturesIsSizedByItsLanguageModel(t *testing.T) {
|
||||
m, err := hfConfig{TextConfig: &hfConfig{NumHiddenLayers: 48, NumAttentionHeads: 40, NumKeyValueHeads: 8,
|
||||
HiddenSize: 5120, NumLocalExperts: 16}}.toModel("meta-llama/Llama-4-Scout-17B-16E-Instruct")
|
||||
if err != nil || m.Layers != 48 || m.KVHeads != 8 || m.Experts != 16 {
|
||||
t.Errorf("%+v %v", m, err)
|
||||
}
|
||||
ds, _ := hfConfig{NumHiddenLayers: 61, NumAttentionHeads: 128, HiddenSize: 7168, NRoutedExperts: 256}.toModel("deepseek-ai/DeepSeek-V3")
|
||||
if ds.Experts != 256 {
|
||||
t.Errorf("DeepSeek's experts: %d", ds.Experts)
|
||||
}
|
||||
}
|
||||
@@ -33,6 +33,8 @@ type Shape struct {
|
||||
KVHeads int
|
||||
// Params is the parameter count, for bits per weight; 0 is unknown.
|
||||
Params int64
|
||||
// Experts is a mixture-of-experts model's routed experts; 0 for dense.
|
||||
Experts int
|
||||
}
|
||||
|
||||
// PoolNode is one ready node, as a plan sees it.
|
||||
@@ -96,11 +98,71 @@ type Option struct {
|
||||
FreeNow bool `json:"free_now"`
|
||||
// BitsPerWeight is the artifact's precision, 0 when unknown.
|
||||
BitsPerWeight float64 `json:"bits_per_weight,omitempty"`
|
||||
// What NeedGB is made of, a card's share (or, on the CPU, the node's
|
||||
// RAM): the weights, the context's KV cache, and the engine's own.
|
||||
WeightsGB float64 `json:"weights_gb,omitempty"`
|
||||
ContextGB float64 `json:"context_gb,omitempty"`
|
||||
EngineGB float64 `json:"engine_gb,omitempty"`
|
||||
// ExpertParallel spreads a mixture-of-experts model's experts over the
|
||||
// cards, whole, rather than slicing each across them: vLLM's
|
||||
// --enable-expert-parallel, for a model of experts on more than one card.
|
||||
ExpertParallel bool `json:"expert_parallel,omitempty"`
|
||||
}
|
||||
|
||||
// GPUs is every card one replica takes.
|
||||
func (o Option) GPUs() int { return o.GPUsPerNode * max(o.Nodes, 1) }
|
||||
|
||||
// TotalGB is what one replica takes of every card together, or of the
|
||||
// node's RAM on the CPU.
|
||||
func (o Option) TotalGB() float64 {
|
||||
if o.CPU {
|
||||
return o.NeedGB
|
||||
}
|
||||
return o.NeedGB * float64(o.GPUs())
|
||||
}
|
||||
|
||||
// Fits is whether a card, or the node, holds what it needs.
|
||||
func (o Option) Fits() bool { return o.NeedGB <= o.HaveGB }
|
||||
|
||||
// WithContext is the option for another context: kvPerToken a token's KV
|
||||
// cache, the whole model; the weights and the engine's share stay.
|
||||
func (o Option) WithContext(kvPerToken int64, tokens int) Option {
|
||||
if kvPerToken <= 0 || tokens <= 0 {
|
||||
return o
|
||||
}
|
||||
ctx := GB(int64(tokens) * kvPerToken)
|
||||
if !o.CPU {
|
||||
ctx /= float64(max(o.GPUs(), 1))
|
||||
}
|
||||
o.NeedGB += ctx - o.ContextGB
|
||||
o.ContextGB = ctx
|
||||
return o
|
||||
}
|
||||
|
||||
// MemoryText says what it needs, part by part.
|
||||
func (o Option) MemoryText() string {
|
||||
parts := fmt.Sprintf("weights %.1f + context %.1f + engine %.1f", o.WeightsGB, o.ContextGB, o.EngineGB)
|
||||
if o.WeightsGB == 0 && o.ContextGB == 0 {
|
||||
parts = ""
|
||||
}
|
||||
if o.CPU {
|
||||
s := fmt.Sprintf("%.1f GB of RAM", o.NeedGB)
|
||||
if parts != "" {
|
||||
s += " (" + parts + ")"
|
||||
}
|
||||
return s + fmt.Sprintf(" of %.0f usable", o.HaveGB)
|
||||
}
|
||||
s := fmt.Sprintf("%.1f GB a card", o.NeedGB)
|
||||
if parts != "" {
|
||||
s += " (" + parts + ")"
|
||||
}
|
||||
s += fmt.Sprintf(" of %.0f usable", o.HaveGB)
|
||||
if o.GPUs() > 1 {
|
||||
s += fmt.Sprintf("; %.0f GB on %d cards", o.TotalGB(), o.GPUs())
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Label says where it runs, in a few words.
|
||||
func (o Option) Label() string {
|
||||
if o.CPU {
|
||||
@@ -151,7 +213,8 @@ func ggufOptions(a Artifact, kv int64, nodes []PoolNode, bpw float64) []Option {
|
||||
// The CPU: the whole replica in a node's RAM.
|
||||
total := a.Bytes + kv + ggufOverhead(a.Bytes)
|
||||
cpu := Option{Artifact: a.Name, Format: a.Format, Engine: "llamacpp", CPU: true,
|
||||
NeedGB: GB(total), BitsPerWeight: bpw}
|
||||
NeedGB: GB(total), BitsPerWeight: bpw,
|
||||
WeightsGB: GB(a.Bytes), ContextGB: GB(kv), EngineGB: GB(ggufOverhead(a.Bytes))}
|
||||
for _, n := range nodes {
|
||||
have := float64(n.HostMemGB) * hostShare
|
||||
if n.HostMemGB <= 0 || GB(total) > have {
|
||||
@@ -175,7 +238,9 @@ func ggufOptions(a Artifact, kv int64, nodes []PoolNode, bpw float64) []Option {
|
||||
continue
|
||||
}
|
||||
o := Option{Artifact: a.Name, Format: a.Format, Engine: "llamacpp", Class: c.class,
|
||||
GPUsPerNode: width, Nodes: 1, NeedGB: GB(per), HaveGB: budget, BitsPerWeight: bpw}
|
||||
GPUsPerNode: width, Nodes: 1, NeedGB: GB(per), HaveGB: budget, BitsPerWeight: bpw,
|
||||
WeightsGB: GB(a.Bytes / int64(width)), ContextGB: GB(kv / int64(width)),
|
||||
EngineGB: GB(ggufOverhead(a.Bytes / int64(width)))}
|
||||
for _, fab := range c.fabrics {
|
||||
for _, n := range fab.nodes {
|
||||
if n.GPUs >= width {
|
||||
@@ -198,13 +263,20 @@ func vllmOptions(a Artifact, kv int64, sh Shape, nodes []PoolNode, bpw float64)
|
||||
perCard := func(width int) int64 {
|
||||
return (a.Bytes+kv)/int64(width) + overheadBytes(a.Bytes/int64(width))
|
||||
}
|
||||
// Tensor parallel within a node must divide the KV heads.
|
||||
divides := func(w int) bool { return sh.KVHeads == 0 || (w <= sh.KVHeads && sh.KVHeads%w == 0) }
|
||||
// Tensor parallel splits the KV heads among the cards, or past their
|
||||
// number gives each card a copy of one: so the width divides the heads,
|
||||
// or the heads divide the width. Qwen3-235B has four and runs on eight.
|
||||
divides := func(w int) bool {
|
||||
return sh.KVHeads == 0 || sh.KVHeads%w == 0 || w%sh.KVHeads == 0
|
||||
}
|
||||
for _, c := range classes(nodes) {
|
||||
budget := float64(c.mem) * gpuShare
|
||||
opt := func(perNode, n int, fabric string) Option {
|
||||
cards := int64(perNode * n)
|
||||
return Option{Artifact: a.Name, Format: a.Format, Engine: "vllm", Class: c.class, Fabric: fabric,
|
||||
GPUsPerNode: perNode, Nodes: n, NeedGB: GB(perCard(perNode * n)), HaveGB: budget, BitsPerWeight: bpw}
|
||||
GPUsPerNode: perNode, Nodes: n, NeedGB: GB(perCard(perNode * n)), HaveGB: budget, BitsPerWeight: bpw,
|
||||
WeightsGB: GB(a.Bytes / cards), ContextGB: GB(kv / cards), EngineGB: GB(overheadBytes(a.Bytes / cards)),
|
||||
ExpertParallel: sh.Experts > 1 && perNode*n > 1}
|
||||
}
|
||||
// One node: the narrowest power of two that holds it.
|
||||
single := false
|
||||
@@ -243,7 +315,10 @@ func vllmOptions(a Artifact, kv int64, sh Shape, nodes []PoolNode, bpw float64)
|
||||
if fab.name == "" {
|
||||
continue
|
||||
}
|
||||
for n := 2; n <= len(fab.nodes); n *= 2 {
|
||||
// Any number of nodes: pipeline stages need not be a power of
|
||||
// two, and a third node is what a model just too big for two
|
||||
// needs.
|
||||
for n := 2; n <= len(fab.nodes); n++ {
|
||||
if GB(perCard(w*n)) > budget {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -123,3 +123,73 @@ func TestOnTheCPUAMiddlingQuantisationIsBestAndOnACardEightBits(t *testing.T) {
|
||||
t.Errorf("GPU: %s first", o[0].Artifact)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of experts spread over more than one card has them spread whole
|
||||
// (expert parallelism); on one card there is nothing to spread.
|
||||
func TestAModelOfExpertsSpreadsItsExpertsOverItsCards(t *testing.T) {
|
||||
nodes := []PoolNode{{ID: "a", Class: "h100-80g", GPUMemGB: 80, GPUs: 8, FreeGPUs: 8}}
|
||||
moe := Shape{KVBytesPerToken: 94 * 4 * 256 * 2, KVHeads: 4, Params: 235_000_000_000, Experts: 128}
|
||||
opts := Options([]Artifact{{Name: "original", Format: "safetensors", Bytes: 470 * gib, Quant: "bf16"}}, moe, nodes, 8192)
|
||||
if len(opts) == 0 || opts[0].GPUsPerNode < 2 || !opts[0].ExpertParallel {
|
||||
t.Fatalf("options %+v", opts)
|
||||
}
|
||||
small := Options([]Artifact{{Name: "original", Format: "safetensors", Bytes: 30 * gib, Quant: "bf16"}},
|
||||
Shape{KVHeads: 4, Experts: 128}, nodes, 8192)
|
||||
if len(small) == 0 || small[0].GPUsPerNode != 1 || small[0].ExpertParallel {
|
||||
t.Errorf("one card: %+v", small)
|
||||
}
|
||||
dense := Options([]Artifact{{Name: "original", Format: "safetensors", Bytes: 140 * gib, Quant: "bf16"}},
|
||||
Shape{KVHeads: 8}, nodes, 8192)
|
||||
if len(dense) == 0 || dense[0].ExpertParallel {
|
||||
t.Errorf("a dense model: %+v", dense)
|
||||
}
|
||||
}
|
||||
|
||||
// Too big for two nodes, it is offered on three, not only on four.
|
||||
func TestAModelTooBigForTwoNodesIsOfferedOnThree(t *testing.T) {
|
||||
var nodes []PoolNode
|
||||
for _, id := range []string{"a", "b", "c", "d"} {
|
||||
nodes = append(nodes, PoolNode{ID: id, Class: "l40s-48g", GPUMemGB: 48, GPUs: 8, FreeGPUs: 8, Fabric: "ib-0"})
|
||||
}
|
||||
// About 900 GB: more than two nodes' 16 cards hold, less than three's 24.
|
||||
opts := Options([]Artifact{{Name: "original", Format: "safetensors", Bytes: 900 * gib, Quant: "bf16"}},
|
||||
Shape{KVHeads: 8}, nodes, 4096)
|
||||
if len(opts) == 0 || opts[0].Nodes != 3 || opts[0].GPUsPerNode != 8 || opts[0].Fabric != "ib-0" {
|
||||
t.Fatalf("options %+v", labels(opts))
|
||||
}
|
||||
}
|
||||
|
||||
// Each way says what it needs part by part, and for another context.
|
||||
func TestAnOptionSaysWhatItNeedsOfEachCardAndOfAll(t *testing.T) {
|
||||
nodes := []PoolNode{{ID: "a", Class: "h100-80g", GPUMemGB: 80, GPUs: 8, FreeGPUs: 8}}
|
||||
kvPerToken := int64(80 * 8 * 128 * 2 * 2) // a 70B: 320 KB a token
|
||||
opts := Options([]Artifact{{Name: "original", Format: "safetensors", Bytes: 140 * gib, Quant: "bf16"}},
|
||||
Shape{KVBytesPerToken: kvPerToken, KVHeads: 8}, nodes, 8192)
|
||||
o := opts[0]
|
||||
if o.GPUsPerNode != 4 {
|
||||
t.Fatalf("expected four cards: %+v", o)
|
||||
}
|
||||
sum := o.WeightsGB + o.ContextGB + o.EngineGB
|
||||
if d := sum - o.NeedGB; d > 0.01 || d < -0.01 || o.WeightsGB != 35 {
|
||||
t.Errorf("parts %.2f+%.2f+%.2f do not make %.2f", o.WeightsGB, o.ContextGB, o.EngineGB, o.NeedGB)
|
||||
}
|
||||
text := o.MemoryText()
|
||||
for _, want := range []string{"GB a card", "weights 35.0", "context 0.6", "of 72 usable", "on 4 cards"} {
|
||||
if !strings.Contains(text, want) {
|
||||
t.Errorf("%q lacks %q", text, want)
|
||||
}
|
||||
}
|
||||
// Sixteen times the context: the weights stay, the context grows, and
|
||||
// it no longer fits.
|
||||
long := o.WithContext(kvPerToken, 8192*16)
|
||||
if long.WeightsGB != o.WeightsGB || long.ContextGB < o.ContextGB*15.9 {
|
||||
t.Errorf("for a longer context: %+v", long)
|
||||
}
|
||||
if !o.Fits() {
|
||||
t.Error("the planned way does not fit")
|
||||
}
|
||||
huge := o.WithContext(kvPerToken, 8192*256)
|
||||
if huge.Fits() {
|
||||
t.Errorf("a 2M-token context fits four cards: %s", huge.MemoryText())
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,10 @@ type Model struct {
|
||||
HeadDim int
|
||||
// MaxContext is the model's trained context length.
|
||||
MaxContext int
|
||||
// Experts is a mixture-of-experts model's routed experts; 0 for a dense
|
||||
// one. Its experts are spread over the cards (expert parallelism) rather
|
||||
// than each sliced across them.
|
||||
Experts int
|
||||
}
|
||||
|
||||
// Precision is how many bytes one parameter weighs.
|
||||
|
||||
@@ -53,6 +53,11 @@ type Machine struct {
|
||||
Remote onboard.Remote
|
||||
// Local is this machine, where sudo is not needed (setup runs as root).
|
||||
Local bool
|
||||
// ServesMB is memory it needs beyond its roles' for the models it will
|
||||
// serve itself, and Serves names them. Short of it is a failure, not a
|
||||
// warning: the install would finish and then have nowhere to run them.
|
||||
ServesMB int
|
||||
Serves string
|
||||
}
|
||||
|
||||
// Level is how bad a finding is.
|
||||
@@ -269,7 +274,17 @@ func Evaluate(m Machine, facts map[string]string, at time.Time) []Finding {
|
||||
}
|
||||
}
|
||||
room("cpus", cpus, need.CPUs, "CPUs")
|
||||
room("memory", mem, need.MemoryMB, "MB")
|
||||
if m.ServesMB > 0 {
|
||||
want := need.MemoryMB + m.ServesMB
|
||||
d := fmt.Sprintf("%d MB (needs %d with %s)", mem, want, m.Serves)
|
||||
if mem >= want {
|
||||
f("memory", OK, d, "")
|
||||
} else {
|
||||
f("memory", Fail, d, "give it more memory, or turn off "+m.Serves+" or This machine as a node")
|
||||
}
|
||||
} else {
|
||||
room("memory", mem, need.MemoryMB, "MB")
|
||||
}
|
||||
room("disk", disk, need.DiskGB, "GB free")
|
||||
|
||||
// Time. Leases and etcd's elections go wrong on clocks that disagree.
|
||||
|
||||
@@ -107,3 +107,16 @@ func TestAgainstRealMachines(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The guide's first recording: 8 GB, a control plane serving the starter and
|
||||
// both judges, every step done and then no room for the starter. Now said at
|
||||
// the checks, as a failure, before anything is installed.
|
||||
func TestAMachineTooSmallForTheModelsItServesFailsTheCheck(t *testing.T) {
|
||||
m := Machine{Name: "llm-1", Roles: []Role{RoleControl, RoleWitness}, ServesMB: 7 * 1024, Serves: "the starter model and 2 judges"}
|
||||
if l, d := levelOf(Evaluate(m, facts(map[string]string{"mem_mb": "7941"}), time.Now()), "memory"); l != Fail || !strings.Contains(d, "2 judges") {
|
||||
t.Errorf("8 GB: %v %q", l, d)
|
||||
}
|
||||
if l, d := levelOf(Evaluate(m, facts(map[string]string{"mem_mb": "11961"}), time.Now()), "memory"); l != OK {
|
||||
t.Errorf("12 GB: %v %q", l, d)
|
||||
}
|
||||
}
|
||||
@@ -72,6 +72,10 @@ type cacheState struct {
|
||||
// project is whose answer it is, and ttl how long it is reused.
|
||||
project string
|
||||
ttl time.Duration
|
||||
// group and vec place it among answers to similar questions, for a
|
||||
// project that reuses those; empty otherwise.
|
||||
group string
|
||||
vec []float32
|
||||
}
|
||||
|
||||
// cachedAnswer is one reusable answer.
|
||||
@@ -89,16 +93,23 @@ type cachedAnswer struct {
|
||||
// project's TTL shortens the life of what it already has.
|
||||
expires time.Time
|
||||
size int64
|
||||
// group is the conversation the question was asked in, everything of
|
||||
// the request but its last user message; vec is that message embedded,
|
||||
// of unit length. Empty for an answer reused only by the same request.
|
||||
group string
|
||||
vec []float32
|
||||
}
|
||||
|
||||
// answerCache is a least-recently-used cache of answers, bounded in bytes.
|
||||
type answerCache struct {
|
||||
maxBytes, maxEntry int64
|
||||
|
||||
mu sync.Mutex
|
||||
lru *list.List // of *cachedAnswer, most recently used at the front
|
||||
by map[string]*list.Element
|
||||
bytes int64
|
||||
mu sync.Mutex
|
||||
lru *list.List // of *cachedAnswer, most recently used at the front
|
||||
by map[string]*list.Element
|
||||
// groups are the answers to similar questions, by conversation.
|
||||
groups map[string]map[*list.Element]struct{}
|
||||
bytes int64
|
||||
// swept is when expired answers were last cleared out.
|
||||
swept time.Time
|
||||
// size reports bytes and answers held, after each change.
|
||||
@@ -117,7 +128,7 @@ func newAnswerCache(maxBytes, maxEntry int64, size func(int64, int)) *answerCach
|
||||
size = func(int64, int) {}
|
||||
}
|
||||
return &answerCache{maxBytes: maxBytes, maxEntry: maxEntry, lru: list.New(),
|
||||
by: map[string]*list.Element{}, size: size}
|
||||
by: map[string]*list.Element{}, groups: map[string]map[*list.Element]struct{}{}, size: size}
|
||||
}
|
||||
|
||||
// get is the answer kept under key, when it is younger than ttl and has
|
||||
@@ -144,7 +155,8 @@ func (c *answerCache) get(key string, ttl time.Duration, now time.Time) *cachedA
|
||||
func (c *answerCache) put(a *cachedAnswer, now time.Time) bool {
|
||||
// What the entry costs besides its body: the key twice, the names, and
|
||||
// the bookkeeping, roughly.
|
||||
a.size = int64(len(a.body) + 2*len(a.key) + len(a.project) + len(a.model) + len(a.path) + 160)
|
||||
a.size = int64(len(a.body) + 2*len(a.key) + len(a.project) + len(a.model) + len(a.path) + 160 +
|
||||
len(a.group) + 4*len(a.vec))
|
||||
if a.size > c.maxEntry {
|
||||
return false
|
||||
}
|
||||
@@ -168,7 +180,14 @@ func (c *answerCache) put(a *cachedAnswer, now time.Time) bool {
|
||||
for c.bytes+a.size > c.maxBytes && c.lru.Len() > 0 {
|
||||
c.remove(c.lru.Back())
|
||||
}
|
||||
c.by[a.key] = c.lru.PushFront(a)
|
||||
el := c.lru.PushFront(a)
|
||||
c.by[a.key] = el
|
||||
if a.group != "" {
|
||||
if c.groups[a.group] == nil {
|
||||
c.groups[a.group] = map[*list.Element]struct{}{}
|
||||
}
|
||||
c.groups[a.group][el] = struct{}{}
|
||||
}
|
||||
c.bytes += a.size
|
||||
c.size(c.bytes, len(c.by))
|
||||
return true
|
||||
@@ -178,9 +197,38 @@ func (c *answerCache) remove(el *list.Element) {
|
||||
a := el.Value.(*cachedAnswer)
|
||||
c.lru.Remove(el)
|
||||
delete(c.by, a.key)
|
||||
if g := c.groups[a.group]; g != nil {
|
||||
delete(g, el)
|
||||
if len(g) == 0 {
|
||||
delete(c.groups, a.group)
|
||||
}
|
||||
}
|
||||
c.bytes -= a.size
|
||||
}
|
||||
|
||||
// nearest is the answer in group whose question is most like vec, when
|
||||
// that reaches least and the answer is younger than ttl, and how alike.
|
||||
func (c *answerCache) nearest(group string, vec []float32, least float64, ttl time.Duration, now time.Time) (*cachedAnswer, float64) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
var best *list.Element
|
||||
bestSim := least
|
||||
for el := range c.groups[group] {
|
||||
a := el.Value.(*cachedAnswer)
|
||||
if now.After(a.expires) || now.Sub(a.at) >= ttl || len(a.vec) != len(vec) {
|
||||
continue
|
||||
}
|
||||
if s := dot(a.vec, vec); s >= bestSim {
|
||||
best, bestSim = el, s
|
||||
}
|
||||
}
|
||||
if best == nil {
|
||||
return nil, 0
|
||||
}
|
||||
c.lru.MoveToFront(best)
|
||||
return best.Value.(*cachedAnswer), bestSim
|
||||
}
|
||||
|
||||
// answerCacheOf is a project's setting, read at most every requestLogTTL
|
||||
// with its other settings. Unreadable, it reuses nothing: asking the model
|
||||
// again is the safe way to be wrong.
|
||||
@@ -238,6 +286,15 @@ func (r *Router) cacheLookup(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
}
|
||||
result = "miss"
|
||||
req.cache = cacheState{key: key, project: req.ProjectID, ttl: set.TTL}
|
||||
if set.Similar > 0 {
|
||||
if a, sim := r.similarLookup(ctx, req, model, set); a != nil {
|
||||
result = "similar"
|
||||
w.Header().Set("Age", strconv.Itoa(int(now.Sub(a.at).Seconds())))
|
||||
w.Header().Set(similarityHeader, strconv.FormatFloat(sim, 'f', 3, 64))
|
||||
req.cache = cacheState{}
|
||||
return a
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -288,15 +345,29 @@ var answerFields = []string{"model", "stream", "stream_options", "user", "metada
|
||||
// the answer taken out and its keys sorted. The project is spelled out
|
||||
// rather than only hashed, so no answer is ever looked up across projects.
|
||||
func answerKey(project, path, model string, body []byte) (string, bool) {
|
||||
m, ok := decodeRequest(body)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
return keyOf(project, path, model, m)
|
||||
}
|
||||
|
||||
// decodeRequest is a request as a map, its numbers kept as written and
|
||||
// what does not change the answer taken out.
|
||||
func decodeRequest(body []byte) (map[string]any, bool) {
|
||||
d := json.NewDecoder(bytes.NewReader(body))
|
||||
d.UseNumber()
|
||||
var m map[string]any
|
||||
if err := d.Decode(&m); err != nil || m == nil {
|
||||
return "", false
|
||||
return nil, false
|
||||
}
|
||||
for _, f := range answerFields {
|
||||
delete(m, f)
|
||||
}
|
||||
return m, true
|
||||
}
|
||||
|
||||
func keyOf(project, path, model string, m map[string]any) (string, bool) {
|
||||
// encoding/json writes map keys sorted, at every depth.
|
||||
norm, err := json.Marshal(canonicalNumbers(m))
|
||||
if err != nil {
|
||||
@@ -420,7 +491,8 @@ func (r *Router) cacheKeep(req *parsedRequest, res dispatchResult, status int, c
|
||||
}
|
||||
now := time.Now()
|
||||
r.answers.put(&cachedAnswer{key: req.cache.key, project: req.cache.project, model: model,
|
||||
path: req.Path, body: body, at: now, expires: now.Add(req.cache.ttl)}, now)
|
||||
path: req.Path, body: body, at: now, expires: now.Add(req.cache.ttl),
|
||||
group: req.cache.group, vec: req.cache.vec}, now)
|
||||
}
|
||||
|
||||
// serveCached answers a request from the cache, as the model would have:
|
||||
|
||||
@@ -27,7 +27,7 @@ import (
|
||||
func cachingRouter(t *testing.T, engineURL string, anyAnswer bool) (*Router, string) {
|
||||
t.Helper()
|
||||
rt, secret := pricedRouter(t, engineURL)
|
||||
if err := rt.store.SetAnswerCache(t.Context(), mustProjectOf(t, rt, "key-1"), time.Hour, anyAnswer); err != nil {
|
||||
if err := rt.store.SetAnswerCache(t.Context(), mustProjectOf(t, rt, "key-1"), store.AnswerCache{TTL: time.Hour, Any: anyAnswer}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rt, secret
|
||||
@@ -135,7 +135,7 @@ func TestAProjectThatReusesAnswersAnswersTheSameRequestWithoutTheModel(t *testin
|
||||
}
|
||||
|
||||
// Turned off, nothing is reused, and nothing is said about it.
|
||||
_ = rt.store.SetAnswerCache(t.Context(), mustProjectOf(t, rt, "key-1"), 0, false)
|
||||
_ = rt.store.SetAnswerCache(t.Context(), mustProjectOf(t, rt, "key-1"), store.AnswerCache{})
|
||||
rt.reqLogs.by = nil
|
||||
calls := e.calls.Load()
|
||||
if rec := post(t, h, ask(0, "what is 2+2"), auth("off")); rec.Header().Get(cacheHeader) != "" || e.calls.Load() != calls+1 {
|
||||
@@ -164,7 +164,7 @@ func TestKeysOfOneProjectNeverSeeAnotherProjectsAnswers(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = rt.store.SetAnswerCache(t.Context(), other, time.Hour, true)
|
||||
_ = rt.store.SetAnswerCache(t.Context(), other, store.AnswerCache{TTL: time.Hour, Any: true})
|
||||
secret2 := store.KeyPrefix + strings.Repeat("b", 43)
|
||||
sum := sha256.Sum256([]byte(secret2))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "other",
|
||||
|
||||
@@ -6,10 +6,12 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/account"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/directory"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -24,6 +26,9 @@ type ldapDirectory struct {
|
||||
every time.Duration
|
||||
label string
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
last *adminapi.DirectorySyncView
|
||||
}
|
||||
|
||||
func newLDAPDirectory(cfg config.LDAP, st *storepkg.DB, log *slog.Logger) (*ldapDirectory, error) {
|
||||
@@ -92,16 +97,26 @@ func (d *ldapDirectory) sync(ctx context.Context, leader func() bool) {
|
||||
func (d *ldapDirectory) syncOnce(ctx context.Context) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
|
||||
defer cancel()
|
||||
last := &adminapi.DirectorySyncView{At: time.Now()}
|
||||
defer func() {
|
||||
d.mu.Lock()
|
||||
d.last = last
|
||||
d.mu.Unlock()
|
||||
}()
|
||||
people, err := d.dir.People(ctx, d.keeper.Mapping)
|
||||
if err != nil {
|
||||
d.log.Warn("could not read the directory; accounts left as they are", "err", err)
|
||||
last.Error = "could not read the directory: " + err.Error()
|
||||
return
|
||||
}
|
||||
last.People = len(people)
|
||||
rep, err := d.keeper.Sync(ctx, people)
|
||||
if err != nil {
|
||||
d.log.Error("the directory sync stopped", "err", err, "people", len(people))
|
||||
last.Error = err.Error()
|
||||
return
|
||||
}
|
||||
last.Accounts, last.Suspended, last.Reinstated, last.RolesChanged = rep.Seen, rep.Suspended, rep.Reinstated, rep.Updated
|
||||
for _, where := range rep.Unmapped {
|
||||
d.log.Warn("auth.ldap.memberships maps a group to a place the tree does not have", "place", where)
|
||||
}
|
||||
@@ -110,3 +125,54 @@ func (d *ldapDirectory) syncOnce(ctx context.Context) {
|
||||
"suspended", rep.Suspended, "reinstated", rep.Reinstated, "roles_changed", rep.Updated)
|
||||
}
|
||||
}
|
||||
|
||||
// lastSync is how the leader's last read of the directory went; nil before
|
||||
// the first, and on a standby.
|
||||
func (d *ldapDirectory) lastSync() *adminapi.DirectorySyncView {
|
||||
d.mu.Lock()
|
||||
defer d.mu.Unlock()
|
||||
return d.last
|
||||
}
|
||||
|
||||
// AuthStatus is how people sign in on this gateway, for the admin API.
|
||||
func (r *Router) AuthStatus() adminapi.AuthView {
|
||||
v := adminapi.AuthView{SCIM: r.scim.TokenFile != "", BaseURL: r.mail.BaseURL}
|
||||
if r.accounts != nil {
|
||||
v.SignupOpen = r.accounts.SignupOpen()
|
||||
}
|
||||
if o := r.oidc; o.Enabled() {
|
||||
v.OIDC = &adminapi.OIDCView{Issuer: o.Issuer, ClientID: o.ClientID, Label: o.Label,
|
||||
CreateAccounts: o.CreateAccounts, NoPasswords: o.NoPasswords}
|
||||
}
|
||||
if l := r.ldap; l.Enabled() {
|
||||
v.LDAP = &adminapi.LDAPView{URLs: l.URLs, BaseDN: l.BaseDN, BindDN: l.BindDN, Label: l.Label, Off: r.ldapOff}
|
||||
if r.directory != nil {
|
||||
v.LDAP.LastSync = r.directory.lastSync()
|
||||
}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// TestDirectory tries an auth.ldap block, given as YAML, before it is kept:
|
||||
// checked as the settings would be, then each step against the directory.
|
||||
func (r *Router) TestDirectory(ctx context.Context, settings, password string) ([]adminapi.CheckView, error) {
|
||||
l, err := config.ParseLDAP([]byte(settings))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
dir, err := directory.NewLDAP(l)
|
||||
if err != nil {
|
||||
return []adminapi.CheckView{{Step: "certificate", Detail: err.Error()}}, nil
|
||||
}
|
||||
if password != "" {
|
||||
dir.WithPassword(password)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, time.Minute)
|
||||
defer cancel()
|
||||
checks := dir.Check(ctx, directory.Mapping{Roles: l.Roles, Memberships: l.Memberships})
|
||||
out := make([]adminapi.CheckView, len(checks))
|
||||
for i, c := range checks {
|
||||
out[i] = adminapi.CheckView(c)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
// auth.ldap is the cluster's: a CA file one machine lacks must not keep its
|
||||
// control plane from starting, where nothing could put the setting right.
|
||||
func TestADirectoryThatCannotBeUsedHereLeavesTheGatewayUp(t *testing.T) {
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: "http://127.0.0.1:1", Models: []string{"m"}})
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.LDAP = config.LDAP{URLs: []string{"ldaps://dc1"}, CAFile: filepath.Join(t.TempDir(), "missing.pem"),
|
||||
BindDN: "CN=svc", BindPasswordFile: "/nonexistent", BaseDN: "DC=corp"}
|
||||
r := newTestRouter(t, cfg)
|
||||
if r.directory != nil {
|
||||
t.Fatal("a directory was set up from a CA file that is not there")
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
r.Handler().ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/login", nil))
|
||||
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `type="email"`) {
|
||||
t.Errorf("/login: %d", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// Settings tried are checked as kept ones are: a plaintext server is
|
||||
// refused before anything is sent to it.
|
||||
func TestDirectorySettingsTriedAreCheckedFirst(t *testing.T) {
|
||||
r := &Router{}
|
||||
if _, err := r.TestDirectory(t.Context(), "urls: [ldap://dc1]\nbind_dn: CN=svc\nbind_password_file: store:auth/ldap-bind\nbase_dn: DC=corp\n", "pw"); err == nil ||
|
||||
!strings.Contains(err.Error(), "plaintext") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
checks, err := r.TestDirectory(t.Context(), "urls: [ldaps://dc1]\nca: \"-----BEGIN CERTIFICATE-----\\nAAAA\\n-----END CERTIFICATE-----\"\n"+
|
||||
"bind_dn: CN=svc\nbind_password_file: store:auth/ldap-bind\nbase_dn: DC=corp\n", "pw")
|
||||
if err != nil || len(checks) != 1 || checks[0].Step != "certificate" || checks[0].OK {
|
||||
t.Errorf("a broken CA: %+v %v", checks, err)
|
||||
}
|
||||
}
|
||||
Loaded 100 of 199 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user