Compare commits
78
Commits
presets-ui
...
v0.15.0
No files matched your search
@@ -34,7 +34,11 @@ jobs:
|
||||
run: go vet ./...
|
||||
|
||||
- name: Test with the race detector
|
||||
run: go test -race ./...
|
||||
# Under the job's limit, so a package that hangs panics with the
|
||||
# test it was in and where every goroutine waited, rather than the
|
||||
# job ending at its deadline with nothing said: from 2026-10-15 every
|
||||
# run did, printing nothing after internal/release.
|
||||
run: go test -race -timeout 12m ./...
|
||||
|
||||
- name: Build
|
||||
run: go build ./...
|
||||
|
||||
@@ -9,3 +9,5 @@ coverage.out
|
||||
/terraform-provider-llmgw/terraform-provider-llmgw
|
||||
/terraform-provider-llmgw/examples/.terraform*
|
||||
/terraform-provider-llmgw/examples/terraform.tfstate*
|
||||
# What a development server run from the checkout writes beside it.
|
||||
/secrets/
|
||||
+375
-4
@@ -15,6 +15,52 @@ Conventions this codebase keeps, which every item here must keep too:
|
||||
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
|
||||
cut and nothing is deployed to the PoC cluster without being told to.
|
||||
|
||||
## Open, in order (2026-10-20)
|
||||
|
||||
What is left, most needed first. The detail is in the item each names.
|
||||
|
||||
**Now: what goes wrong where it is installed.**
|
||||
1. ~~**J8**~~: done 2026-10-20. `ctl guard judge check`, and init times the judge. Left of **J3**: its load test
|
||||
against judges on the control plane, which was stopped half way. Judges on a 4-core node of their own refused
|
||||
0/60 at 2 at once, 13/60 at 4 and 42/60 at 10.
|
||||
2. ~~**I2 P2**~~: done 2026-10-20: the guardrails both ways, a rate per caller and server, a breaker. A portal page
|
||||
for the servers is left, with P3 (stdio, per-person OAuth).
|
||||
3. **K**: fixes from the fabric test the operator runs with docs/large-models.md.
|
||||
|
||||
**Soon: the customer's experience, and HA.**
|
||||
4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
|
||||
forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
|
||||
5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M)
|
||||
6. **F5**: large imports limited by the embedding engine's slots. (M)
|
||||
|
||||
**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's.
|
||||
|
||||
**Governance.**
|
||||
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||
|
||||
**When there is room.**
|
||||
9. **I1**: answers to similar questions from the cache. (M)
|
||||
10. **H1**: tried on the PoC.
|
||||
11. **C4**: Parquet, and a scheduled push for usage export.
|
||||
|
||||
**When needed.**
|
||||
12. **H5**: licences in video and audio. 13. **H3**: replicas shared, the catalog side. 14. **J7**: nginx in front
|
||||
of several control planes. 15. **J6**: rate limits per process (a note).
|
||||
|
||||
**Done since 2026-10-16:**
|
||||
- J1, J5, I3, I4, J2.
|
||||
- Active Directory and Entra ID, tried against a Samba DC, with the console's Sign-in screen.
|
||||
- J3's code.
|
||||
- K's code and docs/large-models.md.
|
||||
- I2 P1 and the cluster's own read-only MCP server (llmgw:admin).
|
||||
- The node console's Overview; a node's page showing its version and the models on each GPU.
|
||||
|
||||
**Not code, the operator's to decide or do:**
|
||||
- Where Alertmanager sends (its receiver is "none"): with a judge down, nobody hears.
|
||||
- The fabric test for K.
|
||||
- The PoC's control planes run a dev build of main (v0.14.9-11-g4eb49d0); its nodes run v0.14.8.
|
||||
|
||||
---
|
||||
|
||||
## A. Security — do first
|
||||
@@ -1385,7 +1431,7 @@ gate, so no token is on the node.
|
||||
- Done when: SF3D runs on cuda01 with nothing installed into the engine's
|
||||
venv by hand (the hand install of 2026-10-10 removed first).
|
||||
|
||||
### H5. Licence on every output (S)
|
||||
### H5. Licence on every output (S) — done 2026-10-14: the whole licence in GLB asset.extras and PNG iTXt; video and audio not yet
|
||||
- `License` gains `id` (SPDX where there is one, else a lowercase name:
|
||||
stability-community, tencent-hunyuan-community), `commercial` (unset is
|
||||
not said), `excluded_regions` (ISO 3166 codes, `EU` for the union),
|
||||
@@ -1398,7 +1444,7 @@ gate, so no token is on the node.
|
||||
audio formats in their metadata where ffmpeg is at hand on the router,
|
||||
otherwise beside it in the job's answer only — said in the docs.
|
||||
|
||||
### H6. Licence choices (S) — the other session's (llmgw-b7), at the user's request
|
||||
### H6. Licence choices (S) — done 2026-10-11 (4bbac08), live on the PoC
|
||||
The user asked (2026-10-11) for the choice to be a person's: on their
|
||||
profile they pick the licences they will not use, and their keys and
|
||||
portal session neither see those models in `/v1/models` nor may call
|
||||
@@ -1407,16 +1453,341 @@ them (403, saying which licence). Matched on `license.id`, and on
|
||||
store table is migration 0046. A site-wide or team-wide default can sit
|
||||
on top of it later.
|
||||
|
||||
### H7. A job worker per model, not one for all (S)
|
||||
### H7. A job worker per model, not one for all (S) — done 2026-10-14: one job per model at a time, eight in all
|
||||
The worker runs one job at a time: a job waiting for the card holds up a
|
||||
job for another model that could run now (seen on the PoC, 2026-10-10).
|
||||
Jobs are claimed and run concurrently up to a limit per model (or per
|
||||
environment, after H3) and a cluster-wide cap, with the lease and
|
||||
leader-only rules of G1 kept.
|
||||
|
||||
### H8. Presets that add themselves (S) — done 2026-10-13 (v0.14.5)
|
||||
flux-schnell and triposr come with llmgw (`ctl model add <preset>`), and
|
||||
the leader adds them on demand where a node fits one, the cluster installs
|
||||
ComfyUI and the catalog makes nothing of that kind (`presets.auto`); a
|
||||
preset removed is declined and not added back. The cluster page and the
|
||||
console show and add them.
|
||||
|
||||
### H9. Small things learned on the PoC (2026-10-14)
|
||||
- Done: `albedo_match` on sf3d (its delighting made dark objects light);
|
||||
`on_demand.evict_after_idle` / `min_lifetime` as live settings; an
|
||||
upgrade that leaves nodes behind says which; `ctl model serve` serves an
|
||||
embedding model for embedding; refreshes wait while a form is filled in.
|
||||
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
|
||||
so a client needs no server of its own (asked by the game team); the
|
||||
shared ComfyUI replica (H3's placement side).
|
||||
- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
|
||||
is and its settings; three machines each run the control plane, with a
|
||||
floating address (VRRP id from the address); a lost store primary costs no
|
||||
request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
|
||||
ttl 20, a standby leaves the lease alone a round after the store returns);
|
||||
a new leader waits for its judge within the budget; a judge is asked again
|
||||
on another machine at half its budget; init sizes the judge (2 GB, no
|
||||
prompt cache, two cores, one judge on up to four cores on one machine).
|
||||
|
||||
## I. What other gateways do that llmgw does not (written 2026-10-16)
|
||||
|
||||
From a comparison with Bifrost (maximhq/bifrost, Apache 2.0, Go), with
|
||||
LiteLLM and Portkey in mind. Bifrost routes to 23+ cloud providers and runs
|
||||
no model; llmgw runs the models, so most of Bifrost is either in llmgw
|
||||
already (keys, budgets, fallbacks, guardrails, tracing, SSO, HA — the last
|
||||
three paid in Bifrost) or does not apply. These four are worth taking for
|
||||
a site that runs its own models. Left until a customer asks: native
|
||||
Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
|
||||
routing by what a request holds. Not taken: a realtime (WebSocket audio)
|
||||
API, prompt management, an `npx` install.
|
||||
|
||||
### I1. Answers to similar questions, not only the same one (M) — done 2026-10-25
|
||||
`keys cache --similar`, `router.cache.embedding_model`, `internal/router/similar.go`. Changed
|
||||
from the plan: the vectors are in the gateway's memory beside the exact answers, not a pgvector
|
||||
table. The exact cache keeps answer text out of the store on purpose (data.md), and a table of
|
||||
answers would undo that; it also works on SQLite. The usage row marks a similar hit as a hit
|
||||
(no column of its own). Open: the 40-rewordings and eval-suite runs on the PoC.
|
||||
The original item follows.
|
||||
The answer cache (`internal/router/cache.go`, C17) answers a request whose
|
||||
body is the same byte for byte. A help desk asked "how do I reset my
|
||||
password" in forty wordings asks the model forty times.
|
||||
- Per project, off by default, beside the exact cache: `llmgw keys cache
|
||||
--project P --similar 0.95` (the cosine similarity a question must reach;
|
||||
higher is stricter). The exact cache is asked first.
|
||||
- The question embedded by a model the operator names
|
||||
(`router.cache.embedding_model`, an embedding model in the catalog), into a
|
||||
pgvector table per project with the answer, the model and the TTL; the
|
||||
store's halfvec and text index work from F applies. SQLite stores keep the
|
||||
exact cache only.
|
||||
- What counts as the question: the last user turn, with the system prompt
|
||||
and the earlier turns hashed and required to match exactly — a similar
|
||||
question in another conversation is not the same question. Tools,
|
||||
`response_format` and the sampling settings also exact.
|
||||
- Said: `x-llmgw-cache: similar` with the similarity, and the usage row
|
||||
says so; guardrails judge the request before a hit, as for the exact
|
||||
cache. A project with `no_store` gets neither.
|
||||
- Done when: forty rewordings of one question on the PoC cost one model call
|
||||
and 39 lookups, and an eval suite run with the cache on scores within a
|
||||
point of the run with it off.
|
||||
|
||||
### I2. MCP gateway: the customer's tool servers behind llmgw (L)
|
||||
P1 done 2026-10-20: HTTP servers registered (`ctl mcp`, `/api/v1/mcp/servers`,
|
||||
store `mcp_servers`), tools as `server.tool` in `/mcp`, `allowed_tools`
|
||||
grants in the tree (deny by default), credentials in the store or by
|
||||
reference, the caller named in `X-Llmgw-*` headers, time and size bounds,
|
||||
progress passed on, audit (Source `mcp`) and metrics. Left: P2, the
|
||||
guardrails both ways, a per-tool rate limit, a breaker, and a portal page;
|
||||
P3, stdio servers and per-person OAuth. The "done when" with Claude Code
|
||||
against a real server is still to try.
|
||||
`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
|
||||
the file server, a database — each an MCP server, each today configured
|
||||
in every client with its own credentials and nobody seeing what was called.
|
||||
- Servers registered once, by an administrator: `llmgw ctl mcp add jira
|
||||
--url https://… --header-file /etc/llmgw/secrets/jira` (HTTP transport;
|
||||
stdio servers run on the control plane under their own user, like an
|
||||
engine). Their credentials stay on the control plane, never in a client.
|
||||
- `/mcp` lists every server's tools, prefixed by its name (`jira.search`),
|
||||
filtered by what the key may use: an allowlist per org, team, project or
|
||||
key, by server or by tool, in the tenancy tree as model allowlists are.
|
||||
- Each call goes to the audit log — who, which tool, the arguments' size,
|
||||
the outcome and how long — and through the guardrails both ways (a tool's
|
||||
answer is prompt input). A per-tool rate limit.
|
||||
- Later, if it earns it: Bifrost's "code mode" (the tools as one typed API
|
||||
the model writes code against), which cuts the tokens of a long tool list.
|
||||
- Done when: Claude Code given only llmgw's `/mcp` reaches a test MCP server
|
||||
through it, a key without the grant does not see the server's tools, and
|
||||
every call is in `ctl audit`.
|
||||
|
||||
### I3. What llmgw adds to a request, measured (S) — done 2026-10-16
|
||||
`llmgw ctl bench overhead` (`internal/cli/bench_overhead.go`): an instant
|
||||
engine and the gateway in one process, four modes (direct, bare, auth, auth
|
||||
and guard patterns) at rising rates. On an Apple M5 the gateway added
|
||||
0.05–0.42 ms at p50 and at most 0.49 ms at p99, in the README.
|
||||
`TestTheGatewaysOverheadIsMeasured` runs a second of it in CI with no bound
|
||||
asserted. Left: the CPU figure is rough with everything in one process; a
|
||||
gateway in a process of its own would measure it properly. The original
|
||||
item follows.
|
||||
|
||||
Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
|
||||
disagree with it). llmgw has none, so any question about its overhead is
|
||||
answered with a guess.
|
||||
- `llmgw ctl bench --overhead`: the same load through the gateway and straight
|
||||
at a `sim-engine` (`internal/cli/simengine.go`) that answers at once, at
|
||||
rising request rates, with auth, accounting and the guardrails' pattern
|
||||
checks on and off. Reports the added latency at p50/p99 and the gateway's
|
||||
CPU per thousand requests.
|
||||
- On the PoC the CPU is oversubscribed (pve1, 28 threads for ~125 vCPUs):
|
||||
report it as a ratio and record the host's load beside it, and repeat on
|
||||
an idle machine before a figure goes into the README.
|
||||
- Done when: the README states the overhead, with how it was measured, and
|
||||
CI runs the short version so a change that doubles it is seen.
|
||||
|
||||
### I4. Several keys for one outside provider (S) — done 2026-10-17
|
||||
`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
|
||||
each in `control.externalEndpoints`, 401 retried on another key and
|
||||
`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
|
||||
still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
|
||||
original item follows.
|
||||
|
||||
An external model has one `api_key_file` (`internal/control/control.go`,
|
||||
where external endpoints are made). A provider's limit is per key; one key
|
||||
answering 429 makes the whole fallback answer 429.
|
||||
- `api_key_files: [a, b, c]`, each with an optional `weight`: one endpoint
|
||||
per key (`external:<model>#<n>`), so the router's scoring, retries and
|
||||
breakers work per key as they already do per replica. A key answering 429
|
||||
or 401 is passed over while its breaker is open; a 401 also notifies
|
||||
(`external_key_refused`).
|
||||
- The usage row says which key answered (by its index, never the key).
|
||||
- Done when: with two keys and one returning 429, every request is answered
|
||||
by the other, and the console shows the one breaker open.
|
||||
|
||||
## J. Found testing HA and single-machine installs (written 2026-10-16)
|
||||
|
||||
Three VMs installed as HA from the setup screens, a fourth as a judge node,
|
||||
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
|
||||
|
||||
### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
|
||||
Not a wait for ever, as first written: on the 4-core VM the judge's restart
|
||||
was *refused* ("fits on no node even with this replica stopped") because the
|
||||
starter, restarted at the same time to reserve fewer cores, still held its
|
||||
old ones. Once refused, nothing tried again, so the judges kept their old
|
||||
figures. Two fixes in `internal/control/reconciler`:
|
||||
- a restart whose replacement fits once other restarts under way have gone
|
||||
(queued, retiring or draining) waits for them, up to DrainGrace, instead
|
||||
of being refused (`fitsAfterRestarts`, restart.go);
|
||||
- a drained replica counts a retiring one that still answers as serving, so
|
||||
one restart after another no longer leaves the first's drained replica,
|
||||
and the room it holds, waiting out its grace (reconciler.go).
|
||||
`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
|
||||
|
||||
### J2. A judge the leader adds is sized as init's is (S)
|
||||
When a policy names a judge the catalog lacks, the leader serves it itself
|
||||
(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
|
||||
`judges_known.go`, through modelserve). That judge reserves 3 GB (the
|
||||
weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
|
||||
with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
|
||||
(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
|
||||
its cores and runs with `--cache-ram 0`.
|
||||
- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
|
||||
request the same memory and cores rule as `cpuPlan`.
|
||||
- Done when: a judge removed from the catalog comes back with the same
|
||||
figures init would have given it.
|
||||
|
||||
### J3. HA setup puts the judges on nodes of their own (S)
|
||||
Under load on a 4-core control plane that was also a node, 38 of 60
|
||||
requests were refused for want of a judgment. With the judges on a node of
|
||||
their own (4 cores, 32 GB, two cores each) the same test refused 2.
|
||||
- The HA setup offers a judge node or two: machines given like the store's,
|
||||
added as nodes with a label (`llmgw.role: judge`), and the judge's
|
||||
profile selects that label.
|
||||
- Done when: the setup installs the judges on the machines given for them,
|
||||
and the load test above refuses no more than it did by hand.
|
||||
|
||||
### J4. The model mirror on more than one machine (M)
|
||||
In an HA install the mirror stays on the first machine. While it is down,
|
||||
models the nodes already have keep serving and new ones wait for it.
|
||||
- A second mirror kept in step (each machine of the store, or one more), the
|
||||
control planes and nodes given both, the first that answers used.
|
||||
- Done when: with the first machine powered off, a model added then is
|
||||
fetched and served.
|
||||
|
||||
### J5. An upgrade corrects an earlier install's judge (S) — done 2026-10-16
|
||||
Built in the judge keeper (`internal/cli/judgekeeper.go`, `refit`), on the
|
||||
leader: a judge whose entry is exactly what init before v0.14.7 wrote is
|
||||
given 2 GB, `--cache-ram 0` and cores by init's rule (`refitCores`: up to
|
||||
four for one judge, two each for more), through `modelsvc.SetProfiles`,
|
||||
and its ready replicas are restarted (after J1). Its replica count is left
|
||||
as it is. One that differs is left alone and said once in the log. The
|
||||
original item follows.
|
||||
|
||||
v0.14.6 and earlier installed two judges reserving no cores or one, with
|
||||
2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
|
||||
missed its 5 s budget and every request was refused. v0.14.7's notes say
|
||||
how to fix it by hand (`ctl model resources`, then `ctl restart`).
|
||||
- On upgrade, or on `server init` run again, a judge whose profile is still
|
||||
exactly what an earlier init wrote is moved to today's figures and
|
||||
restarted (after J1). One changed by hand is left alone and named.
|
||||
- Done when: a v0.14.6 single-machine install upgraded to it judges within
|
||||
budget without anything typed.
|
||||
|
||||
### J6. Rate limits per process (note)
|
||||
Rate limits are counted in each control plane's memory. A failover starts
|
||||
every key's minute afresh. Documented in install.md; worth a shared count in
|
||||
the store only if a customer's limits must hold across a failover.
|
||||
|
||||
### J7. nginx in front of several control planes (S)
|
||||
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
|
||||
nginx's open-source build has no active health checks, so its `upstream`
|
||||
block would send to standbys and to a lost leader alike.
|
||||
- nginx.conf: the floating address as its one upstream, and why; or the
|
||||
`max_fails` passive approach with what it costs.
|
||||
|
||||
### J8. A judge budget that fits the machine (S)
|
||||
A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
|
||||
default: two cores are close to it, and a busy machine passes it.
|
||||
- At install, after the judge is up, time three judgments. If the slowest is
|
||||
more than half the budget, say so in the plan's closing text with the
|
||||
choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
|
||||
open`). The budget itself is not changed without being asked.
|
||||
- Done when: an install on a machine too slow for its judge says so before
|
||||
its first refused request.
|
||||
|
||||
## L. Taken from Kong's AI Gateway (written 2026-10-25)
|
||||
|
||||
From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
|
||||
3.9, and only six of its AI plugins are open (proxy, prompt template,
|
||||
decorator, regex prompt guard, request and response transformer); the rest
|
||||
is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
|
||||
plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
|
||||
prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
|
||||
unshown), prompt templates and decorators (no one has asked), Konnect's
|
||||
entity model. MCP items are llmgw-80's (I2); they are listed for it.
|
||||
|
||||
### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
|
||||
`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
|
||||
operation, request and response model, max_tokens, temperature, top_p, the
|
||||
session as conversation id, input/output/cached tokens, the engine as
|
||||
provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
|
||||
Prometheus already counted tokens by direction, cached apart. Left: the MCP
|
||||
calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
|
||||
|
||||
### L2. Redacted personal data put back in the answer (S) — done 2026-10-25
|
||||
`restore: true` (`guard.Rules.Restore`), `Verdict.Restore`, `internal/router/restore.go`
|
||||
ahead of the output filter. The original item follows.
|
||||
The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
|
||||
`placeholders`), the same value always the same placeholder. A model that
|
||||
answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
|
||||
- Per policy, `restore: true`: the placeholders of one request kept in memory
|
||||
for its life, and swapped back in the answer, streamed too (a placeholder
|
||||
split across two chunks held until it is whole).
|
||||
- Only into the answer to the request that held them; never stored, never in
|
||||
usage rows or traces.
|
||||
- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
|
||||
policy reaches the model as placeholders and the client reads the real name
|
||||
and address back.
|
||||
|
||||
### L3. A guardrail that is someone else's service (S) — done 2026-10-25
|
||||
`guard.Service` (`internal/guard/service.go`), policy `services`, asked
|
||||
before the judge on input and once on the whole answer. The original item follows.
|
||||
Customers may be required to use one: Azure Content Safety, Lakera, NeMo
|
||||
Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
|
||||
templated HTTP call with an expression for "block".
|
||||
- A detector kind `http`: URL, headers from secret files, a body template
|
||||
with the text, a JSON path that says blocked and one that says why;
|
||||
timeout and `fail` as the judge's. Checked on input, output or both.
|
||||
- Done when: a policy with an `http` detector against a test service blocks
|
||||
what it says to, and a service that does not answer is treated as `fail`
|
||||
says.
|
||||
|
||||
### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
|
||||
`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
|
||||
what `fallbacks` already does (local, then an outside model when no room). Left:
|
||||
routing by cost between several outside providers. The original item follows.
|
||||
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
|
||||
and KV cache. Kong also balances by measured latency, by tokens or cost
|
||||
used, and in priority tiers.
|
||||
- A `latency` scorer: a moving average of time to first token per endpoint.
|
||||
- Fallbacks in tiers: local replicas first, then an outside provider, with
|
||||
the tier said in the usage row.
|
||||
- Done when: with two replicas of one model, one on a slower card, `latency`
|
||||
sends most of the traffic to the faster and the slower still gets some.
|
||||
|
||||
### L5. Prices by kind and by cache (S–M) — done 2026-10-25
|
||||
`cached_per_1k`, charged on the answer's `cached_tokens`. A budget in money was already
|
||||
there (`keys limits --budget`). Not done: cache-write and audio/video token prices —
|
||||
no engine llmgw runs reports them, and no provider it reaches is billed by them yet.
|
||||
The original item follows.
|
||||
`catalog` prices are per million input/output tokens and per picture/mesh.
|
||||
Kong prices cached reads and cache writes apart, and audio, image and video
|
||||
tokens apart, and limits by cost.
|
||||
- Price fields for cached input and by modality; the usage row's cost from
|
||||
them; a budget limit in money as well as tokens (`keys limits --cost`).
|
||||
- Done when: a request with 90% cached prompt costs what the cached rate says.
|
||||
|
||||
### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
|
||||
Kong turns OpenAPI operations into MCP tools. A company with an internal
|
||||
REST API and no MCP server could offer it to agents through `/mcp`:
|
||||
`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
|
||||
operation a tool, under the same grants, audit, guardrails and rate as I2.
|
||||
|
||||
### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
|
||||
Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
|
||||
resource metadata, tokens checked by JWKS against the company's identity
|
||||
provider), and a token exchanged for a short-lived one per upstream server
|
||||
rather than one long-lived secret.
|
||||
|
||||
### L8. A2A (S, when asked)
|
||||
Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
|
||||
llmgw's, calls audited and traced. The standard is young; wait for a
|
||||
customer.
|
||||
|
||||
### L9. llmgw behind Kong, written down (S) — done 2026-10-25
|
||||
docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on;
|
||||
Kong 3.9 counts no completion tokens of a stream. The original item follows.
|
||||
A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
|
||||
`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
|
||||
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
|
||||
container, and what Kong's token accounting sees.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
`/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
|
||||
Metrics on `:9091` with `/prometheus/targets` beside them.
|
||||
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
|
||||
the PoC runs PostgreSQL under Patroni.
|
||||
|
||||
+516
-2
@@ -18,13 +18,527 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
## v0.15.0 — 2026-10-25
|
||||
|
||||
The company's tools and editors through the gateway, and guardrails that
|
||||
can lean on others. Its own MCP servers sit behind llmgw's `/mcp` under the
|
||||
guardrails, a rate and a breaker. VS Code gets the gateway's models from an
|
||||
extension the gateway serves. A guardrail policy can ask an outside
|
||||
service, and can put redacted values back in the answer. Chat goes to the
|
||||
replica answering soonest. A similar question can get the answer already
|
||||
given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
|
||||
GenAI conventions, and Kong is written down from trying it. Also an install
|
||||
guide with every screen, at `/install`.
|
||||
|
||||
**Upgrading.**
|
||||
- Migrations 0049 (`mcp_servers`), 0050 (their rate and breaker) and 0051
|
||||
(`projects.cache_similar`), applied when the control plane starts.
|
||||
- `llmgw server init` now refuses a machine that would serve models without
|
||||
the memory for them: the starter model and the judges need 12 GB on one
|
||||
machine, where 8 GB was said before. `--no-model`, `--no-guard-model` or
|
||||
`--no-node` make it fit. A running cluster is not checked again.
|
||||
- Traces carry the GenAI conventions' `gen_ai.*` attributes beside the
|
||||
`llmgw.*` ones, which stay; docs/install.md, "Tracing", lists them.
|
||||
- Nothing else changes until it is turned on: `services` and `restore` in a
|
||||
guardrail policy, `keys cache --similar` with
|
||||
`router.cache.embedding_model`, `cached_per_1k` in a price.
|
||||
|
||||
### Added
|
||||
|
||||
- **Answers to similar questions.** `llmgw keys cache --similar 0.95` also
|
||||
gives a chat question the answer already given to one like it in the same
|
||||
conversation, compared by `router.cache.embedding_model`, as
|
||||
`x-llmgw-cache: similar` with the similarity. Held in memory beside the
|
||||
exact answers; nothing written to the store but the setting.
|
||||
- **A cached prompt costs the cached rate.** `cached_per_1k` in a price
|
||||
charges the prompt tokens an engine or provider read from its cache (the
|
||||
answer's `cached_tokens`) at their own rate; without it they cost what any
|
||||
prompt token does, as before.
|
||||
- **Chat goes to the replica answering soonest.** The router keeps each
|
||||
replica's time to first token, smoothed over its last answers, and the
|
||||
`chat` profile scores by it beside the queue and the KV cache: of two
|
||||
replicas of one model, the one on the slower card or the busier host gets
|
||||
less, and a replica not yet measured is tried.
|
||||
- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9
|
||||
in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw
|
||||
key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
|
||||
no completion tokens of a streamed answer, so bill from llmgw's usage.
|
||||
- **A guardrail that is someone else's service.** A policy's `services`
|
||||
asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
|
||||
HTTP: a body template, keys from files, dotted paths to where the answer
|
||||
says unsafe and why, on input, output or both, block or flag, and the
|
||||
policy's `fail` when it does not answer.
|
||||
- **Redacted values put back in the answer.** `restore: true` on a guardrail
|
||||
policy's input: the model reads `[EMAIL_1]`, the client reads its own
|
||||
address in the reply, streamed or not. Only the request's own values, held
|
||||
in memory for its life; the output rules read the answer as the client
|
||||
will.
|
||||
- **Traces in the OpenTelemetry GenAI conventions.** An inference request
|
||||
and each attempt at an engine carry `gen_ai.operation.name`, the requested
|
||||
and the answering model, `max_tokens`, `temperature` and `top_p` when set,
|
||||
the session as `gen_ai.conversation.id`, and the input, output and cached
|
||||
tokens, and an attempt the engine that answered as `gen_ai.provider.name`.
|
||||
Tools that read the conventions show llmgw's calls as model calls with no
|
||||
dashboard of their own. No prompt or answer text is ever in a span.
|
||||
- **llmgw for VS Code.** An extension of llmgw's own, served by every gateway
|
||||
at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from
|
||||
the binary itself: plain JavaScript, no dependencies, no build, and so no
|
||||
marketplace needed at a site with no internet. It offers the gateway's chat
|
||||
models to VS Code's language-model API — the chat's model picker, agent
|
||||
mode, and every extension that asks VS Code for a model — with the sizes
|
||||
and the tool and vision capabilities the catalog says, streamed, with tool
|
||||
calls and their results both ways. **VS Code** is a tool on the account
|
||||
page's **Connect your editor**: a key, the one-line install, and an **Open
|
||||
in VS Code** link that hands the key to the extension, which keeps it in
|
||||
VS Code's secret storage. **llmgw: Sign In** opens that page with a state
|
||||
of its own; a link without it is confirmed first, the gateway named.
|
||||
- **An install guide with every screen, at `/install`.** From three bare
|
||||
Ubuntu machines to a first answer: what each machine needs and how to
|
||||
check it, the release fetched and checked, the setup screens one by one,
|
||||
the system's state, and a first request with a key from the account page.
|
||||
The screens are a real three-machine install, recorded and kept as
|
||||
terminal frames — sharp at any size and a few kilobytes — with each
|
||||
session played back on request; the administrator's password and every
|
||||
key were masked before it was kept. `/about` says more of what llmgw does
|
||||
(documents, guardrails, sign-in, high availability, pictures and 3D), and
|
||||
its install steps are the ones of today, leading to the guide.
|
||||
- **The company's MCP tools under the guardrails, a rate and a breaker.**
|
||||
- **The answer:** the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
|
||||
ticket is withheld as a pasted one would be, and personal data is redacted.
|
||||
- **The arguments:** they are read before they leave, so a secret pasted into one does not reach the company system.
|
||||
- **Rate:** `ctl mcp add --rate N` allows each caller N calls a minute to a server.
|
||||
- **Breaker:** five failures in a row take a server out for a minute, its tools off the list.
|
||||
|
||||
- **How fast the judge is, measured.** `llmgw ctl guard judge check` (and `POST /api/v1/guard/judges/check`) works
|
||||
out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
|
||||
and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
|
||||
and what to do: more cores, more judges, a GPU judge, a longer wait, or `fail: open`. `server init` runs it once the
|
||||
judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.
|
||||
|
||||
- **The company's own MCP servers through llmgw's `/mcp`.** `llmgw ctl mcp add jira --url … --secret-file
|
||||
Authorization=FILE` registers one; `/mcp` then lists its tools as `jira.search` beside llmgw's own. Only keys granted
|
||||
them see them: `llmgw keys limits --tools 'jira.*'`, patterns at any level of the tenancy tree. None granted anywhere
|
||||
is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
|
||||
server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
|
||||
on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
|
||||
admin API has `/api/v1/mcp/servers`.
|
||||
- **The cluster as a read-only MCP server.** Registered as `llmgw ctl mcp add llmgw --url llmgw:admin` and granted
|
||||
as `llmgw.*`, its tools say what `llmgw ctl` does:
|
||||
- `cluster_status`, `list_nodes`, `list_models`, `list_replicas`, `explain_placement`, `node_logs`;
|
||||
- nothing that changes the cluster.
|
||||
|
||||
It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
|
||||
The admin API serves the same at `POST /api/v1/mcp`. `ctl mcp add --bearer-file` keeps a token, sent as
|
||||
`Authorization: Bearer`.
|
||||
|
||||
- **Large models across cards and nodes.** For a model of experts on more than one card, the plan now adds vLLM's
|
||||
`--enable-expert-parallel`. It reads the expert count from `config.json` under any of its names, and sizes a model
|
||||
that also reads pictures by its `text_config`. Several nodes are offered in any number, not only powers of two. Each
|
||||
way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
|
||||
shows in `ctl model plan`, in the console's Serve a model, and in a serve job's first step. The serve form checks that
|
||||
a longer context still fits. `ctl model serve --env KEY=VALUE` puts engine variables such as `NCCL_DEBUG=INFO` on
|
||||
every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.
|
||||
|
||||
- **Machines of the judges' own in a three-machine install.** The setup screen asks for up to two judge machines after
|
||||
the store's two; `server init` takes `--judge-node user@host` for each. Each is added as a node labelled
|
||||
`llmgw.role: judge`, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
|
||||
control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
|
||||
of their own, 2 were. A node with an `llmgw.role` label takes only profiles whose node selector asks for that role,
|
||||
so no other model is placed on the judges' machine.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A machine too small for the models it would serve is said so first.**
|
||||
The control plane that is also a node, serving the starter model and two
|
||||
judges, needs about 10.5 GB. On 8 GB the install finished every step and
|
||||
then had no room for the starter. Now the setup's checks fail its memory
|
||||
with what it would serve, and `server init` refuses before it installs
|
||||
anything, naming `--no-model`, `--no-guard-model` and `--no-node`.
|
||||
|
||||
- **Turkish pages are Turkish to the end.**
|
||||
- **Errors:** those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
|
||||
catalog", the settings changed meanwhile, and others) are now in the page's language.
|
||||
- **Percent:** a percentage is written the language's way: `%93` in Turkish, `93%` in English.
|
||||
- **Footer:** the time at the foot of a page is in the language's date form.
|
||||
- **Stale forms:** they are refused with a sentence in that language rather than "bad request".
|
||||
|
||||
- **`llmgw keys limits` no longer clears the limits it was not given.** Setting one limit at a scope wrote the whole
|
||||
row, so `--tools 'jira.*'` alone took away that scope's rate, budget and model list. Only what is given changes now.
|
||||
`--replace` writes every limit, as the console's limits form does.
|
||||
|
||||
- **The plan refused tensor parallelism past a model's KV heads.** Qwen3-235B, with four KV heads, had no way to run
|
||||
on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
|
||||
offered, as is one that divides them.
|
||||
- **A big model on a slow mirror was failed while it downloaded.** The startup timeout counted the download, so
|
||||
hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
|
||||
The timeout now runs from when a node's download last moved.
|
||||
|
||||
- **The judge init installs is no longer replaced by the leader's own.** Just after the control plane started, the
|
||||
leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
|
||||
a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
|
||||
after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.
|
||||
- **A judge the leader serves by itself is sized as init's is.** When a guard policy names a judge the catalog lacks,
|
||||
the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
|
||||
3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and `fail: closed` refused the
|
||||
request. It now gets what `server init` gives its judges: 2 GB, two cores each (fewer on a small node), and no
|
||||
prompt cache. Every judge served through `modelserve` runs with `--cache-ram 0`.
|
||||
|
||||
## v0.14.9 — 2026-10-19
|
||||
|
||||
Sign-in set up from the console, with Active Directory tried before it is
|
||||
kept. A node's console now shows what the machine is doing, and a node's web
|
||||
page shows what it runs. Also fixes found trying Active Directory against a
|
||||
Samba domain controller on the test cluster. Sign-in through it, nested
|
||||
groups and a disabled account suspended by the sync all worked there.
|
||||
|
||||
**Upgrading.**
|
||||
- No migration.
|
||||
- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
|
||||
block written into `server.yaml` on a cluster with a store was never
|
||||
read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
|
||||
again from the console's Sign-in screen.
|
||||
- A Samba DC's own autogenerated certificate is refused (a negative serial
|
||||
and no SAN). Give it one from a CA.
|
||||
|
||||
### Added
|
||||
|
||||
- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
|
||||
Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
|
||||
tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
|
||||
reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
|
||||
password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
|
||||
domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
|
||||
has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
|
||||
- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
|
||||
and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
|
||||
GPUs.
|
||||
- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
|
||||
version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
|
||||
here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
|
||||
itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
|
||||
machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
|
||||
Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
|
||||
- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
|
||||
ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
|
||||
control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
|
||||
- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
|
||||
cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
|
||||
domain controller needs.
|
||||
|
||||
## v0.14.8 — 2026-10-18
|
||||
|
||||
Sign-in with an on-premises Active Directory, and Entra ID made safe for
|
||||
people in many groups. Also several keys for one outside provider, and a
|
||||
measure of what the gateway adds to a request.
|
||||
|
||||
**Upgrading.**
|
||||
- Migration 0048 adds the `user_directory` table, applied when the control
|
||||
plane starts.
|
||||
- Nothing changes until `auth.ldap` is set; docs/install.md, "Active
|
||||
Directory", has an example. It has been tested against a fake directory,
|
||||
not yet against a real domain controller.
|
||||
- A judge installed by an earlier `server init` is refitted by the leader
|
||||
by itself (below); the steps given for v0.14.7 are no longer needed.
|
||||
|
||||
### Added
|
||||
|
||||
- **Sign-in with Active Directory** (`auth.ldap`). People sign in on the gateway's form with their AD username or mail
|
||||
and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give
|
||||
staff roles and places in the tree, as with OIDC. Accounts are tied to the `objectGUID` and made at first sign-in
|
||||
when `create_accounts` is on. Every 15 minutes the leader reads the directory. People disabled or deleted there are
|
||||
suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend
|
||||
more than half the accounts is refused. See docs/install.md, "Active Directory".
|
||||
|
||||
- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
|
||||
place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
|
||||
passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
|
||||
`external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
|
||||
|
||||
- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
|
||||
once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
|
||||
engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
|
||||
It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the
|
||||
gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Entra ID's groups past 200 no longer take a role away.** For someone in more than 200 groups Entra sends none in
|
||||
the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next
|
||||
sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app
|
||||
registration. docs/install.md has an Entra ID section.
|
||||
- **An account made by SCIM or OIDC could not be reinstated.** Setting it active was refused as "never approved".
|
||||
- **An upgrade refits the judge an earlier `server init` installed.** Upgrading to v0.14.7 left such a judge as it was:
|
||||
no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was
|
||||
fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no
|
||||
prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's
|
||||
replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone,
|
||||
and the log says so once.
|
||||
- **A restart is no longer refused for room another restart is giving back.** With every core of a node reserved, two
|
||||
models restarted together, one to need more and one less (as after `ctl model resources` on a judge and a starter),
|
||||
had the first refused: "fits on no node even with this replica stopped", though the second was about to give back
|
||||
its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained
|
||||
replica also counts a replica being restarted that still answers as serving, so replicas restarted one after
|
||||
another no longer wait out the five-minute grace each.
|
||||
|
||||
## v0.14.7 — 2026-10-16
|
||||
|
||||
The judge installed by `server init` sized for what it does. On a machine
|
||||
of fewer than eight cores v0.14.6 left every judgment past its budget, so
|
||||
`fail: closed` refused every request. A single-machine install of v0.14.7
|
||||
on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.
|
||||
|
||||
**Upgrading.**
|
||||
- No migration.
|
||||
- **Upgrading does not change a judge already in the catalog.** v0.14.6
|
||||
and earlier installed two judges, reserving no cores or one, and 2 or
|
||||
4 GB. To give them what they need:
|
||||
`llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2`, then
|
||||
`llmgw ctl restart --model qwen3guard-0.6b`. The two then need four cores
|
||||
on a single machine; below that, a longer `latency_budget` or
|
||||
`fail: open`.
|
||||
- On a node whose cores are all reserved, `ctl restart --model` cannot
|
||||
start the new replica beside the old one and waits. Stop the old engine
|
||||
process, and the replica comes back with the new figures.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Both judges fit an 8 GB machine again.** v0.14.6 had `server init` reserve 4 GB a judge replica. That figure came
|
||||
from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default,
|
||||
which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted
|
||||
nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with `--cache-ram 0`, uses about 1.6 GB and
|
||||
stays there, and reserves 2.
|
||||
- **One machine runs one judge, on up to four cores.** A second replica on the same machine shares its cores and is
|
||||
lost with it, so it bought nothing but half the speed of each. `server init` on a single machine now serves one judge
|
||||
on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two
|
||||
on two cores each, as in HA setups. `--judge-replicas` and the setup's Copies still choose.
|
||||
- **A judge gets the two cores it needs.** v0.14.6 gave each judge replica one core on machines under eight. A judgment
|
||||
of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core
|
||||
machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two
|
||||
cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or
|
||||
shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise
|
||||
`latency_budget`, where requests must not wait.
|
||||
|
||||
## v0.14.6 — 2026-10-16
|
||||
|
||||
Setup that says what each part of an install is and lets every one be
|
||||
chosen and set; three machines that each run the gateway, so any one can
|
||||
be lost; a cluster that loses its store's primary without losing a
|
||||
request; one ComfyUI for an environment's models; pictures, 3D and
|
||||
workflows as MCP tools; alerts asked for at setup; extended thinking.
|
||||
|
||||
**Upgrading.**
|
||||
- No migration.
|
||||
- **Control planes**: with `control.lease_ttl` unset the lease is now 10 s
|
||||
(was 15). Upgrade the standbys first and the leader last; a mix of
|
||||
versions meanwhile is fine.
|
||||
- **The store** (PostgreSQL): each member is dialled for 3 s before the
|
||||
next is tried. A store so far away that connecting takes longer needs
|
||||
`connect_timeout=10` in its address.
|
||||
- **Nodes** notice a control plane that died without closing its
|
||||
connection in 8 s (was 15) once they run this release.
|
||||
- **Judges installed by an earlier `server init`** keep reserving no cores
|
||||
and 2 GB. To give them their own:
|
||||
`llmgw ctl model resources qwen3guard-0.6b --memory 4 --cores 2`, then
|
||||
`llmgw ctl restart --model qwen3guard-0.6b`.
|
||||
- **Patroni** replaces a lost primary within 20 s on a new cluster. On an
|
||||
existing one, run `server init --patroni ...` again from the machine with
|
||||
the authority; it changes only that.
|
||||
- **A floating address** keeps VRRP router id 51 until `server add --vip`
|
||||
runs again, which moves it to one of its own; run it for every standby.
|
||||
- A request whose judge is not there yet waits up to the policy's
|
||||
`latency_budget` for one before `fail: closed` refuses it.
|
||||
|
||||
### Added
|
||||
|
||||
- **Setup lets every part be chosen, each with its settings, and says what each is.** `sudo llmgw` on a new machine lists
|
||||
the parts of an install: the control plane, the store, the model mirror, this machine as a node, the starter model,
|
||||
pictures and 3D, the guardrails' judge, the firewall, notifications, limits and keys, on-demand turns, an offline
|
||||
bundle. Beside each is what it will be or what is missing without it. Space turns a part on or off, and Enter opens
|
||||
its settings in a dialog that explains it at more length. The settings are: which starter model (six Qwen sizes, or
|
||||
any GGUF on Hugging Face) and its context; the mirror's directory or another machine's mirror; the judge's
|
||||
categories, copies, wait and what happens when it does not answer; the portal's language; the sign-up form; the
|
||||
default limits; the longest a key lives; the on-demand waits. A plan page then lists what is installed and what is
|
||||
left out, with the same install as a command. Every setting is a `server init` flag too (`--starter`,
|
||||
`--judge-fail`, `--media`, `--max-key-ttl`, ...).
|
||||
- **Three machines are high availability for the gateway, not only the store.** With three machines, setup makes the two
|
||||
database machines control planes too, standing by while the first leads, with an optional floating address
|
||||
(keepalived) that follows whichever leads. Any one machine can now be lost. By command: `server init --patroni ...
|
||||
--standbys --vip 10.0.0.160/24`. The mirror's management API answers the three control planes, so a standby that
|
||||
comes to lead can still fetch models.
|
||||
|
||||
- **Pictures uploaded to work from**: `/v1/files` takes `purpose=vision` or
|
||||
`user_data` besides `batch`, kept as long as job results, so a picture is
|
||||
sent once and named by its `file-…` id in the MCP tools and a workflow's
|
||||
inputs.
|
||||
- **One ComfyUI for an environment's models**: ComfyUI models in one
|
||||
`comfyui.environment` (two or more) share one replica — the catalog
|
||||
makes it a hidden entry, `comfyui-env-<name>`, on the first member's
|
||||
profiles with the most memory any member needs, carrying every member's
|
||||
weights and custom nodes. Asking for any member starts it; it answers for
|
||||
each with that member's workflow and checkpoint, and the members get no
|
||||
replicas of their own. A picture model and a 3D model on one card stop
|
||||
handing it to each other with a cold start each way.
|
||||
- **Pictures, 3D models, vision and workflows as MCP tools** at `/mcp`,
|
||||
beside the memory notes, each offered only where the gateway has a model
|
||||
for it: `generate_image` (with `concept`, `side` and `texture` styles),
|
||||
`generate_mesh` (from a picture or a prompt), `review_images` (a vision
|
||||
model, to a JSON schema), `run_workflow`, `job_status` and
|
||||
`gateway_status`. Each makes its request under the caller's key, so
|
||||
allowlists, limits, licences and budgets apply and it is charged like
|
||||
any request; what it makes is kept as the caller's file and answered by
|
||||
id and download URL. A client that takes a stream gets
|
||||
`notifications/progress` while a mesh is made and waits up to ten
|
||||
minutes; one that does not, 50 seconds, then the job id.
|
||||
- **Setup asks where alerts go**: `llmgw server init --webhook URL`
|
||||
(repeatable) and a **Notification webhooks** field on both setup screens.
|
||||
Slack's, Mattermost's and Rocket.Chat's addresses are sent Slack's text,
|
||||
any other the event as JSON (`slack:` or `json:` in front says which).
|
||||
They are added to the cluster's `notify.webhooks` beside those already
|
||||
there, and each is sent a test once the control plane runs on them.
|
||||
|
||||
- **Extended thinking on `/v1/messages`**: `thinking: {type: "enabled"}`
|
||||
tells a model that thinks by its chat template to (`enable_thinking`),
|
||||
and its reasoning comes back as `thinking` blocks before the text,
|
||||
streamed as `thinking_delta` events; `disabled` tells it not to. Not
|
||||
asked for, the reasoning is left out as before.
|
||||
- **How long on-demand models hold a card** is a setting, in force on the
|
||||
leader within seconds: `on_demand.evict_after_idle` (2m unless set) and
|
||||
`on_demand.min_lifetime` (5m). On one card shared by a picture model and
|
||||
a 3D model, 1m each brings the wait between a picture and its mesh to
|
||||
about a minute.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The judges and the starter model divide a machine's CPUs instead of each taking all of them.** `server init` gave the
|
||||
judge and the starter no cores of their own, so every llama.cpp engine ran on every core: two judges and the starter
|
||||
on a 4-core control plane competed, and under load 38 of 60 requests were refused for want of a judgment. Moved to a
|
||||
node of their own with 2 cores each, the same test refused 2. When this machine is the node, init now gives each judge
|
||||
replica 1 core (2 from 8 cores) and the starter what is left (at most 8); judges bound for other nodes get 2 each. A
|
||||
judge also reserves 4 GB, not 2: it uses about 3.2.
|
||||
|
||||
- **A lost store machine stops costing requests most of a minute.** The store's connections try each member for 3 seconds,
|
||||
not until the kernel gives up, and a connection to a machine that is gone is closed within about 10 seconds (TCP
|
||||
keepalive, and `TCP_USER_TIMEOUT` on Linux). Patroni replaces a lost primary within 20 seconds (`ttl` 20, `loop_wait`
|
||||
5, `retry_timeout` 5). Running `server init --patroni` again applies this to an existing cluster. Measured on three
|
||||
VMs with the primary's power cut: about 45 seconds of hanging requests before.
|
||||
- **Leadership stays put while the store's primary is replaced.** Replacing a lost primary takes longer than the
|
||||
leadership lease lives, so the lease had lapsed for everyone by the time the store answered again. Whoever asked first
|
||||
took it, as often a standby as the leader. A standby that saw the store out of reach now leaves the lease alone for one
|
||||
round after it returns, so the leader renews it, and a leader that is truly gone is replaced a round later. With that,
|
||||
the default lease is 10 seconds (was 15). A node notices a control plane that died without closing its connection in
|
||||
8 seconds (was 15), so it is on its way to the new leader as that one takes over.
|
||||
- **A judge replica that hangs no longer costs the request.** All the tries of a judge request shared the policy's
|
||||
budget, so a replica that hung used it all and the try on another replica had none left: under `fail: closed` the
|
||||
request was refused while a judge stood idle. When the first replica has not answered by half the budget, the same
|
||||
question now goes to a replica on another machine too, and the first answer is taken. The first replica is not cut
|
||||
short, which on a slow machine would refuse answers it was about to give. Replicas on one machine whose CPUs are taken
|
||||
are slow together, so asking both only doubles the work there: measured that way, as many requests failed as before.
|
||||
On such a machine the remedy is room, not a second question: judges on more than one node, more cores, or the judge
|
||||
on a GPU.
|
||||
- **A new leader waits for the judge instead of refusing at once.** The moment a control plane takes the lead it knows no
|
||||
judge yet. When every judge fails at once, they are asked again every 250 ms while the policy's `latency_budget` lasts,
|
||||
so `fail: closed` no longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A
|
||||
judge that is slow is not asked again, and an alarm is raised once per request, not once per try.
|
||||
|
||||
- Node discovery for Prometheus listed only the control plane asked and the
|
||||
store, so a machine that is both was a control plane to one and a store
|
||||
member to the other — scraped twice — and a standby asked alone lost every
|
||||
node. Every control plane lists all of them alike; deploy/monitoring's
|
||||
prometheus.yml says to scrape and ask each.
|
||||
- A model whose weights and context pool did not fit its card was added
|
||||
and then failed to make its context on every start, taking the card from
|
||||
others each time (Apertus 70B with a 65536-token pool, 8B with 131072, on
|
||||
24 GB). `ctl model serve` shrinks the pool to what the card holds, saying
|
||||
so, and refuses weights that do not fit at all; the portal's serve form
|
||||
refuses such a pool and says the largest that fits.
|
||||
- Removing weights from the mirror that a catalog model still loads is
|
||||
refused, naming the model: removed from the mirror alone, the model stayed
|
||||
in the catalog and its replicas went on loading the node's copy.
|
||||
- Jobs ran one at a time, so a 3D job waiting for the card held up another
|
||||
model's job that could have run. They run side by side, one per model at
|
||||
a time and eight in all, on the leader as before.
|
||||
- `llmgw ctl model serve` served an embedding model as a chat model: no
|
||||
`--embedding`, four slots, a 512-token batch, `embeddings` unset and a
|
||||
"say hello" smoke test it could not pass. A model whose GGUF says it
|
||||
pools (or `--embeddings`) is served for embedding: 32 slots of 1024
|
||||
tokens unless sized otherwise, a batch that holds a whole passage, and
|
||||
one vector asked for as its smoke test.
|
||||
- `llmgw ctl upgrade` checks every node's version once the waves are done
|
||||
and fails naming any not on the release, and says when the control plane
|
||||
has no nodes registered; `llmgw upgrade` on a control plane says how many
|
||||
nodes it left behind and the command that upgrades them. A site's nodes
|
||||
stayed a release behind with nothing to say so.
|
||||
- The cluster and node pages swapped their live part every few seconds,
|
||||
forms and all, so filling one in raced the clock: what was typed, an
|
||||
open choice and the focus were lost. The refresh now waits while a field
|
||||
in it has the focus or a form in it has been typed into and not sent;
|
||||
the mirror page's reload during a download waits the same way.
|
||||
|
||||
## v0.14.5 — 2026-10-14
|
||||
|
||||
Picture and 3D models that add themselves where a card fits them, with a
|
||||
card on the portal and a screen in the console to see and add them; a
|
||||
mesh's colour brought to the picture it was made from; and the whole
|
||||
licence written into every mesh.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains a table (migration 0047: preset_state); nothing to do.
|
||||
- **`presets.auto` is on**: once upgraded, the leading control plane adds
|
||||
flux-schnell and triposr, on demand, where a node's card fits one, the
|
||||
cluster installs ComfyUI (`engines.comfyui`) and the catalog makes
|
||||
nothing of that kind. Nothing is fetched until somebody asks for one.
|
||||
Turn it off with `llmgw ctl model presets auto off`.
|
||||
- **Nodes** need this release for `albedo_match` (it is in the save node
|
||||
the gate puts in each replica); a ComfyUI replica takes it when it
|
||||
starts again.
|
||||
|
||||
### Added
|
||||
|
||||
- **A mesh's base colour brought to its picture's**: `LLMGWSaveMesh` takes
|
||||
`albedo_match` with the picture and its mask, and scales the texture in
|
||||
linear light to the object's brightness in the picture. Stable Fast 3D
|
||||
takes the light out of what it sees and read a dark basalt (0.17) as a
|
||||
light stone dimly lit (0.67); matched, it is 0.17. On sf3d it is the
|
||||
`albedo_match` input (off unless asked).
|
||||
- **Picture and 3D models by themselves**: once a node has a card that fits
|
||||
flux-schnell or triposr and the catalog makes nothing of that kind, the
|
||||
leading control plane adds the preset on demand — so a new cluster's
|
||||
portal has its Images and 3D pages without a definition written — where
|
||||
the cluster installs ComfyUI. `presets.auto` turns it off; a preset
|
||||
somebody removes is not added back. `GET /api/v1/presets` says how each
|
||||
stands and why one is not added; `POST /api/v1/presets/{name}` adds one
|
||||
(and sets `engines.comfyui` if the cluster has none); `PUT
|
||||
/api/v1/presets/auto`. Each automatic addition is in the audit log.
|
||||
- **Presets on the portal and in the console**: the cluster page's
|
||||
**Pictures and 3D models** card lists each preset — what it makes, the
|
||||
classes it fits here, whether and by whom it was added, why the leader
|
||||
has not — with an **Add** button that says first when it would install
|
||||
ComfyUI and restart the control planes, and the switch for `presets.auto`.
|
||||
The console's models screen shows the same; **A** adds one, **P** flips
|
||||
the switch. `ctl model add PRESET` and `ctl model presets` go through the
|
||||
API (and `ctl model presets auto on|off`), falling back to the entry as
|
||||
v0.14.4 sent it on an older control plane.
|
||||
|
||||
### Fixed
|
||||
|
||||
- A mesh carried only its licence's name, URL and notice; it carries the
|
||||
whole licence as `/v1/models` gives it — id, commercial use, excluded
|
||||
regions, revenue cap, attribution — for a pipeline that sorts assets by
|
||||
licence.
|
||||
- A workflow's `{{name|false}}` default filled a BOOLEAN input with the
|
||||
word rather than a boolean.
|
||||
|
||||
|
||||
## v0.14.4 — 2026-10-13
|
||||
|
||||
|
||||
@@ -40,7 +40,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -50,6 +51,24 @@ budgets and rate limits decide who may use any of it.
|
||||
|
||||
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
|
||||
|
||||
- **Setup that explains, and high availability that is whole** (v0.14.6).
|
||||
`sudo llmgw` lists the parts of an install, each with what it is for and
|
||||
what is missing without it, and every one can be turned off or set: the
|
||||
starter model and its context, the mirror's directory, the judge's
|
||||
categories and what it does when it does not answer, limits, the
|
||||
portal's language, pictures and 3D. With three machines each runs the
|
||||
control plane, with a floating address on the leader, and any one can
|
||||
be lost: on three VMs, cutting the store primary's power lost no
|
||||
request. One ComfyUI serves an environment's models, pictures, 3D and
|
||||
workflows are MCP tools, and `/v1/messages` gives a model's reasoning as
|
||||
thinking blocks.
|
||||
- **Picture and 3D models that add themselves** (v0.14.5). Where a node's
|
||||
card fits flux-schnell or triposr and the catalog makes nothing of that
|
||||
kind, the leader adds it on demand, so a new cluster's portal has its
|
||||
Images and 3D pages; the cluster page and the console show each preset
|
||||
and add it by hand, and `presets.auto` turns it off. Stable Fast 3D's
|
||||
colours can be brought to the picture's (`albedo_match`), and every mesh
|
||||
carries its whole licence.
|
||||
- **Any ComfyUI workflow, and models by name** (v0.14.4). `POST
|
||||
/v1/workflows/runs` runs a ComfyUI model's workflow as a job, its inputs
|
||||
(text, numbers, choices, images, masks, audio, video, files) declared in
|
||||
@@ -165,6 +184,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
cluster that exists
|
||||
- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
|
||||
Kubernetes with the Helm chart, and why GPU nodes stay outside it
|
||||
- [docs/kong.md](docs/kong.md) — llmgw behind Kong Gateway, or Kong behind
|
||||
llmgw: the configuration, what Kong counts, and which to choose
|
||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||
when a replica spanning nodes will not run
|
||||
@@ -239,6 +260,33 @@ Add `--json` to any command for machine-readable output.
|
||||
says nothing about which of six constraints bit, so it prints every candidate
|
||||
considered and the reason it was rejected.
|
||||
|
||||
## What llmgw adds to a request
|
||||
|
||||
Measured with `llmgw ctl bench overhead`. It starts an engine that answers
|
||||
at once and a gateway in front of it, in one process on loopback. It sends
|
||||
the same load straight to the engine and through the gateway, and reports
|
||||
what the gateway added. Run on an Apple M5 (10 CPUs, load about 2) on
|
||||
2026-10-16, 10 s a step, 72,000 requests with none failed:
|
||||
|
||||
| Through the gateway | added at p50 | added at p99 |
|
||||
|---|---|---|
|
||||
| bare | 0.06–0.24 ms | 0.10–0.33 ms |
|
||||
| with authentication and usage accounting | 0.05–0.32 ms | 0.09–0.37 ms |
|
||||
| with the guardrails' pattern checks as well | 0.10–0.42 ms | 0.15–0.49 ms |
|
||||
|
||||
The range is across 100 to 2,000 requests a second. The overhead was
|
||||
*lowest* at 2,000, most likely because a busier processor keeps its cores
|
||||
awake. The guardrails' judge model is not in these figures: a judgment
|
||||
takes seconds on CPUs (see [install.md](docs/install.md)) and is a model's
|
||||
time, not the gateway's.
|
||||
|
||||
The command also prints the gateway's CPU per thousand requests. Gateway,
|
||||
engine and load share one process there, so that figure is rough (0.04 to
|
||||
0.65 s per thousand in the run above); the latency is the one to quote.
|
||||
Figures from a VM on a busy host move with the host: run it where you will
|
||||
deploy, and keep the load it prints beside the result. CI runs a one-second
|
||||
version on every change (`TestTheGatewaysOverheadIsMeasured`).
|
||||
|
||||
## Running the router
|
||||
|
||||
```sh
|
||||
@@ -266,8 +314,9 @@ within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
||||
admission off.
|
||||
|
||||
Routing profiles select how endpoints are scored: `chat` favours session
|
||||
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
||||
cache. Pick one per model in the config, or per request with the
|
||||
affinity and the replica that has been answering soonest (its time to first
|
||||
token, smoothed), `batch` favours the shortest queue, `long-context` favours
|
||||
free KV cache. Pick one per model in the config, or per request with the
|
||||
`x-llmgw-profile` header.
|
||||
|
||||
## sim-engine
|
||||
|
||||
+37
-12
@@ -1,4 +1,4 @@
|
||||
# Two control planes
|
||||
# More than one control plane
|
||||
|
||||
llmgw handles the hard half of this itself: only one control plane places work,
|
||||
and agents find whichever one is leading. What it does not do is move client
|
||||
@@ -37,7 +37,7 @@ nothing in front of the two had moved.
|
||||
So something has to. The options, in the order most people should consider
|
||||
them:
|
||||
|
||||
**A load balancer you already run.** Health-check `/ready` on each control
|
||||
**A load balancer you already run.** Health-check `/leader` on each control
|
||||
plane and send traffic to whichever answers. This is the right answer if you
|
||||
have one, and it needs nothing else from this directory.
|
||||
|
||||
@@ -50,25 +50,50 @@ in whatever your resolvers actually honour rather than what you set, which on
|
||||
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
||||
one an agent is driving.
|
||||
|
||||
## Check `/ready`, not `/health`
|
||||
## Check `/leader`, not `/health`
|
||||
|
||||
This is the one thing to get right, and the obvious choice is the wrong one.
|
||||
|
||||
`/health` says the process is up. It says nothing about whether it can serve,
|
||||
`/health` says the process is up. It says nothing about whether it leads,
|
||||
and a standby is exactly that case: alive, answering, and holding no endpoints
|
||||
at all — its reconciler placed nothing and it has no agent streams to adopt
|
||||
from. A load balancer checking `/health` would send it half the traffic, and
|
||||
half the traffic would be a 503.
|
||||
from. A standby passes client requests on to the leader, so traffic sent to
|
||||
one still works — until the leader is the machine that was lost, and there
|
||||
is nobody to pass them to.
|
||||
|
||||
`/leader` is 200 on the control plane that leads and 503 on a standby. It is
|
||||
what keepalived follows for the floating address, and what a load balancer
|
||||
should send traffic by: to the one machine that answers 200.
|
||||
|
||||
`/ready` is narrower: 200 only when this process has somewhere to send a
|
||||
request, so 503 on a standby and also on a leader with no model running yet.
|
||||
Checking it takes the portal and the admin pages away with the models — on a
|
||||
new cluster with no node yet, nobody can sign in to add one. Use it to stop
|
||||
sending inference only, if your load balancer can tell the paths apart.
|
||||
|
||||
`/ready` says this process has somewhere to send a request. It is 503 on a
|
||||
standby, 503 on a leader whose replicas have all failed, and 200 when there is
|
||||
something to serve — which is the question a load balancer is actually asking.
|
||||
Both are open, because a load balancer has no key.
|
||||
|
||||
During a failover neither passes for about the length of the lease handover:
|
||||
During a failover nothing passes for about the length of the lease handover:
|
||||
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
||||
is not. That gap is real and there is no way to have it both ways — traffic
|
||||
cannot go to a control plane that has nothing to route to.
|
||||
is not. A proxy that waits that long for a healthy machine rather than
|
||||
failing at once (Caddy's `lb_try_duration`) hides it from clients.
|
||||
|
||||
**Caddy**, in front of three control planes — the end of
|
||||
[deploy/proxy/Caddyfile](../proxy/Caddyfile) has it with the rest of the
|
||||
settings:
|
||||
|
||||
```
|
||||
reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
|
||||
lb_policy first
|
||||
health_uri /leader
|
||||
health_interval 2s
|
||||
lb_try_duration 15s
|
||||
flush_interval -1
|
||||
}
|
||||
```
|
||||
|
||||
With a floating address the address already follows the leader: give Caddy
|
||||
that one address instead.
|
||||
|
||||
Not the admin port, for a different reason: it serves pprof, and pprof can
|
||||
dump the process heap, where API keys sit in plaintext.
|
||||
|
||||
@@ -41,7 +41,9 @@ vrrp_instance llmgw {
|
||||
# MASTER on the machine you would rather serve; BACKUP on the other.
|
||||
state MASTER
|
||||
interface eth0
|
||||
# Any number both machines agree on and nothing else on the segment uses.
|
||||
# Any number both machines agree on and nothing else on the segment uses:
|
||||
# another cluster's keepalived with the same one reads as an invalid
|
||||
# password. `llmgw server add --vip` derives one from the address.
|
||||
virtual_router_id 51
|
||||
# The higher one wins while both are healthy. The weight above is enough
|
||||
# to invert this when the preferred machine's gateway is not answering.
|
||||
|
||||
@@ -21,11 +21,18 @@ scrape_configs:
|
||||
# The control plane's dedicated metrics port. Not its admin listener: that one
|
||||
# is loopback because it also serves pprof, and pprof can dump the heap where
|
||||
# API keys sit in plaintext.
|
||||
#
|
||||
# Every control plane, one line each: the leader moves on restarts and the
|
||||
# router's metrics are the leading one's, so scraping one alone leaves gaps
|
||||
# — and alerts on them — whenever another leads.
|
||||
- job_name: llmgw
|
||||
static_configs:
|
||||
- targets: ['10.0.0.140:9091']
|
||||
labels:
|
||||
instance: control-plane
|
||||
instance: control-plane-140
|
||||
# - targets: ['10.0.0.150:9091']
|
||||
# labels:
|
||||
# instance: control-plane-150
|
||||
|
||||
# Every machine of the cluster, for its node exporter: network and disk
|
||||
# traffic, filesystems, CPU time. The control plane lists them — nodes,
|
||||
@@ -33,10 +40,15 @@ scrape_configs:
|
||||
# nothing edited here, and each carries its llmgw id as the node label.
|
||||
# `llmgw node add` installs the exporter; on a machine added before that,
|
||||
# apt-get install prometheus-node-exporter.
|
||||
# Asked of every control plane too: a standby lists only the control
|
||||
# planes and the store, and they label every machine alike, so the same
|
||||
# target from two of them is scraped once.
|
||||
- job_name: nodes
|
||||
http_sd_configs:
|
||||
- url: http://10.0.0.140:9091/prometheus/targets
|
||||
refresh_interval: 1m
|
||||
# - url: http://10.0.0.150:9091/prometheus/targets
|
||||
# refresh_interval: 1m
|
||||
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
|
||||
@@ -14,6 +14,7 @@ llm.example.com {
|
||||
max_size 32MB
|
||||
}
|
||||
|
||||
# One control plane. With more than one, see the end of this file.
|
||||
reverse_proxy 127.0.0.1:8080 {
|
||||
# Tokens are written the moment they are generated. Without this a
|
||||
# streamed reply can arrive in bursts, and a client measuring time to
|
||||
@@ -39,3 +40,36 @@ llm.example.com {
|
||||
# Access log to stderr, which under systemd is the journal.
|
||||
log
|
||||
}
|
||||
|
||||
# More than one control plane (server init --standbys, server add): one of
|
||||
# them leads, the others stand by. Name them all and send to the one that
|
||||
# leads — /leader is 200 there and 503 on a standby. Not /ready: that is 503
|
||||
# on a leader with no model running yet, and would take the portal away with
|
||||
# it. A standby would pass a request on to the leader anyway, but not while
|
||||
# there is none, and the hop is for nothing.
|
||||
#
|
||||
# With a floating address (--vip) the address already follows the leader:
|
||||
# give reverse_proxy that one address instead, and keep the rest as above.
|
||||
#
|
||||
# llm.example.com {
|
||||
# request_body {
|
||||
# max_size 32MB
|
||||
# }
|
||||
# reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
|
||||
# # The first that is healthy: the leader, whichever it is.
|
||||
# lb_policy first
|
||||
# health_uri /leader
|
||||
# health_interval 2s
|
||||
# health_timeout 2s
|
||||
# # While leadership moves — a few seconds when a leader is lost —
|
||||
# # none is healthy; a request waits that long for one rather than
|
||||
# # failing at once.
|
||||
# lb_try_duration 15s
|
||||
# lb_try_interval 250ms
|
||||
# flush_interval -1
|
||||
# transport http {
|
||||
# response_header_timeout 15m
|
||||
# }
|
||||
# }
|
||||
# log
|
||||
# }
|
||||
File diff suppressed because it is too large.
Load diff
@@ -82,7 +82,7 @@
|
||||
"/mcp": {
|
||||
"post": {
|
||||
"operationId": "postMcp",
|
||||
"summary": "The gateway's MCP server: the memory notes as tools",
|
||||
"summary": "The gateway's MCP server: the memory notes, and pictures, 3D models, vision and workflows, as tools",
|
||||
"tags": [
|
||||
"mcp"
|
||||
],
|
||||
|
||||
@@ -534,6 +534,60 @@ curl https://llm.example.com/mcp \
|
||||
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
|
||||
```
|
||||
|
||||
**Pictures, 3D models, vision and workflows.** Where the gateway has the
|
||||
models for them, the same server offers more tools — only those it has a
|
||||
model for:
|
||||
|
||||
| Tool | Does |
|
||||
|---|---|
|
||||
| `generate_image` | A picture from a prompt. `style` adds what a picture for a mesh wants: `concept` (one object, three-quarter view, grey background), `side` (an exact side view), `texture` (a seamless tile) or `none`. Answers the PNG's `file_id`, `url` and `seed`, with a 512 px preview to look at. |
|
||||
| `generate_mesh` | A GLB from a picture — `image` is a `file-…` id (a `generate_image` result, say), a data URL or base64 — or from a `prompt`, for which a concept picture is made first. `seed`, `resolution`, `texture_resolution`, and any other input the model's workflow declares in `inputs`. Answers the GLB's `file_id`, `url` and licence. |
|
||||
| `review_images` | Asks a vision model about one to four pictures (each shrunk to 1280 px on its long side); `json_schema` makes it answer that JSON. |
|
||||
| `run_workflow` | Runs a ComfyUI workflow from the inputs its catalog entry declares, and answers its result file. |
|
||||
| `job_status` | Where a mesh or a workflow run stands, and its file when done; `wait_seconds` waits for it. |
|
||||
| `gateway_status` | The picture, 3D, vision and workflow models, whether each is loaded on a GPU now, a workflow's inputs, and your last jobs. |
|
||||
|
||||
Each tool makes its request of the gateway under your key, so your
|
||||
allowlist, limits, left-out licences and budget apply, and it is charged
|
||||
and listed like any other request. What it makes is kept as your file for
|
||||
as long as job results are: fetch it from `url` with the same key (`curl -H
|
||||
"Authorization: Bearer $LLMGW_API_KEY" -o rock.glb <url>`). A tool server
|
||||
over HTTP cannot write to your disk, so an agent that needs the file
|
||||
locally — to import it in Blender — fetches it, or goes through a small
|
||||
local bridge that does.
|
||||
|
||||
A picture can be uploaded once and named by its id after: `POST /v1/files`
|
||||
with `purpose=vision` (or `user_data`) keeps it, for as long as job results
|
||||
are kept, as a file of yours that `generate_mesh`, `review_images` and a
|
||||
workflow's file inputs take as `file-…`. A call is at most 64 MB.
|
||||
|
||||
A model that is not loaded is waited for: a picture or a look at one whose
|
||||
model is starting, or waiting for another to give up the GPU, waits as a
|
||||
mesh does, saying so, rather than failing with "model loading".
|
||||
|
||||
A mesh takes from half a minute to a few minutes. A client that takes the
|
||||
answer as a stream (`Accept: text/event-stream`, as Claude Code does) is
|
||||
answered that way: `notifications/progress` for the call's
|
||||
`progressToken` while it runs — queued, waiting for the GPU, the node and
|
||||
its percent — then the result, and the call waits up to ten minutes. One
|
||||
answered in one piece waits 50 seconds; a job not done by then is answered
|
||||
with its `job_id`, to ask after with `job_status`. `wait_seconds` (up to
|
||||
900) sets either.
|
||||
|
||||
```
|
||||
claude mcp add --transport http llmgw https://llm.example.com/mcp \
|
||||
--header "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
**Your company's tools.** Where an administrator has registered the
|
||||
company's own MCP servers (Jira, a file server, a database) and your key is
|
||||
granted their tools, the same `/mcp` lists them too, each as `server.tool`
|
||||
(`jira.search`). Give your agent only llmgw's `/mcp` and your key: the
|
||||
servers' own credentials stay with llmgw. A tool your key is not granted is
|
||||
not listed, and calling it by name is refused. Each call is in the audit
|
||||
log under your key: which tool, the size of what was sent, how it ended and
|
||||
how long it took, never what it said.
|
||||
|
||||
### Licences you leave out
|
||||
|
||||
A model's licence is in `/v1/models` (`license`: its `id`, `name`, whether
|
||||
|
||||
+39
-4
@@ -2,8 +2,8 @@
|
||||
|
||||
`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
|
||||
with nothing installed it opens the setup screens; on a control plane, the
|
||||
console described here; on a node, the node's own console, which edits
|
||||
`agent.yaml`. Everything on these screens can also be done by command — each
|
||||
console described here; on a node, the node's own console, which shows
|
||||
what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
|
||||
screen says which — so a script never needs them and a person never needs
|
||||
to remember the commands.
|
||||
|
||||
@@ -246,6 +246,28 @@ long a key made there may live, and is the expiry of one made without any
|
||||
Limits are requests and tokens a minute, a budget per day or month, and the
|
||||
models allowed. Each level only narrows what is above it.
|
||||
|
||||
**Sign-in** — how people get in besides a password kept here: single
|
||||
sign-on, Active Directory and provisioning, each on or off, with the
|
||||
directory's last sync (how many people, accounts kept in step, suspended,
|
||||
reinstated).
|
||||
- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
|
||||
and the groups for administrators and operators. A provider other than
|
||||
Entra takes its issuer URL in place of the tenant. The form shows the
|
||||
redirect address to register.
|
||||
- **D** sets up Active Directory: the servers, where people are, the
|
||||
service account and its password, the domain's CA certificate as a PEM
|
||||
file on this machine, the group whose members may sign in, and groups
|
||||
for roles and teams.
|
||||
- Before it is kept, the directory is tried from the control plane: each
|
||||
server reached and bound, how many people may sign in, how many are in
|
||||
each group. If any step fails, nothing is kept and the steps are shown.
|
||||
**T** tries the settings in force.
|
||||
- On the provisioning row, Enter makes a SCIM token, shown once.
|
||||
- The password, the client secret and the token are kept in the store,
|
||||
sealed by the master key when one is set. The settings name them as
|
||||
`store:auth/…`, so each control plane has them with no file put on it.
|
||||
The CA is written into the settings themselves.
|
||||
|
||||
**Guardrails…** on a row lists the guardrail policies server.yaml defines.
|
||||
Each policy is marked in one of three ways:
|
||||
|
||||
@@ -335,5 +357,18 @@ to restart it.
|
||||
|
||||
## The node's console
|
||||
|
||||
On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
|
||||
manual. The cluster is watched and changed from a control plane.
|
||||
On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
|
||||
on the machine itself, so it works without an admin token and while the control
|
||||
plane is out of reach. Refreshed every three seconds, it shows
|
||||
|
||||
- llmgw-agent's state and since when, and the version of llmgw;
|
||||
- each control plane the agent dials, from `agent.yaml` and the list the
|
||||
leader last sent, and whether it can be reached;
|
||||
- each GPU: how busy, its memory, its heat, and the models on it;
|
||||
- the engines running here, from the agent's own records: replica, model,
|
||||
engine, GPUs, port, whether the process is still there, and for how long;
|
||||
- the free space where weights and engines are kept;
|
||||
- the agent's warnings and errors of the last hour.
|
||||
|
||||
Below it, `agent.yaml` in the same settings view, and the manual. The cluster
|
||||
is watched and changed from a control plane.
|
||||
+3
-1
@@ -72,7 +72,9 @@ each answer that may be reused, as the client got it, and a hash of the
|
||||
request it answers, in the gateway process's memory — never in the store,
|
||||
never on disk — for the project's TTL or until pushed out by newer ones
|
||||
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
||||
policy. Forgetting a person does not reach into it; the answers age out by
|
||||
policy. A project that reuses answers to similar questions (`--similar`)
|
||||
also has each question's last user message embedded, and the vector kept
|
||||
beside its answer, in the same memory and for as long. Forgetting a person does not reach into it; the answers age out by
|
||||
the TTL. The store keeps only the setting, and a usage record's mark that it
|
||||
was answered from the cache.
|
||||
|
||||
|
||||
@@ -91,6 +91,21 @@ guard:
|
||||
value is the same placeholder throughout a request).
|
||||
- **`block`** stops it.
|
||||
|
||||
**Putting them back.** With `restore: true` in a policy's `input`, the
|
||||
values a request had redacted are put back in its answer: the model reads
|
||||
"write to [PERSON_1] at [EMAIL_1]", the client reads the name and address
|
||||
it sent. Streamed answers too — a placeholder split across two events is
|
||||
held until it is whole. Only that request's own values, only into its
|
||||
answer; the map is kept in memory for the request and nowhere else. The
|
||||
output rules then read the answer as the client will, so a policy that
|
||||
redacts personal data in answers still does.
|
||||
|
||||
```yaml
|
||||
input:
|
||||
pii: {email: redact, phone: redact}
|
||||
restore: true
|
||||
```
|
||||
|
||||
**Shapes.**
|
||||
|
||||
- `system_prompt` is put first.
|
||||
@@ -258,6 +273,39 @@ themselves.
|
||||
context rather than overflowing it and going unjudged.
|
||||
- **Cache:** the same text is not asked about twice within five minutes.
|
||||
|
||||
## An outside service
|
||||
|
||||
A site required to use a particular guardrail — Azure Content Safety,
|
||||
Lakera Guard, NeMo Guardrails, a classifier of its own — names it in a
|
||||
policy's `services`. It is asked over HTTP beside the patterns and before
|
||||
the judge; its "unsafe" blocks or flags like theirs, and a service that does
|
||||
not answer is what the policy's `fail` says.
|
||||
|
||||
```yaml
|
||||
policies:
|
||||
strict:
|
||||
fail: closed
|
||||
services:
|
||||
- name: lakera
|
||||
url: https://api.lakera.ai/v2/guard
|
||||
header_files: {Authorization: /etc/llmgw/secrets/lakera} # "Bearer …", read each call
|
||||
body: '{"messages": [{"role": "user", "content": {{json .Text}}}]}'
|
||||
blocked: flagged # where the answer says unsafe
|
||||
reason: breakdown.0.detector_type
|
||||
on: both # input, output or both
|
||||
action: block # or flag
|
||||
timeout: 2s
|
||||
```
|
||||
|
||||
`body` is a template given `.Text` (`{{json .Text}}` writes it as a JSON
|
||||
string; empty sends `{"input": …}`). `blocked` and `reason` are dotted paths
|
||||
into the JSON answer, numbers for list items; true, a number above zero, a
|
||||
non-empty list or a string other than "false" is unsafe. On input the
|
||||
service reads the latest user turn; on output the whole answer, once, at its
|
||||
end — a streamed answer released a sentence at a time is then stopped or
|
||||
flagged as the judge's is. The rule recorded is `service:<name>:<reason>`.
|
||||
The service is told the text only, never who sent it.
|
||||
|
||||
## Where a policy applies
|
||||
|
||||
`guard.default` applies everywhere. Others are attached to a part of the
|
||||
|
||||
+395
-28
@@ -89,13 +89,36 @@ CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
|
||||
|
||||
**The short way is the screens.** On the machine that will be the control
|
||||
plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
|
||||
whether this is one machine or three for high availability; for three, it asks
|
||||
for the other two — address, user, and an ssh key, the ssh agent, or a
|
||||
password, and the sudo password if sudo asks for one; or, for a machine
|
||||
nothing may log into, a join code to run there with `sudo llmgw join <code>`,
|
||||
which dials out to this one on port 7444 — checks all three before
|
||||
installing anything (system, room, clock, ports, internet, whether they reach
|
||||
one another), shows the plan, and installs. Passwords are held in memory for
|
||||
whether this is one machine or three for high availability, then lists what
|
||||
an install is made of, each part with what it is for and what is missing
|
||||
without it:
|
||||
|
||||
| Part | | Its settings (Enter) |
|
||||
|---|---|---|
|
||||
| Control plane | always | the administrator, other names for the certificate, the portal's language, the sign-up form |
|
||||
| Store | always | SQLite or PostgreSQL on this machine; with three machines, PostgreSQL under Patroni on the other two |
|
||||
| Control plane on all three | three machines, on | a floating address for whichever leads |
|
||||
| Model mirror | on | here, and in which directory, or on another machine |
|
||||
| This machine as a node | on for one machine | |
|
||||
| Starter model | on for one machine | which: Qwen2.5 0.5B, 1.5B or 3B, Qwen3 4B, 8B or 14B, or any GGUF on Hugging Face; its context |
|
||||
| Pictures and 3D | off | |
|
||||
| Guardrails' judge | on | refuse or serve flagged when it does not answer; copies; how long it is waited for; what it refuses |
|
||||
| Firewall | on | |
|
||||
| Notifications | off | the webhooks |
|
||||
| Limits and keys | always | requests and tokens a minute; the longest a key lives |
|
||||
| Taking turns on a card | always | how long an on-demand model goes unused, and is safe after loading |
|
||||
| Offline bundle | off | the bundle and its sha256 |
|
||||
|
||||
Space turns a part on or off; Enter opens its settings, which say more. What
|
||||
cannot go together is said before anything is done — a starter model with no
|
||||
mirror to fetch it into — and closing the message opens the setting that is
|
||||
wrong. For three machines it then asks for the other two — address, user, and
|
||||
an ssh key, the ssh agent, or a password, and the sudo password if sudo asks
|
||||
for one; or, for a machine nothing may log into, a join code to run there
|
||||
with `sudo llmgw join <code>`, which dials out to this one on port 7444 —
|
||||
checks all three before installing anything (system, room, clock, ports,
|
||||
internet, whether they reach one another). Last it shows the plan, part by
|
||||
part, what is left out and what that costs, and installs. Passwords are held in memory for
|
||||
the run only; a sudo password lets the user sudo without one until the install
|
||||
finishes, and is then taken back. At the end it shows the administrator's
|
||||
password and the command that does the same from a script.
|
||||
@@ -129,15 +152,24 @@ That is the whole step. In order, it:
|
||||
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
||||
engine
|
||||
9. closes llmgw's internal ports to everything but the cluster's networks
|
||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
|
||||
replicas on the CPUs of nodes with room — this one, or the first that
|
||||
join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
|
||||
CPUs: one replica on up to four cores when this machine is the only
|
||||
node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||
first thing anyone opens `/chat` to find is a model that answers. A
|
||||
machine without room for it is told so within two minutes, and why.
|
||||
|
||||
`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9,
|
||||
`--no-guard-model` 10 and `--no-model` 11. An NVIDIA card with no working
|
||||
`--no-guard-model` 10 and `--no-model` 11. Every setting the screens ask for
|
||||
is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
|
||||
(or `hf:<owner>/<repo>/<file>.gguf`) and `--starter-context`, `--mirror-dir`,
|
||||
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
|
||||
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
|
||||
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
|
||||
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
|
||||
go into `server.yaml` as init writes it, and become the cluster's first
|
||||
settings when it starts; a `server.yaml` already there is never overwritten,
|
||||
so run again they change nothing — the settings page does. An NVIDIA card with no working
|
||||
driver is reported rather than silently ignored; install the driver and run
|
||||
init again to serve on it.
|
||||
|
||||
@@ -155,15 +187,43 @@ demonstration one:
|
||||
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
||||
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
||||
|
||||
The judge runs as two replicas, each on a CPU — about half a second for a
|
||||
short text on any CPU with AVX2 — and reads a streamed answer every 600
|
||||
characters. The policy fails **closed**: a request the judge cannot read
|
||||
within 5 seconds is refused with 503 rather than served unjudged, and
|
||||
`judge_unavailable` goes to the notification webhooks. Two replicas are what
|
||||
make that bearable; on one small machine, or a VM without AVX2 (see
|
||||
The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
|
||||
cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
|
||||
streamed answer every 600 characters. The policy fails **closed**: a request
|
||||
the judge cannot read within 5 seconds is refused with 503 rather than
|
||||
served unjudged, and `judge_unavailable` goes to the notification webhooks.
|
||||
So a judge gets cores of its own: on one machine one replica on up to four,
|
||||
and on a larger cluster two replicas of two cores each, spread over its nodes
|
||||
so that losing one leaves the other. On a machine of fewer than four cores,
|
||||
a busy one, or a VM without AVX2 (see
|
||||
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
||||
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
|
||||
`--no-guard-model` is the choice there. Defamation, specialized advice,
|
||||
first thing to fail. The choices there are a longer `latency_budget`,
|
||||
`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
|
||||
or `--no-guard-model`.
|
||||
|
||||
init times the judge once it is up. If a judgment takes more than half the
|
||||
wait, or more than the wait when several come at once, it says so before the
|
||||
next steps, with what to do. The wait itself is not changed without being
|
||||
asked. `llmgw ctl guard judge check` does the same at any time:
|
||||
|
||||
```
|
||||
qwen3guard-0.6b, 2 replica(s) ready, a request waits 5.0 s for it
|
||||
alone: 2.4, 2.3, 2.3 s
|
||||
4 at once: slowest 6.1 s; about 0.7 judgments a second, so about 0.3 requests a second with short answers
|
||||
tight: a busy moment passes the budget
|
||||
```
|
||||
|
||||
In a three-machine install the judges are better on machines of their own.
|
||||
On a 4-core control plane that was also a node, 38 of 60 requests under load
|
||||
were refused for want of a judgment. With the judges on a node of their own
|
||||
(4 cores, two judges of two cores each), 2 were refused. The setup screen asks
|
||||
for up to two **judge machines** after the store's two, or give
|
||||
`--judge-node user@host` to `server init`, once for each. Each is added as a
|
||||
node labelled `llmgw.role: judge`, and the judge's profile selects that
|
||||
label, so the judges run only there and nothing else does. Four cores and
|
||||
8 GB are enough for two judges.
|
||||
|
||||
Defamation, specialized advice,
|
||||
intellectual property and elections are left out of what it refuses: they
|
||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||
|
||||
@@ -331,7 +391,11 @@ by itself. Three things are worth knowing before you edit it:
|
||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||
budgets mean anything. Without a price every request costs zero and a budget
|
||||
never binds; without the defaults, a project created without limits is
|
||||
metered by nothing at all.
|
||||
metered by nothing at all. A price may set `cached_per_1k` beside
|
||||
`prompt_per_1k`: the prompt tokens the engine or provider read from its
|
||||
cache (the answer's `cached_tokens`) are charged at it, the rest at the
|
||||
prompt rate. A provider that bills cached reads at a tenth is
|
||||
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
|
||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||
is empty. Without it, a file with no models is refused as having nothing to
|
||||
serve.
|
||||
@@ -1287,6 +1351,26 @@ catalog, and a key rotated in the file is used within seconds. A model whose key
|
||||
file cannot be read is not offered. Everything else applies as to any model:
|
||||
prices and budgets, rate limits, usage records, and key allowlists.
|
||||
|
||||
A provider's limits are per key. With several keys, one answering 429 need not
|
||||
hold up the model: give them all, each with an optional weight (its share
|
||||
against the others'), in place of `api_key_file`:
|
||||
|
||||
```yaml
|
||||
external:
|
||||
url: https://api.openai.com
|
||||
model: gpt-4o-mini
|
||||
api_key_files:
|
||||
- {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
|
||||
- {file: /etc/llmgw/secrets/openai-2.key}
|
||||
```
|
||||
|
||||
Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
|
||||
answers 429 is passed over while its breaker is open, and the request goes to
|
||||
another. One that answers 401 (revoked, expired, mistyped) is passed over too,
|
||||
and `external_key_refused` goes to the notification webhooks, naming the
|
||||
endpoint and never the key. The usage records say which key answered by the
|
||||
same name. A key whose file cannot be read is left out, and the others stay.
|
||||
|
||||
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
@@ -1313,8 +1397,23 @@ A preset's profile says the card it needs, not a class, so it places on
|
||||
whatever cards the site has. When the cluster's settings name no ComfyUI,
|
||||
adding one sets `engines.comfyui` to the version this release installs, and
|
||||
every node with an NVIDIA GPU installs it; a cluster that names a version,
|
||||
or `""` for none, keeps it. In the console it is **A** on the models screen.
|
||||
The rest of this section is for a model of your own.
|
||||
or `""` for none, keeps it; the control planes then start again on that
|
||||
setting, standbys first.
|
||||
|
||||
They also come by themselves. Once a node has a card that fits one, and the
|
||||
catalog has nothing that makes what it makes, the leader adds it, on demand —
|
||||
where the cluster already installs ComfyUI; it never sets `engines.comfyui`
|
||||
by itself. A preset somebody removes stays out until it is added again.
|
||||
`llmgw ctl model presets auto off` (the setting `presets.auto`) stops it.
|
||||
|
||||
The cluster page on the portal has a **Pictures and 3D models** card under
|
||||
the models: each preset, the classes of card here it fits, whether it is in
|
||||
the catalog and who added it, why the leader has not, an **Add** button (one
|
||||
that would also install ComfyUI says so first) and the switch for adding
|
||||
them by themselves. In the console, the models screen shows the same: **A**
|
||||
adds one, **P** turns adding them by themselves on or off.
|
||||
`llmgw ctl model presets` prints it. The rest of this section is for a model
|
||||
of your own.
|
||||
|
||||
The node must offer the engine. Ask for it in the cluster's settings, and every
|
||||
node with an NVIDIA GPU installs it:
|
||||
@@ -1812,6 +1911,23 @@ same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
Setup asks for them: **Notification webhooks** on its first page, or
|
||||
`--webhook` to `llmgw server init`, as often as there are webhooks:
|
||||
|
||||
```
|
||||
sudo llmgw server init --admin ops@example.com \
|
||||
--webhook https://hooks.slack.com/services/… \
|
||||
--webhook json:https://ops.example.com/llmgw
|
||||
```
|
||||
|
||||
An address of Slack's, or one with `/hooks/` in it (Mattermost, Rocket.Chat)
|
||||
or ending in `/slack`, is sent Slack's `{"text": …}`; any other the event as
|
||||
JSON; `slack:` or `json:` in front says which. They are added to the
|
||||
cluster's settings beside any already there — run again, nothing is added
|
||||
twice — and once the control plane runs on them, each is sent a test, whose
|
||||
answer setup prints. Events to choose and a signing secret are set in the
|
||||
settings afterwards (`llmgw ctl config edit`).
|
||||
|
||||
### Tracing
|
||||
|
||||
Metrics say that a model took four minutes to start; a trace says where they
|
||||
@@ -1845,6 +1961,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
An inference request and each `upstream` attempt also carry the
|
||||
[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
|
||||
so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
|
||||
llmgw's calls as model calls without a dashboard made for llmgw:
|
||||
`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
|
||||
`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
|
||||
when the client set them, `gen_ai.conversation.id` (the session),
|
||||
`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
|
||||
`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
|
||||
engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
|
||||
what was answered are never in a span.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
@@ -2030,8 +2158,46 @@ router:
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
### Similar questions
|
||||
|
||||
A help desk asked "how do I reset my password" in forty wordings asks the
|
||||
model forty times under the exact cache. `--similar` also gives a chat
|
||||
question the answer to one like it:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
|
||||
```
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
embedding_model: bge-m3 # an embedding model in the catalog
|
||||
```
|
||||
|
||||
- **What is compared:** the last user message, embedded by
|
||||
`router.cache.embedding_model`, against the questions already answered
|
||||
in the same conversation — the same system prompt, the same earlier turns,
|
||||
tools, `response_format` and sampling settings, the same model and
|
||||
project. A like question in another conversation, or after a tool's
|
||||
result, is not taken for this one. The exact cache is asked first, and an
|
||||
exact hit embeds nothing.
|
||||
- **How alike:** the cosine similarity of the two embeddings, at least the
|
||||
project's `--similar`. 0.95 is strict; below 0.9 different questions start
|
||||
to meet. A question that differs only in a number or a name ("2+2" and
|
||||
"2+3") can be closer than that, so a project whose questions turn on
|
||||
figures should not use it. A question over 2,000 characters is reused only
|
||||
when it is the same.
|
||||
- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
|
||||
and `Age`. Recorded and charged as a hit. The same rules as a hit decide
|
||||
which answers are reused and kept; guardrails judge the request before.
|
||||
- **The embedding** is asked of the model by the gateway on its own account,
|
||||
as a judge is, within 2 seconds; every miss waits for it. Without the
|
||||
model, or when it does not answer, only the same request is reused. The
|
||||
vectors are kept in memory with their answers and counted in
|
||||
`router.cache.max_bytes`.
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
|
||||
and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
@@ -2287,6 +2453,33 @@ Without authentication, `threads.store` and `threads.retain_hours` in
|
||||
`server.yaml` keep both conversations and notes in one shared pool. That is for
|
||||
one trusted machine only; see [development.md](development.md).
|
||||
|
||||
## Three machines, each a control plane
|
||||
|
||||
The screens' three machines are high availability for the gateway as well as
|
||||
the store: the two database machines are made control planes too, standing
|
||||
by while the first leads — what `server add`, below, does for one machine.
|
||||
By command, `--standbys` with `--patroni`, and `--vip` for the floating
|
||||
address:
|
||||
|
||||
```
|
||||
sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --admin you@example.com \
|
||||
--patroni ubuntu@10.0.0.6 --patroni ubuntu@10.0.0.7 \
|
||||
--standbys --vip 10.0.0.160/24 --no-node --no-model
|
||||
```
|
||||
|
||||
| Machine | Runs |
|
||||
|---|---|
|
||||
| the first | control plane, leading at first; model mirror; etcd |
|
||||
| the other two | control plane, standing by; PostgreSQL under Patroni; etcd |
|
||||
|
||||
etcd votes on all three and PostgreSQL is on two, so any one machine can be
|
||||
lost: a standby leads within seconds, Patroni promotes the standby database
|
||||
if it was the primary, and the floating address follows whichever leads. The
|
||||
mirror's management API answers the three control planes' addresses, not
|
||||
only its own machine's, so a standby that comes to lead can still fetch and
|
||||
add models. The mirror itself stays on the first machine: while it is down,
|
||||
models the nodes already have keep serving, and new ones wait for it.
|
||||
|
||||
## Two control planes
|
||||
|
||||
Once the store is replicated (`server init --patroni`, above), one command on
|
||||
@@ -2303,8 +2496,12 @@ and lets it into the store; it starts there as a standby. Every node is told
|
||||
of it within half a minute and fails over to it without being touched.
|
||||
`--vip` keeps a floating address on whichever control plane leads, with
|
||||
keepalived on both: that is the address to give clients, or a proxy in front.
|
||||
Without it, put a load balancer in front that checks `/ready` — not
|
||||
`/health`, which is 200 on a standby that would refuse every request.
|
||||
Without it, put a proxy in front that sends to whichever answers `/leader`
|
||||
with 200 — not `/health`, which is 200 on a standby too, and not `/ready`,
|
||||
which is 503 on a leader with no model yet and would take the portal away
|
||||
with it. The end of [deploy/proxy/Caddyfile](../deploy/proxy/Caddyfile) has
|
||||
Caddy doing that for three control planes, and
|
||||
[deploy/ha/](../deploy/ha/README.md#check-leader-not-health) why.
|
||||
|
||||
- **What a failover costs.** Measured on the reference cluster, stopping the
|
||||
leader: the standby led within five seconds, every node had moved to it
|
||||
@@ -2365,6 +2562,79 @@ the same files and keys as `locales/en/`, and its `_meta.*` entries saying how
|
||||
it writes numbers and dates. The tests refuse a catalogue that lacks a key a
|
||||
page uses.
|
||||
|
||||
## The company's MCP servers
|
||||
|
||||
llmgw's `/mcp` can serve the company's own MCP servers beside its own tools.
|
||||
Each server, a ticket system, a file server or a database, is registered once.
|
||||
Its credentials stay on the control planes, and people's agents are given only
|
||||
`/mcp` and their llmgw key.
|
||||
|
||||
```
|
||||
llmgw ctl mcp add jira --url https://jira.internal/mcp \
|
||||
--description "Acme Jira" --secret-file Authorization=/root/jira-token
|
||||
llmgw ctl mcp test jira # its tools, asked with its credentials
|
||||
llmgw keys limits --scope team --id Acme/support --tools 'jira.*'
|
||||
```
|
||||
|
||||
- **Credentials:**
|
||||
- `--secret-file NAME=FILE` reads a header's value once and keeps it in
|
||||
the store, sealed by the master key, so no file has to be on each
|
||||
control plane.
|
||||
- `--header-ref NAME=REF` reads it at each call from a file, `env:`,
|
||||
`vault:` or `store:` instead.
|
||||
- `--header NAME=VALUE` is for what is not secret.
|
||||
- **Who may call what:** tools are named `server.tool` and granted as
|
||||
patterns (`jira.*`, `db.query`) at any level of the tenancy tree, as
|
||||
models are.
|
||||
- A level that names any narrows what is above it.
|
||||
- Unlike models, **none granted anywhere is none**: a tool reaches into a
|
||||
company system, so it is granted, never assumed.
|
||||
- The console's limits form has the same field.
|
||||
- **The server is told who calls:** `X-Llmgw-Key`, and `X-Llmgw-User` and
|
||||
`X-Llmgw-Project` where there are some. It is called with llmgw's
|
||||
credentials, never the caller's key.
|
||||
- **Bounds:** a call has 60 seconds (`--timeout`) and an answer 4 MB. A
|
||||
server's progress is passed on to a client that takes a stream. `--rate N`
|
||||
allows each caller N calls a minute to a server's tools.
|
||||
- **Guardrails, both ways:** the caller's policies read a tool's answer as
|
||||
the prompt it becomes.
|
||||
- An instruction planted in a ticket is caught as a pasted one would be
|
||||
(`injection: block`).
|
||||
- Personal data is redacted or refused as the policy says. A refused
|
||||
answer does not reach the agent.
|
||||
- The arguments are read before they leave, so a secret pasted into one
|
||||
does not reach the company system.
|
||||
- **A server that keeps failing:** five calls failing in a row take a server
|
||||
out for a minute. Its tools leave the list, and a call is answered at once.
|
||||
- **Record:** every call is in `llmgw ctl audit`, with its arguments' size
|
||||
and a digest, never their text. `llmgw_mcp_tool_calls_total` and
|
||||
`llmgw_mcp_tool_call_seconds` count them by server, tool and outcome.
|
||||
- **Changes:** a server added, changed or removed on one control plane is
|
||||
served by every one within half a minute; its tool list is read again
|
||||
every five minutes. `--disabled` keeps a server but serves none of its
|
||||
tools.
|
||||
|
||||
**This cluster's own, read only.** `llmgw ctl mcp add llmgw --url
|
||||
llmgw:admin` registers the cluster itself as a server: tools that say what
|
||||
`llmgw ctl` does, and change nothing. It is answered in the control plane's
|
||||
own process, with no address and no token to give.
|
||||
|
||||
| Tool | Says |
|
||||
|---|---|
|
||||
| `llmgw.cluster_status` | control planes, store, nodes, models against what they asked for, GPUs, what is unhealthy |
|
||||
| `llmgw.list_nodes` | each node's state, GPUs, load, fabrics, agent version and engines |
|
||||
| `llmgw.list_models` | each model's replicas ready against wanted, profiles, why one is parked |
|
||||
| `llmgw.list_replicas` | each replica's shards: node, GPUs, state (`model` for one model's) |
|
||||
| `llmgw.explain_placement` | why a model is not placed, node by node |
|
||||
| `llmgw.node_logs` | a node's last agent log lines, or an engine's (`source`) |
|
||||
|
||||
Grant it as any other: `llmgw keys limits --scope key --id KEY-ID --tools
|
||||
'llmgw.*'`. It is the same as an administrator's view, so grant it to the
|
||||
people who would have `ctl`.
|
||||
|
||||
Servers over HTTP only for now (the streamable HTTP transport). A server
|
||||
that runs as a local process (stdio) is planned.
|
||||
|
||||
## Single sign-on
|
||||
|
||||
People can sign in through your identity provider — Keycloak, Entra ID,
|
||||
@@ -2372,8 +2642,10 @@ Google, Okta, Authentik, anything that speaks OpenID Connect.
|
||||
|
||||
1. Register a client there with the redirect address
|
||||
`https://<gateway>/login/oidc/callback`.
|
||||
2. Put its secret in a file only root reads.
|
||||
3. Add to `server.yaml`:
|
||||
2. Put its secret in a file the llmgw user reads, on every control plane.
|
||||
3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
|
||||
cluster with a store, `auth` is the cluster's: the same block in
|
||||
`server.yaml` is not read, and the log says so.
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
@@ -2422,6 +2694,99 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
|
||||
join and leave them. Membership in a group is removed when someone leaves
|
||||
it.
|
||||
|
||||
### Entra ID
|
||||
|
||||
The console sets this up: `sudo llmgw`, People → Sign-in, **E**. What
|
||||
follows is what it writes, and what to do in Entra.
|
||||
|
||||
Entra ID is an OpenID Connect provider like the others, with two things to
|
||||
set in the app registration:
|
||||
|
||||
1. **Redirect URI:** under Authentication, add a Web platform with
|
||||
`https://<gateway>/login/oidc/callback`. The issuer is
|
||||
`https://login.microsoftonline.com/<tenant-id>/v2.0`.
|
||||
2. **Groups in the token:** under Token configuration, add a groups claim.
|
||||
Choose **Groups assigned to the application**, and assign the groups you
|
||||
map in `roles` and `memberships` to the enterprise application. Entra
|
||||
sends groups by object id unless you choose to emit names
|
||||
(`sAMAccountName` for groups synced from AD), so write them the same way
|
||||
in the mapping.
|
||||
|
||||
Entra leaves the groups out of the token for someone in more than 200 of
|
||||
them and sends a pointer to Graph instead. llmgw then keeps that person's
|
||||
role and memberships as they were, rather than reading it as no groups, and
|
||||
logs an error saying so. "Groups assigned to the application" avoids this.
|
||||
So does SCIM below, which Entra also speaks: in the enterprise application
|
||||
under Provisioning, set the tenant URL to `https://<gateway>/scim/v2` and
|
||||
the secret token to the contents of `auth.scim.token_file`.
|
||||
|
||||
## Active Directory
|
||||
|
||||
With an on-premises Active Directory, people sign in on the gateway's own
|
||||
form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
|
||||
password. The gateway checks them with the domain controllers over LDAPS.
|
||||
|
||||
The console sets this up and tries it first: `sudo llmgw`, People →
|
||||
Sign-in, **D**. It keeps the service account's password in the store
|
||||
(`bind_password_file: store:auth/ldap-bind`) and writes the CA into the
|
||||
settings (`ca:`), so nothing has to be copied to each control plane.
|
||||
By hand it is this block in the cluster's settings (`llmgw ctl config
|
||||
edit`), where files it names must then be on every control plane:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
enabled: true
|
||||
ldap:
|
||||
urls: [ldaps://dc1.corp.example, ldaps://dc2.corp.example] # tried in order
|
||||
ca_file: /etc/llmgw/corp-ca.pem # the domain's CA, when not in the system's
|
||||
bind_dn: CN=svc-llmgw,OU=Service,DC=corp,DC=example
|
||||
bind_password_file: /etc/llmgw/secrets/ldap-bind # root-only
|
||||
base_dn: OU=Staff,DC=corp,DC=example # where people are
|
||||
user_filter: (memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)
|
||||
label: Acme # "Sign in with your Acme account"
|
||||
create_accounts: true
|
||||
roles: # by group CN or full DN; nested groups count
|
||||
admin: [llmgw-admins]
|
||||
operator: [llmgw-ops]
|
||||
memberships:
|
||||
ml-engineers: "team:Acme/ml"
|
||||
sync_interval: 15m # negative turns the sync off
|
||||
```
|
||||
|
||||
- **Service account:** an ordinary domain user is enough: it only reads.
|
||||
It finds the person and reads their groups. The person's password is
|
||||
checked by binding as them, and is never stored.
|
||||
- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
|
||||
Plaintext LDAP is refused, since passwords would cross the network.
|
||||
- **Certificate:** the domain controller's certificate must name it in
|
||||
its subject alternative names: a name in the CN only is refused, as are
|
||||
certificates with a negative serial number. Samba's own autogenerated
|
||||
certificate is both, so a Samba DC needs one from a CA (`tls certfile`
|
||||
in smb.conf). A CA file a control plane cannot read turns directory
|
||||
sign-in off on that control plane, said in its log. Local accounts still
|
||||
sign in.
|
||||
- **Who may sign in:** enabled user objects under `base_dn` that match
|
||||
`user_filter`. The example admits members of `llmgw-users`, nested groups
|
||||
included.
|
||||
- **Accounts:** an account is tied to the person's `objectGUID`, so a changed
|
||||
mail or name is still the same account. An account that already exists
|
||||
with the person's mail is taken over. Without `create_accounts`, only
|
||||
people who already have an account can sign in. Local accounts, such as
|
||||
the administrator `server init` made, keep signing in with their own
|
||||
password.
|
||||
- **Sync:** every `sync_interval` the leader reads the directory.
|
||||
- Someone disabled or deleted in AD is suspended and their sessions
|
||||
end.
|
||||
- When they are enabled again they are reinstated. An account an
|
||||
administrator suspended by hand stays suspended.
|
||||
- Roles follow groups, and memberships the directory gave are taken away
|
||||
when the person leaves the group.
|
||||
- A sync that would suspend more than half of the directory's accounts is
|
||||
refused and logged, rather than locking everyone out over a wrong
|
||||
`base_dn` or filter.
|
||||
- **Passwords:** a person's password is changed in AD. The gateway sends no
|
||||
reset link for these accounts.
|
||||
|
||||
## Secrets in Vault
|
||||
|
||||
Every setting that names a file holding a secret — the store's `@address`,
|
||||
@@ -2993,6 +3358,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
|
||||
- `docs/choosing-models.md` — which models to run and how many, if the answer
|
||||
to "we have GPUs, now what" is not already obvious
|
||||
- `docs/large-models.md` — a model on several cards or several nodes: how it
|
||||
is split, what each way needs of a card, and bringing one up on a fabric
|
||||
- `docs/data.md` — what is stored and what leaves the machine, for the review
|
||||
that happens before this is allowed near anyone's documents
|
||||
- `docs/reference.md` — every configuration key, generated from the source, so
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
# llmgw and Kong
|
||||
|
||||
For a site that already runs Kong. The two meet over plain HTTP: Kong in
|
||||
front of llmgw, as one more API it governs, or Kong behind llmgw, as an
|
||||
OpenAI-compatible provider. Nothing of either is built into the other.
|
||||
|
||||
Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
|
||||
and later need a Kong licence) in DB-less mode, in front of llmgw.
|
||||
|
||||
## Kong in front of llmgw
|
||||
|
||||
Two ways, both tried.
|
||||
|
||||
**A plain route.** Kong forwards the request as it came. Tried with chat
|
||||
completions, streamed and not; llmgw's other APIs — Anthropic's
|
||||
`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
|
||||
since Kong does not read them. Kong adds its own authentication, rate limits
|
||||
and logging; llmgw still checks the key, the budget and the guardrails.
|
||||
|
||||
```yaml
|
||||
_format_version: "3.0"
|
||||
services:
|
||||
- name: llmgw
|
||||
url: https://llm.example.com # llmgw's client port
|
||||
routes:
|
||||
- name: llmgw
|
||||
paths: [/llmgw] # /llmgw/v1/chat/completions → /v1/chat/completions
|
||||
```
|
||||
|
||||
**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
|
||||
route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
- name: llmgw-chat
|
||||
url: https://llm.example.com
|
||||
routes:
|
||||
- name: chat
|
||||
paths: [/chat]
|
||||
plugins:
|
||||
- name: ai-proxy
|
||||
config:
|
||||
route_type: llm/v1/chat
|
||||
logging: {log_statistics: true, log_payloads: false}
|
||||
model:
|
||||
provider: openai
|
||||
name: qwen-coder # a model llmgw serves
|
||||
options:
|
||||
upstream_url: https://llm.example.com/v1/chat/completions
|
||||
```
|
||||
|
||||
**Keys.** Without an `auth` block, `ai-proxy` passes the client's
|
||||
`Authorization` header on: each team keeps its own llmgw key, and llmgw's
|
||||
budgets, limits, guardrails and usage stay per team. With
|
||||
`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
|
||||
sends one key of its own instead, and llmgw sees every caller as that key —
|
||||
then budgets per team are Kong's to keep.
|
||||
|
||||
**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
|
||||
records prompt and completion tokens as llmgw did. For a streamed answer
|
||||
Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
|
||||
even with `stream_options.include_usage` set — the usage chunk reaches it
|
||||
and is not read. llmgw's own accounting counts both; use its usage export
|
||||
(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
|
||||
|
||||
**Timeouts.** A long answer from a model still loading outlasts Kong's
|
||||
60-second defaults: set the service's `read_timeout` to the time a model may
|
||||
take to start, as for nginx in [install.md](install.md).
|
||||
|
||||
## Kong behind llmgw
|
||||
|
||||
Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
|
||||
it reaches any provider: a model whose source is `external`.
|
||||
|
||||
```yaml
|
||||
- name: kong-gpt
|
||||
source: external
|
||||
external:
|
||||
url: https://kong.example.com/openai # Kong's route, without /v1
|
||||
model: gpt-4o-mini
|
||||
api_key_file: /etc/llmgw/secrets/kong-consumer.key
|
||||
```
|
||||
|
||||
The key is the one Kong's own authentication asks of a consumer. As a
|
||||
fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
|
||||
room.
|
||||
|
||||
## Which to choose
|
||||
|
||||
Kong in front when the site's API policy says every API goes through Kong:
|
||||
llmgw is then one more upstream, and its per-team accounting still holds
|
||||
when each team keeps its own key. Kong behind when the site already reaches
|
||||
outside providers through Kong and llmgw should too. llmgw does not need
|
||||
Kong for anything it does itself — keys, budgets, guardrails, routing to
|
||||
its own replicas.
|
||||
@@ -0,0 +1,145 @@
|
||||
# Large models: several cards, several nodes
|
||||
|
||||
A model too big for one card runs on several. llmgw chooses how from the
|
||||
model's own figures and the cluster as it is; this page is the rules it
|
||||
follows, the commands, and what to look at when a first one is brought up
|
||||
on new hardware.
|
||||
|
||||
## How a model is split
|
||||
|
||||
In order, the first that fits:
|
||||
|
||||
1. **One card.** Nothing to split.
|
||||
2. **Several cards of one node, tensor parallel.** Every layer is cut
|
||||
across the cards, so they talk for every token: they must be on one
|
||||
NVLink domain. The agent reads `nvidia-smi topo -m` and gives a replica
|
||||
the cards with the most NVLink between them
|
||||
([install.md](install.md#nvlink-if-your-gpus-have-it)).
|
||||
3. **Several nodes, pipeline parallel between them.** Each node holds a
|
||||
run of layers on all its cards (tensor parallel inside it), and only a
|
||||
layer's output crosses to the next node. The nodes must share one
|
||||
InfiniBand or RoCE fabric: a replica never spans two, and never a node
|
||||
with none. Any number of nodes, not only two or four.
|
||||
|
||||
The plan takes the fewest cards, then the fewest nodes, then the better
|
||||
precision up to eight bits.
|
||||
|
||||
**Tensor width and the KV heads.** The cards split the model's KV heads
|
||||
among them, or past their number each card holds a copy of one. So the
|
||||
width divides the KV heads or is a multiple of them: a model with 8 KV
|
||||
heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8.
|
||||
|
||||
**Mixture of experts.** A model of experts (Qwen3-235B-A22B, DeepSeek-V3,
|
||||
Llama 4, Mixtral) on more than one card is served with vLLM's
|
||||
`--enable-expert-parallel`: each card holds whole experts rather than a
|
||||
slice of every one, which is less traffic per token. It is read from
|
||||
`config.json` (`num_experts`, `n_routed_experts`, `num_local_experts`).
|
||||
|
||||
**When it does not fit.** A smaller precision first: an FP8 or AWQ
|
||||
repository of the same model, or a GGUF quantisation on llama.cpp if one
|
||||
node's cards hold it. Then more nodes. `llmgw ctl model plan REPO
|
||||
--precision bf16,fp8` shows each precision's fit from the model's config.
|
||||
|
||||
## What a way needs
|
||||
|
||||
Each way the plan offers says what a replica needs of each card, part by
|
||||
part, and of all its cards together:
|
||||
|
||||
```
|
||||
$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768
|
||||
FILE SIZE BEST WAY A CARD OF ALL CARDS NOW
|
||||
original 131 GB 4 × h100-80g 37.4 GB 72 GB 150 GB free
|
||||
|
||||
Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards.
|
||||
The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another).
|
||||
```
|
||||
|
||||
- **weights:** the files, divided over the cards.
|
||||
- **context:** the KV cache for the context planned, divided likewise.
|
||||
- **engine:** the engine's own: 2 GB and a fiftieth of the weights for vLLM.
|
||||
- **usable:** 90% of a card; the rest is the driver's and fragmentation's.
|
||||
|
||||
The console's **Serve a model** shows the same for every way a file runs.
|
||||
A context longer than planned is checked against it before anything is
|
||||
fetched.
|
||||
|
||||
## Serving one
|
||||
|
||||
Without flags the plan's best way is taken:
|
||||
|
||||
```
|
||||
llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768
|
||||
```
|
||||
|
||||
Or a way given by hand, for instance two nodes of eight:
|
||||
|
||||
```
|
||||
llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \
|
||||
--gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO
|
||||
```
|
||||
|
||||
`--gpus` is the cards on each node and `--nodes` the nodes. The pipeline
|
||||
is the nodes and the tensor width the cards of each.
|
||||
|
||||
`--env KEY=VALUE` goes into the engine's environment on every node:
|
||||
`NCCL_DEBUG=INFO` while bringing a cluster up, `NCCL_IB_*` to steer it.
|
||||
Only the variables engines are tuned with are taken. llmgw already sets
|
||||
these:
|
||||
- `NCCL_IB_HCA` to the fabric's adapters;
|
||||
- `NCCL_IB_GID_INDEX` on RoCE;
|
||||
- `NCCL_SOCKET_IFNAME` to the management network;
|
||||
- for an NVLink domain spanning nodes, `NCCL_MNNVL_ENABLE`.
|
||||
|
||||
The job's first step says what it needs of each card. The weights then
|
||||
come to the mirror once and from it to each node. The startup timeout (20
|
||||
minutes unless the profile's `startup_timeout` says otherwise) runs from
|
||||
when a node's download last moved. So a model of hundreds of gigabytes is
|
||||
not failed while it is still arriving, and one whose download stops is
|
||||
failed that long after.
|
||||
|
||||
## Watching it come up
|
||||
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| The replica and its shards, one per node, each with its download | `llmgw ctl replicas` |
|
||||
| Why it is not placed | `llmgw ctl explain-placement MODEL` |
|
||||
| A node's engine output (NCCL's lines with `NCCL_DEBUG=INFO`) | `llmgw ctl logs NODE --source SHARD` |
|
||||
| The bandwidth between two nodes, rail by rail | `llmgw ctl fabric-test NODE NODE` |
|
||||
| Each node's GPUs, NVLink and adapters | `llmgw doctor` on the node; its page under /admin |
|
||||
| Every step of a start, across nodes | the `replica start` trace ([install.md](install.md)) |
|
||||
|
||||
With `NCCL_DEBUG=INFO`, NCCL says which transport it took between nodes:
|
||||
`NET/IB` is the fabric. `NET/Socket` means it fell back to TCP, and the
|
||||
model will run, slowly.
|
||||
|
||||
## Bringing it up on new hardware
|
||||
|
||||
A first run on a fabric, in order. Each step is worth its output if
|
||||
something goes wrong.
|
||||
|
||||
1. **The nodes.** Add each with its fabric:
|
||||
`llmgw node add host --fabric ib-a:mlx5_0,mlx5_1`. Then
|
||||
`llmgw ctl nodes` shows every node ready, on `ib-a`.
|
||||
2. **The fabric.** `llmgw ctl fabric-test node1 node2`: each rail at its
|
||||
rated speed (a 400 Gb/s rail near 45 GB/s).
|
||||
3. **One node, several cards.** A dense 70B at bf16 on four or eight
|
||||
cards of one node, by the plan:
|
||||
`llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors`.
|
||||
Ready, then a question answered.
|
||||
4. **Two nodes.** The same model forced over two:
|
||||
`--gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO`, under
|
||||
another `--name`. Ready, answering, and `NET/IB` in both shards' logs.
|
||||
5. **Experts.** A mixture-of-experts model on one node's eight cards,
|
||||
for instance `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`: its profile
|
||||
carries `--enable-expert-parallel` (`llmgw ctl models`, or its
|
||||
definition from the admin API).
|
||||
6. **A node lost.** With the two-node replica serving, stop one node's
|
||||
agent (`systemctl stop llmgw-agent`): the replica fails and is placed
|
||||
again once the node is back or another has room. Nothing is left
|
||||
holding cards.
|
||||
|
||||
For each, worth keeping:
|
||||
- the time from serve to ready;
|
||||
- tokens a second from `llmgw ctl bench`;
|
||||
- the plan's per-card figure against what `nvidia-smi` shows once loaded;
|
||||
- anything it said that was wrong.
|
||||
+13
-3
@@ -54,10 +54,20 @@ on the page that creates it and never again.
|
||||
|
||||
### Connecting an editor
|
||||
|
||||
The account page's **Connect your editor** makes a key for Continue, Codex or
|
||||
Cline and shows what the tool needs with that key, this gateway's address and a
|
||||
model already in it:
|
||||
The account page's **Connect your editor** makes a key for VS Code, Continue,
|
||||
Codex or Cline and shows what the tool needs with that key, this gateway's
|
||||
address and a model already in it:
|
||||
|
||||
- **VS Code** — llmgw's own extension, downloaded from the gateway at
|
||||
`/setup/vscode/llmgw.vsix` with this gateway's address already in it. It
|
||||
offers every chat model the key may use to VS Code itself: the chat's model
|
||||
picker, agent mode, and any extension that asks VS Code for a language model
|
||||
(Cline's "VS Code LM API" provider among them), with no setup of their own.
|
||||
**llmgw: Sign In** in VS Code opens this page with VS Code chosen; the page's
|
||||
**Open in VS Code** link hands the key back, and the extension keeps it in
|
||||
VS Code's secret storage. A link the extension did not ask for is confirmed
|
||||
first, with the gateway's address shown, since it decides where code is sent.
|
||||
**llmgw: Enter API Key** takes a pasted one instead.
|
||||
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
|
||||
so their `config.yaml` is not touched.
|
||||
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
|
||||
|
||||
@@ -8,7 +8,7 @@ explains every choice.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 12 GB RAM when it is also a node running the starter model and the two judges, which setup checks before it installs anything; four cores or more, or the judges on machines of their own, keep a judgment within its wait. |
|
||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
||||
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
||||
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
||||
|
||||
+330
-1
@@ -37,6 +37,7 @@ rather than repeated.
|
||||
| `default_tpm` | `int` | [Auth](#auth) |
|
||||
| `signup_open` | `bool` | [Auth](#auth) |
|
||||
| `oidc` | `OIDC` | [Auth](#auth) |
|
||||
| `ldap` | `LDAP` | [Auth](#auth) |
|
||||
| `scim` | `SCIM` | [Auth](#auth) |
|
||||
| `max_key_ttl` | `time.Duration` | [Auth](#auth) |
|
||||
| `personal` | `Personal` | [Auth](#auth) |
|
||||
@@ -52,6 +53,7 @@ rather than repeated.
|
||||
| `retain_days` | `int` | [Batch](#batch) |
|
||||
| `max_bytes` | `int64` | [Cache](#cache) |
|
||||
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
||||
| `embedding_model` | `string` | [Cache](#cache) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
@@ -105,10 +107,13 @@ rather than repeated.
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `api_key_files` | `[]ExternalKey` | [External](#external) |
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `output` | `string` | [External](#external) |
|
||||
| `file` | `string` | [ExternalKey](#externalkey) |
|
||||
| `weight` | `float64` | [ExternalKey](#externalkey) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -144,6 +149,21 @@ rather than repeated.
|
||||
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
||||
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `max_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `urls` | `[]string` | [LDAP](#ldap) |
|
||||
| `start_tls` | `bool` | [LDAP](#ldap) |
|
||||
| `ca_file` | `string` | [LDAP](#ldap) |
|
||||
| `ca` | `string` | [LDAP](#ldap) |
|
||||
| `bind_dn` | `string` | [LDAP](#ldap) |
|
||||
| `bind_password_file` | `string` | [LDAP](#ldap) |
|
||||
| `base_dn` | `string` | [LDAP](#ldap) |
|
||||
| `group_base_dn` | `string` | [LDAP](#ldap) |
|
||||
| `user_filter` | `string` | [LDAP](#ldap) |
|
||||
| `label` | `string` | [LDAP](#ldap) |
|
||||
| `create_accounts` | `bool` | [LDAP](#ldap) |
|
||||
| `roles` | `map[string][]string` | [LDAP](#ldap) |
|
||||
| `memberships` | `map[string]string` | [LDAP](#ldap) |
|
||||
| `sync_interval` | `time.Duration` | [LDAP](#ldap) |
|
||||
| `timeout` | `time.Duration` | [LDAP](#ldap) |
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `language` | `string` | [Landing](#landing) |
|
||||
@@ -205,6 +225,8 @@ rather than repeated.
|
||||
| `no_passwords` | `bool` | [OIDC](#oidc) |
|
||||
| `mirror` | `string` | [Offline](#offline) |
|
||||
| `checksums` | `string` | [Offline](#offline) |
|
||||
| `evict_after_idle` | `time.Duration` | [OnDemandSettings](#ondemandsettings) |
|
||||
| `min_lifetime` | `time.Duration` | [OnDemandSettings](#ondemandsettings) |
|
||||
| `nvidia_smi` | `string` | [Options](#options) |
|
||||
| `ib_root` | `string` | [Options](#options) |
|
||||
| `prefer_nvml` | `bool` | [Options](#options) |
|
||||
@@ -236,6 +258,7 @@ rather than repeated.
|
||||
| `require_capabilities` | `bool` | [Policy](#policy) |
|
||||
| `validate_structured` | `bool` | [Policy](#policy) |
|
||||
| `judge` | `*JudgePolicy` | [Policy](#policy) |
|
||||
| `services` | `[]Service` | [Policy](#policy) |
|
||||
| `stream` | `string` | [Policy](#policy) |
|
||||
| `enabled` | `bool` | [Prefix](#prefix) |
|
||||
| `max_nodes` | `int` | [Prefix](#prefix) |
|
||||
@@ -244,6 +267,7 @@ rather than repeated.
|
||||
| `match_ratio` | `float64` | [Prefix](#prefix) |
|
||||
| `balance_abs` | `float64` | [Prefix](#prefix) |
|
||||
| `balance_rel` | `float64` | [Prefix](#prefix) |
|
||||
| `auto` | `bool` | [PresetSettings](#presetsettings) |
|
||||
| `source` | `string` | [PreviousOrigin](#previousorigin) |
|
||||
| `revision` | `string` | [PreviousOrigin](#previousorigin) |
|
||||
| `files` | `[]OriginFile` | [PreviousOrigin](#previousorigin) |
|
||||
@@ -298,6 +322,7 @@ rather than repeated.
|
||||
| `injection` | `string` | [Rules](#rules) |
|
||||
| `deny` | `[]Pattern` | [Rules](#rules) |
|
||||
| `max_chars` | `int` | [Rules](#rules) |
|
||||
| `restore` | `bool` | [Rules](#rules) |
|
||||
| `token_file` | `string` | [SCIM](#scim) |
|
||||
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
||||
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
||||
@@ -344,6 +369,8 @@ rather than repeated.
|
||||
| `mail` | `mail.Config` | [Server](#server) |
|
||||
| `serving` | `ServingSettings` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `presets` | `PresetSettings` | [Server](#server) |
|
||||
| `on_demand` | `OnDemandSettings` | [Server](#server) |
|
||||
| `agents` | `AgentSettings` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
@@ -356,6 +383,15 @@ rather than repeated.
|
||||
| `facility` | `Facility` | [Server](#server) |
|
||||
| `federation` | `Federation` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `name` | `string` | [Service](#service) |
|
||||
| `url` | `string` | [Service](#service) |
|
||||
| `header_files` | `map[string]string` | [Service](#service) |
|
||||
| `body` | `string` | [Service](#service) |
|
||||
| `blocked` | `string` | [Service](#service) |
|
||||
| `reason` | `string` | [Service](#service) |
|
||||
| `on` | `string` | [Service](#service) |
|
||||
| `action` | `string` | [Service](#service) |
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
@@ -389,6 +425,7 @@ rather than repeated.
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `cached_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
@@ -427,6 +464,7 @@ rather than repeated.
|
||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `cached_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
@@ -611,6 +649,15 @@ OIDC is sign-in through an identity provider.
|
||||
|
||||
Takes a [OIDC](#oidc) block.
|
||||
|
||||
### `ldap`
|
||||
|
||||
`LDAP`
|
||||
|
||||
LDAP is sign-in against a directory, Active Directory above all, and
|
||||
the accounts kept in step with it.
|
||||
|
||||
Takes a [LDAP](#ldap) block.
|
||||
|
||||
### `scim`
|
||||
|
||||
`SCIM`
|
||||
@@ -723,6 +770,15 @@ Past it, the answers used longest ago go first.
|
||||
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
unless set.
|
||||
|
||||
### `embedding_model`
|
||||
|
||||
`string`
|
||||
|
||||
EmbeddingModel embeds the questions of projects that reuse answers to
|
||||
similar ones (llmgw keys cache --similar): an embedding model in the
|
||||
catalog. Unset, those projects reuse only the same request's answer.
|
||||
The gateway asks it on its own account, as it asks a judge.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
@@ -943,7 +999,7 @@ work stands without being renewed.
|
||||
Only meaningful with a shared store, where a second control plane can
|
||||
see the claim. Long enough to survive a slow query or a brief network
|
||||
stall, short enough that a standby takes over before anyone has finished
|
||||
reading the alert. Zero takes the default.
|
||||
reading the alert. Zero takes the default, 10s.
|
||||
|
||||
## CustomNode
|
||||
|
||||
@@ -1098,6 +1154,15 @@ APIKeyFile holds the provider's key, on the control plane. A file
|
||||
rather than the key itself, so the key is in neither the catalog, the
|
||||
store nor the audit log; empty for a provider that asks for none.
|
||||
|
||||
### `api_key_files`
|
||||
|
||||
`[]ExternalKey`
|
||||
|
||||
APIKeyFiles are several keys for the provider, in place of
|
||||
APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
|
||||
answers 429 or 401 is passed over while its breaker is open and the
|
||||
request goes to another. A provider's limits are per key.
|
||||
|
||||
### `peer`
|
||||
|
||||
`string`
|
||||
@@ -1130,6 +1195,20 @@ plane.
|
||||
|
||||
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
|
||||
## ExternalKey
|
||||
|
||||
### `file`
|
||||
|
||||
`string`
|
||||
|
||||
File holds the key, on the control plane, as APIKeyFile does.
|
||||
|
||||
### `weight`
|
||||
|
||||
`float64`
|
||||
|
||||
Weight is its share of the requests against the others'; zero is 1.
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
@@ -1396,6 +1475,114 @@ MaxBlocks bounds how many of a prompt's blocks are hashed and looked
|
||||
up. Beyond it a longer match still helps, but the difference no longer
|
||||
decides anything. Zero takes 4096.
|
||||
|
||||
## LDAP
|
||||
|
||||
### `urls`
|
||||
|
||||
`[]string`
|
||||
|
||||
URLs are the directory's servers, tried in turn: ldaps:// (636), or
|
||||
ldap:// (389) with StartTLS.
|
||||
|
||||
### `start_tls`
|
||||
|
||||
`bool`
|
||||
|
||||
StartTLS upgrades an ldap:// connection before anything is sent. A
|
||||
password never crosses a connection without TLS: ldap:// without it
|
||||
is refused.
|
||||
|
||||
### `ca_file`
|
||||
|
||||
`string`
|
||||
|
||||
CAFile is the authority the servers' certificates are checked against,
|
||||
when it is not the system's: an AD Certificate Services root.
|
||||
|
||||
### `ca`
|
||||
|
||||
`string`
|
||||
|
||||
CA is the same authority written out, PEM: kept in the cluster's
|
||||
settings, so no file has to be put on every control plane. Used in
|
||||
place of CAFile when both are set.
|
||||
|
||||
### `bind_dn`
|
||||
|
||||
`string`
|
||||
|
||||
BindDN and the file holding its password: a service account that may
|
||||
read users and groups. Nothing is written to the directory.
|
||||
|
||||
### `bind_password_file`
|
||||
|
||||
`string`
|
||||
|
||||
### `base_dn`
|
||||
|
||||
`string`
|
||||
|
||||
BaseDN is where people are searched for, subtree included.
|
||||
|
||||
### `group_base_dn`
|
||||
|
||||
`string`
|
||||
|
||||
GroupBaseDN is where their groups are; empty is BaseDN's domain root.
|
||||
|
||||
### `user_filter`
|
||||
|
||||
`string`
|
||||
|
||||
UserFilter narrows who may sign in, ANDed with the login match: a
|
||||
group (memberOf=...) or an OU's people only. Empty is every enabled
|
||||
user object.
|
||||
|
||||
### `label`
|
||||
|
||||
`string`
|
||||
|
||||
Label is what the form says to sign in with: "Sign in with your
|
||||
<label> account". Empty is "Active Directory".
|
||||
|
||||
### `create_accounts`
|
||||
|
||||
`bool`
|
||||
|
||||
CreateAccounts makes an account, active, for a person the directory
|
||||
vouches for and the gateway does not know. Off, only existing accounts
|
||||
sign in this way.
|
||||
|
||||
### `roles`
|
||||
|
||||
`map[string][]string`
|
||||
|
||||
Roles maps a staff role (viewer, operator, billing, admin) to the
|
||||
groups that give it, by name (CN) or whole DN. Set at every sign-in
|
||||
and sync: someone taken out of the group loses the role.
|
||||
|
||||
### `memberships`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Memberships maps a group, by CN or DN, to a place in the tree —
|
||||
org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
|
||||
and kept by the sync, which also takes away a place whose group the
|
||||
person has left.
|
||||
|
||||
### `sync_interval`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
SyncInterval is how often the leader reads the directory. Zero is
|
||||
15m; negative turns the sync off, leaving only sign-in.
|
||||
|
||||
### `timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Timeout bounds each round trip to a server. Zero is 10s.
|
||||
|
||||
## Landing
|
||||
|
||||
### `contact`
|
||||
@@ -1910,6 +2097,25 @@ kept by llmgw bundle import and llmgw bundle trust; every engine file
|
||||
a node fetches from the mirror is checked against it. Read again
|
||||
whenever it changes. Empty is /etc/llmgw/offline.sha256.
|
||||
|
||||
## OnDemandSettings
|
||||
|
||||
### `evict_after_idle`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
EvictAfterIdle is how long an on-demand replica must go unasked-for
|
||||
before a model that wants its card may have it. Zero keeps 2m.
|
||||
|
||||
### `min_lifetime`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
MinLifetime is how long a replica that has just come up is safe from
|
||||
that, so two models do not take turns paying for loads. Zero keeps
|
||||
5m. On one card shared by a picture model and a 3D model, 1m each
|
||||
brings the wait between a picture and its mesh to about a minute.
|
||||
Both are in force on the leader within seconds.
|
||||
|
||||
## Options
|
||||
|
||||
### `nvidia_smi`
|
||||
@@ -2112,6 +2318,12 @@ Judge has a judge model on the cluster read the text too.
|
||||
|
||||
Takes a [JudgePolicy](#judgepolicy) block.
|
||||
|
||||
### `services`
|
||||
|
||||
`[]Service`
|
||||
|
||||
Services are outside guardrails asked over HTTP, beside the judge.
|
||||
|
||||
### `stream`
|
||||
|
||||
`string`
|
||||
@@ -2158,6 +2370,20 @@ before affinity is abandoned.
|
||||
|
||||
`float64`
|
||||
|
||||
## PresetSettings
|
||||
|
||||
### `auto`
|
||||
|
||||
`bool`
|
||||
|
||||
Auto has the leading control plane add a picture model (flux-schnell)
|
||||
and a 3D model (triposr) by itself, on demand, once a node has a card
|
||||
that fits one and the catalog has no model that makes the same kind of
|
||||
thing — so a new cluster's portal has its pictures and 3D pages
|
||||
without anybody writing a definition. Only where the cluster already
|
||||
installs ComfyUI (engines.comfyui). A preset somebody removed is not
|
||||
added again by itself. In force on the leader within a minute.
|
||||
|
||||
## PreviousOrigin
|
||||
|
||||
### `source`
|
||||
@@ -2572,6 +2798,14 @@ Deny are patterns, matched against the text as a model reads it.
|
||||
|
||||
MaxChars refuses text longer than this; zero is no limit.
|
||||
|
||||
### `restore`
|
||||
|
||||
`bool`
|
||||
|
||||
Restore, on input, puts what redact rewrote as placeholders back into
|
||||
the answer: the model reads "[EMAIL_1]", the client reads its own
|
||||
address. Only the values that request sent, only into its answer.
|
||||
|
||||
## SCIM
|
||||
|
||||
### `token_file`
|
||||
@@ -2960,6 +3194,23 @@ Firewall turns llmgw's firewall on every machine on or off.
|
||||
|
||||
Takes a [FirewallSettings](#firewallsettings) block.
|
||||
|
||||
### `presets`
|
||||
|
||||
`PresetSettings`
|
||||
|
||||
Presets is how the models that come with llmgw reach the catalog.
|
||||
|
||||
Takes a [PresetSettings](#presetsettings) block.
|
||||
|
||||
### `on_demand`
|
||||
|
||||
`OnDemandSettings`
|
||||
|
||||
OnDemand is how models that load when asked for take turns on the
|
||||
cards.
|
||||
|
||||
Takes a [OnDemandSettings](#ondemandsettings) block.
|
||||
|
||||
### `agents`
|
||||
|
||||
`AgentSettings`
|
||||
@@ -3062,6 +3313,67 @@ Endpoints is the phase-1 static replica list. It is only consulted when
|
||||
no model declares a profile, which is what makes a router-only smoke test
|
||||
against sim-engines possible without a control plane.
|
||||
|
||||
## Service
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name says which service decided, in the verdict's rule.
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
URL is POSTed the body.
|
||||
|
||||
### `header_files`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
HeaderFiles are headers whose values are read from files each time,
|
||||
so a rotated key needs no restart: {Authorization: /etc/…/lakera}.
|
||||
|
||||
### `body`
|
||||
|
||||
`string`
|
||||
|
||||
Body is the request, a template given .Text; {{json .Text}} writes it
|
||||
as a JSON string. Empty is {"input": <text>}.
|
||||
|
||||
### `blocked`
|
||||
|
||||
`string`
|
||||
|
||||
Blocked is where in the answer the service says the text is unsafe:
|
||||
a dotted path, numbers for list items (results.0.flagged). True, a
|
||||
number above zero, a non-empty list or string other than "false" is
|
||||
unsafe.
|
||||
|
||||
### `reason`
|
||||
|
||||
`string`
|
||||
|
||||
Reason is where it says why, for the verdict; optional.
|
||||
|
||||
### `on`
|
||||
|
||||
`string`
|
||||
|
||||
On: input, output or both. Empty is input.
|
||||
|
||||
### `action`
|
||||
|
||||
`string`
|
||||
|
||||
Action on unsafe: block or flag. Empty is block.
|
||||
|
||||
### `timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Timeout bounds one call. Empty is 3s; past it the policy's fail says.
|
||||
|
||||
## ServingSettings
|
||||
|
||||
### `stopped`
|
||||
@@ -3300,6 +3612,15 @@ is sha256=<hex HMAC of the body>.
|
||||
|
||||
`int64`
|
||||
|
||||
### `cached_per_1k`
|
||||
|
||||
`int64`
|
||||
|
||||
CachedPer1k is what a prompt token costs that the engine or provider
|
||||
read from its cache instead of computing: the part of prompt_tokens
|
||||
the answer counts as cached_tokens. Zero charges those at
|
||||
PromptPer1k, as before there was a rate for them.
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
@@ -3574,6 +3895,14 @@ had rather wait behind people than be turned away.
|
||||
|
||||
`int64`
|
||||
|
||||
### `cached_per_1k`
|
||||
|
||||
`int64`
|
||||
|
||||
CachedPer1k is the cost of a prompt token the engine or provider read
|
||||
from its cache (the answer's cached_tokens, counted within the
|
||||
prompt's). Zero charges them at prompt_per_1k.
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
@@ -12,6 +12,7 @@ require (
|
||||
github.com/charmbracelet/x/ansi v0.11.6
|
||||
github.com/coreos/go-oidc/v3 v3.21.0
|
||||
github.com/go-jose/go-jose/v4 v4.1.4
|
||||
github.com/go-ldap/ldap/v3 v3.4.15
|
||||
github.com/go-zeromq/zmq4 v0.17.0
|
||||
github.com/jackc/pgx/v5 v5.11.0
|
||||
github.com/knadh/koanf/parsers/yaml v1.1.1
|
||||
@@ -35,6 +36,7 @@ require (
|
||||
golang.org/x/net v0.59.0
|
||||
golang.org/x/oauth2 v0.37.0
|
||||
golang.org/x/sync v0.23.0
|
||||
golang.org/x/sys v0.48.0
|
||||
golang.org/x/term v0.46.0
|
||||
golang.org/x/text v0.42.0
|
||||
google.golang.org/protobuf v1.36.12
|
||||
@@ -44,6 +46,7 @@ require (
|
||||
require (
|
||||
filippo.io/edwards25519 v1.2.0 // indirect
|
||||
filippo.io/hpke v0.4.0 // indirect
|
||||
github.com/Azure/go-ntlmssp v0.1.1 // indirect
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||
@@ -56,6 +59,7 @@ require (
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
||||
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
||||
github.com/go-asn1-ber/asn1-ber v1.5.8 // indirect
|
||||
github.com/go-logr/logr v1.4.4 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
||||
@@ -88,7 +92,6 @@ require (
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.46.0 // indirect
|
||||
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
|
||||
golang.org/x/sys v0.48.0 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||
google.golang.org/grpc v1.83.1 // indirect
|
||||
|
||||
@@ -8,8 +8,12 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
||||
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
|
||||
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
|
||||
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktpoUAgOJK3OTFc/xug0PCXYCqU0FgDKI=
|
||||
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
||||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||
@@ -46,8 +50,12 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
|
||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||
github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
|
||||
github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
||||
github.com/go-ldap/ldap/v3 v3.4.15 h1:eRB4f4Ecn1POL1mO86+uUDlecmyWsdiJwh60bAUS4UY=
|
||||
github.com/go-ldap/ldap/v3 v3.4.15/go.mod h1:jeD0c98EdZQCKKKKe4kTWs5aCf6y+j8KuYsYLznZSQk=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
|
||||
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
@@ -69,6 +77,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
|
||||
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
|
||||
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||
@@ -81,6 +91,18 @@ github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
|
||||
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
|
||||
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
|
||||
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
|
||||
github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
|
||||
github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
|
||||
github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
|
||||
github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
|
||||
github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
|
||||
github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
|
||||
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
|
||||
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
|
||||
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package account
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeDirectory struct {
|
||||
asked []string
|
||||
u store.User
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeDirectory) SignIn(_ context.Context, login, _ string) (store.User, error) {
|
||||
f.asked = append(f.asked, login)
|
||||
return f.u, f.err
|
||||
}
|
||||
|
||||
// With a directory, a name the store has no password for is the
|
||||
// directory's to check; an account with a password here — the
|
||||
// administrator init made — is still checked here, so a directory that is
|
||||
// down locks nobody out.
|
||||
func TestPasswordsKeptElsewhereAreTheDirectorysToCheck(t *testing.T) {
|
||||
svc, st := newService(t)
|
||||
ctx := t.Context()
|
||||
hash, _ := HashPassword("local-secret")
|
||||
admin := store.User{ID: "usr-admin", Email: "admin@example.com", Name: "a", PasswordHash: hash,
|
||||
State: store.UserActive, CreatedAt: time.Now()}
|
||||
ada := store.User{ID: "usr-ada", Email: "ada@corp.example", Name: "Ada", PasswordHash: "!ldap",
|
||||
State: store.UserActive, CreatedAt: time.Now()}
|
||||
for _, u := range []store.User{admin, ada} {
|
||||
if err := st.CreateUser(ctx, u); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
dir := &fakeDirectory{u: ada}
|
||||
svc.SetDirectory(dir)
|
||||
|
||||
if _, _, err := svc.SignIn(ctx, "admin@example.com", "local-secret"); err != nil || len(dir.asked) != 0 {
|
||||
t.Fatalf("the local administrator: %v, directory asked %v", err, dir.asked)
|
||||
}
|
||||
// A name, not an address, and an account kept by the directory.
|
||||
for _, login := range []string{`CORP\ada`, "ada@corp.example"} {
|
||||
if tok, _, err := svc.SignIn(ctx, login, "ad-secret"); err != nil || tok == "" {
|
||||
t.Errorf("%s: %v", login, err)
|
||||
}
|
||||
}
|
||||
if len(dir.asked) != 2 || dir.asked[0] != `CORP\ada` {
|
||||
t.Errorf("the directory was asked %v", dir.asked)
|
||||
}
|
||||
dir.err = ErrBadCredentials
|
||||
if _, _, err := svc.SignIn(ctx, "ada", "wrong"); !errors.Is(err, ErrBadCredentials) {
|
||||
t.Errorf("a password the directory refused: %v", err)
|
||||
}
|
||||
// A reset link would give a directory's account a password here.
|
||||
if _, _, err := svc.RequestReset(ctx, st, "ada@corp.example"); !errors.Is(err, ErrNoReset) {
|
||||
t.Errorf("a directory's account was sent a reset link: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -80,8 +80,26 @@ type Service struct {
|
||||
|
||||
mu sync.Mutex
|
||||
attempts map[string]*attemptWindow
|
||||
|
||||
// dir checks the passwords of accounts this store keeps none for.
|
||||
dir Directory
|
||||
}
|
||||
|
||||
// Directory checks a password against an outside directory — Active
|
||||
// Directory over LDAP — and answers the account to sign in: made or kept in
|
||||
// step with the directory. Its errors are this package's ErrBadCredentials
|
||||
// and ErrSuspended for the person's own, and anything else for the
|
||||
// directory's.
|
||||
type Directory interface {
|
||||
SignIn(ctx context.Context, login, password string) (store.User, error)
|
||||
}
|
||||
|
||||
// SetDirectory has passwords for accounts with none here, and names that are
|
||||
// not addresses, checked by the directory. An account with a password here
|
||||
// — the administrator init made — is still checked here, so a directory that
|
||||
// is down locks nobody out.
|
||||
func (s *Service) SetDirectory(d Directory) { s.dir = d }
|
||||
|
||||
type attemptWindow struct {
|
||||
count int
|
||||
since time.Time
|
||||
@@ -137,6 +155,7 @@ func (s *Service) SignUp(ctx context.Context, email, name, password string) erro
|
||||
// SignIn checks a password and opens a session, returning the session token and
|
||||
// the CSRF token that must accompany the forms in it.
|
||||
func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
|
||||
login := strings.TrimSpace(email)
|
||||
email = store.NormalizeEmail(email)
|
||||
// Counted per address and the client trying it, so somebody guessing
|
||||
// from elsewhere cannot lock the owner out of their own account; and per
|
||||
@@ -151,6 +170,15 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
|
||||
}
|
||||
|
||||
u, err := s.store.UserByEmail(ctx, email)
|
||||
// No password kept here — the directory's, or nobody's — and a
|
||||
// directory to ask: it decides.
|
||||
if s.dir != nil && (errors.Is(err, store.ErrNotFound) || (err == nil && strings.HasPrefix(u.PasswordHash, "!"))) {
|
||||
if u, err = s.dir.SignIn(ctx, login, password); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
s.clearAttempts(byClient)
|
||||
return s.open(ctx, u)
|
||||
}
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
// Verify anyway, against a hash that cannot match. Returning early here
|
||||
// would let a stopwatch enumerate every account on the system.
|
||||
@@ -180,7 +208,11 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
|
||||
case store.UserSuspended:
|
||||
return "", "", ErrSuspended
|
||||
}
|
||||
return s.open(ctx, u)
|
||||
}
|
||||
|
||||
// open makes a session for a person whose password was right.
|
||||
func (s *Service) open(ctx context.Context, u store.User) (token, csrf string, err error) {
|
||||
if token, err = store.NewSessionToken(); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
@@ -402,7 +434,10 @@ func (s *Service) RequestReset(ctx context.Context, rs ResetStore, email string)
|
||||
return "", store.User{}, ErrTooManyTries
|
||||
}
|
||||
u, err := rs.UserByEmail(ctx, email)
|
||||
if err != nil || u.PasswordHash == "" || u.State != store.UserActive {
|
||||
// An account whose password is kept elsewhere — "!oidc", "!scim",
|
||||
// "!ldap" — gets no link: setting one here would give it a second way
|
||||
// in that its directory knows nothing of, and outlives its leaving.
|
||||
if err != nil || u.PasswordHash == "" || strings.HasPrefix(u.PasswordHash, "!") || u.State != store.UserActive {
|
||||
return "", store.User{}, ErrNoReset
|
||||
}
|
||||
token, err := rs.CreatePasswordReset(ctx, u.ID, ResetLinkTTL)
|
||||
|
||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CachedTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,20 +39,37 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0, 0, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
|
||||
if got := p.Cost(1000, 0, 1000, 0, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0, 0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACachedPromptCostsTheCachedRate(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, CachedPer1k: 100, CompletionPer1k: 3000}
|
||||
// 10,000 prompt tokens of which 9,000 were read from the cache: 1,000 at
|
||||
// the prompt rate and 9,000 at the cached one.
|
||||
if got, want := p.Cost(10000, 9000, 0, 0, 0), int64(1000+900); got != want {
|
||||
t.Errorf("a 90%% cached prompt cost %d, want %d", got, want)
|
||||
}
|
||||
// More cached than prompt is an engine's miscount, not a refund.
|
||||
if got := p.Cost(100, 500, 0, 0, 0); got != 10 {
|
||||
t.Errorf("cached beyond the prompt cost %d, want 10", got)
|
||||
}
|
||||
// No cached rate: cached tokens cost what any prompt token does.
|
||||
if got := (Price{PromptPer1k: 1000}).Cost(10000, 9000, 0, 0, 0); got != 10000 {
|
||||
t.Errorf("without a cached rate a cached prompt cost %d, want 10000", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
|
||||
if got := p.Cost(20, 0, 0, 3, 0); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
|
||||
@@ -18,6 +18,11 @@ import (
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// CachedPer1k is what a prompt token costs that the engine or provider
|
||||
// read from its cache instead of computing: the part of prompt_tokens
|
||||
// the answer counts as cached_tokens. Zero charges those at
|
||||
// PromptPer1k, as before there was a rate for them.
|
||||
CachedPer1k int64 `koanf:"cached_per_1k"`
|
||||
// PerImage is what one picture costs, for a model that makes them: a
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
@@ -33,15 +38,24 @@ type Price struct {
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool {
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.CachedPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
}
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
//
|
||||
// cachedTokens are counted within promptTokens, as OpenAI's usage counts
|
||||
// them, and are charged at the cached rate instead of the prompt rate.
|
||||
func (p Price) Cost(promptTokens, cachedTokens, completionTokens, images, meshes int64) int64 {
|
||||
cachedTokens = min(max(cachedTokens, 0), max(promptTokens, 0))
|
||||
cachedRate := p.CachedPer1k
|
||||
if cachedRate == 0 {
|
||||
cachedRate = p.PromptPer1k
|
||||
}
|
||||
return divUp((promptTokens-cachedTokens)*p.PromptPer1k+cachedTokens*cachedRate, 1000) +
|
||||
divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
||||
}
|
||||
|
||||
|
||||
+33
-4
@@ -6,6 +6,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -1131,6 +1132,31 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
stop()
|
||||
spec.Profile.ModelPath = path
|
||||
}
|
||||
// A shared ComfyUI replica's members' weights, each fetched like the
|
||||
// model's, and handed to the gate where they landed.
|
||||
if raw := spec.Profile.Env[engines.SharedModelsEnv]; raw != "" {
|
||||
var ms []map[string]string
|
||||
if err := json.Unmarshal([]byte(raw), &ms); err != nil {
|
||||
report(supervisor.Failed, "the shared replica's models are not readable: "+err.Error(), nil)
|
||||
return
|
||||
}
|
||||
for i, m := range ms {
|
||||
stop := a.watchFetch(m["source"], func(p weights.Progress) {
|
||||
send(&llmgwv1.ShardStatus{State: agentsvc.StateToWire(supervisor.Starting),
|
||||
Fetch: &llmgwv1.FetchProgress{Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal}})
|
||||
})
|
||||
path, err := a.weights().Ensure(ctx, m["source"])
|
||||
stop()
|
||||
if err != nil {
|
||||
report(supervisor.Failed, fmt.Sprintf("model %s: %v", m["name"], err), nil)
|
||||
return
|
||||
}
|
||||
ms[i]["path"] = path
|
||||
}
|
||||
b, _ := json.Marshal(ms)
|
||||
spec.Profile.Env = maps.Clone(spec.Profile.Env)
|
||||
spec.Profile.Env[engines.SharedModelsEnv] = string(b)
|
||||
}
|
||||
// Adapters are fetched like the model, and before the engine starts: it
|
||||
// loads them at start, and one missing is a replica that answers for the
|
||||
// base model under the adapter's name.
|
||||
@@ -1391,11 +1417,14 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
|
||||
// off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
|
||||
// in the socket's buffer until the kernel gives up retransmitting, fifteen
|
||||
// minutes later. Found by switching off a leader: the standby took over in 14
|
||||
// seconds and the node stayed with the dead one. An HTTP/2 ping after ten
|
||||
// quiet seconds, given five to answer, finds it in a quarter of a minute.
|
||||
// seconds and the node stayed with the dead one. An HTTP/2 ping after five
|
||||
// quiet seconds, given three to answer, finds it in eight: by the time the
|
||||
// standby leads, as it does within the lease's ten seconds, the node is on
|
||||
// its way to it. At fifteen, with the lease at fifteen too, the new leader
|
||||
// went without its nodes, and so its judges, for half a minute.
|
||||
const (
|
||||
streamIdle = 10 * time.Second
|
||||
streamPing = 5 * time.Second
|
||||
streamIdle = 5 * time.Second
|
||||
streamPing = 3 * time.Second
|
||||
)
|
||||
|
||||
// localizeSpec fills in what only this node knows.
|
||||
|
||||
@@ -74,6 +74,13 @@ func NewSensorSampler() *SensorSampler {
|
||||
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
||||
}
|
||||
|
||||
// SampleGPUs reads every card's heat, power and how busy it is now, without
|
||||
// a sampler's ten seconds of memory: for a screen that asks every few seconds
|
||||
// and wants what it asked for. Nil on a machine without nvidia-smi or a card.
|
||||
func SampleGPUs(ctx context.Context) []GPUSensor {
|
||||
return (&SensorSampler{smi: runSMI}).gpus(ctx)
|
||||
}
|
||||
|
||||
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
@@ -129,7 +129,7 @@ func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
||||
// so taken before baseEnv drops them. Without the repository the gate
|
||||
// links no Hugging Face cache, and sf3d's node went to the network for
|
||||
// its gated repository and was refused with a 401.
|
||||
for _, k := range []string{comfyNodesEnv, comfyHFRepoEnv} {
|
||||
for _, k := range []string{comfyNodesEnv, comfyHFRepoEnv, SharedModelsEnv} {
|
||||
if v := p.Env[k]; v != "" {
|
||||
env[k] = v
|
||||
}
|
||||
@@ -144,3 +144,8 @@ const comfyNodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
// comfyHFRepoEnv is the repository a custom node loads the weights by name
|
||||
// from; catalog.HFRepoEnv and comfy.HFRepoEnv by the same name.
|
||||
const comfyHFRepoEnv = "LLMGW_COMFYUI_HF_REPO"
|
||||
|
||||
// SharedModelsEnv is a shared replica's members — name, source, and the
|
||||
// path the agent fetched it to — as JSON; catalog.SharedModelsEnv and
|
||||
// comfy.SharedModelsEnv by the same name.
|
||||
const SharedModelsEnv = "LLMGW_COMFYUI_MODELS"
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
@@ -12,6 +14,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
project string
|
||||
ttl time.Duration
|
||||
anyAnswer bool
|
||||
similar float64
|
||||
off bool
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
@@ -28,15 +31,25 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
"a request continuing a stored conversation or under a guardrails no_store\n" +
|
||||
"policy. The guardrails judge a reused answer as they judge any other.\n\n" +
|
||||
"Off for every project until this is run. Without --ttl or --off, says how\n" +
|
||||
"it stands. The answers are held in memory, router.cache bounds how much.",
|
||||
"it stands. The answers are held in memory, router.cache bounds how much.\n\n" +
|
||||
"--similar also answers a chat question that is like one already answered:\n" +
|
||||
"its last user message embedded by router.cache.embedding_model, and given\n" +
|
||||
"the answer whose question reaches that cosine similarity (0.95 is strict,\n" +
|
||||
"0.9 looser), as x-llmgw-cache: similar. The system prompt, the earlier\n" +
|
||||
"turns, tools and the sampling settings must still be the same.",
|
||||
Example: " llmgw keys cache --project Acme/ml/train --ttl 1h\n" +
|
||||
" llmgw keys cache --project Acme/ml/eval --ttl 24h --any\n" +
|
||||
" llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95\n" +
|
||||
" llmgw keys cache --project Acme/ml/train --off",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
switch {
|
||||
case off && (ttl != 0 || anyAnswer):
|
||||
return fmt.Errorf("--off, or --ttl with --any: not both")
|
||||
case off && (ttl != 0 || anyAnswer || similar != 0):
|
||||
return fmt.Errorf("--off, or --ttl with --any or --similar: not both")
|
||||
case similar < 0 || similar > 1:
|
||||
return fmt.Errorf("--similar is a cosine similarity between 0 and 1, like 0.95")
|
||||
case similar != 0 && ttl == 0:
|
||||
return fmt.Errorf("--similar goes with --ttl: say how long answers are reused")
|
||||
case ttl < 0:
|
||||
return fmt.Errorf("--ttl is how long an answer is reused, like 1h")
|
||||
case ttl > 0 && ttl < time.Second:
|
||||
@@ -54,7 +67,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
return err
|
||||
}
|
||||
if off || ttl > 0 {
|
||||
if err := s.SetAnswerCache(cmd.Context(), p.ID, ttl, anyAnswer); err != nil {
|
||||
if err := s.SetAnswerCache(cmd.Context(), p.ID, store.AnswerCache{TTL: ttl, Any: anyAnswer, Similar: similar}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
@@ -71,6 +84,9 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s reuses answers for %s: those asked at temperature 0 or with a seed, "+
|
||||
"and embeddings, rerank and score.\n", p.Path, c.TTL)
|
||||
}
|
||||
if c.TTL > 0 && c.Similar > 0 {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Chat questions reaching %.2f similarity to one answered get its answer too.\n", c.Similar)
|
||||
}
|
||||
if off || ttl > 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Gateways see the change within 30 seconds.")
|
||||
}
|
||||
@@ -80,6 +96,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||
cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
|
||||
cmd.Flags().DurationVar(&ttl, "ttl", 0, "how long an answer is reused, like 1h")
|
||||
cmd.Flags().BoolVar(&anyAnswer, "any", false, "reuse sampled answers too, not only reproducible ones")
|
||||
cmd.Flags().Float64Var(&similar, "similar", 0, "also answer chat questions this similar to one answered, like 0.95")
|
||||
cmd.Flags().BoolVar(&off, "off", false, "stop reusing answers for this project")
|
||||
return cmd
|
||||
}
|
||||
@@ -150,6 +150,7 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
f.DurationVar(&cfg.SLO.TPOTp99, "tpot-p99", cfg.SLO.TPOTp99, "per-output-token target")
|
||||
f.BoolVar(&save, "save", false,
|
||||
"record the capacity against the model and profile, so the number outlives this terminal")
|
||||
cmd.AddCommand(newBenchOverheadCmd(asJSON))
|
||||
return cmd
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,421 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"text/tabwriter"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// What llmgw adds to a request, measured: the same load straight at an
|
||||
// engine that answers at once, and through the gateway in front of it —
|
||||
// bare, with authentication and usage accounting, and with the guardrails'
|
||||
// pattern checks besides. The engine's own time is the same on both paths,
|
||||
// so what differs is the gateway's.
|
||||
//
|
||||
// Everything runs in this process, on loopback: no cluster, no GPU, and the
|
||||
// numbers are this machine's. On a machine shared with other work (a VM on
|
||||
// an oversubscribed host) they move with it; compare modes run together,
|
||||
// not figures from different runs.
|
||||
|
||||
// overheadMode is one way a request reaches the engine.
|
||||
type overheadMode struct {
|
||||
name string
|
||||
gateway bool
|
||||
auth, guard bool
|
||||
}
|
||||
|
||||
var overheadModes = []overheadMode{
|
||||
{name: "direct"},
|
||||
{name: "gateway", gateway: true},
|
||||
{name: "gateway+auth", gateway: true, auth: true},
|
||||
{name: "gateway+auth+guard", gateway: true, auth: true, guard: true},
|
||||
}
|
||||
|
||||
// overheadStep is one mode at one request rate.
|
||||
type overheadStep struct {
|
||||
Mode string `json:"mode"`
|
||||
Rate int `json:"rate"`
|
||||
Sent int `json:"sent"`
|
||||
Failed int `json:"failed"`
|
||||
P50 time.Duration `json:"p50"`
|
||||
P99 time.Duration `json:"p99"`
|
||||
AddedP50 time.Duration `json:"added_p50"`
|
||||
AddedP99 time.Duration `json:"added_p99"`
|
||||
// CPUPer1k is the process's CPU time per thousand requests beyond what
|
||||
// the direct mode spent at the same rate: the gateway's.
|
||||
CPUPer1k time.Duration `json:"cpu_per_1k"`
|
||||
cpu time.Duration
|
||||
}
|
||||
|
||||
func newBenchOverheadCmd(asJSON *bool) *cobra.Command {
|
||||
var (
|
||||
rates []int
|
||||
each time.Duration
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
Use: "overhead",
|
||||
Short: "Measure what the gateway adds to a request, against an engine that answers at once",
|
||||
Long: "Runs an engine that answers at once and a gateway in front of it, in this\n" +
|
||||
"process on loopback, and sends the same load at each rate straight to the\n" +
|
||||
"engine and through the gateway: bare, with authentication and usage\n" +
|
||||
"accounting, and with the guardrails' pattern checks too. Reports each\n" +
|
||||
"request's latency at p50 and p99, what the gateway added to it, and the\n" +
|
||||
"gateway's CPU time per thousand requests.\n\n" +
|
||||
"Needs no cluster. The figures are this machine's: on one shared with other\n" +
|
||||
"work they move with it, so compare the modes of one run, and note the\n" +
|
||||
"machine's load beside any figure you keep.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
steps, err := runOverhead(cmd.Context(), rates, each)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(steps)
|
||||
}
|
||||
printOverhead(cmd.OutOrStdout(), steps)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.IntSliceVar(&rates, "rates", []int{100, 500, 1000, 2000}, "requests a second to send, one step each")
|
||||
f.DurationVar(&each, "each", 3*time.Second, "how long each step sends for")
|
||||
return cmd
|
||||
}
|
||||
|
||||
func printOverhead(w io.Writer, steps []overheadStep) {
|
||||
tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "MODE\tRATE/S\tSENT\tFAILED\tP50\tP99\t+P50\t+P99\tCPU/1K REQ")
|
||||
for _, s := range steps {
|
||||
added := func(d time.Duration) string {
|
||||
if s.Mode == "direct" {
|
||||
return "-"
|
||||
}
|
||||
return msOf(d)
|
||||
}
|
||||
cpu := "-"
|
||||
if s.Mode != "direct" {
|
||||
cpu = msOf(s.CPUPer1k)
|
||||
}
|
||||
fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%s\t%s\t%s\t%s\t%s\n", s.Mode, s.Rate, s.Sent, s.Failed,
|
||||
msOf(s.P50), msOf(s.P99), added(s.AddedP50), added(s.AddedP99), cpu)
|
||||
}
|
||||
_ = tw.Flush()
|
||||
fmt.Fprintf(w, "\n%d CPUs, %s. Loopback, in one process: the figures are this machine's.\n",
|
||||
runtime.NumCPU(), loadAverage())
|
||||
}
|
||||
|
||||
func msOf(d time.Duration) string { return fmt.Sprintf("%.2fms", float64(d)/float64(time.Millisecond)) }
|
||||
|
||||
// loadAverage is the machine's, where it can be read, to keep beside the
|
||||
// figures.
|
||||
func loadAverage() string {
|
||||
if b, err := os.ReadFile("/proc/loadavg"); err == nil {
|
||||
if f := strings.Fields(string(b)); len(f) >= 3 {
|
||||
return "load " + strings.Join(f[:3], " ")
|
||||
}
|
||||
}
|
||||
// macOS: "{ 1.23 1.10 0.98 }".
|
||||
if b, err := exec.Command("sysctl", "-n", "vm.loadavg").Output(); err == nil {
|
||||
if f := strings.Fields(strings.Trim(strings.TrimSpace(string(b)), "{}")); len(f) >= 3 {
|
||||
return "load " + strings.Join(f[:3], " ")
|
||||
}
|
||||
}
|
||||
return "load not read"
|
||||
}
|
||||
|
||||
// runOverhead measures every mode at every rate.
|
||||
func runOverhead(ctx context.Context, rates []int, each time.Duration) ([]overheadStep, error) {
|
||||
engine, err := instantEngine()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer engine.Close()
|
||||
dir, err := os.MkdirTemp("", "llmgw-overhead-")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
|
||||
type target struct {
|
||||
url, key string
|
||||
stop func()
|
||||
}
|
||||
targets := map[string]target{}
|
||||
defer func() {
|
||||
for _, t := range targets {
|
||||
if t.stop != nil {
|
||||
t.stop()
|
||||
}
|
||||
}
|
||||
}()
|
||||
for _, m := range overheadModes {
|
||||
if !m.gateway {
|
||||
targets[m.name] = target{url: engine.URL}
|
||||
continue
|
||||
}
|
||||
url, key, stop, err := overheadGateway(ctx, engine.URL, filepath.Join(dir, m.name+".db"), m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", m.name, err)
|
||||
}
|
||||
targets[m.name] = target{url: url, key: key, stop: stop}
|
||||
}
|
||||
|
||||
client := &http.Client{Transport: &http.Transport{MaxIdleConns: 1024, MaxIdleConnsPerHost: 1024}, Timeout: 30 * time.Second}
|
||||
var out []overheadStep
|
||||
for _, rate := range rates {
|
||||
var direct overheadStep
|
||||
for _, m := range overheadModes {
|
||||
t := targets[m.name]
|
||||
s := loadStep(ctx, client, t.url, t.key, rate, each)
|
||||
s.Mode = m.name
|
||||
if m.name == "direct" {
|
||||
direct = s
|
||||
} else {
|
||||
s.AddedP50, s.AddedP99 = max(0, s.P50-direct.P50), max(0, s.P99-direct.P99)
|
||||
if s.Sent > 0 {
|
||||
s.CPUPer1k = max(0, s.cpu-direct.cpu) * 1000 / time.Duration(s.Sent)
|
||||
}
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// loadStep sends rate requests a second for d, each in its own goroutine,
|
||||
// and measures each one's latency and the process's CPU meanwhile.
|
||||
func loadStep(ctx context.Context, client *http.Client, url, key string, rate int, d time.Duration) overheadStep {
|
||||
s := overheadStep{Rate: rate}
|
||||
body := []byte(`{"model":"bench","max_tokens":8,"messages":[{"role":"user","content":"What is the capital of France?"}]}`)
|
||||
var (
|
||||
mu sync.Mutex
|
||||
lat []time.Duration
|
||||
wg sync.WaitGroup
|
||||
total = int(float64(rate) * d.Seconds())
|
||||
)
|
||||
interval := time.Second / time.Duration(max(rate, 1))
|
||||
cpu0 := processCPU()
|
||||
start := time.Now()
|
||||
for i := range total {
|
||||
if ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
if wait := time.Until(start.Add(time.Duration(i) * interval)); wait > 0 {
|
||||
time.Sleep(wait)
|
||||
}
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url+"/v1/chat/completions", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+key)
|
||||
}
|
||||
t := time.Now()
|
||||
resp, err := client.Do(req)
|
||||
ok := err == nil && resp.StatusCode == http.StatusOK
|
||||
if resp != nil {
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
}
|
||||
el := time.Since(t)
|
||||
mu.Lock()
|
||||
s.Sent++
|
||||
if ok {
|
||||
lat = append(lat, el)
|
||||
} else {
|
||||
s.Failed++
|
||||
}
|
||||
mu.Unlock()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
s.cpu = processCPU() - cpu0
|
||||
s.P50, s.P99 = percentile(lat, 0.50), percentile(lat, 0.99)
|
||||
return s
|
||||
}
|
||||
|
||||
func percentile(ds []time.Duration, p float64) time.Duration {
|
||||
if len(ds) == 0 {
|
||||
return 0
|
||||
}
|
||||
ds = slices.Clone(ds)
|
||||
sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
|
||||
i := int(p * float64(len(ds)-1))
|
||||
return ds[i]
|
||||
}
|
||||
|
||||
// processCPU is the CPU time this process has used, user and system.
|
||||
func processCPU() time.Duration {
|
||||
var ru syscall.Rusage
|
||||
if err := syscall.Getrusage(syscall.RUSAGE_SELF, &ru); err != nil {
|
||||
return 0
|
||||
}
|
||||
return time.Duration(ru.Utime.Nano() + ru.Stime.Nano())
|
||||
}
|
||||
|
||||
// instantEngine is an OpenAI-compatible engine that answers at once, with
|
||||
// the metrics the router scrapes.
|
||||
func instantEngine() (*httpServer, error) {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, _ *http.Request) {
|
||||
fmt.Fprint(w, "# TYPE vllm:num_requests_running gauge\nvllm:num_requests_running 0\n"+
|
||||
"# TYPE vllm:num_requests_waiting gauge\nvllm:num_requests_waiting 0\n"+
|
||||
"# TYPE vllm:kv_cache_usage_perc gauge\nvllm:kv_cache_usage_perc 0.1\n")
|
||||
})
|
||||
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
|
||||
mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprint(w, `{"id":"b","object":"chat.completion","model":"bench","choices":[{"index":0,"message":{"role":"assistant","content":"Paris."},"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":2,"total_tokens":16}}`)
|
||||
})
|
||||
return serveLoopback(mux)
|
||||
}
|
||||
|
||||
// httpServer is a handler served on a loopback port.
|
||||
type httpServer struct {
|
||||
URL string
|
||||
srv *http.Server
|
||||
}
|
||||
|
||||
func (s *httpServer) Close() { _ = s.srv.Close() }
|
||||
|
||||
func serveLoopback(h http.Handler) (*httpServer, error) {
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
srv := &http.Server{Handler: h, ReadHeaderTimeout: 10 * time.Second}
|
||||
go func() { _ = srv.Serve(ln) }()
|
||||
return &httpServer{URL: "http://" + ln.Addr().String(), srv: srv}, nil
|
||||
}
|
||||
|
||||
// overheadGateway starts a gateway in front of the engine as the mode asks,
|
||||
// and returns its address and, with auth, a key.
|
||||
func overheadGateway(ctx context.Context, engineURL, storePath string, m overheadMode) (string, string, func(), error) {
|
||||
cfg := config.Default()
|
||||
cfg.Listen = "127.0.0.1:0"
|
||||
cfg.Router.ScrapeInterval = 200 * time.Millisecond
|
||||
cfg.Endpoints = []config.Endpoint{{ID: "bench", URL: engineURL, Models: []string{"bench"}}}
|
||||
key := ""
|
||||
if m.auth {
|
||||
var err error
|
||||
if key, err = overheadKey(ctx, storePath); err != nil {
|
||||
return "", "", nil, err
|
||||
}
|
||||
cfg.Auth.Enabled, cfg.Auth.Store = true, storePath
|
||||
// Limits would measure themselves refusing: off.
|
||||
cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM = 0, 0
|
||||
}
|
||||
rt, err := router.New(cfg, slog.New(slog.DiscardHandler))
|
||||
if err != nil {
|
||||
return "", "", nil, err
|
||||
}
|
||||
if m.guard {
|
||||
g, err := baselineGuard()
|
||||
if err != nil {
|
||||
return "", "", nil, err
|
||||
}
|
||||
rt.SetGuard(g)
|
||||
}
|
||||
rctx, cancel := context.WithCancel(ctx)
|
||||
rt.Start(rctx)
|
||||
srv, err := serveLoopback(rt.Handler())
|
||||
if err != nil {
|
||||
cancel()
|
||||
return "", "", nil, err
|
||||
}
|
||||
stop := func() { srv.Close(); cancel(); rt.Stop() }
|
||||
// Ready once a request goes through: the endpoint is scraped first.
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for {
|
||||
body := `{"model":"bench","messages":[{"role":"user","content":"hi"}]}`
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, srv.URL+"/v1/chat/completions", strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+key)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err == nil {
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
return srv.URL, key, stop, nil
|
||||
}
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
stop()
|
||||
status := "no answer"
|
||||
if err == nil {
|
||||
status = resp.Status
|
||||
}
|
||||
return "", "", nil, fmt.Errorf("the gateway did not serve within 10s (%s)", status)
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// overheadKey makes a store with one project and a key in it.
|
||||
func overheadKey(ctx context.Context, path string) (string, error) {
|
||||
st, err := storepkg.Open(ctx, path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer st.Close()
|
||||
org, err := st.CreateOrg(ctx, "bench")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
team, err := st.CreateTeam(ctx, org, "bench")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
project, err := st.CreateProject(ctx, team, "bench")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
secret, k, err := storepkg.NewKey("key-bench", "bench", project)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
k.CreatedAt = time.Now()
|
||||
return secret, st.CreateKey(ctx, k)
|
||||
}
|
||||
|
||||
// baselineGuard is the policy init installs, without its judge: the pattern
|
||||
// checks every request passes.
|
||||
func baselineGuard() (guard.Guard, error) {
|
||||
m, err := config.Parse([]byte(guardSection(false)))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg, err := config.FromMap(m)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return guard.NewGate(cfg.Guard, nil, slog.New(slog.DiscardHandler))
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The short version, so CI sees the gateway's overhead on every change: every
|
||||
// mode answers every request, and what the gateway adds is measured, not
|
||||
// guessed. No bound is asserted on it, as CI's machines vary; the table is
|
||||
// in the test's log.
|
||||
func TestTheGatewaysOverheadIsMeasured(t *testing.T) {
|
||||
steps, err := runOverhead(t.Context(), []int{50}, time.Second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(steps) != len(overheadModes) {
|
||||
t.Fatalf("%d steps for %d modes", len(steps), len(overheadModes))
|
||||
}
|
||||
for _, s := range steps {
|
||||
if s.Sent == 0 || s.Failed > 0 || s.P50 <= 0 || s.P99 < s.P50 {
|
||||
t.Errorf("%s: sent %d failed %d p50 %s p99 %s", s.Mode, s.Sent, s.Failed, s.P50, s.P99)
|
||||
}
|
||||
}
|
||||
var b bytes.Buffer
|
||||
printOverhead(&b, steps)
|
||||
if !strings.Contains(b.String(), "gateway+auth+guard") {
|
||||
t.Errorf("table:\n%s", b.String())
|
||||
}
|
||||
t.Logf("\n%s", b.String())
|
||||
}
|
||||
@@ -243,6 +243,10 @@ var liveSettings = []string{
|
||||
"firewall", "firewall_networks",
|
||||
// Sent to the nodes' agents as they change.
|
||||
"agents",
|
||||
// Read by the leader's presets pass each time it runs.
|
||||
"presets",
|
||||
// Handed to the reconciler as they change.
|
||||
"on_demand",
|
||||
// Read by server init, node add and backups, not by the server.
|
||||
"backup_recipients", "backup_copy_to",
|
||||
}
|
||||
|
||||
@@ -186,6 +186,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newCapacityCmd(client, &asJSON),
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newGuardCmd(client, &asJSON),
|
||||
newMCPCmd(client, &asJSON),
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
func newMCPCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "mcp",
|
||||
Short: "The company's own MCP servers, served through llmgw's /mcp",
|
||||
Long: "An MCP server registered here — Jira, a file server, a database — is served\n" +
|
||||
"through llmgw's /mcp beside llmgw's own tools, its tools named SERVER.tool.\n" +
|
||||
"Its credentials stay on the control planes: a client gives only its llmgw\n" +
|
||||
"key. A key sees and calls only the tools granted it, as patterns, with\n" +
|
||||
"`llmgw keys limits --tools 'jira.*'` at any level of the tenancy tree;\n" +
|
||||
"none granted anywhere is none. Every call is in `llmgw ctl audit`.",
|
||||
}
|
||||
c.AddCommand(newMCPAddCmd(client, asJSON), newMCPListCmd(client, asJSON), newMCPRemoveCmd(client),
|
||||
newMCPTestCmd(client, asJSON))
|
||||
return c
|
||||
}
|
||||
|
||||
func newMCPAddCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var req adminapi.MCPServerRequest
|
||||
var headers, refs, secretFiles []string
|
||||
var bearerFile string
|
||||
var disabled bool
|
||||
c := &cobra.Command{
|
||||
Use: "add NAME --url URL",
|
||||
Short: "Register an MCP server, or change one",
|
||||
Example: " llmgw ctl mcp add jira --url https://jira.internal/mcp --bearer-file /root/jira-token\n" +
|
||||
" llmgw ctl mcp add llmgw --url llmgw:admin --description \"this cluster, read only\"\n" +
|
||||
" llmgw ctl mcp add wiki --url https://wiki.internal/mcp --header-ref Authorization=vault:secret/mcp/wiki#token\n" +
|
||||
" llmgw ctl mcp add jira --disabled # off, kept",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var err error
|
||||
if req.Headers, err = pairs(headers, "--header"); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.HeaderRefs, err = pairs(refs, "--header-ref"); err != nil {
|
||||
return err
|
||||
}
|
||||
files, err := pairs(secretFiles, "--secret-file")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// A secret read here and kept in the store, sealed: the file
|
||||
// need not be on every control plane, and is not read again.
|
||||
for header, file := range files {
|
||||
b, err := os.ReadFile(file) //nolint:gosec // a file the operator names
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if req.Secrets == nil {
|
||||
req.Secrets = map[string]string{}
|
||||
}
|
||||
req.Secrets[header] = strings.TrimSpace(string(b))
|
||||
}
|
||||
if bearerFile != "" {
|
||||
b, err := os.ReadFile(bearerFile) //nolint:gosec // a file the operator names
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if req.Secrets == nil {
|
||||
req.Secrets = map[string]string{}
|
||||
}
|
||||
req.Secrets["Authorization"] = "Bearer " + strings.TrimSpace(string(b))
|
||||
}
|
||||
if cmd.Flags().Changed("disabled") {
|
||||
on := !disabled
|
||||
req.Enabled = &on
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
var v adminapi.MCPServerView
|
||||
if err := client().put(cmd.Context(), "/api/v1/mcp/servers/"+args[0], body, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||
}
|
||||
state := "on"
|
||||
if !v.Enabled {
|
||||
state = "off"
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s, %s. Its tools are %s.TOOL in /mcp, to keys granted them "+
|
||||
"(llmgw keys limits --tools '%s.*'). Check it: llmgw ctl mcp test %s\n", v.Name, v.URL, state, v.Name, v.Name, v.Name)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := c.Flags()
|
||||
f.StringVar(&req.URL, "url", "", "the server's streamable HTTP address, https://…/mcp; llmgw:admin for this cluster, read only")
|
||||
f.StringVar(&req.Description, "description", "", "what it is, shown beside its tools")
|
||||
f.StringArrayVar(&headers, "header", nil, "NAME=VALUE sent as written, for what is not secret; repeatable")
|
||||
f.StringArrayVar(&refs, "header-ref", nil, "NAME=REF read at each call: a file on every control plane, env:, vault: or store:; repeatable")
|
||||
f.StringArrayVar(&secretFiles, "secret-file", nil, "NAME=FILE read now and kept in the store, sealed: a token; repeatable")
|
||||
f.StringVar(&bearerFile, "bearer-file", "", "a token FILE, read now and kept sealed, sent as Authorization: Bearer")
|
||||
f.BoolVar(&disabled, "disabled", false, "keep it but serve none of its tools")
|
||||
f.IntVar(&req.TimeoutSeconds, "timeout", 0, "seconds one call may take (default 60)")
|
||||
f.IntVar(&req.RatePerMinute, "rate", 0, "calls a minute each caller may make to its tools (default: no bound; -1 takes one away)")
|
||||
return c
|
||||
}
|
||||
|
||||
func newMCPListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "The MCP servers /mcp serves",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var list []adminapi.MCPServerView
|
||||
if err := client().get(cmd.Context(), "/api/v1/mcp/servers", &list); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(list)
|
||||
}
|
||||
if len(list) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "no MCP servers; add one: llmgw ctl mcp add NAME --url URL")
|
||||
return nil
|
||||
}
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "NAME\tSTATE\tURL\tCREDENTIALS\tDESCRIPTION")
|
||||
for _, s := range list {
|
||||
state := "on"
|
||||
if !s.Enabled {
|
||||
state = "off"
|
||||
}
|
||||
var creds []string
|
||||
for h := range s.HeaderRefs {
|
||||
creds = append(creds, h)
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", s.Name, state, s.URL, dash(strings.Join(creds, ",")), s.Description)
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newMCPRemoveCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "remove NAME",
|
||||
Short: "Remove an MCP server: its tools leave /mcp",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := client().delete(cmd.Context(), "/api/v1/mcp/servers/"+args[0], nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s removed; its tools leave /mcp within half a minute\n", args[0])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newMCPTestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "test NAME",
|
||||
Short: "Ask an MCP server for its tools, with its credentials, as /mcp would",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var v adminapi.MCPTestView
|
||||
if err := client().post(cmd.Context(), "/api/v1/mcp/servers/"+args[0]+"/test", nil, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||
}
|
||||
if !v.OK {
|
||||
return errors.New(args[0] + " did not answer: " + v.Error)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s answers with %d tool(s):\n", args[0], len(v.Tools))
|
||||
for _, t := range v.Tools {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " "+t)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// pairs reads NAME=VALUE flags.
|
||||
func pairs(list []string, flag string) (map[string]string, error) {
|
||||
if len(list) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
out := map[string]string{}
|
||||
for _, p := range list {
|
||||
k, v, ok := strings.Cut(p, "=")
|
||||
if !ok || strings.TrimSpace(k) == "" {
|
||||
return nil, fmt.Errorf("%s %q is not NAME=VALUE", flag, p)
|
||||
}
|
||||
out[strings.TrimSpace(k)] = v
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -268,6 +268,26 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
}
|
||||
fmt.Fprintf(out, "%d node(s) upgraded to %s in %s, %d already on it\n",
|
||||
done, v, time.Since(start).Round(time.Second), skipped)
|
||||
// Checked at the end, against what the control plane says now: a node
|
||||
// that is not on the version is said by name, and the command fails. A
|
||||
// site's nodes stayed a release behind with nothing printed to say so.
|
||||
if len(nodes) == 0 {
|
||||
fmt.Fprintln(out, "This control plane has no nodes registered, so none were upgraded.")
|
||||
}
|
||||
var now []adminapi.NodeView
|
||||
if err := client().get(ctx, "/api/v1/nodes", &now); err == nil {
|
||||
var behind []string
|
||||
for _, n := range now {
|
||||
if (len(only) == 0 || slices.Contains(only, n.ID)) && n.AgentVersion != v {
|
||||
behind = append(behind, fmt.Sprintf("%s (%s, %s)", n.ID, orUnknown(n.AgentVersion), n.State))
|
||||
}
|
||||
}
|
||||
if len(behind) > 0 {
|
||||
slices.Sort(behind)
|
||||
return fmt.Errorf("not on %s: %s\nrun this again to try them, or `llmgw ctl logs NODE` for why",
|
||||
v, strings.Join(behind, ", "))
|
||||
}
|
||||
}
|
||||
// The one machine this does not reach. Said every time, because it is
|
||||
// exactly the kind of thing that stays three versions behind unnoticed.
|
||||
fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
|
||||
|
||||
@@ -150,3 +150,54 @@ func TestABinaryInstalledByHandIsNotAServerRunningIt(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpgradeThatFindsNoNodesSaysSo(t *testing.T) {
|
||||
// A site's nodes stayed a release behind with nothing printed to say
|
||||
// that none had been reached.
|
||||
f := &fakeFleet{versions: map[string]string{}}
|
||||
srv := f.server(t)
|
||||
var out bytes.Buffer
|
||||
if err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, time.Second, 1); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "no nodes registered") {
|
||||
t.Errorf("output:\n%s", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpgradeFailsNamingANodeLeftBehind(t *testing.T) {
|
||||
// A node that joins, or comes back on its old binary, while the waves
|
||||
// run is found at the end rather than passed over.
|
||||
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0"}}
|
||||
srv := f.server(t)
|
||||
late := func() *ctlClient {
|
||||
f.mu.Lock()
|
||||
if len(f.told) == 1 {
|
||||
f.versions["late"] = "v1.0.0"
|
||||
}
|
||||
f.mu.Unlock()
|
||||
return newCtlClient(srv.URL)
|
||||
}
|
||||
var out bytes.Buffer
|
||||
err := upgradeNodes(t.Context(), &out, late, "v1.0.1", nil, 5*time.Second, 1)
|
||||
if err == nil || !strings.Contains(err.Error(), "late (v1.0.0, ready)") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpgradingAControlPlaneAloneSaysItsNodesWereNot(t *testing.T) {
|
||||
f := &fakeFleet{versions: map[string]string{"p-g1": "v1.0.0", "p-g3": "v1.0.1"}}
|
||||
srv := f.server(t)
|
||||
var out bytes.Buffer
|
||||
nodesLeftBehind(t.Context(), &out, newCtlClient(srv.URL), "v1.0.1")
|
||||
if !strings.Contains(out.String(), "1 of 2 node(s) are not on v1.0.1") ||
|
||||
!strings.Contains(out.String(), "ctl upgrade v1.0.1 --skip-control-plane") {
|
||||
t.Errorf("output:\n%s", out.String())
|
||||
}
|
||||
out.Reset()
|
||||
f.versions["p-g1"] = "v1.0.1"
|
||||
nodesLeftBehind(t.Context(), &out, newCtlClient(srv.URL), "v1.0.1")
|
||||
if out.Len() != 0 {
|
||||
t.Errorf("said something with every node on the version: %s", out.String())
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -33,7 +34,7 @@ func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
"then fetches and serves by itself: " + knownJudgeNames() + ".",
|
||||
}
|
||||
c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client),
|
||||
newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON))
|
||||
newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON), newGuardJudgeCheckCmd(client, asJSON))
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -383,3 +384,70 @@ func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command
|
||||
c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
|
||||
return c
|
||||
}
|
||||
|
||||
func newGuardJudgeCheckCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var model string
|
||||
var burst int
|
||||
c := &cobra.Command{
|
||||
Use: "check",
|
||||
Short: "How a judge keeps up: a judgment alone and among others, against the time a request waits",
|
||||
Long: "Times three judgments of a prompt of about 280 tokens, one after another, then\n" +
|
||||
"a burst of them at once (two for each ready replica unless --burst says),\n" +
|
||||
"and sets both against the policies' latency budget. A judgment past the\n" +
|
||||
"budget is refused under fail: closed; past half of it alone, or past it\n" +
|
||||
"among others, a busy moment refuses requests. Each request is judged once\n" +
|
||||
"for its prompt and again every 600 characters of its answer.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
v, err := checkJudge(cmd.Context(), client(), model, burst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(v)
|
||||
}
|
||||
fmt.Fprint(cmd.OutOrStdout(), judgeCheckText(v))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
|
||||
c.Flags().IntVar(&burst, "burst", 0, "judgments at once (default: two for each ready replica)")
|
||||
return c
|
||||
}
|
||||
|
||||
func checkJudge(ctx context.Context, c *ctlClient, model string, burst int) (adminapi.JudgeCheckView, error) {
|
||||
body, _ := json.Marshal(adminapi.JudgeCheckRequest{Model: model, Burst: burst})
|
||||
var v adminapi.JudgeCheckView
|
||||
err := c.patient(3*time.Minute).post(ctx, "/api/v1/guard/judges/check", body, &v)
|
||||
return v, err
|
||||
}
|
||||
|
||||
// judgeCheckText says a check in a few lines, and what to do when it is not
|
||||
// fine.
|
||||
func judgeCheckText(v adminapi.JudgeCheckView) string {
|
||||
var b strings.Builder
|
||||
took := make([]string, len(v.TookMs))
|
||||
for i, t := range v.TookMs {
|
||||
took[i] = fmt.Sprintf("%.1f", float64(t)/1000)
|
||||
}
|
||||
fmt.Fprintf(&b, "%s, %d replica(s) ready, a request waits %.1f s for it\n", v.Model, v.Ready, float64(v.BudgetMs)/1000)
|
||||
fmt.Fprintf(&b, " alone: %s s\n", strings.Join(took, ", "))
|
||||
fmt.Fprintf(&b, " %d at once: slowest %.1f s", v.Burst, float64(v.BurstSlowestMs)/1000)
|
||||
if v.BurstFailed > 0 {
|
||||
fmt.Fprintf(&b, ", %d gave no verdict", v.BurstFailed)
|
||||
}
|
||||
fmt.Fprintf(&b, "; about %.1f judgments a second, so about %.1f requests a second with short answers\n",
|
||||
v.PerSecond, v.PerSecond/2)
|
||||
switch v.Verdict {
|
||||
case "ok":
|
||||
b.WriteString("ok: well inside the budget\n")
|
||||
case "tight":
|
||||
b.WriteString("tight: a busy moment passes the budget\n")
|
||||
default:
|
||||
b.WriteString("slow: a judgment alone passes the budget\n")
|
||||
}
|
||||
for _, a := range v.Advice {
|
||||
b.WriteString(" - " + a + "\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -32,12 +32,34 @@ type judgeKeeper struct {
|
||||
// tried is when each judge was last asked for, so one that cannot be
|
||||
// fetched is not asked for again on every look.
|
||||
tried map[string]time.Time
|
||||
// grace is how long after its first look the keeper serves no judge
|
||||
// itself: init adds its own judge just after the control plane starts,
|
||||
// sized and placed as the install says, and one the leader added first
|
||||
// was what init then kept. Zero waits for nothing.
|
||||
grace time.Duration
|
||||
first time.Time
|
||||
|
||||
// What refitting an earlier install's judge needs: the catalog to change
|
||||
// its profile in, the replicas to restart, and the nodes' CPUs. Any nil
|
||||
// leaves judges as they are.
|
||||
profiles interface {
|
||||
SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error)
|
||||
}
|
||||
restart func(ids ...string) error
|
||||
replicas func(model string) []string
|
||||
nodeCPUs func() []int
|
||||
// looked is each judge already refitted or found changed by hand, so
|
||||
// each is said once.
|
||||
looked map[string]bool
|
||||
}
|
||||
|
||||
// judgeRetry is how long a judge whose serving failed waits before it is
|
||||
// asked for again.
|
||||
const judgeRetry = 10 * time.Minute
|
||||
|
||||
// judgeGrace is the keeper's grace on a control plane just started.
|
||||
const judgeGrace = 10 * time.Minute
|
||||
|
||||
func (k *judgeKeeper) run(ctx context.Context, every time.Duration) {
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
@@ -69,6 +91,13 @@ func (k *judgeKeeper) check() {
|
||||
have := map[string]bool{}
|
||||
for _, m := range k.models() {
|
||||
have[m.Name] = true
|
||||
k.refit(m)
|
||||
}
|
||||
if k.first.IsZero() {
|
||||
k.first = now()
|
||||
}
|
||||
if now().Sub(k.first) < k.grace {
|
||||
return
|
||||
}
|
||||
busy := map[string]bool{}
|
||||
for _, j := range k.serve.Jobs() {
|
||||
@@ -96,6 +125,15 @@ func (k *judgeKeeper) check() {
|
||||
}
|
||||
k.tried[name] = now()
|
||||
var opts judgeServeOptions
|
||||
if !kj.GPU {
|
||||
// The cores init would give the same number of judges.
|
||||
var cpus []int
|
||||
if k.nodeCPUs != nil {
|
||||
cpus = k.nodeCPUs()
|
||||
}
|
||||
opts.Replicas = 2
|
||||
opts.Cores = refitCores(opts.Replicas, cpus)
|
||||
}
|
||||
if kj.GPU {
|
||||
if opts.GPUClass = roomiestGPUClass(k.gpus()); opts.GPUClass == "" {
|
||||
k.log.Warn("a policy names a judge that runs on a GPU, and this cluster has none",
|
||||
@@ -105,6 +143,8 @@ func (k *judgeKeeper) check() {
|
||||
}
|
||||
req := kj.serveRequest(opts)
|
||||
req.By = "llmgw (a guard policy names it)"
|
||||
// init, or an operator, may add it while this fetches: theirs stays.
|
||||
req.IfAbsent = true
|
||||
job, err := k.serve.Start(req)
|
||||
if err != nil {
|
||||
k.log.Error("could not start serving a judge a policy names", "judge", name, "err", err)
|
||||
@@ -114,3 +154,88 @@ func (k *judgeKeeper) check() {
|
||||
"judge", name, "repo", kj.Repo, "file", kj.File, "job", job.ID)
|
||||
}
|
||||
}
|
||||
|
||||
// refit moves a judge an earlier `server init` installed to what init
|
||||
// installs now, and restarts its replicas onto it. Those judges reserved no
|
||||
// cores or one, 2 or 4 GB, and ran llama.cpp's prompt cache: on fewer than
|
||||
// eight cores every judgment missed its 5 s budget and fail: closed refused
|
||||
// every request, until fixed by hand. A judge whose profile differs in any
|
||||
// other way was set by somebody, and is left as it is, said once.
|
||||
func (k *judgeKeeper) refit(m catalog.Model) {
|
||||
if k.profiles == nil || k.restart == nil || k.replicas == nil || m.Name != defaultJudge || !m.IsGuard() {
|
||||
return
|
||||
}
|
||||
if k.looked == nil {
|
||||
k.looked = map[string]bool{}
|
||||
}
|
||||
if k.looked[m.Name] {
|
||||
return
|
||||
}
|
||||
k.looked[m.Name] = true
|
||||
if !earlierInitJudge(m) {
|
||||
if len(m.Profiles) != 1 || !slices.Contains(m.Profiles[0].Args, "--cache-ram") {
|
||||
k.log.Info("the judge's profile is not one init wrote; left as it is",
|
||||
"judge", m.Name, "see", "llmgw ctl model resources "+m.Name)
|
||||
}
|
||||
return
|
||||
}
|
||||
var cpus []int
|
||||
if k.nodeCPUs != nil {
|
||||
cpus = k.nodeCPUs()
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
p.MinMemoryGB = judgeMemoryGB
|
||||
p.CPUCores = refitCores(m.Scaling.Min, cpus)
|
||||
p.Args = []string{"--ctx-size", "8192", "--cache-ram", "0"}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if _, err := k.profiles.SetProfiles(ctx, m.Name, []catalog.Profile{p}, "llmgw (the judge an earlier init installed)"); err != nil {
|
||||
k.log.Warn("could not refit the judge an earlier init installed", "judge", m.Name, "err", err)
|
||||
k.looked[m.Name] = false
|
||||
return
|
||||
}
|
||||
ids := k.replicas(m.Name)
|
||||
if len(ids) > 0 {
|
||||
if err := k.restart(ids...); err != nil {
|
||||
k.log.Warn("refitted the judge, but could not restart it", "judge", m.Name, "err", err)
|
||||
}
|
||||
}
|
||||
k.log.Info("refitted the judge an earlier init installed", "judge", m.Name,
|
||||
"memory_gb", p.MinMemoryGB, "cpu_cores", p.CPUCores, "replicas_restarted", len(ids))
|
||||
}
|
||||
|
||||
// earlierInitJudge is whether a judge's entry is exactly what `server init`
|
||||
// before v0.14.7 wrote: one llama.cpp profile on CPUs, an 8192 context and
|
||||
// no other args, 2 or 4 GB, none to two cores.
|
||||
func earlierInitJudge(m catalog.Model) bool {
|
||||
if len(m.Profiles) != 1 {
|
||||
return false
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
return p.Name == "cpu" && p.Engine == "llamacpp" && p.GPUClass == "" && p.GPU == nil &&
|
||||
len(p.NodeSelector) == 0 && slices.Equal(p.Args, []string{"--ctx-size", "8192"}) &&
|
||||
(p.MinMemoryGB == 2 || p.MinMemoryGB == 4) && p.CPUCores >= 0 && p.CPUCores <= 2
|
||||
}
|
||||
|
||||
// refitCores is a judge replica's cores by init's rule: up to four for the
|
||||
// one judge of a cluster, two each where there are more, fewer where the
|
||||
// smallest node cannot hold them all.
|
||||
func refitCores(replicas int, cpus []int) int {
|
||||
smallest := 0
|
||||
for _, c := range cpus {
|
||||
if c > 0 && (smallest == 0 || c < smallest) {
|
||||
smallest = c
|
||||
}
|
||||
}
|
||||
if replicas <= 1 {
|
||||
if smallest == 0 {
|
||||
return judgeCores
|
||||
}
|
||||
return min(smallest, judgeCoresAlone)
|
||||
}
|
||||
cores := judgeCores
|
||||
if len(cpus) == 1 && smallest > 0 && replicas*cores > smallest {
|
||||
cores = max(1, smallest/replicas)
|
||||
}
|
||||
return cores
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"slices"
|
||||
@@ -51,7 +52,8 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
|
||||
// The GPU judge alone: the CPU one is there, and one llmgw does not know
|
||||
// is not guessed at.
|
||||
if len(fs.started) != 1 || fs.started[0].Name != "qwen3guard-4b" || fs.started[0].GPUClass != "6000-24g" ||
|
||||
fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU {
|
||||
fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU ||
|
||||
!fs.started[0].IfAbsent {
|
||||
t.Fatalf("started %+v", fs.started)
|
||||
}
|
||||
|
||||
@@ -90,8 +92,12 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
|
||||
lib = guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
|
||||
inCatalog = nil
|
||||
k.gpus = func() []adminapi.ClassUse { return nil }
|
||||
k.nodeCPUs = func() []int { return []int{8, 4} }
|
||||
k.check()
|
||||
if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 {
|
||||
// Sized as init sizes its two judges: 2 GB, and two cores each on a
|
||||
// cluster whose smallest node has four.
|
||||
if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 ||
|
||||
last.MemoryGB != judgeMemoryGB || last.CPUCores != refitCores(2, []int{8, 4}) || last.CPUCores == 0 {
|
||||
t.Errorf("CPU judge: %+v", last)
|
||||
}
|
||||
fs.err = errors.New("the mirror is down")
|
||||
@@ -124,3 +130,108 @@ func TestCategoriesFollowTheJudges(t *testing.T) {
|
||||
t.Errorf("no categories (every one counts) became %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
type fakeProfiles struct {
|
||||
set map[string][]catalog.Profile
|
||||
}
|
||||
|
||||
func (f *fakeProfiles) SetProfiles(_ context.Context, name string, ps []catalog.Profile, _ string) (catalog.Model, error) {
|
||||
if f.set == nil {
|
||||
f.set = map[string][]catalog.Profile{}
|
||||
}
|
||||
f.set[name] = ps
|
||||
return catalog.Model{Name: name, Profiles: ps}, nil
|
||||
}
|
||||
|
||||
// A judge that v0.14.6 or earlier's init installed is moved to today's
|
||||
// figures, once, and its replicas restarted onto them; one somebody changed
|
||||
// is left alone.
|
||||
func TestAnEarlierInstallsJudgeIsRefittedOnce(t *testing.T) {
|
||||
lib := guard.Config{Policies: map[string]guard.Policy{
|
||||
"baseline": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}},
|
||||
}}
|
||||
earlier := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 1, nil)
|
||||
earlier.Profiles[0].MinMemoryGB, earlier.Profiles[0].Args = 4, []string{"--ctx-size", "8192"}
|
||||
inCatalog := []catalog.Model{earlier}
|
||||
fp := &fakeProfiles{}
|
||||
var restarted []string
|
||||
k := &judgeKeeper{
|
||||
library: func() guard.Config { return lib },
|
||||
models: func() []catalog.Model { return inCatalog },
|
||||
serve: &fakeServe{},
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
profiles: fp,
|
||||
restart: func(ids ...string) error { restarted = append(restarted, ids...); return nil },
|
||||
replicas: func(string) []string { return []string{"j-1", "j-2"} },
|
||||
nodeCPUs: func() []int { return []int{4} },
|
||||
}
|
||||
k.check()
|
||||
got := fp.set["qwen3guard-0.6b"]
|
||||
if len(got) != 1 || got[0].MinMemoryGB != 2 || got[0].CPUCores != 2 || !slices.Contains(got[0].Args, "--cache-ram") {
|
||||
t.Fatalf("refitted to %+v", got)
|
||||
}
|
||||
if !slices.Equal(restarted, []string{"j-1", "j-2"}) {
|
||||
t.Errorf("restarted %v", restarted)
|
||||
}
|
||||
k.check()
|
||||
if len(restarted) != 2 {
|
||||
t.Error("refitted twice")
|
||||
}
|
||||
|
||||
// Changed by somebody: an extra arg. Left as it is.
|
||||
mine := guardModel("http://x/g.gguf", 2, 3, nil)
|
||||
mine.Profiles[0].Args = []string{"--ctx-size", "16384"}
|
||||
inCatalog = []catalog.Model{mine}
|
||||
fp.set, k.looked = nil, nil
|
||||
k.check()
|
||||
if fp.set != nil {
|
||||
t.Errorf("a judge set by hand was changed: %+v", fp.set)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJudgeIsRefittedToTheCoresInitWouldGive(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
replicas int
|
||||
cpus []int
|
||||
want int
|
||||
}{
|
||||
{1, []int{4}, 4},
|
||||
{1, []int{2}, 2},
|
||||
{1, []int{16}, 4},
|
||||
{2, []int{4}, 2},
|
||||
{2, []int{2}, 1},
|
||||
{2, []int{4, 32}, 2},
|
||||
{2, nil, 2},
|
||||
} {
|
||||
if got := refitCores(c.replicas, c.cpus); got != c.want {
|
||||
t.Errorf("%d judges on %v: %d cores, want %d", c.replicas, c.cpus, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Just after the control plane starts, the judge init is adding is left to
|
||||
// init: the keeper serves one itself only after its grace.
|
||||
func TestTheKeeperLeavesAJustStartedInstallsJudgeToInit(t *testing.T) {
|
||||
lib := guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
|
||||
now := time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC)
|
||||
fs := &fakeServe{}
|
||||
k := &judgeKeeper{
|
||||
library: func() guard.Config { return lib },
|
||||
models: func() []catalog.Model { return nil },
|
||||
serve: fs, grace: judgeGrace,
|
||||
gpus: func() []adminapi.ClassUse { return nil },
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
now: func() time.Time { return now },
|
||||
}
|
||||
k.check()
|
||||
now = now.Add(judgeGrace / 2)
|
||||
k.check()
|
||||
if len(fs.started) != 0 {
|
||||
t.Fatalf("served within its grace: %+v", fs.started)
|
||||
}
|
||||
now = now.Add(judgeGrace)
|
||||
k.check()
|
||||
if len(fs.started) != 1 {
|
||||
t.Errorf("never served after its grace: %d", len(fs.started))
|
||||
}
|
||||
}
|
||||
@@ -142,7 +142,9 @@ func (k knownJudge) serveRequest(o judgeServeOptions) modelserve.Request {
|
||||
}
|
||||
return req
|
||||
}
|
||||
req.CPU, req.CPUCores = true, o.Cores
|
||||
// Sized as init sizes its judge: the memory it measured to need with
|
||||
// no prompt cache, and the cores its rule gives.
|
||||
req.CPU, req.CPUCores, req.MemoryGB = true, o.Cores, judgeMemoryGB
|
||||
if req.Replicas == 0 {
|
||||
req.Replicas = 2
|
||||
}
|
||||
|
||||
+27
-1
@@ -7,6 +7,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"path"
|
||||
"slices"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
@@ -301,7 +302,8 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
var perMember bool
|
||||
var rpm, tpm int
|
||||
var budget float64
|
||||
var models []string
|
||||
var models, tools []string
|
||||
var replace bool
|
||||
|
||||
cmd := &cobra.Command{
|
||||
Use: "limits",
|
||||
@@ -357,7 +359,19 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
return errors.New("--per-member is for --scope org, team or project")
|
||||
}
|
||||
}
|
||||
// Only what is given changes: setting a tool grant must not
|
||||
// clear the scope's rate or budget. --replace writes the row
|
||||
// whole, a flag not given then no limit, as the console's
|
||||
// form does with every field it shows.
|
||||
l := store.Limits{Scope: level, ScopeID: scopeID}
|
||||
if !replace {
|
||||
switch cur, err := s.LimitsFor(cmd.Context(), level, scopeID); {
|
||||
case err == nil:
|
||||
l = cur
|
||||
case !errors.Is(err, store.ErrNotFound):
|
||||
return err
|
||||
}
|
||||
}
|
||||
if cmd.Flags().Changed("rpm") {
|
||||
l.RPM = &rpm
|
||||
}
|
||||
@@ -379,6 +393,14 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
}
|
||||
l.AllowedModels = models
|
||||
}
|
||||
if cmd.Flags().Changed("tools") {
|
||||
for _, p := range tools {
|
||||
if _, err := path.Match(p, "x.y"); err != nil {
|
||||
return fmt.Errorf("--tools %q is not a pattern: %w", p, err)
|
||||
}
|
||||
}
|
||||
l.AllowedTools = tools
|
||||
}
|
||||
if err := s.PutLimits(cmd.Context(), l); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -395,6 +417,10 @@ func newLimitsCmd(open opener) *cobra.Command {
|
||||
cmd.Flags().Float64Var(&budget, "budget", 0, "budget per period, in whole units as the quotas page shows them")
|
||||
cmd.Flags().StringVar(&period, "period", "month", "day | month")
|
||||
cmd.Flags().StringSliceVar(&models, "models", nil, "allowed models; empty allows none")
|
||||
cmd.Flags().BoolVar(&replace, "replace", false,
|
||||
"write every limit at the scope: one not given is then no limit there (default: only what is given changes)")
|
||||
cmd.Flags().StringSliceVar(&tools, "tools", nil,
|
||||
"the company MCP servers' tools granted here, as patterns: jira.*,db.query; none granted anywhere is none")
|
||||
return cmd
|
||||
}
|
||||
|
||||
|
||||
+16
-1
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/user"
|
||||
"slices"
|
||||
@@ -32,7 +33,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
Use: "model",
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelPresetsCmd(), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
c.AddCommand(newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newTargetCmd(client, asJSON), newAdaptersCmd(client),
|
||||
newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
|
||||
@@ -73,6 +74,20 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
return fmt.Errorf("no preset %q: there are %s; any other model is added from a file (-f)",
|
||||
args[0], strings.Join(catalog.PresetNames(), ", "))
|
||||
}
|
||||
// The control plane adds it, and keeps that it was added as
|
||||
// a preset, by whom; one from before it could is sent the
|
||||
// entry, as it was then.
|
||||
var res adminapi.ModelAdded
|
||||
err := client().post(cmd.Context(), "/api/v1/presets/"+url.PathEscape(p.Model.Name), []byte("{}"), &res)
|
||||
if err == nil {
|
||||
fmt.Fprintf(cmd.OutOrStdout(),
|
||||
"%s added, and ComfyUI with it where the cluster had none. It is placed on the next\n"+
|
||||
"reconcile pass and loads when first asked for; watch it with:\n llmgw ctl models\n", p.Model.Name)
|
||||
return nil
|
||||
}
|
||||
if !olderControlPlane(err) {
|
||||
return err
|
||||
}
|
||||
m = p.Model
|
||||
case path == "":
|
||||
return fmt.Errorf("a preset (%s) or --file, the model definition as YAML",
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
@@ -11,25 +13,106 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func newModelPresetsCmd() *cobra.Command {
|
||||
return &cobra.Command{
|
||||
func newModelPresetsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "presets",
|
||||
Short: "The models that come with llmgw, added by name",
|
||||
Short: "The models that come with llmgw: whether each fits and is here",
|
||||
Long: "Each is a catalog entry whose workflow, custom nodes, packages and licence\n" +
|
||||
"were found and tried, and whose profile says the card it needs rather than\n" +
|
||||
"a class, so it places on any site's cards. Add one with\n" +
|
||||
"`llmgw ctl model add NAME`.",
|
||||
"`llmgw ctl model add NAME`.\n\n" +
|
||||
"While presets.auto is on, the leader adds one by itself when a card fits it\n" +
|
||||
"and the catalog has nothing that makes what it makes; one somebody removed\n" +
|
||||
"is not added again by itself.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "NAME\tMAKES\tCARD\tLICENCE\tWHAT IT IS FOR")
|
||||
for _, p := range catalog.Presets() {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", p.Model.Name, p.Makes, presetCard(p.Model),
|
||||
presetLicence(p.Model), p.Model.Description)
|
||||
var v adminapi.PresetsView
|
||||
err := client().get(cmd.Context(), "/api/v1/presets", &v)
|
||||
if err != nil && !olderControlPlane(err) {
|
||||
return err
|
||||
}
|
||||
return w.Flush()
|
||||
if *asJSON && err == nil {
|
||||
return emitJSON(v)
|
||||
}
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
if err != nil {
|
||||
// A control plane from before it kept them: what this
|
||||
// release has, with nothing about the cluster.
|
||||
fmt.Fprintln(w, "NAME\tMAKES\tCARD\tLICENCE\tWHAT IT IS FOR")
|
||||
for _, p := range catalog.Presets() {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", p.Model.Name, p.Makes, presetCard(p.Model),
|
||||
presetLicence(p.Model), p.Model.Description)
|
||||
}
|
||||
return w.Flush()
|
||||
}
|
||||
fmt.Fprintln(w, "NAME\tMAKES\tCARD\tLICENCE\tFITS\tHERE\tWHAT IT IS FOR")
|
||||
for _, p := range v.Presets {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", p.Name, p.Makes, p.Card, licenceText(p.Licence),
|
||||
orDash(strings.Join(p.FitsClasses, ",")), presetState(p), p.Description)
|
||||
}
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if v.Auto {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "\nAdded by themselves where a card fits (presets.auto); "+
|
||||
"llmgw ctl model presets auto off stops it.")
|
||||
} else {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "\nNot added by themselves; llmgw ctl model presets auto on starts it.")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "auto on|off",
|
||||
Short: "Whether the leader adds a preset by itself where a card fits it",
|
||||
Args: cobra.ExactArgs(1),
|
||||
ValidArgs: []string{"on", "off"},
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if args[0] != "on" && args[0] != "off" {
|
||||
return fmt.Errorf("on or off, not %q", args[0])
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.PresetsAutoRequest{On: args[0] == "on"})
|
||||
var v adminapi.PresetsView
|
||||
if err := client().put(cmd.Context(), "/api/v1/presets/auto", body, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if v.Auto {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Presets are added by themselves where a card fits.")
|
||||
} else {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Presets are no longer added by themselves.")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
})
|
||||
return c
|
||||
}
|
||||
|
||||
// olderControlPlane is an error from a control plane that has no such path:
|
||||
// one from before it was added.
|
||||
func olderControlPlane(err error) bool {
|
||||
s := err.Error()
|
||||
return strings.Contains(s, "answered with a page") || strings.HasPrefix(s, "404 Not Found: 404 page not found")
|
||||
}
|
||||
|
||||
func presetState(p adminapi.PresetView) string {
|
||||
switch {
|
||||
case p.InCatalog && p.AddedBy == "auto":
|
||||
return "added by itself"
|
||||
case p.InCatalog && p.AddedBy != "":
|
||||
return "added by " + p.AddedBy
|
||||
case p.InCatalog:
|
||||
return "here"
|
||||
case p.Declined:
|
||||
return "removed; stays out"
|
||||
}
|
||||
return "—"
|
||||
}
|
||||
|
||||
func licenceText(l catalog.License) string {
|
||||
if l.Commercial != nil && !*l.Commercial {
|
||||
return l.Name + ", not commercial"
|
||||
}
|
||||
return l.Name
|
||||
}
|
||||
|
||||
func presetCard(m catalog.Model) string {
|
||||
|
||||
@@ -17,6 +17,10 @@ import (
|
||||
// presetAdmin is an admin API holding settings, which records the models
|
||||
// added and the settings put.
|
||||
type presetAdmin struct {
|
||||
// api is a control plane that adds presets itself; without it, one
|
||||
// from before, which answers their paths with a 404.
|
||||
api bool
|
||||
byAPI []string
|
||||
mu sync.Mutex
|
||||
settings string
|
||||
added []catalog.Model
|
||||
@@ -30,6 +34,10 @@ func (a *presetAdmin) client(t *testing.T) func() *ctlClient {
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch {
|
||||
case a.api && r.Method == http.MethodPost && strings.HasPrefix(r.URL.Path, "/api/v1/presets/"):
|
||||
name := strings.TrimPrefix(r.URL.Path, "/api/v1/presets/")
|
||||
a.byAPI = append(a.byAPI, name)
|
||||
_ = json.NewEncoder(w).Encode(adminapi.ModelAdded{Added: name})
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/api/v1/models":
|
||||
var m catalog.Model
|
||||
_ = json.Unmarshal(body, &m)
|
||||
@@ -62,6 +70,19 @@ func runModelAdd(t *testing.T, client func() *ctlClient, args ...string) (string
|
||||
return out.String(), err
|
||||
}
|
||||
|
||||
func TestAPresetIsAddedByTheControlPlane(t *testing.T) {
|
||||
a := &presetAdmin{api: true}
|
||||
out, err := runModelAdd(t, a.client(t), "triposr")
|
||||
if err != nil {
|
||||
t.Fatalf("%v\n%s", err, out)
|
||||
}
|
||||
if len(a.byAPI) != 1 || a.byAPI[0] != "triposr" || len(a.added) != 0 || len(a.put) != 0 {
|
||||
t.Errorf("by the API %v; sent the entry %v; settings %v", a.byAPI, a.added, a.put)
|
||||
}
|
||||
}
|
||||
|
||||
// A control plane from before presets were its own is sent the entry, and
|
||||
// ComfyUI set beside it, as the release that brought presets did.
|
||||
func TestAPresetIsAddedByNameAndHasTheNodesInstallComfyUI(t *testing.T) {
|
||||
a := &presetAdmin{settings: "router:\n retries: 2\n"}
|
||||
out, err := runModelAdd(t, a.client(t), "triposr")
|
||||
|
||||
@@ -60,6 +60,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
f.StringVar(&req.Quant, "quant", "", "the quantisation to serve (default: the one that suits the cluster best)")
|
||||
f.StringVar(&req.File, "file", "", "the file to serve, instead of choosing by --quant")
|
||||
f.StringVar(&req.Projector, "projector", "", "a vision model's mmproj file, fetched and loaded beside it")
|
||||
f.BoolVar(&req.Embeddings, "embeddings", false, "serve it as an embedding model (its GGUF header says so for most): "+
|
||||
"--embedding, 32 slots of 1024 tokens unless --ctx-pool or --context-length says otherwise")
|
||||
f.StringVar(&req.Format, "format", "gguf", "gguf, run by llama.cpp, or safetensors: the original weights, run by vLLM on GPUs")
|
||||
f.BoolVar(&req.CPU, "cpu", false, "run it on the CPU, whatever the plan says")
|
||||
f.StringVar(&req.GPUClass, "gpu-class", "", "run it on GPUs of this class, whatever the plan says")
|
||||
@@ -67,6 +69,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
|
||||
f.IntVar(&req.TP, "tp", 0, "the older name of --gpus")
|
||||
_ = f.MarkHidden("tp")
|
||||
f.StringToStringVar(&req.Env, "env", nil,
|
||||
"KEY=VALUE in the engine's environment on every node, e.g. NCCL_DEBUG=INFO; repeatable")
|
||||
f.BoolVar(&req.Fallbacks, "fallbacks", false, "also add the next best ways to run it, used when the first is busy")
|
||||
f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
|
||||
f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
|
||||
|
||||
@@ -291,14 +291,14 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
}
|
||||
fmt.Fprintln(out)
|
||||
w := tabTo(out)
|
||||
fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tHOLDS\tOF\tNOW")
|
||||
fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tA CARD\tOF\tALL CARDS\tNOW")
|
||||
for _, a := range p.Artifacts {
|
||||
name := a.Name
|
||||
if a.Shards > 1 && a.Format == "gguf" {
|
||||
name += fmt.Sprintf(" (%d parts)", a.Shards)
|
||||
}
|
||||
if a.Best == nil {
|
||||
fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\n", name, bytesShort(a.Bytes))
|
||||
fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\t\n", name, bytesShort(a.Bytes))
|
||||
continue
|
||||
}
|
||||
b := a.Best
|
||||
@@ -306,11 +306,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
if b.FreeNow {
|
||||
now = "free"
|
||||
}
|
||||
holds := fmt.Sprintf("%.1f GB a card", b.NeedGB)
|
||||
holds, all := fmt.Sprintf("%.1f GB", b.NeedGB), fmt.Sprintf("%.0f GB", b.TotalGB())
|
||||
if b.CPU {
|
||||
holds = fmt.Sprintf("%.1f GB of RAM", b.NeedGB)
|
||||
holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, now)
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
|
||||
}
|
||||
_ = w.Flush()
|
||||
rec, ok := p.Recommended()
|
||||
@@ -318,7 +318,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
|
||||
return errDoesNotFit
|
||||
}
|
||||
fmt.Fprintf(out, "\nBest: %s on %s.\n", rec.Artifact, rec.Label())
|
||||
fmt.Fprintf(out, "\nBest: %s on %s: %s.\n", rec.Artifact, rec.Label(), rec.MemoryText())
|
||||
if p.KVBytesPerToken > 0 {
|
||||
fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n",
|
||||
modelplan.GB(p.KVBytesPerToken*1000))
|
||||
}
|
||||
if len(p.Options) > 1 {
|
||||
fmt.Fprintln(out, "\nEvery way, best first:")
|
||||
w = tabTo(out)
|
||||
@@ -327,7 +331,7 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
if o.FreeNow {
|
||||
now = "free"
|
||||
}
|
||||
fmt.Fprintf(w, " %s\t%s\t%.1f of %.0f GB\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.NeedGB, o.HaveGB, o.Hosts, now)
|
||||
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now)
|
||||
}
|
||||
_ = w.Flush()
|
||||
}
|
||||
|
||||
@@ -131,3 +131,33 @@ func TestKeyBudgetsAreWholeUnits(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A limit given alone changes that limit alone: granting a tool keeps the
|
||||
// scope's rate. --replace writes the row whole, as the console's form does.
|
||||
func TestALimitGivenAloneLeavesTheOthersAtTheScope(t *testing.T) {
|
||||
db := filepath.Join(t.TempDir(), "s.db")
|
||||
mustRun(t, db, "org", "add", "org", "Acme")
|
||||
mustRun(t, db, "org", "add", "team", "platform")
|
||||
mustRun(t, db, "org", "add", "project", "gateway")
|
||||
mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--rpm", "30")
|
||||
mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--tools", "jira.*")
|
||||
read := func() store.Limits {
|
||||
s, err := store.Open(context.Background(), db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
all, _ := s.AllLimits(context.Background())
|
||||
for _, l := range all {
|
||||
return l
|
||||
}
|
||||
return store.Limits{}
|
||||
}
|
||||
if l := read(); l.RPM == nil || *l.RPM != 30 || len(l.AllowedTools) != 1 {
|
||||
t.Errorf("after a tool grant: rpm %v, tools %v", l.RPM, l.AllowedTools)
|
||||
}
|
||||
mustRun(t, db, "keys", "limits", "--replace", "--scope", "project", "--id", "gateway", "--tpm", "1000")
|
||||
if l := read(); l.RPM != nil || l.AllowedTools != nil || l.TPM == nil {
|
||||
t.Errorf("after --replace: %+v", l)
|
||||
}
|
||||
}
|
||||
@@ -1,8 +1,11 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
@@ -70,6 +73,9 @@ func newSelfUpgradeCmd() *cobra.Command {
|
||||
if strings.TrimSpace(string(b)) == "active" {
|
||||
if steady++; steady >= 3 {
|
||||
fmt.Fprintf(out, "%s restarted and running\n", service)
|
||||
if service == "llmgw-server" {
|
||||
nodesLeftBehind(cmd.Context(), out, newCtlClient("127.0.0.1:9090"), v)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
} else {
|
||||
@@ -85,3 +91,25 @@ func newSelfUpgradeCmd() *cobra.Command {
|
||||
c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
|
||||
return c
|
||||
}
|
||||
|
||||
// nodesLeftBehind says, after a control plane alone was upgraded, that its
|
||||
// nodes were not, and how they are: this command looks like the cluster's
|
||||
// upgrade and is not, and a site's nodes stayed a release behind after it.
|
||||
func nodesLeftBehind(ctx context.Context, out io.Writer, c *ctlClient, v string) {
|
||||
var nodes []adminapi.NodeView
|
||||
_ = c.patient(20*time.Second).get(ctx, "/api/v1/nodes", &nodes)
|
||||
behind := 0
|
||||
for _, n := range nodes {
|
||||
if n.AgentVersion != v {
|
||||
behind++
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case len(nodes) == 0:
|
||||
fmt.Fprintf(out, "This upgraded this machine only. The nodes are upgraded from the leading "+
|
||||
"control plane with:\n sudo llmgw ctl upgrade %s --skip-control-plane\n", v)
|
||||
case behind > 0:
|
||||
fmt.Fprintf(out, "This upgraded this machine only: %d of %d node(s) are not on %s. On the leading "+
|
||||
"control plane, run:\n sudo llmgw ctl upgrade %s --skip-control-plane\n", behind, len(nodes), v, v)
|
||||
}
|
||||
}
|
||||
+85
-15
@@ -76,6 +76,7 @@ func newServerCmd() *cobra.Command {
|
||||
|
||||
cmd := &cobra.Command{
|
||||
Use: "server",
|
||||
Args: cobra.NoArgs,
|
||||
Short: "Run the router (phase 1; the control plane arrives in phase 3)",
|
||||
Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
|
||||
"In this phase the replica list is static: give it in the config file, or\n" +
|
||||
@@ -138,6 +139,26 @@ func newServerCmd() *cobra.Command {
|
||||
_ = shutdownTracing(sctx)
|
||||
}()
|
||||
|
||||
// The ports before anything else: a second server on this machine,
|
||||
// whose ports are taken, must stop here, not go on to compete for
|
||||
// the lease and lead a cluster no request can reach.
|
||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||
}
|
||||
if cfg.MaxConnections > 0 {
|
||||
// Past this, new connections wait in the kernel's backlog
|
||||
// rather than each taking a goroutine and its buffers.
|
||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||
if err != nil {
|
||||
_ = apiLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||
}
|
||||
defer apiLn.Close()
|
||||
defer adminLn.Close()
|
||||
|
||||
rt, err := router.New(cfg, log)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -264,6 +285,7 @@ func newServerCmd() *cobra.Command {
|
||||
agentSvc = svc
|
||||
// Stopped by the operators (llmgw stop): nothing placed.
|
||||
ctl.Reconciler().SetStopped(cfg.Serving.Stopped)
|
||||
ctl.Reconciler().SetEvictionTimes(cfg.OnDemand.EvictAfterIdle, cfg.OnDemand.MinLifetime)
|
||||
// Whether nodes keep llmgw's firewall, as the settings say now.
|
||||
svc.SetFirewall(func() *llmgwv1.Firewall {
|
||||
c := current.Load()
|
||||
@@ -417,6 +439,9 @@ func newServerCmd() *cobra.Command {
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
api.SetMCP(st)
|
||||
rt.SetAdminMCP(http.HandlerFunc(api.MCP))
|
||||
api.SetAuth(&adminapi.AuthAdmin{Status: rt.AuthStatus, TestLDAP: rt.TestDirectory, Secrets: st, Log: log})
|
||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||
Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
|
||||
// And the same document on the portal, for administrators.
|
||||
@@ -465,6 +490,14 @@ func newServerCmd() *cobra.Command {
|
||||
}
|
||||
api.SetModels(svc)
|
||||
catalogSvc = svc
|
||||
// The models that come with llmgw: added by the leader
|
||||
// when a node fits one and the catalog makes nothing of
|
||||
// its kind, unless somebody removed it or turned it off.
|
||||
api.SetPresets(st, func() adminapi.PresetConfig {
|
||||
c := current.Load()
|
||||
return adminapi.PresetConfig{Auto: c.Presets.Auto, ComfyUI: c.Engines[catalog.EngineComfyUI]}
|
||||
})
|
||||
go runPresetsPass(ctx, api, admin.Leadership(), log)
|
||||
// And the mirror page, so a model that has just been
|
||||
// fetched can be served without writing YAML by hand.
|
||||
rt.SetCatalogAdmin(svc)
|
||||
@@ -576,6 +609,7 @@ func newServerCmd() *cobra.Command {
|
||||
// A judge a policy names and the catalog lacks is
|
||||
// fetched and served, on the leader.
|
||||
keeper := &judgeKeeper{
|
||||
grace: judgeGrace,
|
||||
library: gate.Library,
|
||||
models: func() []catalog.Model { return admin.Catalog().Models() },
|
||||
serve: serve,
|
||||
@@ -584,6 +618,25 @@ func newServerCmd() *cobra.Command {
|
||||
},
|
||||
gpus: func() []adminapi.ClassUse { return adminAPI.Summary().GPUs },
|
||||
log: log,
|
||||
// An earlier install's judge, moved to today's figures.
|
||||
profiles: catalogSvc,
|
||||
restart: admin.Reconciler().Restart,
|
||||
replicas: func(model string) []string {
|
||||
var ids []string
|
||||
for _, r := range admin.Reconciler().ReadyReplicas() {
|
||||
if r.Model == model {
|
||||
ids = append(ids, r.ID)
|
||||
}
|
||||
}
|
||||
return ids
|
||||
},
|
||||
nodeCPUs: func() []int {
|
||||
var out []int
|
||||
for _, n := range admin.Nodes() {
|
||||
out = append(out, n.CPUCount)
|
||||
}
|
||||
return out
|
||||
},
|
||||
}
|
||||
go keeper.run(ctx, 30*time.Second)
|
||||
}
|
||||
@@ -633,21 +686,6 @@ func newServerCmd() *cobra.Command {
|
||||
ReadTimeout: time.Minute,
|
||||
})
|
||||
|
||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||
}
|
||||
if cfg.MaxConnections > 0 {
|
||||
// Past this, new connections wait in the kernel's backlog
|
||||
// rather than each taking a goroutine and its buffers.
|
||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||
if err != nil {
|
||||
_ = apiLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||
}
|
||||
|
||||
// A metrics-only listener, when one is configured. Started after
|
||||
// AdminHandler, which is where the registry it serves is built.
|
||||
var metricsSrv *http.Server
|
||||
@@ -770,6 +808,9 @@ func newServerCmd() *cobra.Command {
|
||||
fwPub.Kick()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
|
||||
admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
|
||||
// A moment apart per node, so off this goroutine.
|
||||
go agentSvc.BroadcastAgentSettings()
|
||||
@@ -1281,3 +1322,32 @@ func embedder(replay func(ctx context.Context, path, model string, body []byte)
|
||||
return res.Data[0].Embedding, nil
|
||||
}
|
||||
}
|
||||
|
||||
// presetsEvery is how often the leader looks for presets to add; the first
|
||||
// look waits as long, for the nodes to have registered their cards.
|
||||
const presetsEvery = time.Minute
|
||||
|
||||
// runPresetsPass has the leader add the presets a node fits and the catalog
|
||||
// lacks, now and then.
|
||||
func runPresetsPass(ctx context.Context, api *adminapi.API, lead *control.Leadership, log *slog.Logger) {
|
||||
t := time.NewTicker(presetsEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
if lead != nil && !lead.IsLeader() {
|
||||
continue
|
||||
}
|
||||
added, err := api.PresetsPass(ctx)
|
||||
for _, name := range added {
|
||||
log.Info("added a preset by itself: a node fits it and the catalog made nothing of its kind",
|
||||
"model", name, "off_switch", "presets.auto")
|
||||
}
|
||||
if err != nil {
|
||||
log.Warn("could not add a preset", "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// `llmgw server check` once ran a whole second server: words after server
|
||||
// that name no subcommand are refused.
|
||||
func TestServerTakesNoWordsItDoesNotKnow(t *testing.T) {
|
||||
cmd := newServerCmd()
|
||||
cmd.SetArgs([]string{"check", "--config", "/nonexistent"})
|
||||
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||
if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "unknown command") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A second server whose port is taken stops at once, before the router or
|
||||
// any control plane is built.
|
||||
func TestAServerWhosePortIsTakenStopsFirst(t *testing.T) {
|
||||
taken, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer taken.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
cmd := newServerCmd()
|
||||
cmd.SetArgs([]string{"--listen", taken.Addr().String(), "--admin-listen", "127.0.0.1:0",
|
||||
"--endpoint", "a=http://127.0.0.1:1=m"})
|
||||
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||
if err := cmd.ExecuteContext(ctx); err == nil || !strings.Contains(err.Error(), "listening on") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"hash/fnv"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
@@ -44,6 +45,11 @@ type serverAdd struct {
|
||||
// addr is the machine's address as given; ip what the others reach it at.
|
||||
addr, ip, name string
|
||||
out io.Writer
|
||||
// priority is its keepalived priority, under this machine's 110; 0 is
|
||||
// 100. Two standbys get two, so which takes the address is not a tie.
|
||||
priority int
|
||||
// quiet leaves out the closing sentence, for init, which says its own.
|
||||
quiet bool
|
||||
}
|
||||
|
||||
func newServerAddCmd() *cobra.Command {
|
||||
@@ -148,7 +154,9 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
||||
return fmt.Errorf("%s: %w", s.name, err)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(sa.out, "\n%s is a control plane of this cluster, standing by while this one leads.\n", sa.name)
|
||||
if !sa.quiet {
|
||||
fmt.Fprintf(sa.out, "\n%s is a control plane of this cluster, standing by while this one leads.\n", sa.name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -511,18 +519,31 @@ func (sa *serverAdd) floating(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
here := pgha.Local{}
|
||||
priority := sa.priority
|
||||
if priority <= 0 {
|
||||
priority = 100
|
||||
}
|
||||
for _, m := range []struct {
|
||||
r onboard.Remote
|
||||
name string
|
||||
priority int
|
||||
}{{here, "this machine", 110}, {sa.remote, sa.name, 100}} {
|
||||
}{{here, "this machine", 110}, {sa.remote, sa.name, priority}} {
|
||||
// The interface on the address's network, by its route: not the
|
||||
// route to the address, which is lo on the machine holding it
|
||||
// already, as a run after the first finds it.
|
||||
out, err := m.r.Run(ctx, fmt.Sprintf(`set -e
|
||||
command -v keepalived >/dev/null 2>&1 || { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y keepalived >/dev/null; }
|
||||
ip -o route get %s | awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}'`, vip), nil)
|
||||
dev() { awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}'; }
|
||||
d=$(ip -o -4 route show %s | dev)
|
||||
[ -n "$d" ] && [ "$d" != lo ] || d=$(ip -o route get %s | dev)
|
||||
echo "$d"`, nw.String(), vip), nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
iface := strings.TrimSpace(string(out))
|
||||
if iface == "" || iface == "lo" {
|
||||
return fmt.Errorf("on %s: no interface on %s's network", m.name, sa.vip)
|
||||
}
|
||||
conf := keepalivedConf(iface, vip.String(), prefix, m.priority, pass)
|
||||
if out, err := m.r.Run(ctx, `set -e
|
||||
sudo install -d -m 0755 /etc/keepalived
|
||||
@@ -560,6 +581,17 @@ func vrrpPassword(etcDir string) (string, error) {
|
||||
return pass, nil
|
||||
}
|
||||
|
||||
// vrrpRouterID is the VRRP router id for a floating address: from the
|
||||
// address, so two clusters on one network, each with its own address, do
|
||||
// not answer each other's adverts — with one id for all, each logged the
|
||||
// other's as an invalid password, and dropped its own address's
|
||||
// announcements with them.
|
||||
func vrrpRouterID(vip string) int {
|
||||
h := fnv.New32a()
|
||||
_, _ = h.Write([]byte(vip))
|
||||
return int(h.Sum32()%255) + 1
|
||||
}
|
||||
|
||||
func keepalivedConf(iface, vip string, prefix, priority int, pass string) string {
|
||||
return fmt.Sprintf(`# llmgw: the gateway's floating address, on whichever control plane leads.
|
||||
vrrp_script llmgw_leader {
|
||||
@@ -572,7 +604,7 @@ vrrp_script llmgw_leader {
|
||||
vrrp_instance llmgw {
|
||||
state BACKUP
|
||||
interface %s
|
||||
virtual_router_id 51
|
||||
virtual_router_id %d
|
||||
priority %d
|
||||
advert_int 1
|
||||
authentication {
|
||||
@@ -586,7 +618,7 @@ vrrp_instance llmgw {
|
||||
llmgw_leader
|
||||
}
|
||||
}
|
||||
`, iface, priority, pass, vip, prefix)
|
||||
`, iface, vrrpRouterID(vip), priority, pass, vip, prefix)
|
||||
}
|
||||
|
||||
// clientCapable says a control plane's certificate can also present itself
|
||||
|
||||
+139
-18
@@ -73,10 +73,20 @@ type serverInit struct {
|
||||
// patroni are user@host of the machines to keep the store on under
|
||||
// Patroni, replicated; this machine is the third etcd vote for two.
|
||||
patroni []string
|
||||
// webhooks are where the cluster's alerts go: put into its settings
|
||||
// once it runs, and each sent a test.
|
||||
webhooks []string
|
||||
// patroniNodes are the same machines already connected to, which the
|
||||
// setup screen does with the password it was given; when set, patroni
|
||||
// only names them.
|
||||
patroniNodes []pgha.Node
|
||||
// judgeMachines are machines for the guardrails' judges alone, added as
|
||||
// nodes labelled for them; judgeCPUs their cores, once added.
|
||||
judgeMachines []judgeMachine
|
||||
judgeCPUs []int
|
||||
// judgeNote is what timing the judge found, said before the next
|
||||
// steps when it is not fine.
|
||||
judgeNote string
|
||||
// noMirror, noNode and noModel turn off the parts of a first install that
|
||||
// make it serve something: a mirror, this machine as a node, and a small
|
||||
// model on it.
|
||||
@@ -92,9 +102,34 @@ type serverInit struct {
|
||||
// bundle is an offline bundle to install from, and bundleSHA its
|
||||
// manifest's sha256: the cluster is then set up offline.
|
||||
bundle, bundleSHA string
|
||||
// mirrorPath is the mirror's directory here; "" is /srv/models. A seam
|
||||
// for tests.
|
||||
// mirrorPath is the mirror's directory here; "" is /srv/models.
|
||||
mirrorPath string
|
||||
|
||||
// What setup lets a person choose besides the parts: zero is the
|
||||
// default (serverinit_choices.go).
|
||||
starter starterSpec
|
||||
starterCtx int
|
||||
judge judgeSpec
|
||||
// language is the portal's for a browser that asks for neither of its
|
||||
// own; "" leaves it English.
|
||||
language string
|
||||
signupOpen bool
|
||||
defaultRPM, defaultTPM int
|
||||
maxKeyTTL time.Duration
|
||||
// media has the nodes install ComfyUI and the leader add a picture model
|
||||
// and a 3D model once a card fits one.
|
||||
media bool
|
||||
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
|
||||
evictAfterIdle, minLifetime time.Duration
|
||||
// standbys makes each machine of the replicated store a control plane
|
||||
// too, standing by while this one leads: three machines, each able to
|
||||
// take over. vip is a floating address, as 10.0.0.160/24, kept on
|
||||
// whichever leads.
|
||||
standbys bool
|
||||
vip string
|
||||
// cpus is this machine's CPU count for dividing it between init's
|
||||
// models; zero asks the machine. A seam for tests.
|
||||
cpus int
|
||||
// pgSetup prepares PostgreSQL and returns the store address; a seam for
|
||||
// tests. Nil does it for real.
|
||||
pgSetup func(context.Context) (string, error)
|
||||
@@ -119,7 +154,10 @@ func newDefaultServerInit() serverInit {
|
||||
|
||||
func newServerInitCmd() *cobra.Command {
|
||||
in := newDefaultServerInit()
|
||||
var plain bool
|
||||
var (
|
||||
plain bool
|
||||
starter, judgeCats string
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
Use: "init",
|
||||
Short: "Set this machine up as a control plane and start it",
|
||||
@@ -155,6 +193,9 @@ func newServerInitCmd() *cobra.Command {
|
||||
if os.Geteuid() != 0 {
|
||||
return errors.New("init creates a user, writes /etc and starts a service: run it as root")
|
||||
}
|
||||
if err := in.readChoices(starter, judgeCats); err != nil {
|
||||
return err
|
||||
}
|
||||
in.out = cmd.OutOrStdout()
|
||||
if !plain && isTerminal(in.out) {
|
||||
return in.runInitTUI(cmd.Context(), in.out)
|
||||
@@ -182,9 +223,42 @@ func newServerInitCmd() *cobra.Command {
|
||||
"leave the firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
|
||||
f.BoolVar(&in.postgres, "postgres", false,
|
||||
"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
|
||||
f.StringArrayVar(&in.webhooks, "webhook", nil,
|
||||
"where alerts go — a model parked, a node lost, a budget spent: an http(s) address, Slack's, Mattermost's "+
|
||||
"and Rocket.Chat's sent Slack's text, any other the event as JSON (slack: or json: in front says which); repeatable")
|
||||
f.StringArrayVar(&in.patroni, "patroni", nil,
|
||||
"user@host of a machine to keep the store on, replicated under Patroni; give two or more. "+
|
||||
"Installs PostgreSQL, Patroni and etcd there over ssh, and moves the existing store in")
|
||||
|
||||
// What server.yaml is written with; each is in it only when init writes
|
||||
// it, never into one already there.
|
||||
f.StringVar(&in.mirrorPath, "mirror-dir", mirrorDir, "the directory the mirror on this machine keeps model files in")
|
||||
f.StringVar(&starter, "starter", starterName,
|
||||
"the starter model: "+strings.Join(starterNames(), ", ")+", or hf:<owner>/<repo>/<file>.gguf")
|
||||
f.IntVar(&in.starterCtx, "starter-context", defaultContext, "the context, in tokens, a client may use of the starter model")
|
||||
f.StringVar(&in.judge.Fail, "judge-fail", "closed",
|
||||
"a request no judge read in time: closed refuses it, open serves it flagged")
|
||||
f.StringVar(&judgeCats, "judge-categories", strings.Join(defaultJudgeCategories, ","),
|
||||
"what the judge refuses, comma-separated, of: "+strings.Join(judgeCategories, ", "))
|
||||
f.IntVar(&in.judge.Replicas, "judge-replicas", 2, "how many copies of the judge run, each on CPUs")
|
||||
f.Var(judgeMachineFlag{&in.judgeMachines}, "judge-node",
|
||||
"user@host of a machine for the guardrails' judges alone: added as a node, and the judges run only there; repeatable")
|
||||
f.DurationVar(&in.judge.Budget, "judge-budget", 5*time.Second, "how long a request waits for the judge")
|
||||
f.StringVar(&in.language, "language", "", "the portal's language for a browser that asks for neither: en or tr")
|
||||
f.BoolVar(&in.signupOpen, "signup-open", false, "let anyone ask for an account on the portal, to be approved by an administrator")
|
||||
f.IntVar(&in.defaultRPM, "default-rpm", defaultRPM, "requests a minute a key may make when nothing gives it a limit")
|
||||
f.IntVar(&in.defaultTPM, "default-tpm", defaultTPM, "tokens a minute a key may use when nothing gives it a limit")
|
||||
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
|
||||
f.BoolVar(&in.media, "media", false,
|
||||
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
|
||||
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
|
||||
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
|
||||
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
|
||||
"how long a model just loaded on demand is safe from that (0: 5m)")
|
||||
f.BoolVar(&in.standbys, "standbys", false,
|
||||
"with --patroni, make each of those machines a control plane too, standing by while this one leads")
|
||||
f.StringVar(&in.vip, "vip", "",
|
||||
"with --standbys, a floating address for whichever control plane leads, as 10.0.0.160/24 (keepalived on each)")
|
||||
return cmd
|
||||
}
|
||||
|
||||
@@ -192,6 +266,9 @@ func (in *serverInit) run(ctx context.Context) error {
|
||||
if err := in.checkBundle(); err != nil {
|
||||
return fmt.Errorf("offline bundle: %w", err)
|
||||
}
|
||||
if err := in.checkMemory(); err != nil {
|
||||
return err
|
||||
}
|
||||
steps := []struct {
|
||||
name string
|
||||
fn func(context.Context) error
|
||||
@@ -249,6 +326,14 @@ func (in *serverInit) run(ctx context.Context) error {
|
||||
fmt.Fprintf(in.out, " !! firewall: %v; llmgw's ports answer every network this machine is on\n", err)
|
||||
}
|
||||
}
|
||||
if err := in.ensureWebhooks(ctx); err != nil {
|
||||
fmt.Fprintf(in.out, " !! webhooks: %v\n", err)
|
||||
}
|
||||
if hasSystemd && len(in.judgeMachines) > 0 {
|
||||
if err := in.ensureJudgeNodes(ctx); err != nil {
|
||||
return fmt.Errorf("judge nodes: %w", err)
|
||||
}
|
||||
}
|
||||
if in.judgeWanted() && in.configJudgesWith(guardModelName) {
|
||||
// Not fatal: without its judge the baseline policy still checks for
|
||||
// secrets and injection, and a judged request is served flagged.
|
||||
@@ -265,6 +350,14 @@ func (in *serverInit) run(ctx context.Context) error {
|
||||
return fmt.Errorf("starter model: %w", err)
|
||||
}
|
||||
}
|
||||
if in.standbys {
|
||||
if err := in.ensureStandbys(ctx); err != nil {
|
||||
return fmt.Errorf("standby control planes: %w", err)
|
||||
}
|
||||
}
|
||||
if hasSystemd && in.judgeWanted() && in.configJudgesWith(guardModelName) {
|
||||
in.checkJudge(ctx)
|
||||
}
|
||||
in.nextSteps()
|
||||
return nil
|
||||
}
|
||||
@@ -301,29 +394,45 @@ func (in *serverInit) configJudgesWith(model string) bool {
|
||||
}
|
||||
|
||||
// guardSection is server.yaml's guard: the baseline every request passes,
|
||||
// with the judge model when there is one.
|
||||
// with the judge model, as init sets it up unless told otherwise, when there
|
||||
// is one.
|
||||
func guardSection(judge bool) string {
|
||||
j := ` # A judge model reads what patterns cannot: violence, weapons, abuse,
|
||||
if !judge {
|
||||
return guardSectionFor(nil)
|
||||
}
|
||||
j := (&serverInit{}).judgeFor()
|
||||
return guardSectionFor(&j)
|
||||
}
|
||||
|
||||
// guardSectionFor is server.yaml's guard with this judge, or none.
|
||||
func guardSectionFor(j *judgeSpec) string {
|
||||
fail := "closed"
|
||||
jb := ` # A judge model reads what patterns cannot: violence, weapons, abuse,
|
||||
# self-harm. Installed by init; --no-guard-model leaves it out.
|
||||
# judge: {model: ` + guardModelName + `, input: block, output: block, ...}
|
||||
`
|
||||
if judge {
|
||||
j = ` # A judge model reads what patterns cannot: violence, crime, sexual
|
||||
# content, self-harm, jailbreaks. Two replicas run on CPUs, about two
|
||||
if j != nil {
|
||||
fail = j.Fail
|
||||
copies := map[int]string{1: "One replica runs", 2: "Two replicas run", 3: "Three replicas run"}[j.Replicas]
|
||||
if copies == "" {
|
||||
copies = fmt.Sprintf("%d replicas run", j.Replicas)
|
||||
}
|
||||
jb = ` # A judge model reads what patterns cannot: violence, crime, sexual
|
||||
# content, self-harm, jailbreaks. ` + copies + ` on CPUs, about two
|
||||
# seconds for a short text; an answer is read every 600 characters as
|
||||
# it streams. With fail: closed, a request no judge can read within
|
||||
# latency_budget is refused (503); open serves it, flagged, unjudged.
|
||||
# PII is the patterns' (above), and copyright, unethical acts and
|
||||
# politically sensitive topics are left out: they refuse too much
|
||||
# ordinary work. Controversial counts as safe; controversial: unsafe
|
||||
# makes it count. With a GPU: llmgw ctl guard judge enable --gpu, then
|
||||
# llmgw ctl guard judge use ` + defaultGPUJudge + ` ` + guardModelName + `.
|
||||
# politically sensitive topics are left out unless chosen: they refuse
|
||||
# too much ordinary work. Controversial counts as safe; controversial:
|
||||
# unsafe makes it count. With a GPU: llmgw ctl guard judge enable --gpu,
|
||||
# then llmgw ctl guard judge use ` + defaultGPUJudge + ` ` + guardModelName + `.
|
||||
judge:
|
||||
models: [` + guardModelName + `]
|
||||
input: block
|
||||
output: block
|
||||
categories: [Violent, Non-violent Illegal Acts, Sexual Content or Sexual Acts, Suicide & Self-Harm, Jailbreak]
|
||||
latency_budget: 5s
|
||||
categories: [` + strings.Join(j.Categories, ", ") + `]
|
||||
latency_budget: ` + durationYAML(j.Budget) + `
|
||||
`
|
||||
}
|
||||
return `# Guardrails (docs/guardrails.md). The baseline applies to every request;
|
||||
@@ -333,13 +442,13 @@ guard:
|
||||
default: [baseline]
|
||||
policies:
|
||||
baseline:
|
||||
fail: closed
|
||||
fail: ` + fail + `
|
||||
input:
|
||||
pii: {secret: block, card: redact}
|
||||
injection: flag
|
||||
output:
|
||||
pii: {secret: block, card: redact}
|
||||
` + j
|
||||
` + jb
|
||||
}
|
||||
|
||||
func (in *serverInit) done(format string, a ...any) {
|
||||
@@ -518,6 +627,10 @@ func (in *serverInit) certHosts() []string {
|
||||
for _, h := range in.hosts {
|
||||
add(h)
|
||||
}
|
||||
// The floating address is this machine's whenever it leads.
|
||||
if ip, _, err := net.ParseCIDR(in.vip); err == nil {
|
||||
add(ip.String())
|
||||
}
|
||||
if name, err := os.Hostname(); err == nil {
|
||||
add(name)
|
||||
}
|
||||
@@ -556,8 +669,13 @@ func (in *serverInit) ensureConfig(context.Context) error {
|
||||
if contact == "" {
|
||||
contact = "platform@example.com"
|
||||
}
|
||||
body := fmt.Sprintf(serverConfigTemplate, in.etcDir, in.etcDir, in.etcDir,
|
||||
in.storeAddress(), contact, guardSection(in.judgeWanted()))
|
||||
var judge *judgeSpec
|
||||
if in.judgeWanted() {
|
||||
j := in.judgeFor()
|
||||
judge = &j
|
||||
}
|
||||
body := in.tuneConfig(fmt.Sprintf(serverConfigTemplate, in.etcDir, in.etcDir, in.etcDir,
|
||||
in.storeAddress(), contact, guardSectionFor(judge)))
|
||||
if err := os.WriteFile(path, []byte(body), 0o640); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -804,6 +922,9 @@ func (in *serverInit) waitHealthy(ctx context.Context) error {
|
||||
}
|
||||
|
||||
func (in *serverInit) nextSteps() {
|
||||
if in.judgeNote != "" {
|
||||
fmt.Fprint(in.out, "\n"+in.judgeNote)
|
||||
}
|
||||
fmt.Fprint(in.out, `
|
||||
The control plane is up. Next:
|
||||
|
||||
|
||||
@@ -0,0 +1,473 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"net"
|
||||
"os"
|
||||
"path"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||
)
|
||||
|
||||
// What a first install lets a person choose beyond which parts it has: which
|
||||
// starter model, how the judge decides, what the portal speaks, the limits a
|
||||
// key starts with, and whether pictures and 3D come by themselves. Each zero
|
||||
// value is what init did before it could be chosen, so `llmgw server init`
|
||||
// with no flags installs what it always did.
|
||||
//
|
||||
// All of it is written into server.yaml when init writes it, and the cluster's
|
||||
// keys among it become the cluster's first settings when the control plane
|
||||
// first starts. A server.yaml already there is never overwritten, so a second
|
||||
// run changes none of it: the settings page and `llmgw ctl config` do.
|
||||
|
||||
// starterSpec is a model init can fetch and serve so there is something to ask.
|
||||
type starterSpec struct {
|
||||
Name, Alias, Source string
|
||||
// SizeGB is the weights' size, for the memory a node needs; 0 is not known.
|
||||
SizeGB float64
|
||||
About string
|
||||
}
|
||||
|
||||
// starterChoices are the starters offered, smallest first: the first is the
|
||||
// default. Each is a single GGUF file, Q4_K_M.
|
||||
var starterChoices = []starterSpec{
|
||||
{starterName, starterAlias, starterSource, 0.5, "answers at once on any CPU; fast, not clever"},
|
||||
{"qwen2.5-1.5b-instruct", "qwen-1.5b", "hf:Qwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf", 1.1,
|
||||
"still quick on a CPU, and noticeably better"},
|
||||
{"qwen2.5-3b-instruct", "qwen-3b", "hf:Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf", 2.1,
|
||||
"slow on a CPU, quick on any GPU"},
|
||||
{"qwen3-4b", "", "hf:Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf", 2.5,
|
||||
"thinks before it answers; a GPU is worth having"},
|
||||
{"qwen3-8b", "", "hf:Qwen/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf", 5.0,
|
||||
"a real assistant; a GPU with 8 GB or more"},
|
||||
{"qwen3-14b", "", "hf:Qwen/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf", 9.0,
|
||||
"better still; a GPU with 16 GB or more"},
|
||||
}
|
||||
|
||||
// starterFrom reads --starter: a name from starterChoices, or the hf: source
|
||||
// of any GGUF file, named after the file.
|
||||
func starterFrom(s string) (starterSpec, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
for _, c := range starterChoices {
|
||||
if s == c.Name || (c.Alias != "" && s == c.Alias) {
|
||||
return c, nil
|
||||
}
|
||||
}
|
||||
if rest, ok := strings.CutPrefix(s, "hf:"); ok && strings.Count(rest, "/") >= 2 && strings.HasSuffix(strings.ToLower(rest), ".gguf") {
|
||||
name := strings.ToLower(strings.TrimSuffix(path.Base(rest), path.Ext(rest)))
|
||||
return starterSpec{Name: name, Source: s, About: "chosen at install"}, nil
|
||||
}
|
||||
var names []string
|
||||
for _, c := range starterChoices {
|
||||
names = append(names, c.Name)
|
||||
}
|
||||
return starterSpec{}, fmt.Errorf("starter %q: one of %s, or hf:<owner>/<repo>/<file>.gguf", s, strings.Join(names, ", "))
|
||||
}
|
||||
|
||||
func starterNames() []string {
|
||||
var out []string
|
||||
for _, c := range starterChoices {
|
||||
out = append(out, c.Name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// readChoices takes the flags that are not read straight into init: the
|
||||
// starter by its name, the judge's categories by commas. Then it checks them
|
||||
// all.
|
||||
func (in *serverInit) readChoices(starter, categories string) error {
|
||||
if strings.TrimSpace(starter) != "" {
|
||||
s, err := starterFrom(starter)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
in.starter = s
|
||||
}
|
||||
in.judge.Categories = nil
|
||||
for _, c := range strings.Split(categories, ",") {
|
||||
if c = strings.TrimSpace(c); c != "" {
|
||||
in.judge.Categories = append(in.judge.Categories, c)
|
||||
}
|
||||
}
|
||||
return in.checkChoices()
|
||||
}
|
||||
|
||||
// checkChoices says what among the choices cannot be installed.
|
||||
func (in *serverInit) checkChoices() error {
|
||||
switch in.language {
|
||||
case "", "en", "tr":
|
||||
default:
|
||||
return fmt.Errorf("language %q: en or tr", in.language)
|
||||
}
|
||||
switch in.judge.Fail {
|
||||
case "", "closed", "open":
|
||||
default:
|
||||
return fmt.Errorf("judge fail %q: closed or open", in.judge.Fail)
|
||||
}
|
||||
for _, c := range in.judge.Categories {
|
||||
if !slices.Contains(judgeCategories, c) {
|
||||
return fmt.Errorf("judge category %q: one of %s", c, strings.Join(judgeCategories, ", "))
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case in.judge.Replicas < 0 || in.judge.Replicas > 8:
|
||||
return fmt.Errorf("judge replicas %d: from 1 to 8", in.judge.Replicas)
|
||||
case in.judge.Budget < 0 || in.judge.Budget > time.Minute:
|
||||
return fmt.Errorf("judge budget %s: at most a minute", in.judge.Budget)
|
||||
case in.starterCtx < 0 || in.starterCtx > 262144:
|
||||
return fmt.Errorf("starter context %d: at most 262144 tokens", in.starterCtx)
|
||||
case in.starterCtx > 0 && in.starterCtx < 512:
|
||||
return fmt.Errorf("starter context %d: at least 512 tokens", in.starterCtx)
|
||||
case in.defaultRPM < 0 || in.defaultTPM < 0:
|
||||
return fmt.Errorf("default limits cannot be negative")
|
||||
case in.maxKeyTTL < 0 || (in.maxKeyTTL > 0 && in.maxKeyTTL < time.Hour):
|
||||
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
|
||||
case in.evictAfterIdle < 0 || in.minLifetime < 0:
|
||||
return fmt.Errorf("on-demand times cannot be negative")
|
||||
case in.vip != "" && !in.standbys:
|
||||
return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
|
||||
}
|
||||
if in.vip != "" {
|
||||
ip, _, err := net.ParseCIDR(in.vip)
|
||||
if err != nil || ip.To4() == nil {
|
||||
return fmt.Errorf("floating address %q: an IPv4 address with its prefix, as 10.0.0.160/24", in.vip)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// starterModelSpec is the starter this install serves.
|
||||
func (in *serverInit) starterModelSpec() starterSpec {
|
||||
if in.starter.Source == "" {
|
||||
return starterChoices[0]
|
||||
}
|
||||
return in.starter
|
||||
}
|
||||
|
||||
// minMemoryGB is what a node needs free for the starter: its weights and
|
||||
// room for its context, never less than the 2 GB it always asked.
|
||||
func (s starterSpec) minMemoryGB() int {
|
||||
return max(2, int(math.Ceil(s.SizeGB+1.5)))
|
||||
}
|
||||
|
||||
// servingMemoryMB is what this machine needs, beyond what the control plane
|
||||
// does, for the models it will serve itself: the starter's and the judges'
|
||||
// reservations, and a gigabyte more because the scheduler counts a machine's
|
||||
// memory in whole gigabytes, rounded down — 7.75 GB is 7. Zero when it serves
|
||||
// none of them; what names those it does.
|
||||
//
|
||||
// Asked before anything is installed: the 8 GB machine the guide was first
|
||||
// recorded on finished every step and then had no room for the starter, the
|
||||
// one thing a person asks first.
|
||||
func (in *serverInit) servingMemoryMB(judgesElsewhere bool) (mb int, what string) {
|
||||
if in.noNode {
|
||||
return 0, ""
|
||||
}
|
||||
gb := 0
|
||||
var parts []string
|
||||
if !in.noModel && (!in.noMirror || in.mirrorURL != "") {
|
||||
gb += in.starterModelSpec().minMemoryGB()
|
||||
parts = append(parts, "the starter model")
|
||||
}
|
||||
if !in.noGuardModel && !judgesElsewhere {
|
||||
n := in.judgeFor().Replicas
|
||||
gb += n * judgeMemoryGB
|
||||
parts = append(parts, fmt.Sprintf("%d judges", n))
|
||||
}
|
||||
if gb == 0 {
|
||||
return 0, ""
|
||||
}
|
||||
return (gb + 1) * 1024, strings.Join(parts, " and ")
|
||||
}
|
||||
|
||||
// checkMemory refuses, before anything is installed, a machine too small for
|
||||
// the models it would serve as well as the control plane.
|
||||
func (in *serverInit) checkMemory() error {
|
||||
if in.unitPath == "" {
|
||||
return nil // a development run, served by nothing
|
||||
}
|
||||
meminfo, err := os.ReadFile("/proc/meminfo")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
total, ok := pgha.MemTotal(string(meminfo))
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
extra, what := in.servingMemoryMB(len(in.judgeMachines) > 0)
|
||||
if extra == 0 {
|
||||
return nil
|
||||
}
|
||||
have, need := int(total>>20), preflight.Needs[preflight.RoleControl].MemoryMB+extra
|
||||
if have >= need {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("memory: this machine has %d MB; the control plane and %s it would serve need %d. "+
|
||||
"Give it more, or leave them off it: --no-model, --no-guard-model, or --no-node to serve models only on nodes added later", have, what, need)
|
||||
}
|
||||
|
||||
// defaultContext is the context a client may use of the starter.
|
||||
const defaultContext = 4096
|
||||
|
||||
// starterContext is the context a client may use of the starter.
|
||||
func (in *serverInit) starterContext() int {
|
||||
if in.starterCtx > 0 {
|
||||
return in.starterCtx
|
||||
}
|
||||
return defaultContext
|
||||
}
|
||||
|
||||
// judgeSpec is how the guardrails' judge decides.
|
||||
type judgeSpec struct {
|
||||
// Fail is what is done with a request no judge read in time: closed
|
||||
// refuses it, open serves it flagged.
|
||||
Fail string
|
||||
Categories []string
|
||||
Replicas int
|
||||
Budget time.Duration
|
||||
}
|
||||
|
||||
// judgeCategories are the ones a judge can be asked about, as Qwen3Guard
|
||||
// names them; defaultJudgeCategories those it is asked about unless told.
|
||||
var (
|
||||
judgeCategories = []string{"Violent", "Non-violent Illegal Acts", "Sexual Content or Sexual Acts",
|
||||
"Suicide & Self-Harm", "Jailbreak", "PII", "Unethical Acts", "Politically Sensitive Topics", "Copyright Violation"}
|
||||
defaultJudgeCategories = judgeCategories[:5]
|
||||
)
|
||||
|
||||
// judgeFor is the judge as this install sets it up, the defaults filled in.
|
||||
func (in *serverInit) judgeFor() judgeSpec {
|
||||
j := in.judge
|
||||
if j.Fail == "" {
|
||||
j.Fail = "closed"
|
||||
}
|
||||
if len(j.Categories) == 0 {
|
||||
j.Categories = defaultJudgeCategories
|
||||
}
|
||||
if j.Replicas <= 0 {
|
||||
j.Replicas = 2
|
||||
if in.singleMachine() {
|
||||
j.Replicas = 1
|
||||
}
|
||||
}
|
||||
if j.Budget <= 0 {
|
||||
j.Budget = 5 * time.Second
|
||||
}
|
||||
return j
|
||||
}
|
||||
|
||||
// singleMachine is an install where this machine is the only node there
|
||||
// will be, as far as init knows: a second judge replica there shares the
|
||||
// same cores and is lost with the same machine, so it buys nothing but
|
||||
// half the speed of each.
|
||||
func (in *serverInit) singleMachine() bool {
|
||||
return !in.noNode && !in.standbys && len(in.patroni) == 0 && len(in.patroniNodes) == 0
|
||||
}
|
||||
|
||||
// The configuration's limits, as init writes them unless told otherwise.
|
||||
const (
|
||||
defaultRPM = 60
|
||||
defaultTPM = 200000
|
||||
defaultMaxKeyTTL = 2160 * time.Hour
|
||||
)
|
||||
|
||||
// tuneConfig writes this install's choices into the server.yaml template:
|
||||
// the lines the template has, changed, and the blocks it has not, added.
|
||||
// Unchosen, the template comes back as it was.
|
||||
func (in *serverInit) tuneConfig(body string) string {
|
||||
set := func(old, now string) {
|
||||
if old != now {
|
||||
body = strings.Replace(body, old, now, 1)
|
||||
}
|
||||
}
|
||||
if in.signupOpen {
|
||||
set(" signup_open: false", " signup_open: true")
|
||||
}
|
||||
if in.defaultRPM > 0 {
|
||||
set(" default_rpm: 60", " default_rpm: "+strconv.Itoa(in.defaultRPM))
|
||||
}
|
||||
if in.defaultTPM > 0 {
|
||||
set(" default_tpm: 200000", " default_tpm: "+strconv.Itoa(in.defaultTPM))
|
||||
}
|
||||
if in.maxKeyTTL > 0 && in.maxKeyTTL != defaultMaxKeyTTL {
|
||||
set(" max_key_ttl: 2160h # 90 days", fmt.Sprintf(" max_key_ttl: %s # %s", durationYAML(in.maxKeyTTL), days(in.maxKeyTTL)))
|
||||
}
|
||||
if in.language != "" {
|
||||
set(" # language: tr", " language: "+in.language)
|
||||
}
|
||||
var extra []string
|
||||
if in.media {
|
||||
extra = append(extra, "# Pictures and 3D, chosen at install: the nodes install ComfyUI, and the\n"+
|
||||
"# leader adds a picture model and a 3D model by itself once a card fits\n"+
|
||||
"# one ('llmgw ctl model presets auto off' stops it).\n"+
|
||||
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
|
||||
}
|
||||
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
|
||||
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
|
||||
if in.evictAfterIdle > 0 {
|
||||
b += " evict_after_idle: " + durationYAML(in.evictAfterIdle) + "\n"
|
||||
}
|
||||
if in.minLifetime > 0 {
|
||||
b += " min_lifetime: " + durationYAML(in.minLifetime) + "\n"
|
||||
}
|
||||
extra = append(extra, b)
|
||||
}
|
||||
if len(extra) > 0 {
|
||||
body += "\n" + strings.Join(extra, "\n")
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
|
||||
func durationYAML(d time.Duration) string {
|
||||
switch {
|
||||
case d == 0:
|
||||
return "0s"
|
||||
case d%time.Hour == 0:
|
||||
return fmt.Sprintf("%dh", d/time.Hour)
|
||||
case d%time.Minute == 0:
|
||||
return fmt.Sprintf("%dm", d/time.Minute)
|
||||
}
|
||||
return d.String()
|
||||
}
|
||||
|
||||
func days(d time.Duration) string {
|
||||
n := int(d.Hours() / 24)
|
||||
if n == 1 {
|
||||
return "1 day"
|
||||
}
|
||||
return fmt.Sprintf("%d days", n)
|
||||
}
|
||||
|
||||
// initFlags is this install as `llmgw server init`'s flags, for doing it
|
||||
// again from a script; what is left at its default is left out.
|
||||
func (in *serverInit) initFlags() []string {
|
||||
var out []string
|
||||
add := func(flag string, v ...string) {
|
||||
if len(v) == 0 {
|
||||
out = append(out, flag)
|
||||
return
|
||||
}
|
||||
out = append(out, flag+" "+shellWord(v[0]))
|
||||
}
|
||||
if in.admin != "" {
|
||||
add("--admin", in.admin)
|
||||
}
|
||||
for _, h := range in.hosts {
|
||||
add("--host", h)
|
||||
}
|
||||
if in.language != "" {
|
||||
add("--language", in.language)
|
||||
}
|
||||
if in.signupOpen {
|
||||
add("--signup-open")
|
||||
}
|
||||
if in.defaultRPM > 0 && in.defaultRPM != defaultRPM {
|
||||
add("--default-rpm", strconv.Itoa(in.defaultRPM))
|
||||
}
|
||||
if in.defaultTPM > 0 && in.defaultTPM != defaultTPM {
|
||||
add("--default-tpm", strconv.Itoa(in.defaultTPM))
|
||||
}
|
||||
if in.maxKeyTTL > 0 && in.maxKeyTTL != defaultMaxKeyTTL {
|
||||
add("--max-key-ttl", durationYAML(in.maxKeyTTL))
|
||||
}
|
||||
if in.postgres {
|
||||
add("--postgres")
|
||||
}
|
||||
for _, t := range in.patroni {
|
||||
add("--patroni", t)
|
||||
}
|
||||
for _, m := range in.judgeMachines {
|
||||
add("--judge-node", m.target)
|
||||
}
|
||||
switch {
|
||||
case in.mirrorURL != "":
|
||||
add("--mirror", in.mirrorURL)
|
||||
case in.noMirror:
|
||||
add("--no-mirror")
|
||||
case in.mirrorPath != "" && in.mirrorPath != mirrorDir:
|
||||
add("--mirror-dir", in.mirrorPath)
|
||||
}
|
||||
if in.bundle != "" {
|
||||
add("--bundle", in.bundle)
|
||||
add("--bundle-sha256", in.bundleSHA)
|
||||
}
|
||||
if in.noNode {
|
||||
add("--no-node")
|
||||
}
|
||||
if in.noModel {
|
||||
add("--no-model")
|
||||
} else {
|
||||
if s := in.starterModelSpec(); s.Name != starterName {
|
||||
v := s.Name
|
||||
if !slices.ContainsFunc(starterChoices, func(c starterSpec) bool { return c.Name == s.Name }) {
|
||||
v = s.Source
|
||||
}
|
||||
add("--starter", v)
|
||||
}
|
||||
if in.starterCtx > 0 && in.starterCtx != defaultContext {
|
||||
add("--starter-context", strconv.Itoa(in.starterCtx))
|
||||
}
|
||||
}
|
||||
if in.noGuardModel {
|
||||
add("--no-guard-model")
|
||||
} else {
|
||||
// Against the defaults for an install of the same shape: one judge
|
||||
// on one machine, two otherwise.
|
||||
shape := serverInit{noNode: in.noNode, standbys: in.standbys, patroni: in.patroni, patroniNodes: in.patroniNodes}
|
||||
j, d := in.judgeFor(), shape.judgeFor()
|
||||
if j.Fail != d.Fail {
|
||||
add("--judge-fail", j.Fail)
|
||||
}
|
||||
if !slices.Equal(j.Categories, d.Categories) {
|
||||
add("--judge-categories", strings.Join(j.Categories, ","))
|
||||
}
|
||||
if j.Replicas != d.Replicas {
|
||||
add("--judge-replicas", strconv.Itoa(j.Replicas))
|
||||
}
|
||||
if j.Budget != d.Budget {
|
||||
add("--judge-budget", durationYAML(j.Budget))
|
||||
}
|
||||
}
|
||||
if in.noFirewall {
|
||||
add("--no-firewall")
|
||||
}
|
||||
if in.standbys {
|
||||
add("--standbys")
|
||||
}
|
||||
if in.vip != "" {
|
||||
add("--vip", in.vip)
|
||||
}
|
||||
for _, w := range in.webhooks {
|
||||
for _, s := range strings.Split(w, ",") {
|
||||
if s = strings.TrimSpace(s); s != "" {
|
||||
add("--webhook", s)
|
||||
}
|
||||
}
|
||||
}
|
||||
if in.media {
|
||||
add("--media")
|
||||
}
|
||||
if in.evictAfterIdle > 0 {
|
||||
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
|
||||
}
|
||||
if in.minLifetime > 0 {
|
||||
add("--min-lifetime", durationYAML(in.minLifetime))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// shellWord quotes a word for a shell when it needs it.
|
||||
func shellWord(s string) string {
|
||||
if s != "" && !strings.ContainsAny(s, " \t\n'\"\\$`&;|<>(){}*?[]#~!") {
|
||||
return s
|
||||
}
|
||||
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
)
|
||||
|
||||
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
in := &serverInit{
|
||||
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
|
||||
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
|
||||
}
|
||||
j := in.judgeFor()
|
||||
body := in.tuneConfig(fmt.Sprintf(serverConfigTemplate, "/etc/llmgw", "/etc/llmgw", "/etc/llmgw",
|
||||
"/var/lib/llmgw/llmgw.db", "ops@example.com", guardSectionFor(&j)))
|
||||
path := filepath.Join(t.TempDir(), "server.yaml")
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg, err := config.Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("%v\n%s", err, body)
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
switch {
|
||||
case cfg.Landing.Language != "tr" || !cfg.Auth.SignupOpen:
|
||||
t.Errorf("language %q signup %v", cfg.Landing.Language, cfg.Auth.SignupOpen)
|
||||
case cfg.Auth.DefaultRPM != 120 || cfg.Auth.DefaultTPM != 50000 || cfg.Auth.MaxKeyTTL != 30*24*time.Hour:
|
||||
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
|
||||
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
|
||||
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
|
||||
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
|
||||
t.Errorf("on demand %+v", cfg.OnDemand)
|
||||
}
|
||||
base := cfg.Guard.Policies["baseline"]
|
||||
if base.Fail != "open" || base.Judge == nil || !slices.Equal(base.Judge.Categories, []string{"Violent", "PII"}) ||
|
||||
base.Judge.LatencyBudget != 3*time.Second {
|
||||
t.Errorf("the judge as written: fail %q %+v", base.Fail, base.Judge)
|
||||
}
|
||||
if !strings.Contains(body, "Three replicas run on CPUs") {
|
||||
t.Error("the comment says another number of judges than run")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNothingChosenIsTheTemplateAsItWas(t *testing.T) {
|
||||
body := fmt.Sprintf(serverConfigTemplate, "/e", "/e", "/e", "/s.db", "a@b", guardSection(true))
|
||||
if got := (&serverInit{}).tuneConfig(body); got != body {
|
||||
t.Error("an install with nothing chosen writes another configuration than before")
|
||||
}
|
||||
// Each line tuneConfig changes is in the template, or a choice would be
|
||||
// lost without a word.
|
||||
for _, line := range []string{" signup_open: false", " default_rpm: 60", " default_tpm: 200000",
|
||||
" max_key_ttl: 2160h # 90 days", " # language: tr"} {
|
||||
if !strings.Contains(serverConfigTemplate, line) {
|
||||
t.Errorf("the template has no %q", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
|
||||
s, err := starterFrom("qwen3-8b")
|
||||
if err != nil || s.SizeGB != 5.0 || !strings.HasPrefix(s.Source, "hf:Qwen/Qwen3-8B-GGUF/") {
|
||||
t.Errorf("by name: %+v %v", s, err)
|
||||
}
|
||||
s, err = starterFrom("hf:bartowski/Phi-4-GGUF/Phi-4-Q4_K_M.gguf")
|
||||
if err != nil || s.Name != "phi-4-q4_k_m" {
|
||||
t.Errorf("by file: %+v %v", s, err)
|
||||
}
|
||||
if _, err := starterFrom("gpt-5"); err == nil {
|
||||
t.Error("a model nobody offers was taken")
|
||||
}
|
||||
m := starterModel(s, 8192, 0, "http://x/p.gguf", []string{"6000-24g"}, true)
|
||||
if m.Name != "phi-4-q4_k_m" || m.ContextLength != 8192 || len(m.Aliases) != 0 ||
|
||||
!slices.Equal(m.Profiles[0].Args, []string{"--ctx-size", "32768"}) {
|
||||
t.Errorf("entry %+v", m)
|
||||
}
|
||||
big, _ := starterFrom("qwen3-14b")
|
||||
if m := starterModel(big, 16384, 0, "http://x/q.gguf", nil, true); m.Profiles[0].MinMemoryGB != 11 ||
|
||||
!slices.Equal(m.Profiles[0].Args, []string{"--ctx-size", "65536"}) {
|
||||
t.Errorf("a 9 GB starter asks for %d GB, %v", m.Profiles[0].MinMemoryGB, m.Profiles[0].Args)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
|
||||
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
|
||||
mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
|
||||
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
|
||||
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
|
||||
in.starter, _ = starterFrom("qwen3-4b")
|
||||
got := strings.Join(in.initFlags(), " ")
|
||||
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
|
||||
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
|
||||
if got != want {
|
||||
t.Errorf("flags\n got %s\nwant %s", got, want)
|
||||
}
|
||||
if got := strings.Join((&serverInit{}).initFlags(), " "); got != "" {
|
||||
t.Errorf("nothing chosen is %q", got)
|
||||
}
|
||||
if w := shellWord("Non-violent Illegal Acts,PII"); w != "'Non-violent Illegal Acts,PII'" {
|
||||
t.Errorf("quoted %s", w)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
|
||||
for _, in := range []serverInit{
|
||||
{language: "de"},
|
||||
{judge: judgeSpec{Fail: "maybe"}},
|
||||
{judge: judgeSpec{Categories: []string{"Rudeness"}}},
|
||||
{judge: judgeSpec{Replicas: 9}},
|
||||
{starterCtx: 100},
|
||||
{maxKeyTTL: time.Minute},
|
||||
{vip: "10.0.0.160/24"},
|
||||
{standbys: true, vip: "10.0.0.160"},
|
||||
} {
|
||||
if err := in.checkChoices(); err == nil {
|
||||
t.Errorf("taken: %+v", in)
|
||||
}
|
||||
}
|
||||
if err := (&serverInit{standbys: true, vip: "10.0.0.160/24"}).checkChoices(); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithStandbysTheMirrorAnswersEveryControlPlane(t *testing.T) {
|
||||
in := &serverInit{hosts: []string{"10.0.0.5"}, patroniNodes: []pgha.Node{
|
||||
{Member: pgha.Member{Name: "db1", IP: "10.0.0.6"}}, {Member: pgha.Member{Name: "db2", IP: "10.0.0.7"}},
|
||||
}}
|
||||
if u := in.mirrorUnit(); !strings.Contains(u, "--api-listen 127.0.0.1:8082") || strings.Contains(u, "--api-allow") {
|
||||
t.Errorf("one control plane: the API is for this machine only:\n%s", u)
|
||||
}
|
||||
in.standbys = true
|
||||
u := in.mirrorUnit()
|
||||
for _, want := range []string{"--api-listen :8082", "--api-allow 10.0.0.5", "--api-allow 10.0.0.6", "--api-allow 10.0.0.7"} {
|
||||
if !strings.Contains(u, want) {
|
||||
t.Errorf("the unit lacks %q:\n%s", want, u)
|
||||
}
|
||||
}
|
||||
if got := in.mirrorAPIURL(); got != "http://10.0.0.5:8082" {
|
||||
t.Errorf("the control planes reach the mirror at %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFloatingAddressIsInTheCertificate(t *testing.T) {
|
||||
in := &serverInit{vip: "10.0.0.160/24"}
|
||||
if !slices.Contains(in.certHosts(), "10.0.0.160") {
|
||||
t.Errorf("hosts %v", in.certHosts())
|
||||
}
|
||||
}
|
||||
|
||||
func TestStandbysWantTheStoresMachines(t *testing.T) {
|
||||
in := &serverInit{standbys: true, out: &strings.Builder{}}
|
||||
if err := in.ensureStandbys(context.Background()); err == nil || !strings.Contains(err.Error(), "--patroni") {
|
||||
t.Errorf("standbys with nowhere to go: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTwoClustersOnOneNetworkKeepApartFloatingAddresses(t *testing.T) {
|
||||
a, b := vrrpRouterID("10.0.0.161"), vrrpRouterID("10.0.0.164")
|
||||
if a == b || a < 1 || a > 255 || b < 1 || b > 255 {
|
||||
t.Errorf("router ids %d and %d", a, b)
|
||||
}
|
||||
conf := keepalivedConf("ens18", "10.0.0.164", 24, 100, "secret12")
|
||||
if !strings.Contains(conf, fmt.Sprintf("virtual_router_id %d\n", b)) || !strings.Contains(conf, "interface ens18") {
|
||||
t.Errorf("conf:\n%s", conf)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheJudgesAndTheStarterShareTheCPUsRatherThanEachTakingAll(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
in serverInit
|
||||
judge, starter int
|
||||
}{
|
||||
// One machine: one judge, on up to four cores; the starter has the rest.
|
||||
{serverInit{cpus: 4}, 4, 0},
|
||||
{serverInit{cpus: 8}, 4, 4},
|
||||
{serverInit{cpus: 32}, 4, 8},
|
||||
{serverInit{cpus: 2}, 2, 0},
|
||||
{serverInit{cpus: 1}, 0, 0},
|
||||
{serverInit{cpus: 4, noGuardModel: true}, judgeCores, 4}, // no judge: all for the starter
|
||||
// Two asked for on one machine: two cores each, one each where two
|
||||
// would not fit both.
|
||||
{serverInit{cpus: 8, judge: judgeSpec{Replicas: 2}}, 2, 4},
|
||||
{serverInit{cpus: 2, judge: judgeSpec{Replicas: 2}}, 1, 0},
|
||||
{serverInit{cpus: 8, judge: judgeSpec{Replicas: 3}}, 2, 2},
|
||||
{serverInit{cpus: 4, noNode: true}, 2, 0}, // on nodes not known yet
|
||||
} {
|
||||
got := c.in.cpuPlan()
|
||||
if got.judge != c.judge || got.starter != c.starter {
|
||||
t.Errorf("%+v: judge %d starter %d, want %d %d", c.in, got.judge, got.starter, c.judge, c.starter)
|
||||
}
|
||||
}
|
||||
m := guardModel("http://x/g.gguf", 2, 1, nil)
|
||||
if p := m.Profiles[0]; p.CPUCores != 1 || p.MinMemoryGB != 2 || !slices.Contains(p.Args, "--cache-ram") {
|
||||
t.Errorf("judge profile: %d cores, %d GB, args %v", p.CPUCores, p.MinMemoryGB, p.Args)
|
||||
}
|
||||
s := starterModel(starterChoices[0], defaultContext, 2, "http://x/q.gguf", []string{"6000-24g"}, true)
|
||||
if cpu := s.Profiles[len(s.Profiles)-1]; cpu.Name != "cpu" || cpu.CPUCores != 2 || s.Profiles[0].CPUCores != 0 {
|
||||
t.Errorf("starter: GPU profile %d cores, CPU %+v", s.Profiles[0].CPUCores, cpu)
|
||||
}
|
||||
}
|
||||
|
||||
// A second judge on the same machine shares its cores and is lost with it:
|
||||
// one machine has one, more machines two.
|
||||
func TestOneMachineHasOneJudgeAndMoreHaveTwo(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
in serverInit
|
||||
want int
|
||||
}{
|
||||
{serverInit{}, 1},
|
||||
{serverInit{noNode: true}, 2},
|
||||
{serverInit{patroni: []string{"a@db1", "b@db2"}}, 2},
|
||||
{serverInit{standbys: true}, 2},
|
||||
{serverInit{judge: judgeSpec{Replicas: 3}}, 3},
|
||||
} {
|
||||
if got := c.in.judgeFor().Replicas; got != c.want {
|
||||
t.Errorf("%+v: %d judges, want %d", c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
// The defaults are left out of the command; a choice against them is in.
|
||||
ha := serverInit{noNode: true, patroni: []string{"a@db1", "b@db2"}, judge: judgeSpec{Replicas: 2}}
|
||||
if f := strings.Join(ha.initFlags(), " "); strings.Contains(f, "--judge-replicas") {
|
||||
t.Errorf("two judges on three machines named: %s", f)
|
||||
}
|
||||
one := serverInit{judge: judgeSpec{Replicas: 2}}
|
||||
if f := strings.Join(one.initFlags(), " "); !strings.Contains(f, "--judge-replicas 2") {
|
||||
t.Errorf("two judges on one machine not named: %s", f)
|
||||
}
|
||||
}
|
||||
@@ -9,7 +9,9 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -120,10 +122,36 @@ func (in *serverInit) writeUnit(path, body string) (bool, error) {
|
||||
|
||||
func (in *serverInit) mirrorUnit() string {
|
||||
base := "http://" + net.JoinHostPort(in.reachableAddr(), "8081")
|
||||
return mirrorUnitFor(mirrorDir, mirrorAPI, base, nil,
|
||||
if in.standbys {
|
||||
return mirrorUnitFor(in.mirrorRoot(), ":8082", base, in.mirrorAllow(),
|
||||
"'llmgw server init'. Bytes on :8081 for every node; the\n# management API for the control planes only, whichever leads.")
|
||||
}
|
||||
return mirrorUnitFor(in.mirrorRoot(), mirrorAPI, base, nil,
|
||||
"'llmgw server init'. Bytes on :8081 for every node; the\n# management API on loopback only, for the control plane beside it.")
|
||||
}
|
||||
|
||||
// localMirrorAPI is the management API of the mirror beside this control
|
||||
// plane: on loopback, or, with standbys that may lead instead, on an address
|
||||
// they reach.
|
||||
func (in *serverInit) localMirrorAPI() string {
|
||||
if in.standbys {
|
||||
return "http://" + net.JoinHostPort(in.reachableAddr(), "8082")
|
||||
}
|
||||
return "http://" + mirrorAPI
|
||||
}
|
||||
|
||||
// mirrorAllow are the control planes the mirror's management API answers
|
||||
// besides loopback: this one by its own address, and each standby.
|
||||
func (in *serverInit) mirrorAllow() []string {
|
||||
out := []string{in.reachableAddr()}
|
||||
for _, n := range in.patroniNodes {
|
||||
if !slices.Contains(out, n.IP) {
|
||||
out = append(out, n.IP)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// mirrorUnitFor is a mirror's systemd unit. allow is who, besides the
|
||||
// machine itself, may use the management API.
|
||||
func mirrorUnitFor(dir, apiListen, base string, allow []string, about string) string {
|
||||
@@ -160,7 +188,7 @@ func (in *serverInit) mirrorAPIURL() string {
|
||||
if in.mirrorURL != "" {
|
||||
return in.mirrorURL
|
||||
}
|
||||
return "http://" + mirrorAPI
|
||||
return in.localMirrorAPI()
|
||||
}
|
||||
|
||||
// useMirror points the control plane at a mirror on another machine, set up
|
||||
@@ -202,15 +230,16 @@ func (in *serverInit) ensureMirror(ctx context.Context) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg.Mirror.API != "" && cfg.Mirror.API != "http://"+mirrorAPI {
|
||||
if cfg.Mirror.API != "" && cfg.Mirror.API != "http://"+mirrorAPI && cfg.Mirror.API != in.localMirrorAPI() {
|
||||
in.kept("mirror at %s, as server.yaml says", cfg.Mirror.API)
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := os.MkdirAll(mirrorDir, 0o755); err != nil {
|
||||
dir := in.mirrorRoot()
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := in.chown(mirrorDir, true); err != nil {
|
||||
if err := in.chown(dir, true); err != nil {
|
||||
return err
|
||||
}
|
||||
changed, err := in.writeUnit(mirrorUnitPath, in.mirrorUnit())
|
||||
@@ -227,21 +256,25 @@ func (in *serverInit) ensureMirror(ctx context.Context) error {
|
||||
if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
|
||||
return err
|
||||
}
|
||||
in.done("model mirror in %s, serving on :8081", mirrorDir)
|
||||
in.done("model mirror in %s, serving on :8081", dir)
|
||||
} else {
|
||||
in.kept("model mirror in %s", mirrorDir)
|
||||
in.kept("model mirror in %s", dir)
|
||||
}
|
||||
|
||||
if cfg.Mirror.API == "" {
|
||||
if err := appendMirrorBlockFor(in.configPath(), "http://"+mirrorAPI,
|
||||
"# The model mirror on this machine, added by 'llmgw server init'. Its\n"+
|
||||
"# management API is on loopback, so it needs no certificate."); err != nil {
|
||||
comment := "# The model mirror on this machine, added by 'llmgw server init'. Its\n" +
|
||||
"# management API is on loopback, so it needs no certificate."
|
||||
if in.standbys {
|
||||
comment = "# The model mirror on this machine, added by 'llmgw server init'. Its\n" +
|
||||
"# management API answers the control planes' addresses only."
|
||||
}
|
||||
if err := appendMirrorBlockFor(in.configPath(), in.localMirrorAPI(), comment); err != nil {
|
||||
return err
|
||||
}
|
||||
in.done("server.yaml points at the mirror")
|
||||
}
|
||||
|
||||
m, err := mirror.NewClient("http://"+mirrorAPI, tlsconf.Config{}, 5*time.Second)
|
||||
m, err := mirror.NewClient(in.localMirrorAPI(), tlsconf.Config{}, 5*time.Second)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -414,8 +447,9 @@ func hasNVIDIADevice() bool {
|
||||
// --- the starter model ---
|
||||
|
||||
// starterModel is the catalog entry: a GPU profile first when there is a card,
|
||||
// and a CPU profile always, both pinned to this machine.
|
||||
func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model {
|
||||
// and a CPU profile always, both pinned to this machine. ctx is the context a
|
||||
// client may use; each engine holds four times that, for four at once.
|
||||
func starterModel(spec starterSpec, ctx, cores int, source string, gpuClasses []string, pinned bool) catalog.Model {
|
||||
var pin map[string]string
|
||||
if pinned {
|
||||
pin = map[string]string{localLabelKey: localLabelValue}
|
||||
@@ -424,21 +458,25 @@ func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model
|
||||
for _, class := range gpuClasses {
|
||||
profiles = append(profiles, catalog.Profile{
|
||||
Name: "gpu-" + class, Engine: "llamacpp", GPUClass: class,
|
||||
Priority: 200, MinMemoryGB: 2, NodeSelector: pin,
|
||||
Args: []string{"--ctx-size", "32768"},
|
||||
Priority: 200, MinMemoryGB: spec.minMemoryGB(), NodeSelector: pin,
|
||||
Args: []string{"--ctx-size", strconv.Itoa(max(32768, 4*ctx))},
|
||||
})
|
||||
}
|
||||
profiles = append(profiles, catalog.Profile{
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2, NodeSelector: pin,
|
||||
Args: []string{"--ctx-size", "16384"},
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: spec.minMemoryGB(), CPUCores: cores, NodeSelector: pin,
|
||||
Args: []string{"--ctx-size", strconv.Itoa(max(16384, 4*ctx))},
|
||||
})
|
||||
var aliases []string
|
||||
if spec.Alias != "" {
|
||||
aliases = []string{spec.Alias}
|
||||
}
|
||||
return catalog.Model{
|
||||
Name: starterName,
|
||||
Aliases: []string{starterAlias},
|
||||
Description: "A small model init put here so there is something to try; fast, not clever.",
|
||||
Name: spec.Name,
|
||||
Aliases: aliases,
|
||||
Description: "The model init put here so there is something to try: " + spec.About + ".",
|
||||
Source: source,
|
||||
RoutingProfile: "chat",
|
||||
ContextLength: 4096,
|
||||
ContextLength: ctx,
|
||||
MaxOutput: 1024,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: profiles,
|
||||
@@ -446,8 +484,8 @@ func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model
|
||||
}
|
||||
|
||||
// guardModel is the judge's catalog entry: CPU only, so it takes no card from
|
||||
// the models people use, on whichever node has room.
|
||||
func guardModel(source string) catalog.Model {
|
||||
// the models people use, on whichever node has room; replicas copies of it.
|
||||
func guardModel(source string, replicas, cores int, selector map[string]string) catalog.Model {
|
||||
return catalog.Model{
|
||||
Name: guardModelName,
|
||||
Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
|
||||
@@ -456,10 +494,17 @@ func guardModel(source string) catalog.Model {
|
||||
Guard: &catalog.GuardModel{Format: "qwen3guard"},
|
||||
// Two: every judged request waits on it, and with fail: closed a
|
||||
// judge that is not there refuses everything.
|
||||
Scaling: catalog.Scaling{Min: 2, Max: 2},
|
||||
Scaling: catalog.Scaling{Min: replicas, Max: replicas},
|
||||
Profiles: []catalog.Profile{{
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2,
|
||||
Args: []string{"--ctx-size", "8192"},
|
||||
// About 1.6 GB with its context, which stays so with no prompt
|
||||
// cache: llama.cpp keeps up to 8 GB of prompts by default, and a
|
||||
// judge's, each a different text, are not asked again. With the
|
||||
// cache one grew to 3.2 GB over a day of traffic; reserving 4 for
|
||||
// that left a 7 GB machine room for only one of the two.
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: judgeMemoryGB, CPUCores: cores,
|
||||
Args: []string{"--ctx-size", "8192", "--cache-ram", "0"},
|
||||
// The judge machines, when the install was given some.
|
||||
NodeSelector: selector,
|
||||
}},
|
||||
}
|
||||
}
|
||||
@@ -469,15 +514,21 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
|
||||
// Served here when this machine is a node, pinned so it does not wander
|
||||
// to the first GPU node that joins; otherwise on whichever node joins.
|
||||
pinned := !in.noNode
|
||||
return in.ensureServed(ctx, starterName, starterSource, starterVersion, pinned,
|
||||
func(url string, classes []string) catalog.Model { return starterModel(url, classes, pinned) })
|
||||
spec, n := in.starterModelSpec(), in.starterContext()
|
||||
return in.ensureServed(ctx, spec.Name, spec.Source, starterVersion, pinned,
|
||||
func(url string, classes []string) catalog.Model {
|
||||
return starterModel(spec, n, in.cpuPlan().starter, url, classes, pinned)
|
||||
})
|
||||
}
|
||||
|
||||
// ensureGuardModel fetches the guardrails' judge model into the mirror and
|
||||
// adds it to the catalog, to run on any node with room.
|
||||
func (in *serverInit) ensureGuardModel(ctx context.Context) error {
|
||||
return in.ensureServed(ctx, guardModelName, guardModelSource, guardModelVersion, false,
|
||||
func(url string, _ []string) catalog.Model { return guardModel(url) })
|
||||
func(url string, _ []string) catalog.Model {
|
||||
cores, selector := in.judgePlacement()
|
||||
return guardModel(url, in.judgeFor().Replicas, cores, selector)
|
||||
})
|
||||
}
|
||||
|
||||
// ensureServed fetches a model into the mirror, adds it to the catalog, and
|
||||
@@ -661,3 +712,55 @@ func waitFor(ctx context.Context, limit time.Duration, what string, done func()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// cpuSplit is the cores init's models reserve: each judge replica's, and
|
||||
// the starter's on CPU.
|
||||
type cpuSplit struct{ judge, starter int }
|
||||
|
||||
// cpuPlan divides the CPUs between the judges and the starter, so that the
|
||||
// engines do not each take every core. Without it, two judge replicas and
|
||||
// the starter on one 4-core control plane all ran on all four: under load
|
||||
// most judgments missed their budget and the requests were refused.
|
||||
//
|
||||
// A judge needs cores: a judgment of about 280 tokens took 8.0 s on one,
|
||||
// 4.2 s on two and 2.3 s on four, against a 5 s budget. On one machine its
|
||||
// one replica gets up to four; where there are two, two each (one each only
|
||||
// where two would not fit both). The starter has what is left, up to 8, or
|
||||
// no reservation at all when nothing is, rather than a starter that fits
|
||||
// nowhere. A machine that is not the node reserves two a judge on whichever
|
||||
// node takes it, and the starter, on a node not known yet, none.
|
||||
func (in *serverInit) cpuPlan() cpuSplit {
|
||||
if in.noNode {
|
||||
return cpuSplit{judge: judgeCores}
|
||||
}
|
||||
n := in.cpus
|
||||
if n <= 0 {
|
||||
n = runtime.NumCPU()
|
||||
}
|
||||
if n < 2 {
|
||||
return cpuSplit{}
|
||||
}
|
||||
judges := 0
|
||||
if !in.noGuardModel {
|
||||
judges = in.judgeFor().Replicas
|
||||
}
|
||||
judge := judgeCores
|
||||
switch {
|
||||
case judges == 1:
|
||||
judge = min(n, judgeCoresAlone)
|
||||
case judges > 0 && judges*judge > n:
|
||||
judge = max(1, n/judges)
|
||||
}
|
||||
left := max(n-judges*judge, 0)
|
||||
return cpuSplit{judge: judge, starter: min(left, 8)}
|
||||
}
|
||||
|
||||
// judgeCores and judgeMemoryGB are what a judge replica needs: two cores
|
||||
// for a judgment within its budget, and 2 GB for its 1.6 with no prompt
|
||||
// cache. judgeCoresAlone is what the one judge of a single machine takes,
|
||||
// past which a model this small hardly gets faster.
|
||||
const (
|
||||
judgeCores = 2
|
||||
judgeCoresAlone = 4
|
||||
judgeMemoryGB = 2
|
||||
)
|
||||
@@ -28,7 +28,7 @@ func TestTheInitAgentUnitIsTheShippedUnit(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
|
||||
m := starterModel("http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"}, true)
|
||||
m := starterModel(starterChoices[0], defaultContext, 0, "http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"}, true)
|
||||
c, err := catalog.New([]catalog.Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("the starter is not a valid catalog entry: %v", err)
|
||||
@@ -53,7 +53,7 @@ func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
|
||||
t.Error("the starter should be running, not waiting to be asked for")
|
||||
}
|
||||
|
||||
if m := starterModel("http://x/q.gguf", nil, true); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
|
||||
if m := starterModel(starterChoices[0], defaultContext, 0, "http://x/q.gguf", nil, true); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
|
||||
t.Errorf("without a GPU only the CPU profile: %+v", m.Profiles)
|
||||
}
|
||||
}
|
||||
@@ -96,7 +96,7 @@ func TestTheMirrorUnitAdvertisesAnAddressNodesCanReach(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestWithNoNodeHereTheStarterGoesToTheFirstNodeThatJoins(t *testing.T) {
|
||||
m := starterModel("http://x/q.gguf", nil, false)
|
||||
m := starterModel(starterChoices[0], defaultContext, 0, "http://x/q.gguf", nil, false)
|
||||
for _, p := range m.Profiles {
|
||||
if len(p.NodeSelector) > 0 {
|
||||
t.Errorf("profile %s is pinned to a machine that is not a node: %v", p.Name, p.NodeSelector)
|
||||
@@ -171,7 +171,7 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestTheGuardModelIsAValidJudge(t *testing.T) {
|
||||
m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf")
|
||||
m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 2, nil)
|
||||
c, err := catalog.New([]catalog.Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("not a valid catalog entry: %v", err)
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// Machines of the judges' own. Under load on a 4-core control plane that was
|
||||
// also a node, 38 of 60 requests were refused for want of a judgment; with the
|
||||
// judges on a node of their own the same test refused 2. A judge machine is
|
||||
// added as a node carrying judgeRoleLabel, and the judge's profile selects
|
||||
// that label, so nothing else is placed beside it and it on nothing else.
|
||||
|
||||
// judgeRoleLabel marks a node as the judges'.
|
||||
const (
|
||||
judgeRoleLabel = placement.RoleLabel
|
||||
judgeRoleValue = "judge"
|
||||
)
|
||||
|
||||
// judgeMachine is a machine for the judges: user@host, and the connection the
|
||||
// setup screen already made to it, when it made one.
|
||||
type judgeMachine struct {
|
||||
target string
|
||||
remote onboard.Remote
|
||||
}
|
||||
|
||||
// ensureJudgeNodes adds each judge machine as a node labelled for the
|
||||
// judges, and says how many cores the smallest of them has.
|
||||
func (in *serverInit) ensureJudgeNodes(ctx context.Context) error {
|
||||
if len(in.judgeMachines) == 0 {
|
||||
return nil
|
||||
}
|
||||
na := nodeAdd{serverCfg: in.configPath(), plain: true}
|
||||
na.opts.PKI = in.pkiDir
|
||||
na.opts.NoFirewall = in.noFirewall
|
||||
if err := na.prepare(); err != nil {
|
||||
return err
|
||||
}
|
||||
in.judgeCPUs = nil
|
||||
for _, jm := range in.judgeMachines {
|
||||
o := na.target(jm.target)
|
||||
fmt.Fprintf(in.out, " .. adding %s as a node for the judges\n", o.Addr)
|
||||
r := onboard.New(o, nil)
|
||||
if jm.remote != nil {
|
||||
r.Remote = jm.remote
|
||||
}
|
||||
res, err := r.Run(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %w", o.Addr, err)
|
||||
}
|
||||
if err := na.watch(ctx, res.Node, func(onboard.Event) {}); err != nil && err != errNoAdmin {
|
||||
return fmt.Errorf("%s: %w", res.Node, err)
|
||||
}
|
||||
if out, err := r.Remote.Run(ctx, "nproc", nil); err == nil {
|
||||
if n, err := strconv.Atoi(strings.TrimSpace(string(out))); err == nil {
|
||||
in.judgeCPUs = append(in.judgeCPUs, n)
|
||||
}
|
||||
}
|
||||
if _, err := changeStoredSettings(in.configPath(), "a node for the judges, from init", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"agents", "nodes", res.Node, "labels", judgeRoleLabel}, judgeRoleValue)
|
||||
}); err != nil {
|
||||
return fmt.Errorf("labelling %s for the judges: %w", res.Node, err)
|
||||
}
|
||||
in.done("%s is a node for the judges alone (%s: %s)", res.Node, judgeRoleLabel, judgeRoleValue)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// judgePlacement is the judge's cores and where it may run: on the judge
|
||||
// machines when there are some, by init's rule over their cores; otherwise
|
||||
// on any node, with this machine's plan.
|
||||
func (in *serverInit) judgePlacement() (cores int, selector map[string]string) {
|
||||
if len(in.judgeMachines) == 0 {
|
||||
return in.cpuPlan().judge, nil
|
||||
}
|
||||
return refitCores(in.judgeFor().Replicas, in.judgeCPUs), map[string]string{judgeRoleLabel: judgeRoleValue}
|
||||
}
|
||||
|
||||
// judgeMachineFlag is --judge-node, given again for each machine.
|
||||
type judgeMachineFlag struct{ to *[]judgeMachine }
|
||||
|
||||
func (f judgeMachineFlag) String() string {
|
||||
if f.to == nil {
|
||||
return ""
|
||||
}
|
||||
var out []string
|
||||
for _, m := range *f.to {
|
||||
out = append(out, m.target)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
func (f judgeMachineFlag) Set(v string) error {
|
||||
if strings.TrimSpace(v) == "" {
|
||||
return fmt.Errorf("--judge-node wants user@host")
|
||||
}
|
||||
*f.to = append(*f.to, judgeMachine{target: strings.TrimSpace(v)})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (judgeMachineFlag) Type() string { return "user@host" }
|
||||
|
||||
// judgeReadyWait is how long init waits for its judge to load before timing
|
||||
// it: the weights come from the mirror, and llama.cpp loads in seconds.
|
||||
const judgeReadyWait = 15 * time.Minute
|
||||
|
||||
// checkJudge times the judge once it is up, and keeps what it found for the
|
||||
// closing text: a machine too slow for its judge is said before its first
|
||||
// refused request, not after. The budget is not changed without asking.
|
||||
func (in *serverInit) checkJudge(ctx context.Context) {
|
||||
c := in.adminAPI()
|
||||
deadline := time.Now().Add(judgeReadyWait)
|
||||
for {
|
||||
var judges []adminapi.JudgeView
|
||||
if err := c.get(ctx, "/api/v1/guard/judges", &judges); err == nil {
|
||||
ready := false
|
||||
for _, j := range judges {
|
||||
ready = ready || (j.Model == guardModelName && j.Ready > 0)
|
||||
}
|
||||
if ready {
|
||||
break
|
||||
}
|
||||
}
|
||||
if time.Now().After(deadline) || ctx.Err() != nil {
|
||||
in.judgeNote = " !! the judge was not up to be timed; once it is: llmgw ctl guard judge check\n"
|
||||
return
|
||||
}
|
||||
time.Sleep(10 * time.Second)
|
||||
}
|
||||
fmt.Fprintf(in.out, " .. timing the judge\n")
|
||||
v, err := checkJudge(ctx, c, guardModelName, 0)
|
||||
if err != nil {
|
||||
in.judgeNote = " !! the judge could not be timed (" + err.Error() + "); later: llmgw ctl guard judge check\n"
|
||||
return
|
||||
}
|
||||
if v.Verdict == "ok" {
|
||||
in.done("the judge reads a prompt in %.1f s, of the %.1f s a request waits; about %.1f requests a second",
|
||||
float64(v.SlowestMs)/1000, float64(v.BudgetMs)/1000, v.PerSecond/2)
|
||||
return
|
||||
}
|
||||
in.judgeNote = "The judge is " + v.Verdict + " on this cluster:\n" + indent(judgeCheckText(v), " ") + "\n"
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestJudgesGoOnTheirOwnMachinesWhenTheInstallHasSome(t *testing.T) {
|
||||
in := newDefaultServerInit()
|
||||
if _, sel := in.judgePlacement(); sel != nil {
|
||||
t.Errorf("no judge machines, yet a selector: %v", sel)
|
||||
}
|
||||
if err := (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.21"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.22")
|
||||
in.judge.Replicas, in.judgeCPUs = 2, []int{4, 4}
|
||||
cores, sel := in.judgePlacement()
|
||||
if sel[judgeRoleLabel] != judgeRoleValue || cores != refitCores(2, []int{4, 4}) {
|
||||
t.Errorf("cores %d, selector %v", cores, sel)
|
||||
}
|
||||
m := guardModel("http://x/g.gguf", 2, cores, sel)
|
||||
if m.Profiles[0].NodeSelector[judgeRoleLabel] != judgeRoleValue {
|
||||
t.Errorf("the judge's profile does not select its machines: %+v", m.Profiles[0])
|
||||
}
|
||||
// The same from a script.
|
||||
flags := strings.Join(in.initFlags(), " ")
|
||||
if !strings.Contains(flags, "--judge-node ubuntu@10.0.0.21") || !strings.Contains(flags, "--judge-node ubuntu@10.0.0.22") {
|
||||
t.Errorf("flags %s", flags)
|
||||
}
|
||||
if err := (judgeMachineFlag{&in.judgeMachines}).Set(" "); err == nil {
|
||||
t.Error("an empty --judge-node was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSetupAsksForJudgeMachinesWhenChosen(t *testing.T) {
|
||||
m := newSetupModel("ctl", []string{"10.0.0.5"})
|
||||
m.choice, m.page = 1, pMachines
|
||||
m.mfocus = m.shown() * mfCount // the judges' choice
|
||||
m = press(m, "right").(setupModel)
|
||||
if m.judges != 1 || m.shown() != 3 || m.mfocus != 3*mfCount {
|
||||
t.Fatalf("judges %d, shown %d, focus %d", m.judges, m.shown(), m.mfocus)
|
||||
}
|
||||
if v := m.machinesView(); !strings.Contains(v, "Judge machine 1") || !strings.Contains(v, "Check the 4 machines") {
|
||||
t.Errorf("the page lacks the judge machine:\n%s", v)
|
||||
}
|
||||
for i, addr := range []string{"10.0.0.11", "10.0.0.12", ""} {
|
||||
m.db[i].addr.SetValue(addr)
|
||||
m.db[i].user.SetValue("ubuntu")
|
||||
m.db[i].auth = authAgent
|
||||
}
|
||||
if err := m.validateMachines(); !strings.Contains(err, "judge machine 1 has no address") {
|
||||
t.Errorf("err %q", err)
|
||||
}
|
||||
m.db[2].addr.SetValue("10.0.0.21")
|
||||
if err := m.validateMachines(); err != "" {
|
||||
t.Errorf("err %q", err)
|
||||
}
|
||||
if cmd := m.command(); !strings.Contains(cmd, "--judge-node ubuntu@10.0.0.21") || strings.Contains(cmd, "--patroni ubuntu@10.0.0.21") {
|
||||
t.Errorf("command %s", cmd)
|
||||
}
|
||||
// None again: the judge machine is no longer asked for.
|
||||
m = press(m, "left").(setupModel)
|
||||
if m.judges != 0 || strings.Contains(m.machinesView(), "Judge machine 1") {
|
||||
t.Errorf("judges %d after choosing none", m.judges)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||
)
|
||||
|
||||
func TestTheMemoryAMachineNeedsCountsWhatItServes(t *testing.T) {
|
||||
in := &serverInit{}
|
||||
mb, what := in.servingMemoryMB(false)
|
||||
need := preflight.Needs[preflight.RoleControl].MemoryMB + mb
|
||||
if what == "" || need <= 7941 || need > 11961 {
|
||||
t.Errorf("node, starter and judges here: need %d MB (%q); an 8 GB machine must fail and a 12 GB one pass", need, what)
|
||||
}
|
||||
// Judges on machines of their own take their share off this one.
|
||||
if mb2, _ := in.servingMemoryMB(true); mb2 >= mb {
|
||||
t.Errorf("with judge machines %d MB, without %d", mb2, mb)
|
||||
}
|
||||
// A control plane that serves nothing needs only its own.
|
||||
if mb, what := (&serverInit{noNode: true}).servingMemoryMB(false); mb != 0 || what != "" {
|
||||
t.Errorf("no node: %d %q", mb, what)
|
||||
}
|
||||
if mb, _ := (&serverInit{noModel: true, noGuardModel: true}).servingMemoryMB(false); mb != 0 {
|
||||
t.Errorf("nothing served: %d", mb)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,185 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.yaml.in/yaml/v3"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// Where the cluster says what went wrong: a model parked, a node lost, a
|
||||
// budget spent. notify.webhooks existed, and setup never asked for one, so a
|
||||
// site learnt of a lost node from its users.
|
||||
|
||||
// parseWebhook reads a webhook as setup takes it: an address, with "slack:"
|
||||
// or "json:" in front to say what it is sent, else what its address says.
|
||||
func parseWebhook(s string) (notify.Webhook, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
format := ""
|
||||
for _, f := range []string{"slack", "json"} {
|
||||
if rest, ok := strings.CutPrefix(s, f+":"); ok && !strings.HasPrefix(rest, "//") {
|
||||
format, s = f, rest
|
||||
}
|
||||
}
|
||||
u, err := url.Parse(s)
|
||||
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
|
||||
return notify.Webhook{}, fmt.Errorf("webhook %q: an http(s) address, with slack: or json: in front to say its format", s)
|
||||
}
|
||||
if format == "" {
|
||||
format = webhookFormat(u)
|
||||
}
|
||||
return notify.Webhook{URL: s, Format: format}, nil
|
||||
}
|
||||
|
||||
// webhookFormat is what an address takes: Slack's own, and the incoming
|
||||
// webhooks that read Slack's {"text": …} — Mattermost's and Rocket.Chat's
|
||||
// /hooks/, Discord's …/slack — are sent that; anything else the event as
|
||||
// JSON.
|
||||
func webhookFormat(u *url.URL) string {
|
||||
switch {
|
||||
case u.Host == "hooks.slack.com",
|
||||
strings.Contains(u.Path, "/hooks/"),
|
||||
strings.HasSuffix(u.Path, "/slack"):
|
||||
return "slack"
|
||||
}
|
||||
return "json"
|
||||
}
|
||||
|
||||
// parseWebhooks reads a list of them, by commas or as given.
|
||||
func parseWebhooks(list []string) ([]notify.Webhook, error) {
|
||||
var out []notify.Webhook
|
||||
for _, item := range list {
|
||||
for _, s := range strings.Split(item, ",") {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
continue
|
||||
}
|
||||
w, err := parseWebhook(s)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, w)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// webhookEntry is one of notify.webhooks as the settings document has it.
|
||||
type webhookEntry struct {
|
||||
URL string `yaml:"url"`
|
||||
Format string `yaml:"format,omitempty"`
|
||||
Events []string `yaml:"events,omitempty,flow"`
|
||||
SecretFile string `yaml:"secret_file,omitempty"`
|
||||
}
|
||||
|
||||
// withWebhooks adds these webhooks to a settings document's notify.webhooks,
|
||||
// and says whether that changed it: one already there, by address, is left
|
||||
// as it is, with whatever events and secret it was given.
|
||||
func withWebhooks(doc *settings.Doc, add []notify.Webhook) (bool, error) {
|
||||
path := []string{"notify", "webhooks"}
|
||||
var list []webhookEntry
|
||||
if n := doc.Node(path); n != nil {
|
||||
if err := n.Decode(&list); err != nil {
|
||||
return false, fmt.Errorf("notify.webhooks in the cluster's settings: %w", err)
|
||||
}
|
||||
}
|
||||
changed := false
|
||||
for _, w := range add {
|
||||
if !slices.ContainsFunc(list, func(e webhookEntry) bool { return e.URL == w.URL }) {
|
||||
list = append(list, webhookEntry{URL: w.URL, Format: w.Format})
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
if !changed {
|
||||
return false, nil
|
||||
}
|
||||
b, err := yaml.Marshal(list)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, doc.SetYAML(path, string(b))
|
||||
}
|
||||
|
||||
// ensureWebhooks puts the webhooks given into the cluster's settings and
|
||||
// sends each a test, once the control plane runs on them. Said, not fatal: a
|
||||
// cluster without its alerts still serves.
|
||||
func (in *serverInit) ensureWebhooks(ctx context.Context) error {
|
||||
if len(in.webhooks) == 0 {
|
||||
return nil
|
||||
}
|
||||
hooks, err := parseWebhooks(in.webhooks)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c := in.adminAPI()
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
changed, err := withWebhooks(doc, hooks)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
b, err := doc.Bytes()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(b), Note: "notification webhooks, from setup"})
|
||||
if err := c.put(ctx, "/api/v1/settings", req, &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
in.done("notification webhooks in the cluster's settings (version %d)", cur.Version)
|
||||
} else {
|
||||
in.kept("notification webhooks (already in the cluster's settings)")
|
||||
}
|
||||
|
||||
// The control planes start again on the change; the test is sent by
|
||||
// one that knows every webhook asked for.
|
||||
var res []notify.Result
|
||||
err = waitFor(ctx, 2*time.Minute, "the control plane to run on its webhooks", func() (bool, error) {
|
||||
res = nil
|
||||
if c.post(ctx, "/api/v1/notify/test", nil, &res) != nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, w := range hooks {
|
||||
// Answered by host: the path of a webhook is its secret.
|
||||
if !slices.ContainsFunc(res, func(r notify.Result) bool { return r.URL == notify.Redact(w.URL) }) {
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var failed []string
|
||||
for _, r := range res {
|
||||
if !slices.ContainsFunc(hooks, func(w notify.Webhook) bool { return notify.Redact(w.URL) == r.URL }) {
|
||||
continue
|
||||
}
|
||||
if r.Err != "" {
|
||||
failed = append(failed, r.URL+": "+r.Err)
|
||||
continue
|
||||
}
|
||||
in.done("a test notification reached %s", r.URL)
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return errors.New("a test notification did not arrive: " + strings.Join(failed, "; ") +
|
||||
" (kept in the settings: 'llmgw ctl notify test' tries again)")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func TestAWebhooksFormatIsWhatItsAddressTakes(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"https://hooks.slack.com/services/T1/B2/abc": "slack",
|
||||
"https://chat.example.com/hooks/xyz": "slack", // Mattermost, Rocket.Chat
|
||||
"https://discord.com/api/webhooks/1/abc/slack": "slack",
|
||||
"https://alerts.example.com/llmgw": "json",
|
||||
"json:https://hooks.slack.com/services/T1/B2/abc": "json",
|
||||
"slack:https://alerts.example.com/llmgw": "slack",
|
||||
" https://events.pagerduty.com/integration/x/enqueue ": "json",
|
||||
} {
|
||||
w, err := parseWebhook(in)
|
||||
if err != nil || w.Format != want || strings.HasPrefix(w.URL, "slack:") || strings.HasPrefix(w.URL, "json:") {
|
||||
t.Errorf("%q: %+v %v, want %s", in, w, err, want)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"hooks.slack.com/x", "ftp://example.com/x", "slack:", "https://"} {
|
||||
if _, err := parseWebhook(bad); err == nil {
|
||||
t.Errorf("%q was taken", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWebhooksAreAddedToTheSettingsBesideTheOnesThere(t *testing.T) {
|
||||
doc, err := settings.FromBytes("settings", []byte("router:\n retries: 2\nnotify:\n webhooks:\n"+
|
||||
" - url: https://alerts.example.com/a\n events: [node_lost]\n secret_file: /etc/llmgw/secrets/hook\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hooks, _ := parseWebhooks([]string{"https://alerts.example.com/a, https://hooks.slack.com/services/T/B/c"})
|
||||
changed, err := withWebhooks(doc, hooks)
|
||||
if err != nil || !changed {
|
||||
t.Fatalf("changed %v: %v", changed, err)
|
||||
}
|
||||
b, _ := doc.Bytes()
|
||||
got := string(b)
|
||||
for _, want := range []string{"retries: 2", "url: https://alerts.example.com/a", "events: [node_lost]",
|
||||
"secret_file: /etc/llmgw/secrets/hook", "url: https://hooks.slack.com/services/T/B/c", "format: slack"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("%q not in:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
if strings.Count(got, "alerts.example.com/a") != 1 {
|
||||
t.Errorf("a webhook already there was added again:\n%s", got)
|
||||
}
|
||||
if changed, _ := withWebhooks(doc, hooks); changed {
|
||||
t.Error("adding the same webhooks again changed the settings")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetupPutsItsWebhooksInTheSettingsAndTestsThemOnceInForce(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
body, tests := "router:\n retries: 2\n", 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 1, Body: body})
|
||||
case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
|
||||
var req adminapi.SettingsRequest
|
||||
_ = json.NewDecoder(r.Body).Decode(&req)
|
||||
body = req.Body
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 2, Body: body})
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/api/v1/notify/test":
|
||||
tests++
|
||||
// The first answer comes from the control plane before it
|
||||
// started again on the change: it knows no webhooks.
|
||||
res := []notify.Result{}
|
||||
if tests > 1 {
|
||||
res = []notify.Result{{URL: notify.Redact("https://hooks.slack.com/services/T/B/c")},
|
||||
{URL: notify.Redact("https://ops.example.com/in"), Err: "502 Bad Gateway"}}
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(res)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "server.yaml"),
|
||||
[]byte("admin_listen: "+strings.TrimPrefix(srv.URL, "http://")+"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out bytes.Buffer
|
||||
in := &serverInit{etcDir: dir, out: &out,
|
||||
webhooks: []string{"https://hooks.slack.com/services/T/B/c", "https://ops.example.com/in"}}
|
||||
err := in.ensureWebhooks(t.Context())
|
||||
if err == nil || !strings.Contains(err.Error(), "502") || !strings.Contains(err.Error(), "ctl notify test") {
|
||||
t.Errorf("a webhook that did not answer: %v", err)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if !strings.Contains(body, "format: slack") || !strings.Contains(body, "url: https://ops.example.com/in") ||
|
||||
!strings.Contains(body, "retries: 2") || tests < 2 {
|
||||
t.Errorf("settings %q after %d tests", body, tests)
|
||||
}
|
||||
if strings.Contains(out.String(), "/services/T/B/c") {
|
||||
t.Errorf("a webhook's secret path was printed:\n%s", out.String())
|
||||
}
|
||||
if !strings.Contains(out.String(), "reached https://hooks.slack.com/…") {
|
||||
t.Errorf("the one that answered is not said:\n%s", out.String())
|
||||
}
|
||||
}
|
||||
@@ -51,6 +51,8 @@ func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
|
||||
Remote: remote,
|
||||
})
|
||||
}
|
||||
// Kept: the standbys are made control planes on the same machines.
|
||||
in.patroniNodes = setup.DB
|
||||
self, err := sourceIPToward(setup.DB[0].IP)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
)
|
||||
|
||||
// Three machines, each able to take over: the store's two machines are made
|
||||
// control planes too, standing by while this one leads, as 'llmgw server add'
|
||||
// makes any machine one. With the store replicated across them and etcd's
|
||||
// three votes on the three, losing any one machine loses neither the store
|
||||
// nor the gateway. With a floating address, clients keep the one address.
|
||||
|
||||
// ensureStandbys makes each machine of the replicated store a control plane,
|
||||
// one after the other.
|
||||
func (in *serverInit) ensureStandbys(ctx context.Context) error {
|
||||
if len(in.patroniNodes) == 0 {
|
||||
return errors.New("the standbys go on the store's machines: give them with --patroni")
|
||||
}
|
||||
var hosts []string
|
||||
if ip, _, err := net.ParseCIDR(in.vip); err == nil {
|
||||
hosts = []string{ip.String()}
|
||||
}
|
||||
for i, n := range in.patroniNodes {
|
||||
fmt.Fprintf(in.out, " .. making %s a control plane too\n", n.Name)
|
||||
sa := serverAdd{
|
||||
etcDir: in.etcDir, pkiDir: in.pkiDir, hosts: hosts, vip: in.vip,
|
||||
remote: n.Remote, noFirewall: in.noFirewall, addr: n.IP, ip: n.IP,
|
||||
out: in.out, priority: 100 - 10*i, quiet: true,
|
||||
}
|
||||
if err := sa.run(ctx); err != nil {
|
||||
return fmt.Errorf("%s: %w", n.Name, err)
|
||||
}
|
||||
in.done("%s (%s) is a control plane too, standing by while this one leads", n.Name, n.IP)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,751 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/mail"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
)
|
||||
|
||||
// The parts of an install, each chosen and each with its settings: what the
|
||||
// setup screens show before anything is installed, for one machine and for
|
||||
// three alike. Beside each part is what it is for and what is missing
|
||||
// without it; Enter opens its settings, which say the same at more length.
|
||||
|
||||
type installPart struct {
|
||||
title string
|
||||
// about is what the part is, at the length its dialog has room for.
|
||||
about string
|
||||
// fixed parts are always installed: they have settings, not a box.
|
||||
fixed bool
|
||||
on bool
|
||||
// fields are its settings; none is a part only to tick.
|
||||
fields []*setupField
|
||||
// summary is a line of what it will be; off what is missing without it.
|
||||
summary func() string
|
||||
off string
|
||||
// needs, when set, says why the part cannot be had with the rest as
|
||||
// chosen: a starter model with no mirror to fetch it into.
|
||||
needs func() string
|
||||
}
|
||||
|
||||
type installForm struct {
|
||||
ha bool
|
||||
host string
|
||||
found []string
|
||||
parts []*installPart
|
||||
// focus is a part, or len(parts) for the button that goes on.
|
||||
focus int
|
||||
// open is the part whose settings are open in a dialog.
|
||||
open *installPart
|
||||
dialog *setupDialog
|
||||
// err is said in a dialog of its own; when errField is set, closing it
|
||||
// opens that field's part with the focus on it.
|
||||
err string
|
||||
errPart *installPart
|
||||
errField *setupField
|
||||
height int
|
||||
|
||||
standby *installPart
|
||||
vipF *setupField
|
||||
|
||||
control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart
|
||||
|
||||
admin, hosts, language, signup *setupField
|
||||
storeKind *setupField
|
||||
mirrorWhere, mirrorDirF, mirrorURL *setupField
|
||||
starterPick, starterSource, starterCtx *setupField
|
||||
judgeFail, judgeCopies, judgeBudget *setupField
|
||||
judgeCats []*setupField
|
||||
webhooks *setupField
|
||||
rpm, tpm, keyDays *setupField
|
||||
idle, safe *setupField
|
||||
bundlePath, bundleSHA *setupField
|
||||
}
|
||||
|
||||
// newInstallForm is the parts as in has them: its flags, or the defaults.
|
||||
func newInstallForm(in *serverInit, host string, found []string, ha bool) *installForm {
|
||||
f := &installForm{ha: ha, host: host, found: found, height: 24}
|
||||
|
||||
// --- the control plane ---
|
||||
f.admin = textSetupField("Administrator's email", in.admin, "you@example.com",
|
||||
"A password is made for this account and shown once, at the end.")
|
||||
if !ha {
|
||||
f.admin.Hint += " Leave it empty to make accounts later with 'llmgw users create'."
|
||||
}
|
||||
known := "none found"
|
||||
if len(found) > 0 {
|
||||
known = strings.Join(found, ", ")
|
||||
}
|
||||
f.hosts = textSetupField("Other names nodes reach this machine by", strings.Join(in.hosts, ", "),
|
||||
"llm.example.com, 203.0.113.10",
|
||||
"Already in its certificate: "+known+". Add a DNS name, a NAT address or a floating address, comma-separated.")
|
||||
lang := 0
|
||||
if in.language == "tr" {
|
||||
lang = 1
|
||||
}
|
||||
f.language = choiceField("The portal speaks", []string{"English", "Türkçe"}, lang,
|
||||
"To a browser that asks for neither. Each person can switch at the foot of any page.")
|
||||
f.signup = choiceField("Sign-up form", []string{"closed", "open"}, map[bool]int{false: 0, true: 1}[in.signupOpen], "")
|
||||
f.signup.Notes = []string{
|
||||
"Accounts are made by an administrator, or come from single sign-on.",
|
||||
"Anyone who reaches the portal can ask for an account; it waits for an administrator's approval.",
|
||||
}
|
||||
f.control = &installPart{
|
||||
title: "Control plane", fixed: true, on: true,
|
||||
about: "The gateway every request comes through, the admin API, the web portal on port 8080, and the " +
|
||||
"certificate authority that signs every node's identity. Always installed: it is what llmgw is.\n\n" +
|
||||
"Its authority's key stays in " + orText(in.pkiDir, defaultPKIDir) + " on this machine. Back it up: nothing recovers it.",
|
||||
fields: []*setupField{f.admin, f.hosts, f.language, f.signup},
|
||||
summary: func() string {
|
||||
s := []string{}
|
||||
if a := f.admin.Value(); a != "" {
|
||||
s = append(s, "administrator "+a)
|
||||
} else {
|
||||
s = append(s, "no administrator")
|
||||
}
|
||||
s = append(s, f.language.Options[f.language.Choice], "sign-up "+f.signup.Options[f.signup.Choice])
|
||||
return strings.Join(s, " · ")
|
||||
},
|
||||
}
|
||||
|
||||
// --- the store ---
|
||||
f.storeKind = choiceField("Kept in", []string{"SQLite", "PostgreSQL"}, map[bool]int{false: 0, true: 1}[in.postgres], "")
|
||||
f.storeKind.Notes = []string{
|
||||
"A file on this machine. Nothing to look after; enough for one control plane and a team's traffic.",
|
||||
"Installed here, with its own role and a generated password. For a larger store, or before adding a second control plane.",
|
||||
}
|
||||
f.store = &installPart{
|
||||
title: "Store", fixed: true, on: true,
|
||||
about: "Accounts, API keys, the catalog of models, usage and the cluster's settings live here.",
|
||||
summary: func() string {
|
||||
if ha {
|
||||
return "PostgreSQL under Patroni, on the other two"
|
||||
}
|
||||
return f.storeKind.Options[f.storeKind.Choice] + " on this machine"
|
||||
},
|
||||
}
|
||||
if ha {
|
||||
f.store.about += "\n\nWith three machines it is PostgreSQL on the other two, replicated under Patroni: " +
|
||||
"if the primary is lost the standby takes over in about thirty seconds, and the control plane follows. " +
|
||||
"The machines are asked for on the next page."
|
||||
} else {
|
||||
f.store.fields = []*setupField{f.storeKind}
|
||||
}
|
||||
|
||||
// --- control planes on all three ---
|
||||
if ha {
|
||||
f.vipF = textSetupField("Floating address", in.vip, "10.0.0.160/24",
|
||||
"Optional. An unused address on these machines' network, with its prefix: keepalived keeps it on "+
|
||||
"whichever control plane leads, and clients use it. Without one, put a load balancer in front, "+
|
||||
"or give clients the three addresses.")
|
||||
f.standby = &installPart{
|
||||
title: "Control plane on all three", on: in.standbys,
|
||||
about: "The two database machines become control planes too, standing by while this one leads. " +
|
||||
"If the leader is lost, one of them takes over within seconds: it places models, and every node " +
|
||||
"is told of it and fails over without being touched. With the store on two machines and etcd's " +
|
||||
"three votes on the three, losing any one machine loses neither the store nor the gateway.\n\n" +
|
||||
"The model mirror stays on this machine: while it is down, models the nodes already have keep " +
|
||||
"serving, and new ones wait for it.",
|
||||
fields: []*setupField{f.vipF},
|
||||
off: "the gateway runs on this machine only: if it is lost, the store survives but nothing is served until it is back",
|
||||
summary: func() string {
|
||||
if v := f.vipF.Value(); v != "" {
|
||||
return "two standbys; " + v + " follows whichever leads"
|
||||
}
|
||||
return "two standbys; no floating address"
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// --- the mirror ---
|
||||
where := 0
|
||||
if in.mirrorURL != "" {
|
||||
where = 1
|
||||
}
|
||||
f.mirrorWhere = choiceField("Where", []string{"on this machine", "on another machine"}, where, "")
|
||||
f.mirrorWhere.Notes = []string{
|
||||
"Serves model files on port 8081 to every node; its management API is on loopback only.",
|
||||
"Set up there first with 'llmgw mirror init --control-plane <this machine>'.",
|
||||
}
|
||||
f.mirrorDirF = textSetupField("Directory", orText(in.mirrorPath, mirrorDir), mirrorDir,
|
||||
"Models are large: a 7B model is about 5 GB, a 70B about 40 GB. Put it on the biggest disk.")
|
||||
f.mirrorDirF.Show = func() bool { return f.mirrorWhere.Choice == 0 }
|
||||
f.mirrorURL = textSetupField("Its management API", in.mirrorURL, "http://10.0.0.20:8082", "")
|
||||
f.mirrorURL.Show = func() bool { return f.mirrorWhere.Choice == 1 }
|
||||
f.mirror = &installPart{
|
||||
title: "Model mirror", on: !in.noMirror || in.mirrorURL != "",
|
||||
about: "Downloads each model once, from Hugging Face or wherever it comes from, and serves the files to " +
|
||||
"every node on the network, so ten nodes do not download the same 40 GB ten times. " +
|
||||
"The starter model, the judge and an offline bundle go into it.",
|
||||
fields: []*setupField{f.mirrorWhere, f.mirrorDirF, f.mirrorURL},
|
||||
off: "each model's files must already be on a mirror somewhere; no starter model, no judge",
|
||||
summary: func() string {
|
||||
if f.mirrorWhere.Choice == 1 {
|
||||
return "at " + orText(f.mirrorURL.Value(), "(an address)")
|
||||
}
|
||||
return "on this machine, in " + orText(f.mirrorDirF.Value(), mirrorDir)
|
||||
},
|
||||
}
|
||||
|
||||
// --- this machine as a node ---
|
||||
f.node = &installPart{
|
||||
title: "This machine as a node", on: !in.noNode,
|
||||
about: "Runs models here as well, on this machine's NVIDIA GPUs when it has them and on its CPU otherwise. " +
|
||||
"The node installs its own engines (llama.cpp first, vLLM after where the card can run it).\n\n" +
|
||||
"Leave it off when this machine has neither the memory nor the cards to spare: models then run on " +
|
||||
"the nodes added later with 'llmgw node add'.",
|
||||
off: "models run only on nodes added later",
|
||||
summary: func() string { return "serves models here, on its GPUs or its CPU" },
|
||||
}
|
||||
if ha {
|
||||
f.node.about += " With three machines the control plane usually serves none."
|
||||
}
|
||||
|
||||
// --- the starter model ---
|
||||
var opts, notes []string
|
||||
pick := 0
|
||||
cur := in.starterModelSpec()
|
||||
for i, c := range starterChoices {
|
||||
opts = append(opts, strings.TrimSuffix(c.Name, "-instruct"))
|
||||
notes = append(notes, fmt.Sprintf("%s, about %.1f GB: %s.", c.Name, c.SizeGB, c.About))
|
||||
if c.Name == cur.Name {
|
||||
pick = i
|
||||
}
|
||||
}
|
||||
opts = append(opts, "other")
|
||||
notes = append(notes, "Any single GGUF file on Hugging Face.")
|
||||
custom := ""
|
||||
if in.starter.Source != "" && !slices.ContainsFunc(starterChoices, func(c starterSpec) bool { return c.Name == in.starter.Name }) {
|
||||
pick, custom = len(starterChoices), in.starter.Source
|
||||
}
|
||||
f.starterPick = choiceField("Model", opts, pick, "")
|
||||
f.starterPick.Notes = notes
|
||||
f.starterSource = textSetupField("Its file", custom, "hf:owner/repo/file.gguf", "")
|
||||
f.starterSource.Show = func() bool { return f.starterPick.Choice == len(starterChoices) }
|
||||
f.starterCtx = textSetupField("Context a client may use, in tokens", strconv.Itoa(in.starterContext()), "4096",
|
||||
"The engine holds four times this, for four requests at once; more context takes more memory.")
|
||||
f.starter = &installPart{
|
||||
title: "Starter model", on: !in.noModel,
|
||||
about: "A model fetched into the mirror and served on this machine, so there is something to ask the " +
|
||||
"moment the install finishes. Small ones answer on any CPU; the larger want a GPU. More models are " +
|
||||
"added later from /admin/mirror or with 'llmgw ctl model add'.",
|
||||
fields: []*setupField{f.starterPick, f.starterSource, f.starterCtx},
|
||||
off: "nothing to ask until a model is added",
|
||||
summary: func() string {
|
||||
name := opts[f.starterPick.Choice]
|
||||
if f.starterPick.Choice == len(starterChoices) {
|
||||
name = orText(f.starterSource.Value(), "(a file)")
|
||||
} else {
|
||||
name = fmt.Sprintf("%s (%.1f GB)", starterChoices[f.starterPick.Choice].Name, starterChoices[f.starterPick.Choice].SizeGB)
|
||||
}
|
||||
return name + ", " + f.starterCtx.Value() + " tokens"
|
||||
},
|
||||
needs: func() string {
|
||||
if !f.mirror.on {
|
||||
return "the starter model is fetched into the mirror: turn the mirror on, or the starter model off"
|
||||
}
|
||||
return ""
|
||||
},
|
||||
}
|
||||
if ha {
|
||||
f.starter.about += " Unless this machine is a node, it runs on the first node that joins with room for it."
|
||||
}
|
||||
|
||||
// --- pictures and 3D ---
|
||||
f.media = &installPart{
|
||||
title: "Pictures and 3D", on: in.media,
|
||||
about: "The nodes install ComfyUI, and once a node has a card that fits one the control plane adds a " +
|
||||
"picture model (FLUX.1 schnell) and a 3D model (TripoSR) by itself, so the portal's pictures and 3D " +
|
||||
"pages and their APIs work without anybody writing a definition. They load when asked for and give " +
|
||||
"their card back when idle.\n\n" +
|
||||
"Each comes under its own licence, kept with every file it makes. A preset someone removes is not " +
|
||||
"added again. Later: 'llmgw ctl model presets auto off|on'.",
|
||||
off: "no pictures or 3D until a preset is added from the console or /admin",
|
||||
summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
|
||||
}
|
||||
|
||||
// --- the guardrails' judge ---
|
||||
j := in.judgeFor()
|
||||
f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
|
||||
map[string]int{"closed": 0, "open": 1}[j.Fail], "")
|
||||
f.judgeFail.Notes = []string{
|
||||
"The request is refused (503). Safe, and why the judge runs twice.",
|
||||
"The request is served unjudged and marked so in the logs. Nothing is refused for the judge's sake.",
|
||||
}
|
||||
f.judgeCopies = textSetupField("Copies", strconv.Itoa(j.Replicas), "2",
|
||||
"On one machine one copy, on up to four cores; with more machines two, two cores each, on different nodes. Each reserves 2 GB.")
|
||||
f.judgeBudget = textSetupField("Waits at most", durationYAML(j.Budget), "5s",
|
||||
"A short text is judged in about two seconds on CPUs.")
|
||||
judgeFields := []*setupField{f.judgeFail, f.judgeCopies, f.judgeBudget, headingField("Refuses")}
|
||||
catNotes := map[string]string{
|
||||
"PII": "the patterns already catch secrets and card numbers",
|
||||
"Unethical Acts": "refuses much ordinary work",
|
||||
"Politically Sensitive Topics": "refuses much ordinary work",
|
||||
"Copyright Violation": "refuses much ordinary work",
|
||||
}
|
||||
for _, c := range judgeCategories {
|
||||
t := toggleField(c, slices.Contains(j.Categories, c), catNotes[c])
|
||||
f.judgeCats = append(f.judgeCats, t)
|
||||
judgeFields = append(judgeFields, t)
|
||||
}
|
||||
f.judge = &installPart{
|
||||
title: "Guardrails' judge", on: !in.noGuardModel,
|
||||
about: "Qwen3Guard 0.6B reads every prompt and every answer for what patterns cannot see: violence, " +
|
||||
"crime, sexual content, self-harm, jailbreaks. It runs on CPUs and takes no card. Without it the " +
|
||||
"baseline still blocks secrets and card numbers and flags prompt injection.",
|
||||
fields: judgeFields,
|
||||
off: "requests are checked by patterns only: secrets, card numbers, injection",
|
||||
summary: func() string {
|
||||
n := 0
|
||||
for _, c := range f.judgeCats {
|
||||
if c.On {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("Qwen3Guard ×%s on CPUs, %d categories, unanswered: %s",
|
||||
f.judgeCopies.Value(), n, f.judgeFail.Options[f.judgeFail.Choice])
|
||||
},
|
||||
needs: func() string {
|
||||
if !f.mirror.on {
|
||||
return "the judge's model is fetched into the mirror: turn the mirror on, or the judge off"
|
||||
}
|
||||
return ""
|
||||
},
|
||||
}
|
||||
|
||||
// --- the firewall ---
|
||||
f.firewall = &installPart{
|
||||
title: "Firewall", on: !in.noFirewall,
|
||||
about: "An nftables table of llmgw's own keeps its internal ports — the agents' 7443, the mirror's 8081, " +
|
||||
"the store's — to the networks the cluster is on. The portal and API on 8080 stay open. " +
|
||||
"Other rules on the machine are left as they are.",
|
||||
off: "llmgw's internal ports answer every network this machine is on",
|
||||
summary: func() string { return "internal ports to the cluster's networks only" },
|
||||
}
|
||||
|
||||
// --- notifications ---
|
||||
f.webhooks = textSetupField("Webhooks", strings.Join(in.webhooks, ", "), "https://hooks.slack.com/services/…",
|
||||
"Slack, Mattermost, Rocket.Chat, or any address taking JSON; comma-separated. slack: or json: in front says which. A test is sent to each.")
|
||||
f.alerts = &installPart{
|
||||
title: "Notifications", on: len(in.webhooks) > 0,
|
||||
about: "Where the cluster says what went wrong: a model parked because it fits nowhere, a node lost, " +
|
||||
"a judge not answering, a budget spent.",
|
||||
fields: []*setupField{f.webhooks},
|
||||
off: "nothing is said outside the dashboards",
|
||||
summary: func() string {
|
||||
hooks, err := parseWebhooks([]string{f.webhooks.Value()})
|
||||
if err != nil || len(hooks) == 0 {
|
||||
return "(no webhook yet)"
|
||||
}
|
||||
var where []string
|
||||
for _, h := range hooks {
|
||||
where = append(where, notify.Redact(h.URL))
|
||||
}
|
||||
return strings.Join(where, ", ")
|
||||
},
|
||||
}
|
||||
|
||||
// --- limits ---
|
||||
f.rpm = textSetupField("Requests a minute", strconv.Itoa(orInt(in.defaultRPM, defaultRPM)), "60", "")
|
||||
f.tpm = textSetupField("Tokens a minute", strconv.Itoa(orInt(in.defaultTPM, defaultTPM)), "200000", "")
|
||||
ttl := in.maxKeyTTL
|
||||
if ttl <= 0 {
|
||||
ttl = defaultMaxKeyTTL
|
||||
}
|
||||
f.keyDays = textSetupField("The longest a key lives, in days", strconv.Itoa(int(ttl.Hours()/24)), "90",
|
||||
"A key made without an expiry gets this. Shorter per organization, team or project later.")
|
||||
f.limits = &installPart{
|
||||
title: "Limits and keys", fixed: true, on: true,
|
||||
about: "What a key may use when nothing more particular gives it a limit: a project made without limits " +
|
||||
"would otherwise be metered by nothing. Organizations, teams and projects get their own later.",
|
||||
fields: []*setupField{f.rpm, f.tpm, f.keyDays},
|
||||
summary: func() string {
|
||||
return fmt.Sprintf("%s requests and %s tokens a minute; keys live %s days", f.rpm.Value(), f.tpm.Value(), f.keyDays.Value())
|
||||
},
|
||||
}
|
||||
|
||||
// --- on-demand turns ---
|
||||
f.idle = textSetupField("Unused for", durationYAML(orDur(in.evictAfterIdle, 2*time.Minute)), "2m",
|
||||
"before a model loaded on demand gives its card to another that wants it.")
|
||||
f.safe = textSetupField("Safe after loading for", durationYAML(orDur(in.minLifetime, 5*time.Minute)), "5m",
|
||||
"so two models do not take turns paying for loads. 1m each suits one card shared by a picture model and a 3D model.")
|
||||
f.turns = &installPart{
|
||||
title: "Taking turns on a card", fixed: true, on: true,
|
||||
about: "Models that load when asked for share the cards: on a site with fewer cards than such models, " +
|
||||
"one waits for another to go idle. These are the waits.",
|
||||
fields: []*setupField{f.idle, f.safe},
|
||||
summary: func() string {
|
||||
return "after " + f.idle.Value() + " unused; safe for " + f.safe.Value() + " after loading"
|
||||
},
|
||||
}
|
||||
|
||||
// --- offline bundle ---
|
||||
f.bundlePath = textSetupField("Bundle", in.bundle, "/media/usb/llmgw-bundle.tar",
|
||||
"Made on a machine with internet by 'llmgw bundle create'.")
|
||||
f.bundleSHA = textSetupField("Its sha256", in.bundleSHA, "the sha256 'llmgw bundle create' printed", "")
|
||||
f.bundleSHA.Text.Width = 64
|
||||
f.bundle = &installPart{
|
||||
title: "Offline bundle", on: in.bundle != "",
|
||||
about: "For a site with no internet: engines, releases and models carried in on a disk. They go into the " +
|
||||
"mirror, and the cluster is set to reach nothing outside.",
|
||||
fields: []*setupField{f.bundlePath, f.bundleSHA},
|
||||
off: "engines and models are downloaded from the internet",
|
||||
summary: func() string { return orText(f.bundlePath.Value(), "(a file)") },
|
||||
needs: func() string {
|
||||
if !f.mirror.on {
|
||||
return "an offline bundle goes into the model mirror: turn the mirror on"
|
||||
}
|
||||
return ""
|
||||
},
|
||||
}
|
||||
|
||||
f.parts = []*installPart{f.control, f.store}
|
||||
if ha {
|
||||
f.parts = append(f.parts, f.standby)
|
||||
}
|
||||
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall,
|
||||
f.alerts, f.limits, f.turns, f.bundle)
|
||||
return f
|
||||
}
|
||||
|
||||
func orText(s, d string) string {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return d
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func orInt(n, d int) int {
|
||||
if n <= 0 {
|
||||
return d
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func orDur(v, d time.Duration) time.Duration {
|
||||
if v <= 0 {
|
||||
return d
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// openPart opens a part's settings.
|
||||
func (f *installForm) openPart(p *installPart) {
|
||||
f.open = p
|
||||
f.dialog = &setupDialog{Title: p.title, About: p.about, Fields: p.fields}
|
||||
f.dialog.first()
|
||||
}
|
||||
|
||||
// fail says what is wrong, and with what, so that closing the message opens it.
|
||||
func (f *installForm) fail(p *installPart, field *setupField, msg string) error {
|
||||
f.err, f.errPart, f.errField = msg, p, field
|
||||
return fmt.Errorf("%s", msg)
|
||||
}
|
||||
|
||||
// key takes a key; next is whether the person asked to go on, the choices
|
||||
// already put into in.
|
||||
func (f *installForm) key(k tea.KeyMsg, in *serverInit) (next bool) {
|
||||
if f.err != "" {
|
||||
p, field := f.errPart, f.errField
|
||||
f.err, f.errPart, f.errField = "", nil, nil
|
||||
if p != nil {
|
||||
f.openPart(p)
|
||||
if field != nil {
|
||||
f.dialog.focusOn(field)
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
if f.dialog != nil {
|
||||
if f.dialog.key(k) {
|
||||
f.dialog, f.open = nil, nil
|
||||
}
|
||||
return false
|
||||
}
|
||||
n := len(f.parts) + 1
|
||||
switch k.String() {
|
||||
case "down", "tab", "j":
|
||||
f.focus = (f.focus + 1) % n
|
||||
case "up", "shift+tab", "k":
|
||||
f.focus = (f.focus + n - 1) % n
|
||||
case " ", "x":
|
||||
if f.focus < len(f.parts) {
|
||||
p := f.parts[f.focus]
|
||||
if p.fixed {
|
||||
break
|
||||
}
|
||||
p.on = !p.on
|
||||
// A part that is nothing until it is told something is told
|
||||
// at once.
|
||||
if p.on && (p == f.alerts && f.webhooks.Value() == "" || p == f.bundle && f.bundlePath.Value() == "") {
|
||||
f.openPart(p)
|
||||
}
|
||||
}
|
||||
case "enter":
|
||||
if f.focus < len(f.parts) {
|
||||
f.openPart(f.parts[f.focus])
|
||||
return false
|
||||
}
|
||||
return f.apply(in) == nil
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// apply checks the choices and puts them into in; what is wrong is kept to
|
||||
// be said.
|
||||
func (f *installForm) apply(in *serverInit) error {
|
||||
for _, p := range f.parts {
|
||||
if p.on && p.needs != nil {
|
||||
if why := p.needs(); why != "" {
|
||||
return f.fail(nil, nil, why)
|
||||
}
|
||||
}
|
||||
}
|
||||
admin := f.admin.Value()
|
||||
if admin != "" || f.ha {
|
||||
if _, err := mail.ParseAddress(admin); err != nil {
|
||||
msg := "the administrator's email is not an address"
|
||||
if admin == "" {
|
||||
msg = "the administrator's email is needed"
|
||||
}
|
||||
return f.fail(f.control, f.admin, msg)
|
||||
}
|
||||
}
|
||||
var hosts []string
|
||||
for _, h := range strings.Split(f.hosts.Value(), ",") {
|
||||
if h = strings.TrimSpace(h); h != "" {
|
||||
hosts = append(hosts, h)
|
||||
}
|
||||
}
|
||||
|
||||
mirrorURL := ""
|
||||
if f.mirror.on && f.mirrorWhere.Choice == 1 {
|
||||
u, err := url.Parse(f.mirrorURL.Value())
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return f.fail(f.mirror, f.mirrorURL, "the other machine's mirror: its management API, as http://host:8082")
|
||||
}
|
||||
mirrorURL = f.mirrorURL.Value()
|
||||
}
|
||||
mirrorPath := ""
|
||||
if f.mirror.on && f.mirrorWhere.Choice == 0 {
|
||||
mirrorPath = f.mirrorDirF.Value()
|
||||
if !strings.HasPrefix(mirrorPath, "/") {
|
||||
return f.fail(f.mirror, f.mirrorDirF, "the mirror's directory: a path from /")
|
||||
}
|
||||
}
|
||||
|
||||
var starter starterSpec
|
||||
ctx := 0
|
||||
if f.starter.on {
|
||||
if f.starterPick.Choice < len(starterChoices) {
|
||||
starter = starterChoices[f.starterPick.Choice]
|
||||
} else {
|
||||
s, err := starterFrom(f.starterSource.Value())
|
||||
if err != nil || !strings.HasPrefix(s.Source, "hf:") {
|
||||
return f.fail(f.starter, f.starterSource, "the starter model's file: hf:<owner>/<repo>/<file>.gguf")
|
||||
}
|
||||
starter = s
|
||||
}
|
||||
n, err := strconv.Atoi(f.starterCtx.Value())
|
||||
if err != nil || n < 512 || n > 262144 {
|
||||
return f.fail(f.starter, f.starterCtx, "the starter's context: a number of tokens from 512 to 262144")
|
||||
}
|
||||
ctx = n
|
||||
}
|
||||
|
||||
var judge judgeSpec
|
||||
if f.judge.on {
|
||||
judge.Fail = []string{"closed", "open"}[f.judgeFail.Choice]
|
||||
n, err := strconv.Atoi(f.judgeCopies.Value())
|
||||
if err != nil || n < 1 || n > 8 {
|
||||
return f.fail(f.judge, f.judgeCopies, "the judge's copies: from 1 to 8")
|
||||
}
|
||||
judge.Replicas = n
|
||||
d, err := time.ParseDuration(f.judgeBudget.Value())
|
||||
if err != nil || d < 500*time.Millisecond || d > time.Minute {
|
||||
return f.fail(f.judge, f.judgeBudget, "how long the judge is waited for: a duration such as 5s, at most 1m")
|
||||
}
|
||||
judge.Budget = d
|
||||
for i, c := range f.judgeCats {
|
||||
if c.On {
|
||||
judge.Categories = append(judge.Categories, judgeCategories[i])
|
||||
}
|
||||
}
|
||||
if len(judge.Categories) == 0 {
|
||||
return f.fail(f.judge, f.judgeCats[0], "the judge refuses nothing: tick a category, or turn the judge off")
|
||||
}
|
||||
}
|
||||
|
||||
var hooks []string
|
||||
if f.alerts.on {
|
||||
h := f.webhooks.Value()
|
||||
parsed, err := parseWebhooks([]string{h})
|
||||
if err != nil {
|
||||
return f.fail(f.alerts, f.webhooks, err.Error())
|
||||
}
|
||||
if len(parsed) == 0 {
|
||||
return f.fail(f.alerts, f.webhooks, "give a webhook, or turn notifications off")
|
||||
}
|
||||
hooks = []string{h}
|
||||
}
|
||||
|
||||
rpm, err := strconv.Atoi(f.rpm.Value())
|
||||
if err != nil || rpm < 1 {
|
||||
return f.fail(f.limits, f.rpm, "requests a minute: a whole number, 1 or more")
|
||||
}
|
||||
tpm, err := strconv.Atoi(f.tpm.Value())
|
||||
if err != nil || tpm < 1 {
|
||||
return f.fail(f.limits, f.tpm, "tokens a minute: a whole number, 1 or more")
|
||||
}
|
||||
keyDays, err := strconv.Atoi(f.keyDays.Value())
|
||||
if err != nil || keyDays < 1 || keyDays > 3650 {
|
||||
return f.fail(f.limits, f.keyDays, "the longest a key lives: days, from 1 to 3650")
|
||||
}
|
||||
idle, err := time.ParseDuration(f.idle.Value())
|
||||
if err != nil || idle < 0 {
|
||||
return f.fail(f.turns, f.idle, "a duration such as 2m or 90s")
|
||||
}
|
||||
safe, err := time.ParseDuration(f.safe.Value())
|
||||
if err != nil || safe < 0 {
|
||||
return f.fail(f.turns, f.safe, "a duration such as 5m or 90s")
|
||||
}
|
||||
|
||||
bundle, sum := "", ""
|
||||
if f.bundle.on {
|
||||
bundle, sum = f.bundlePath.Value(), strings.ToLower(f.bundleSHA.Value())
|
||||
switch {
|
||||
case bundle == "":
|
||||
return f.fail(f.bundle, f.bundlePath, "give the bundle's path, or turn the offline bundle off")
|
||||
case sum == "":
|
||||
return f.fail(f.bundle, f.bundleSHA, "give the bundle's sha256, as 'llmgw bundle create' printed it")
|
||||
}
|
||||
}
|
||||
|
||||
in.admin, in.hosts = admin, hosts
|
||||
in.language = map[int]string{0: "", 1: "tr"}[f.language.Choice]
|
||||
in.signupOpen = f.signup.Choice == 1
|
||||
in.postgres = !f.ha && f.storeKind.Choice == 1
|
||||
in.noMirror = !f.mirror.on || mirrorURL != ""
|
||||
in.mirrorURL, in.mirrorPath = mirrorURL, mirrorPath
|
||||
in.noNode = !f.node.on
|
||||
in.noModel = !f.starter.on
|
||||
in.starter, in.starterCtx = starter, ctx
|
||||
in.media = f.media.on
|
||||
in.noGuardModel = !f.judge.on
|
||||
in.judge = judge
|
||||
in.noFirewall = !f.firewall.on
|
||||
in.webhooks = hooks
|
||||
in.defaultRPM, in.defaultTPM = rpm, tpm
|
||||
in.maxKeyTTL = time.Duration(keyDays) * 24 * time.Hour
|
||||
in.evictAfterIdle, in.minLifetime = 0, 0
|
||||
if idle != 2*time.Minute {
|
||||
in.evictAfterIdle = idle
|
||||
}
|
||||
if safe != 5*time.Minute {
|
||||
in.minLifetime = safe
|
||||
}
|
||||
in.bundle, in.bundleSHA = bundle, sum
|
||||
in.standbys, in.vip = false, ""
|
||||
if f.standby != nil && f.standby.on {
|
||||
in.standbys, in.vip = true, f.vipF.Value()
|
||||
}
|
||||
if err := in.checkChoices(); err != nil {
|
||||
if strings.HasPrefix(err.Error(), "floating address") {
|
||||
return f.fail(f.standby, f.vipF, err.Error())
|
||||
}
|
||||
return f.fail(nil, nil, err.Error())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// view is the list of parts, and what the focused one is, under it.
|
||||
func (f *installForm) view(width int) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n " + sDim.Render("Space turns a part on or off · Enter opens its settings and says more") + "\n\n")
|
||||
titleW := 0
|
||||
for _, p := range f.parts {
|
||||
titleW = max(titleW, len(p.title))
|
||||
}
|
||||
for i, p := range f.parts {
|
||||
box := "[ ]"
|
||||
switch {
|
||||
case p.fixed:
|
||||
box = " • "
|
||||
case p.on:
|
||||
box = "[" + sOK.Render("x") + "]"
|
||||
}
|
||||
title := fmt.Sprintf("%-*s", titleW, p.title)
|
||||
line := ""
|
||||
if p.on {
|
||||
line = p.summary()
|
||||
} else {
|
||||
line = sDim.Render(p.off)
|
||||
}
|
||||
if i == f.focus {
|
||||
title = fSelected.Render(title)
|
||||
b.WriteString(fSelected.Render("› ") + box + " " + title + " " + truncateWidth(line, max(20, width-titleW-8)) + "\n")
|
||||
} else {
|
||||
b.WriteString(" " + box + " " + title + " " + truncateWidth(line, max(20, width-titleW-8)) + "\n")
|
||||
}
|
||||
}
|
||||
b.WriteString("\n" + button(f.focus == len(f.parts), "Next") + "\n")
|
||||
if f.focus < len(f.parts) {
|
||||
p := f.parts[f.focus]
|
||||
b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render(p.title) + "\n")
|
||||
b.WriteString(paragraphs(p.about, max(40, width-6), " ") + "\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// overlay is the dialog over the list, if one is open: its title and body.
|
||||
func (f *installForm) overlay() (string, string) {
|
||||
if f.err != "" {
|
||||
return "Not yet", f.err + "\n\n" + sDim.Render("any key to go back")
|
||||
}
|
||||
if f.dialog != nil {
|
||||
return f.dialog.Title, f.dialog.view(max(8, f.height-8))
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// keys is the key bar for the list or its dialog.
|
||||
func (f *installForm) keys() string {
|
||||
switch {
|
||||
case f.err != "":
|
||||
return "any key back"
|
||||
case f.dialog != nil:
|
||||
return setupDialogKeys(f.dialog)
|
||||
}
|
||||
return "↑↓ move · Space on/off · Enter settings, or on Next go on · Ctrl+C leave"
|
||||
}
|
||||
|
||||
// planView says what will be installed and what will not, part by part, in
|
||||
// words: the last look before anything is done.
|
||||
func (f *installForm) planView(width int) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render("Installed on "+f.host) + "\n\n")
|
||||
var left []*installPart
|
||||
for _, p := range f.parts {
|
||||
if !p.on {
|
||||
left = append(left, p)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, " %s %s\n %s\n", sOK.Render("✓"), p.title, wrap(p.summary(), max(40, width-8), " "))
|
||||
}
|
||||
if len(left) > 0 {
|
||||
b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render("Left out") + "\n\n")
|
||||
for _, p := range left {
|
||||
fmt.Fprintf(&b, " %s %s\n %s\n", sDim.Render("-"), p.title, sDim.Render(wrap(p.off, max(40, width-8), " ")))
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
+61
-17
@@ -53,6 +53,7 @@ const (
|
||||
cAccounts
|
||||
cKeys
|
||||
cOrgs
|
||||
cSignIn
|
||||
cAudit
|
||||
cManual
|
||||
cSettings
|
||||
@@ -69,7 +70,7 @@ const (
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
@@ -90,7 +91,10 @@ type consoleData struct {
|
||||
// federation is the other llmgw clusters whose models are offered
|
||||
// here; empty on a control plane without a store.
|
||||
federation adminapi.FederationView
|
||||
err error
|
||||
// presets is the models that come with llmgw as they stand here; nil
|
||||
// from a control plane that cannot say.
|
||||
presets *adminapi.PresetsView
|
||||
err error
|
||||
}
|
||||
|
||||
type (
|
||||
@@ -137,8 +141,8 @@ type consoleModel struct {
|
||||
detail bool
|
||||
scroll int
|
||||
|
||||
// node is the console of a node: its agent.yaml and nothing to watch,
|
||||
// since the cluster is watched from the control plane.
|
||||
// node is the console of a node: what runs on the machine and its
|
||||
// agent.yaml, since the cluster is watched from the control plane.
|
||||
node bool
|
||||
|
||||
// The settings.
|
||||
@@ -178,6 +182,7 @@ type consoleModel struct {
|
||||
quotas []adminapi.QuotaView
|
||||
quotasErr error
|
||||
guard guardState
|
||||
signIn signInState
|
||||
kb kbState
|
||||
|
||||
st *storeHolder
|
||||
@@ -187,6 +192,11 @@ type consoleModel struct {
|
||||
docs docViewer
|
||||
mon monitorState
|
||||
det monitorState
|
||||
|
||||
// nodeOv is a node's overview as last read; nodeBusy says a reading is
|
||||
// under way.
|
||||
nodeOv nodeOverview
|
||||
nodeBusy bool
|
||||
}
|
||||
|
||||
var usageGroupings = []string{"team", "project", "org"}
|
||||
@@ -195,9 +205,6 @@ func newConsoleModel(addr, file string) consoleModel {
|
||||
m := consoleModel{client: newCtlClient(addr), spin: newSpinner(), width: 100, height: 30, file: file,
|
||||
st: &storeHolder{file: file}, docs: newDocViewer()}
|
||||
m.node = strings.HasSuffix(file, "agent.yaml")
|
||||
if m.node {
|
||||
m.menu = cSettings
|
||||
}
|
||||
m.loadSettings()
|
||||
return m
|
||||
}
|
||||
@@ -254,7 +261,7 @@ func (m *consoleModel) loadSettings() {
|
||||
|
||||
func (m consoleModel) Init() tea.Cmd {
|
||||
if m.node {
|
||||
return m.spin.Tick
|
||||
return tea.Batch(m.fetchNodeOverview(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
|
||||
}
|
||||
return tea.Batch(m.fetch(), m.spin.Tick, tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} }))
|
||||
}
|
||||
@@ -275,6 +282,10 @@ func (m consoleModel) fetch() tea.Cmd {
|
||||
_ = c.get(ctx, "/api/v1/jobs", &d.jobs)
|
||||
_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
|
||||
_ = c.get(ctx, "/api/v1/federation", &d.federation)
|
||||
var presets adminapi.PresetsView
|
||||
if c.get(ctx, "/api/v1/presets", &presets) == nil {
|
||||
d.presets = &presets
|
||||
}
|
||||
return consoleDataMsg(d)
|
||||
}
|
||||
}
|
||||
@@ -313,7 +324,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
return m, cmd
|
||||
case consoleTickMsg:
|
||||
if m.node {
|
||||
return m, nil
|
||||
return m.nodeTick()
|
||||
}
|
||||
cmds := []tea.Cmd{m.fetch(), tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })}
|
||||
if m.menu == cUsage {
|
||||
@@ -325,6 +336,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
if m.menu == cGuard {
|
||||
cmds = append(cmds, m.fetchGuard())
|
||||
}
|
||||
if m.menu == cSignIn {
|
||||
cmds = append(cmds, m.fetchSignIn())
|
||||
}
|
||||
if m.menu == cKB {
|
||||
cmds = append(cmds, m.fetchKB())
|
||||
}
|
||||
@@ -344,6 +358,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
cmds = append(cmds, m.fetchLogs(m.logs.node))
|
||||
}
|
||||
return m, tea.Batch(cmds...)
|
||||
case nodeOverviewMsg:
|
||||
m.nodeOv, m.nodeBusy = nodeOverview(msg), false
|
||||
case consoleDataMsg:
|
||||
m.loaded = true
|
||||
if msg.err != nil {
|
||||
@@ -388,6 +404,8 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
m.quotas, m.quotasErr = msg.rows, msg.err
|
||||
case consoleKBMsg:
|
||||
m.kb = kbState{view: msg.view, err: msg.err, loaded: true}
|
||||
case consoleSignInMsg:
|
||||
m.signIn = signInState{view: msg.view, err: msg.err, loaded: true, ldap: msg.ldap, oidc: msg.oidc}
|
||||
case consoleGuardMsg:
|
||||
window := m.guard.window
|
||||
m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
|
||||
@@ -416,7 +434,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
|
||||
func (m consoleModel) entryOK(i int) bool {
|
||||
if m.node {
|
||||
return i == cSettings || i == cManual || i == cWipe || i == cLeave
|
||||
return i == cOverview || i == cSettings || i == cManual || i == cWipe || i == cLeave
|
||||
}
|
||||
return i != cReplicate || !m.storeReplicated()
|
||||
}
|
||||
@@ -546,6 +564,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.menuD = m.quotaMenu(m.quotas[m.row])
|
||||
case cGuard:
|
||||
m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
|
||||
case cSignIn:
|
||||
m.menuD = m.signInMenu(m.signInRowKey())
|
||||
case cKB:
|
||||
m.menuD = m.kbMenu(m.kb.view.Collections[m.row])
|
||||
case cMirror:
|
||||
@@ -599,6 +619,10 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.hf.models, m.hf.err, m.row = nil, nil, 0
|
||||
return m, m.fetchHF()
|
||||
}
|
||||
case "p":
|
||||
if m.menu == cModels && m.inPanel && m.data.presets != nil {
|
||||
m.autoPresetsConfirm()
|
||||
}
|
||||
case "a":
|
||||
switch m.menu {
|
||||
case cQuotas:
|
||||
@@ -638,6 +662,19 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.usageExportForm()
|
||||
return m, nil
|
||||
}
|
||||
if m.menu == cSignIn && m.signIn.loaded {
|
||||
m.entraForm()
|
||||
return m, nil
|
||||
}
|
||||
case "d":
|
||||
if m.menu == cSignIn && m.signIn.loaded {
|
||||
m.directoryForm()
|
||||
return m, nil
|
||||
}
|
||||
case "t":
|
||||
if m.menu == cSignIn && m.signIn.view.LDAP != nil {
|
||||
return m, m.tryDirectory()
|
||||
}
|
||||
case "right", "l":
|
||||
if m.menu == cUsage {
|
||||
m.usageBy = (m.usageBy + 1) % len(usageGroupings)
|
||||
@@ -664,6 +701,8 @@ func (m consoleModel) onEnter() tea.Cmd {
|
||||
return m.fetchQuotas()
|
||||
case cGuard:
|
||||
return m.fetchGuard()
|
||||
case cSignIn:
|
||||
return m.fetchSignIn()
|
||||
case cKB:
|
||||
return m.fetchKB()
|
||||
case cMirror:
|
||||
@@ -1042,6 +1081,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.quotas)
|
||||
case cGuard:
|
||||
return len(m.guard.view.Attachments)
|
||||
case cSignIn:
|
||||
return len(signInRows)
|
||||
case cKB:
|
||||
return len(m.kb.view.Collections)
|
||||
case cMirror:
|
||||
@@ -1069,9 +1110,6 @@ func (m consoleModel) rowCount() int {
|
||||
func (m consoleModel) View() string {
|
||||
host, _ := os.Hostname()
|
||||
nav := []navItem{{Label: "Watch", Heading: true}}
|
||||
if m.node {
|
||||
nav = nil
|
||||
}
|
||||
for i := 0; i < cCount; i++ {
|
||||
if m.node && !m.entryOK(i) {
|
||||
continue
|
||||
@@ -1166,6 +1204,9 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
}
|
||||
switch m.menu {
|
||||
case cOverview:
|
||||
if m.node {
|
||||
return m.nodeOverviewPanel(pw)
|
||||
}
|
||||
var b bytes.Buffer
|
||||
c := &cobra.Command{}
|
||||
c.SetOut(&b)
|
||||
@@ -1238,8 +1279,12 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
}
|
||||
if m.inPanel {
|
||||
keys = "↑↓ row · Enter actions · A add a model · ← back · Q leave"
|
||||
if m.data.presets != nil {
|
||||
keys = "↑↓ row · Enter actions · A add a model · P pictures and 3D added by themselves · ← back · Q leave"
|
||||
}
|
||||
}
|
||||
return "Models", "\n" + table([]string{"MODEL", "READY", "STATUS", "CLASS", "PROFILES", "UPDATE", ""}, rows, m.row, pw, m.inPanel), keys
|
||||
return "Models", "\n" + table([]string{"MODEL", "READY", "STATUS", "CLASS", "PROFILES", "UPDATE", ""}, rows, m.row, pw, m.inPanel) +
|
||||
presetsBlock(m.data.presets, pw), keys
|
||||
case cReplicas:
|
||||
var rows [][]string
|
||||
for _, r := range m.data.replicas {
|
||||
@@ -1277,6 +1322,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return m.quotasPanel(pw)
|
||||
case cGuard:
|
||||
return m.guardPanel(pw)
|
||||
case cSignIn:
|
||||
return m.signInPanel(pw)
|
||||
case cKB:
|
||||
return m.kbPanel(pw)
|
||||
case cAccounts, cKeys, cOrgs:
|
||||
@@ -1633,9 +1680,6 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
|
||||
}
|
||||
}
|
||||
menu := cOverview
|
||||
if strings.HasSuffix(file, "agent.yaml") {
|
||||
menu = cSettings
|
||||
}
|
||||
for {
|
||||
m := newConsoleModel(addr, file)
|
||||
m.menu = menu
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
"github.com/charmbracelet/x/ansi"
|
||||
)
|
||||
|
||||
// A form in a dialog: text to type, one of a few choices, a thing on or off,
|
||||
// and headings between them. The setup's settings for each part are one of
|
||||
// these, opened over the list of parts.
|
||||
|
||||
type fieldKind int
|
||||
|
||||
const (
|
||||
fkText fieldKind = iota
|
||||
// fkChoice is one of Options, changed with ← →.
|
||||
fkChoice
|
||||
// fkToggle is on or off, changed with Space.
|
||||
fkToggle
|
||||
// fkHeading is a title between fields; it takes no focus.
|
||||
fkHeading
|
||||
)
|
||||
|
||||
type setupField struct {
|
||||
Label string
|
||||
// Hint is said under the field; Notes, for a choice, under each option
|
||||
// as it is chosen.
|
||||
Hint string
|
||||
Notes []string
|
||||
Kind fieldKind
|
||||
|
||||
Text textField
|
||||
Options []string
|
||||
Choice int
|
||||
On bool
|
||||
// Show, when set, says whether the field is shown, and so whether it
|
||||
// counts: a directory for a mirror kept elsewhere is neither.
|
||||
Show func() bool
|
||||
}
|
||||
|
||||
func textSetupField(label, value, placeholder, hint string) *setupField {
|
||||
t := newTextField()
|
||||
t.SetValue(value)
|
||||
t.Placeholder, t.CharLimit, t.Width = placeholder, 800, 40
|
||||
return &setupField{Label: label, Hint: hint, Kind: fkText, Text: t}
|
||||
}
|
||||
|
||||
func choiceField(label string, options []string, choice int, hint string) *setupField {
|
||||
return &setupField{Label: label, Hint: hint, Kind: fkChoice, Options: options, Choice: choice}
|
||||
}
|
||||
|
||||
func toggleField(label string, on bool, hint string) *setupField {
|
||||
return &setupField{Label: label, Hint: hint, Kind: fkToggle, On: on}
|
||||
}
|
||||
|
||||
func headingField(label string) *setupField {
|
||||
return &setupField{Label: label, Kind: fkHeading}
|
||||
}
|
||||
|
||||
func (f *setupField) shown() bool { return f.Show == nil || f.Show() }
|
||||
|
||||
func (f *setupField) Value() string { return strings.TrimSpace(f.Text.Value()) }
|
||||
|
||||
// setupDialog is a form over the screen, with what it is for above it.
|
||||
type setupDialog struct {
|
||||
Title string
|
||||
About string
|
||||
Fields []*setupField
|
||||
focus int
|
||||
}
|
||||
|
||||
// focusable is whether a field can have the focus now.
|
||||
func (d *setupDialog) focusable(i int) bool {
|
||||
f := d.Fields[i]
|
||||
return f.Kind != fkHeading && f.shown()
|
||||
}
|
||||
|
||||
// first puts the focus on the first field that takes it.
|
||||
func (d *setupDialog) first() {
|
||||
d.focus = -1
|
||||
d.move(1)
|
||||
}
|
||||
|
||||
// focusOn puts the focus on a field, as when it is what is wrong.
|
||||
func (d *setupDialog) focusOn(f *setupField) {
|
||||
for i, x := range d.Fields {
|
||||
if x == f {
|
||||
d.focus = i
|
||||
}
|
||||
}
|
||||
d.blurAll()
|
||||
}
|
||||
|
||||
func (d *setupDialog) move(dir int) {
|
||||
n := len(d.Fields)
|
||||
for range n {
|
||||
d.focus = (d.focus + dir + n) % n
|
||||
if d.focusable(d.focus) {
|
||||
break
|
||||
}
|
||||
}
|
||||
d.blurAll()
|
||||
}
|
||||
|
||||
// blurAll leaves only the focused text field with a cursor.
|
||||
func (d *setupDialog) blurAll() {
|
||||
for i, f := range d.Fields {
|
||||
if f.Kind != fkText {
|
||||
continue
|
||||
}
|
||||
if i == d.focus {
|
||||
f.Text.Focus()
|
||||
} else {
|
||||
f.Text.Blur()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// hasFields is whether there is anything to set, rather than only to read.
|
||||
func (d *setupDialog) hasFields() bool {
|
||||
for i := range d.Fields {
|
||||
if d.focusable(i) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// lastFocusable is the last field that takes the focus now.
|
||||
func (d *setupDialog) lastFocusable() int {
|
||||
for i := len(d.Fields) - 1; i >= 0; i-- {
|
||||
if d.focusable(i) {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
// key takes a key and says whether the dialog is done: Esc, or Enter on
|
||||
// the last field.
|
||||
func (d *setupDialog) key(k tea.KeyMsg) (done bool) {
|
||||
switch k.Type {
|
||||
case tea.KeyEsc:
|
||||
return true
|
||||
case tea.KeyEnter:
|
||||
if !d.hasFields() || d.focus == d.lastFocusable() {
|
||||
return true
|
||||
}
|
||||
d.move(1)
|
||||
return false
|
||||
case tea.KeyTab, tea.KeyDown:
|
||||
if d.hasFields() {
|
||||
d.move(1)
|
||||
}
|
||||
return false
|
||||
case tea.KeyShiftTab, tea.KeyUp:
|
||||
if d.hasFields() {
|
||||
d.move(-1)
|
||||
}
|
||||
return false
|
||||
}
|
||||
if d.focus < 0 || d.focus >= len(d.Fields) || !d.focusable(d.focus) {
|
||||
return false
|
||||
}
|
||||
f := d.Fields[d.focus]
|
||||
switch f.Kind {
|
||||
case fkText:
|
||||
f.Text, _ = f.Text.Update(k)
|
||||
case fkChoice:
|
||||
switch k.String() {
|
||||
case "left", "h":
|
||||
f.Choice = (f.Choice + len(f.Options) - 1) % len(f.Options)
|
||||
case "right", "l", " ":
|
||||
f.Choice = (f.Choice + 1) % len(f.Options)
|
||||
}
|
||||
case fkToggle:
|
||||
if k.String() == " " || k.String() == "x" {
|
||||
f.On = !f.On
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// dialogWidth is how wide the text in a dialog is wrapped.
|
||||
const dialogWidth = 66
|
||||
|
||||
// view draws the form in at most maxLines lines, the part around the focus
|
||||
// when it is longer: a long form scrolls as the focus moves.
|
||||
func (d *setupDialog) view(maxLines int) string {
|
||||
var lines []string
|
||||
add := func(s string) { lines = append(lines, strings.Split(s, "\n")...) }
|
||||
if d.About != "" {
|
||||
add(paragraphs(d.About, dialogWidth, ""))
|
||||
}
|
||||
if !d.hasFields() {
|
||||
add("")
|
||||
add(sDim.Render("Nothing to set for this part. Esc or Enter to go back."))
|
||||
return clipLines(lines, 0, 0, maxLines)
|
||||
}
|
||||
add("")
|
||||
focusStart, focusEnd := 0, 0
|
||||
for i, f := range d.Fields {
|
||||
if !f.shown() {
|
||||
continue
|
||||
}
|
||||
focused := i == d.focus
|
||||
if focused {
|
||||
focusStart = len(lines)
|
||||
}
|
||||
label := func(s string) string {
|
||||
if focused {
|
||||
return fSelected.Render("› " + s)
|
||||
}
|
||||
return " " + s
|
||||
}
|
||||
switch f.Kind {
|
||||
case fkHeading:
|
||||
add("")
|
||||
add(" " + lipgloss.NewStyle().Bold(true).Render(f.Label))
|
||||
continue
|
||||
case fkText:
|
||||
add(label(f.Label))
|
||||
add(" " + f.Text.View())
|
||||
case fkChoice:
|
||||
var opts []string
|
||||
for j, o := range f.Options {
|
||||
if j == f.Choice {
|
||||
opts = append(opts, "("+sOK.Render("•")+") "+o)
|
||||
} else {
|
||||
opts = append(opts, "( ) "+sDim.Render(o))
|
||||
}
|
||||
}
|
||||
add(label(f.Label))
|
||||
// Side by side while they fit, else one a line.
|
||||
if row := strings.Join(opts, " "); ansi.StringWidth(row) <= dialogWidth-4 {
|
||||
add(" " + row)
|
||||
} else {
|
||||
for _, o := range opts {
|
||||
add(" " + o)
|
||||
}
|
||||
}
|
||||
if f.Choice < len(f.Notes) && f.Notes[f.Choice] != "" {
|
||||
add(" " + sDim.Render(wrap(f.Notes[f.Choice], dialogWidth-4, " ")))
|
||||
}
|
||||
case fkToggle:
|
||||
box := "[ ]"
|
||||
if f.On {
|
||||
box = "[" + sOK.Render("x") + "]"
|
||||
}
|
||||
add(label(box) + " " + f.Label)
|
||||
}
|
||||
if f.Hint != "" {
|
||||
add(" " + sDim.Render(wrap(f.Hint, dialogWidth-4, " ")))
|
||||
}
|
||||
if focused {
|
||||
focusEnd = len(lines) - 1
|
||||
}
|
||||
if f.Kind != fkToggle {
|
||||
add("")
|
||||
}
|
||||
}
|
||||
for len(lines) > 0 && lines[len(lines)-1] == "" {
|
||||
lines = lines[:len(lines)-1]
|
||||
}
|
||||
return clipLines(lines, focusStart, focusEnd, maxLines)
|
||||
}
|
||||
|
||||
// clipLines is at most n of lines, keeping lines from to through in sight
|
||||
// as far as they fit, with a mark at an edge that has more beyond it.
|
||||
func clipLines(lines []string, from, through, n int) string {
|
||||
if n <= 2 || len(lines) <= n {
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
top := 0
|
||||
// The last line shown is a mark when there is more below: the focus
|
||||
// ends a line above it.
|
||||
if through > n-2 {
|
||||
top = through - (n - 2)
|
||||
}
|
||||
// Its first line wins over its last, under the mark above.
|
||||
if from > 0 && top > from-1 {
|
||||
top = from - 1
|
||||
}
|
||||
top = max(0, min(top, len(lines)-n))
|
||||
out := append([]string{}, lines[top:top+n]...)
|
||||
if top > 0 {
|
||||
out[0] = sDim.Render(" ↑ more")
|
||||
}
|
||||
if top+n < len(lines) {
|
||||
out[n-1] = sDim.Render(" ↓ more")
|
||||
}
|
||||
return strings.Join(out, "\n")
|
||||
}
|
||||
|
||||
// paragraphs wraps text paragraph by paragraph, blank lines kept between.
|
||||
func paragraphs(s string, width int, indent string) string {
|
||||
var out []string
|
||||
for _, p := range strings.Split(s, "\n\n") {
|
||||
out = append(out, indent+wrap(p, width, indent))
|
||||
}
|
||||
return strings.Join(out, "\n\n")
|
||||
}
|
||||
|
||||
// setupDialogKeys is the key bar under a dialog.
|
||||
func setupDialogKeys(d *setupDialog) string {
|
||||
if !d.hasFields() {
|
||||
return "Esc or Enter back"
|
||||
}
|
||||
keys := "Tab/↑↓ move"
|
||||
if d.focus >= 0 && d.focus < len(d.Fields) {
|
||||
switch d.Fields[d.focus].Kind {
|
||||
case fkChoice:
|
||||
keys += " · ←→ choose"
|
||||
case fkToggle:
|
||||
keys += " · Space tick"
|
||||
}
|
||||
}
|
||||
return keys + " · Esc done"
|
||||
}
|
||||
+14
-7
@@ -107,11 +107,7 @@ func optionLabel(o modelplan.Option) string {
|
||||
if o.FreeNow {
|
||||
now = sOK.Render("free")
|
||||
}
|
||||
need := fmt.Sprintf("%.1f of %.0f GB a card", o.NeedGB, o.HaveGB)
|
||||
if o.CPU {
|
||||
need = fmt.Sprintf("%.1f GB of RAM", o.NeedGB)
|
||||
}
|
||||
return fmt.Sprintf("%s · %s · %s", o.Label(), need, now)
|
||||
return fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
|
||||
}
|
||||
|
||||
// repoMenu offers a repository's files, each with the best way it runs here;
|
||||
@@ -197,8 +193,15 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
if p.Note != "" {
|
||||
note += "\n" + sWarn.Render("! ") + p.Note
|
||||
}
|
||||
note += fmt.Sprintf("\nPlanned for a %d-token context. The mirror fetches it and checks it, it joins the\n"+
|
||||
"catalog, and a replica is started and asked to say hello.", p.Context)
|
||||
note += fmt.Sprintf("\nMemory for a %d-token context, each way it runs:", p.Context)
|
||||
for _, o := range opts {
|
||||
note += "\n " + o.Label() + ": " + o.MemoryText()
|
||||
}
|
||||
if p.KVBytesPerToken > 0 {
|
||||
note += fmt.Sprintf("\nThe context costs %.2f GB per 1000 tokens, the whole model; a longer one is checked\n"+
|
||||
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
|
||||
}
|
||||
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
|
||||
ctxDefault := "4096"
|
||||
if a.Format == "safetensors" {
|
||||
ctxDefault = strconv.Itoa(p.Context)
|
||||
@@ -234,6 +237,10 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
req.File = a.Name
|
||||
}
|
||||
o := opts[slices.Index(where, v[4])]
|
||||
// Asked for a context other than planned: does it still fit?
|
||||
if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
|
||||
return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
|
||||
}
|
||||
if o.CPU {
|
||||
req.CPU = true
|
||||
} else {
|
||||
|
||||
+60
-196
@@ -4,7 +4,6 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/mail"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -47,33 +46,16 @@ func (w *lineWriter) Write(p []byte) (int, error) {
|
||||
}
|
||||
|
||||
// Form fields, in order.
|
||||
const (
|
||||
fAdmin = iota
|
||||
fHosts
|
||||
fPostgres
|
||||
fMirror
|
||||
fNode
|
||||
fModel
|
||||
fGuard
|
||||
fBundle
|
||||
fBundleSHA
|
||||
fStart
|
||||
fCount
|
||||
)
|
||||
|
||||
type initModel struct {
|
||||
in *serverInit
|
||||
found []string // this machine's addresses
|
||||
|
||||
// The form, until it is submitted.
|
||||
form bool
|
||||
focus int
|
||||
admin textField
|
||||
hosts textField
|
||||
bundle textField
|
||||
bundleSHA textField
|
||||
toggles [fCount]bool
|
||||
formError string
|
||||
// The parts to install, until they are chosen and the plan agreed to.
|
||||
form bool
|
||||
plan bool
|
||||
choices *installForm
|
||||
// scroll is how far the panel is scrolled, back to the top on each page.
|
||||
scroll int
|
||||
|
||||
// Then the run.
|
||||
running bool
|
||||
@@ -93,38 +75,9 @@ type initModel struct {
|
||||
}
|
||||
|
||||
func newInitModel(in *serverInit, found []string, askForm bool) initModel {
|
||||
admin := newTextField()
|
||||
admin.Placeholder = "you@example.com"
|
||||
admin.SetValue(in.admin)
|
||||
admin.CharLimit = 200
|
||||
admin.Width = 40
|
||||
hosts := newTextField()
|
||||
hosts.Placeholder = "llm.example.com, 203.0.113.10"
|
||||
hosts.SetValue(strings.Join(in.hosts, ", "))
|
||||
hosts.CharLimit = 400
|
||||
hosts.Width = 40
|
||||
bundle := newTextField()
|
||||
bundle.Placeholder = "/media/usb/llmgw-bundle.tar"
|
||||
bundle.SetValue(in.bundle)
|
||||
bundle.CharLimit = 400
|
||||
bundle.Width = 40
|
||||
bundleSHA := newTextField()
|
||||
bundleSHA.Placeholder = "the sha256 'llmgw bundle create' printed"
|
||||
bundleSHA.SetValue(in.bundleSHA)
|
||||
bundleSHA.CharLimit = 64
|
||||
bundleSHA.Width = 64
|
||||
sp := newSpinner()
|
||||
m := initModel{in: in, found: found, form: askForm, admin: admin, hosts: hosts,
|
||||
bundle: bundle, bundleSHA: bundleSHA, spin: sp, width: 100}
|
||||
m.toggles[fPostgres] = in.postgres
|
||||
m.toggles[fMirror] = !in.noMirror
|
||||
m.toggles[fNode] = !in.noNode
|
||||
m.toggles[fModel] = !in.noModel
|
||||
m.toggles[fGuard] = !in.noGuardModel
|
||||
if askForm {
|
||||
m.admin.Focus()
|
||||
}
|
||||
return m
|
||||
host, _ := os.Hostname()
|
||||
return initModel{in: in, found: found, form: askForm, choices: newInstallForm(in, host, found, false),
|
||||
spin: newSpinner(), width: 100, height: 30}
|
||||
}
|
||||
|
||||
func (m initModel) Init() tea.Cmd {
|
||||
@@ -140,6 +93,7 @@ func (m initModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
switch msg := msg.(type) {
|
||||
case tea.WindowSizeMsg:
|
||||
m.width, m.height = msg.Width, msg.Height
|
||||
m.choices.height = msg.Height
|
||||
if m.framed {
|
||||
m.width = msg.Width - navWidth - 6
|
||||
}
|
||||
@@ -202,100 +156,32 @@ func (m *initModel) addLine(l string) {
|
||||
|
||||
func (m initModel) formKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
switch k.Type {
|
||||
case tea.KeyTab, tea.KeyDown:
|
||||
m.moveFocus(1)
|
||||
case tea.KeyPgDown:
|
||||
m.scroll += max(1, m.height-8)
|
||||
return m, nil
|
||||
case tea.KeyShiftTab, tea.KeyUp:
|
||||
m.moveFocus(-1)
|
||||
case tea.KeyPgUp:
|
||||
m.scroll = max(0, m.scroll-max(1, m.height-8))
|
||||
return m, nil
|
||||
case tea.KeyEnter:
|
||||
if m.focus != fStart {
|
||||
m.moveFocus(1)
|
||||
return m, nil
|
||||
}
|
||||
admin := strings.TrimSpace(m.admin.Value())
|
||||
if admin != "" {
|
||||
if _, err := mail.ParseAddress(admin); err != nil {
|
||||
m.formError = "the administrator's email is not an address"
|
||||
m.focus = fAdmin
|
||||
m.admin.Focus()
|
||||
return m, nil
|
||||
}
|
||||
}
|
||||
bundle, sum := strings.TrimSpace(m.bundle.Value()), strings.ToLower(strings.TrimSpace(m.bundleSHA.Value()))
|
||||
switch {
|
||||
case bundle != "" && sum == "":
|
||||
m.formError = "give the bundle's sha256, as 'llmgw bundle create' printed it"
|
||||
m.focus = fBundleSHA
|
||||
m.moveFocus(0)
|
||||
return m, nil
|
||||
case bundle != "" && !m.toggles[fMirror] && m.in.mirrorURL == "":
|
||||
m.formError = "an offline bundle goes into the model mirror: tick it"
|
||||
m.focus = fMirror
|
||||
m.moveFocus(0)
|
||||
return m, nil
|
||||
}
|
||||
m.in.bundle, m.in.bundleSHA = bundle, sum
|
||||
m.in.admin = admin
|
||||
m.in.hosts = nil
|
||||
for _, h := range strings.Split(m.hosts.Value(), ",") {
|
||||
if h = strings.TrimSpace(h); h != "" {
|
||||
m.in.hosts = append(m.in.hosts, h)
|
||||
}
|
||||
}
|
||||
m.in.postgres = m.toggles[fPostgres]
|
||||
m.in.noMirror = !m.toggles[fMirror]
|
||||
m.in.noNode = !m.toggles[fNode]
|
||||
m.in.noModel = !m.toggles[fModel]
|
||||
m.in.noGuardModel = !m.toggles[fGuard]
|
||||
m.form = false
|
||||
return m, tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
|
||||
}
|
||||
switch m.focus {
|
||||
case fAdmin:
|
||||
var cmd tea.Cmd
|
||||
m.admin, cmd = m.admin.Update(k)
|
||||
return m, cmd
|
||||
case fHosts:
|
||||
var cmd tea.Cmd
|
||||
m.hosts, cmd = m.hosts.Update(k)
|
||||
return m, cmd
|
||||
case fBundle:
|
||||
var cmd tea.Cmd
|
||||
m.bundle, cmd = m.bundle.Update(k)
|
||||
return m, cmd
|
||||
case fBundleSHA:
|
||||
var cmd tea.Cmd
|
||||
m.bundleSHA, cmd = m.bundleSHA.Update(k)
|
||||
return m, cmd
|
||||
case fPostgres, fMirror, fNode, fModel, fGuard:
|
||||
if k.String() == " " || k.String() == "x" {
|
||||
m.toggles[m.focus] = !m.toggles[m.focus]
|
||||
if m.plan {
|
||||
switch k.Type {
|
||||
case tea.KeyEsc:
|
||||
m.plan, m.scroll = false, 0
|
||||
case tea.KeyEnter:
|
||||
m.form, m.plan, m.scroll = false, false, 0
|
||||
return m, tea.Batch(m.spin.Tick, func() tea.Msg { return startInitMsg{} })
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
if m.choices.key(k, m.in) {
|
||||
m.plan, m.scroll = true, 0
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (m *initModel) moveFocus(d int) {
|
||||
m.focus = (m.focus + d + fCount) % fCount
|
||||
// With --mirror there is no mirror to run here, so nothing to tick.
|
||||
if m.focus == fMirror && m.in.mirrorURL != "" {
|
||||
m.focus = (m.focus + d + fCount) % fCount
|
||||
}
|
||||
m.admin.Blur()
|
||||
m.hosts.Blur()
|
||||
m.bundle.Blur()
|
||||
m.bundleSHA.Blur()
|
||||
switch m.focus {
|
||||
case fAdmin:
|
||||
m.admin.Focus()
|
||||
case fHosts:
|
||||
m.hosts.Focus()
|
||||
case fBundle:
|
||||
m.bundle.Focus()
|
||||
case fBundleSHA:
|
||||
m.bundleSHA.Focus()
|
||||
}
|
||||
// command is this install as a command line, for doing it again from a script.
|
||||
func (in *serverInit) command() string {
|
||||
return strings.Join(append([]string{"sudo llmgw server init"}, in.initFlags()...), " \\\n ")
|
||||
}
|
||||
|
||||
func (m initModel) View() string {
|
||||
@@ -306,27 +192,50 @@ func (m initModel) View() string {
|
||||
if i := strings.Index(body, "\n\n"); i >= 0 {
|
||||
body = body[i+1:]
|
||||
}
|
||||
steps := []navItem{{Label: "Start", State: "done"}, {Label: "Settings", State: "current"}, {Label: "Install", State: "todo"}}
|
||||
keys := "Tab/↑↓ move · Space tick · Enter on Set up · Ctrl+C leave"
|
||||
if !m.form {
|
||||
steps := []navItem{{Label: "Start", State: "done"}, {Label: "Choices", State: "current"},
|
||||
{Label: "Plan", State: "todo"}, {Label: "Install", State: "todo"}}
|
||||
keys := m.choices.keys()
|
||||
title := "What to install"
|
||||
switch {
|
||||
case m.form && m.plan:
|
||||
steps[1].State, steps[2].State = "done", "current"
|
||||
keys = "Ctrl+C stop"
|
||||
keys, title = "Enter install · Esc back to the choices · PgUp/PgDn scroll · Ctrl+C leave", "Plan"
|
||||
case !m.form:
|
||||
steps[1].State, steps[2].State, steps[3].State = "done", "done", "current"
|
||||
keys, title = "Ctrl+C stop", "Installing"
|
||||
if m.done {
|
||||
steps[2].State = "done"
|
||||
steps[3].State = "done"
|
||||
keys = "Enter close"
|
||||
}
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
return frame{Width: m.width + navWidth + 6, Height: m.height, Title: "llmgw setup", Where: host,
|
||||
NavTitle: "Steps", Nav: steps, PanelTitle: "This machine as a control plane", Body: body, Keys: keys}.render()
|
||||
f := frame{Width: m.width + navWidth + 6, Height: m.height, Title: "llmgw setup",
|
||||
NavTitle: "Steps", Nav: steps, PanelTitle: title, Body: body, Keys: keys, Scroll: m.scroll}
|
||||
if m.form && !m.plan {
|
||||
f.DialogTitle, f.Dialog = m.choices.overlay()
|
||||
}
|
||||
f.Where, _ = os.Hostname()
|
||||
return f.render()
|
||||
}
|
||||
|
||||
// plainView is the screen as text, for a panel or a terminal that is not one.
|
||||
func (m initModel) plainView() string {
|
||||
var b strings.Builder
|
||||
b.WriteString(sTitle.Render("llmgw · setting up this machine as a control plane") + "\n\n")
|
||||
if m.form && m.plan {
|
||||
b.WriteString(m.choices.planView(m.width))
|
||||
b.WriteString("\n " + sDim.Render("The same from a script:") + "\n")
|
||||
for _, l := range strings.Split(m.in.command(), "\n") {
|
||||
b.WriteString(" " + l + "\n")
|
||||
}
|
||||
b.WriteString("\n" + button(true, "Enter to install") + "\n")
|
||||
return b.String()
|
||||
}
|
||||
if m.form {
|
||||
return b.String() + m.formView()
|
||||
b.WriteString(m.choices.view(m.width))
|
||||
if title, d := m.choices.overlay(); d != "" && !m.framed {
|
||||
b.WriteString("\n" + sTitle.Render(title) + "\n" + d + "\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
for _, l := range m.visibleLines() {
|
||||
b.WriteString(m.lineView(l) + "\n")
|
||||
@@ -386,51 +295,6 @@ func (m initModel) lineView(l string) string {
|
||||
return " " + t
|
||||
}
|
||||
|
||||
func (m initModel) formView() string {
|
||||
var b strings.Builder
|
||||
label := func(i int, s string) string {
|
||||
if m.focus == i {
|
||||
return sTitle.Render("› " + s)
|
||||
}
|
||||
return " " + s
|
||||
}
|
||||
check := func(i int, s, hint string) string {
|
||||
box := "[ ]"
|
||||
if m.toggles[i] {
|
||||
box = "[" + sOK.Render("x") + "]"
|
||||
}
|
||||
return fmt.Sprintf("%s %s %s\n", label(i, box), s, sDim.Render(hint))
|
||||
}
|
||||
fmt.Fprintf(&b, "%s\n %s\n", label(fAdmin, "Administrator's email"), m.admin.View())
|
||||
b.WriteString(" " + sDim.Render("A password is generated and shown once. Leave empty to create accounts later.") + "\n\n")
|
||||
fmt.Fprintf(&b, "%s\n %s\n", label(fHosts, "Other names nodes will reach this machine by"), m.hosts.View())
|
||||
found := "none found"
|
||||
if len(m.found) > 0 {
|
||||
found = strings.Join(m.found, ", ")
|
||||
}
|
||||
b.WriteString(" " + sDim.Render("Already in the certificate: "+found+". Add a DNS name or NAT address here.") + "\n\n")
|
||||
b.WriteString(check(fPostgres, "PostgreSQL", "rather than SQLite; installed here, for several control planes or a larger store"))
|
||||
if m.in.mirrorURL != "" {
|
||||
b.WriteString(" Model mirror " + sDim.Render("at "+m.in.mirrorURL+", from --mirror") + "\n")
|
||||
} else {
|
||||
b.WriteString(check(fMirror, "Model mirror", "downloads model files once for every node"))
|
||||
}
|
||||
b.WriteString(check(fNode, "This machine as a node", "serves models here too, on its GPUs or CPU"))
|
||||
b.WriteString(check(fModel, "Starter model", "a small Qwen, so there is something to ask at once"))
|
||||
b.WriteString(check(fGuard, "Guardrails' judge", "Llama Guard 3 1B on a CPU, refusing violent, criminal and abusive requests"))
|
||||
fmt.Fprintf(&b, "\n%s\n %s\n", label(fBundle, "Offline bundle"), m.bundle.View())
|
||||
b.WriteString(" " + sDim.Render("For a site with no internet: engines, releases and models from 'llmgw bundle create'. Empty downloads them.") + "\n")
|
||||
fmt.Fprintf(&b, "%s\n %s\n", label(fBundleSHA, "Its sha256"), m.bundleSHA.View())
|
||||
b.WriteString("\n" + label(fStart, lipgloss.NewStyle().Bold(true).Render("Set up")) + "\n")
|
||||
if m.formError != "" {
|
||||
b.WriteString("\n" + sBad.Render(m.formError) + "\n")
|
||||
}
|
||||
if !m.framed { // the frame's key bar says it
|
||||
b.WriteString("\n" + sDim.Render("tab or ↑↓ to move · space to tick · enter on Set up to start · ctrl+c to leave") + "\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// runInitTUI runs init with the screen above, asking the form first when
|
||||
// the flags left the administrator unset.
|
||||
func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
|
||||
|
||||
+175
-30
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
)
|
||||
@@ -11,21 +12,54 @@ func typeIn(m tea.Model, s string) tea.Model {
|
||||
return send(m, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(s)})
|
||||
}
|
||||
|
||||
// toPart moves the focus in the list of parts to the one titled so.
|
||||
func toPart(t *testing.T, m tea.Model, title string) tea.Model {
|
||||
t.Helper()
|
||||
f := m.(initModel).choices
|
||||
for i := 0; i <= len(f.parts); i++ {
|
||||
if f.focus < len(f.parts) && f.parts[f.focus].title == title {
|
||||
return m
|
||||
}
|
||||
m = press(m, "down")
|
||||
}
|
||||
t.Fatalf("no part %q", title)
|
||||
return m
|
||||
}
|
||||
|
||||
// toNext moves the focus to the button that goes on, and presses it.
|
||||
func toNext(m tea.Model) tea.Model {
|
||||
f := m.(initModel).choices
|
||||
for f.focus != len(f.parts) {
|
||||
m = press(m, "down")
|
||||
}
|
||||
return press(m, "enter")
|
||||
}
|
||||
|
||||
func TestTheInitFormBecomesInitsSettings(t *testing.T) {
|
||||
in := &serverInit{pkiDir: "/etc/llmgw/pki"}
|
||||
var m tea.Model = newInitModel(in, []string{"10.0.0.140"}, true)
|
||||
if v := m.View(); !strings.Contains(v, "Already in the certificate: 10.0.0.140") {
|
||||
m = press(toPart(t, m, "Control plane"), "enter")
|
||||
m = typeIn(m, "ops@example.com")
|
||||
m = press(m, "tab")
|
||||
if v := m.View(); !strings.Contains(v, "Already in its certificate: 10.0.0.140") {
|
||||
t.Fatalf("the addresses found are not shown:\n%s", v)
|
||||
}
|
||||
m = typeIn(m, "ops@example.com")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab})
|
||||
m = typeIn(m, "llm.example.com, 203.0.113.10")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // PostgreSQL on
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab},
|
||||
tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // starter model off
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune(" ")}) // judge off
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}) // no bundle
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyEnter})
|
||||
m = press(m, "esc")
|
||||
m = press(toPart(t, m, "Store"), "enter", "right", "esc") // PostgreSQL
|
||||
// Turning notifications on asks for the webhook at once.
|
||||
m = press(toPart(t, m, "Notifications"), "space")
|
||||
if m.(initModel).choices.dialog == nil {
|
||||
t.Fatal("notifications turned on without asking where")
|
||||
}
|
||||
m = typeIn(m, "https://hooks.slack.com/services/T1/B2/x, json:https://alerts.example.com/in")
|
||||
m = press(m, "esc")
|
||||
m = press(toPart(t, m, "Starter model"), "space") // off
|
||||
m = press(toPart(t, m, "Guardrails' judge"), "space") // off
|
||||
m = toNext(m)
|
||||
if !m.(initModel).plan {
|
||||
t.Fatalf("Next did not lead to the plan: %s", m.(initModel).choices.err)
|
||||
}
|
||||
|
||||
if in.admin != "ops@example.com" || strings.Join(in.hosts, "|") != "llm.example.com|203.0.113.10" {
|
||||
t.Errorf("admin %q hosts %q", in.admin, in.hosts)
|
||||
@@ -33,8 +67,36 @@ func TestTheInitFormBecomesInitsSettings(t *testing.T) {
|
||||
if !in.postgres || in.noMirror || in.noNode || !in.noModel || !in.noGuardModel {
|
||||
t.Errorf("toggles: postgres %v mirror %v node %v model %v", in.postgres, !in.noMirror, !in.noNode, !in.noModel)
|
||||
}
|
||||
if hooks, err := parseWebhooks(in.webhooks); err != nil || len(hooks) != 2 || hooks[0].Format != "slack" || hooks[1].Format != "json" {
|
||||
t.Errorf("webhooks %q: %+v %v", in.webhooks, hooks, err)
|
||||
}
|
||||
v := m.View()
|
||||
for _, want := range []string{"✓ Store", "PostgreSQL on this machine", "Left out", "Starter model", "--postgres", "--no-model"} {
|
||||
if !strings.Contains(v, want) {
|
||||
t.Errorf("the plan lacks %q:\n%s", want, v)
|
||||
}
|
||||
}
|
||||
m = press(m, "enter")
|
||||
if m.(initModel).form {
|
||||
t.Error("the form stayed up after Set up")
|
||||
t.Error("the form stayed up after the plan was agreed to")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWebhookThatIsNoAddressKeepsTheFormUp(t *testing.T) {
|
||||
in := &serverInit{}
|
||||
var m tea.Model = newInitModel(in, nil, true)
|
||||
m = press(toPart(t, m, "Notifications"), "space")
|
||||
m = typeIn(m, "hooks.slack.com/services/x")
|
||||
m = toNext(press(m, "esc"))
|
||||
im := m.(initModel)
|
||||
if !im.form || im.plan || !strings.Contains(m.View(), "http(s) address") {
|
||||
t.Fatalf("an address without its scheme was taken:\n%s", m.View())
|
||||
}
|
||||
// Closing the message opens the field that is wrong.
|
||||
m = press(m, "q")
|
||||
d := m.(initModel).choices.dialog
|
||||
if d == nil || d.Fields[d.focus] != m.(initModel).choices.webhooks {
|
||||
t.Errorf("the webhook was not given the cursor")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -44,34 +106,119 @@ func TestAnOfflineBundleIsGivenInTheFormWithItsDigest(t *testing.T) {
|
||||
if v := m.View(); !strings.Contains(v, "Offline bundle") {
|
||||
t.Fatalf("the form offers no bundle:\n%s", v)
|
||||
}
|
||||
for range fBundle {
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab})
|
||||
}
|
||||
m = press(toPart(t, m, "Offline bundle"), "space")
|
||||
m = typeIn(m, "/media/usb/b.tar")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyEnter})
|
||||
if !m.(initModel).form || !strings.Contains(m.View(), "sha256") || m.(initModel).focus != fBundleSHA {
|
||||
m = toNext(press(m, "esc"))
|
||||
if m.(initModel).plan || !strings.Contains(m.View(), "sha256") {
|
||||
t.Fatalf("a bundle without its digest was accepted:\n%s", m.View())
|
||||
}
|
||||
m = press(m, "q") // the message closed, the digest has the cursor
|
||||
m = typeIn(m, strings.Repeat("AB", 32))
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyEnter})
|
||||
if m.(initModel).form || in.bundle != "/media/usb/b.tar" || in.bundleSHA != strings.Repeat("ab", 32) {
|
||||
t.Errorf("bundle %q sha %q, form up %v", in.bundle, in.bundleSHA, m.(initModel).form)
|
||||
m = toNext(press(m, "esc"))
|
||||
if !m.(initModel).plan || in.bundle != "/media/usb/b.tar" || in.bundleSHA != strings.Repeat("ab", 32) {
|
||||
t.Errorf("bundle %q sha %q, plan %v: %s", in.bundle, in.bundleSHA, m.(initModel).plan, m.(initModel).choices.err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEmailThatIsNotOneIsSaidSo(t *testing.T) {
|
||||
in := &serverInit{}
|
||||
var m tea.Model = newInitModel(in, nil, true)
|
||||
m = press(m, "enter") // the control plane, first
|
||||
m = typeIn(m, "not an address")
|
||||
for range fStart {
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab})
|
||||
}
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyEnter})
|
||||
if !m.(initModel).form || !strings.Contains(m.View(), "not an address") {
|
||||
m = toNext(press(m, "esc"))
|
||||
if m.(initModel).plan || !strings.Contains(m.View(), "not an address") {
|
||||
t.Errorf("a bad email was accepted:\n%s", m.View())
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachPartsSettingsReachInit(t *testing.T) {
|
||||
in := &serverInit{}
|
||||
var m tea.Model = newInitModel(in, nil, true)
|
||||
m = press(toPart(t, m, "Control plane"), "enter", "tab", "tab", "right", "tab", "right", "esc") // Türkçe, sign-up open
|
||||
// The starter: qwen3-8b, the fifth, with 8192 tokens.
|
||||
m = press(toPart(t, m, "Starter model"), "enter", "right", "right", "right", "right", "tab")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = press(typeIn(m, "8192"), "esc")
|
||||
// The judge: serving flagged, three copies, 3s, and PII too.
|
||||
m = press(toPart(t, m, "Guardrails' judge"), "enter", "right", "tab")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = press(typeIn(m, "3"), "tab")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = typeIn(m, "3s")
|
||||
for range 6 { // past the five ticked, to PII
|
||||
m = press(m, "tab")
|
||||
}
|
||||
m = press(m, "space", "esc")
|
||||
m = press(toPart(t, m, "Pictures and 3D"), "space")
|
||||
m = press(toPart(t, m, "Limits and keys"), "enter")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = press(typeIn(m, "120"), "tab", "tab")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = press(typeIn(m, "30"), "esc")
|
||||
m = press(toPart(t, m, "Taking turns on a card"), "enter")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = press(typeIn(m, "1m"), "tab")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = press(typeIn(m, "1m"), "esc")
|
||||
m = press(toPart(t, m, "Model mirror"), "enter", "tab")
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyCtrlU})
|
||||
m = press(typeIn(m, "/data/models"), "esc")
|
||||
m = toNext(m)
|
||||
if !m.(initModel).plan {
|
||||
t.Fatalf("not taken: %s", m.(initModel).choices.err)
|
||||
}
|
||||
switch {
|
||||
case in.language != "tr" || !in.signupOpen:
|
||||
t.Errorf("language %q signup %v", in.language, in.signupOpen)
|
||||
case in.starter.Name != "qwen3-8b" || in.starterCtx != 8192:
|
||||
t.Errorf("starter %q ctx %d", in.starter.Name, in.starterCtx)
|
||||
case in.judge.Fail != "open" || in.judge.Replicas != 3 || in.judge.Budget != 3*time.Second ||
|
||||
len(in.judge.Categories) != 6 || in.judge.Categories[5] != "PII":
|
||||
t.Errorf("judge %+v", in.judge)
|
||||
case !in.media:
|
||||
t.Error("pictures and 3D not taken")
|
||||
case in.defaultRPM != 120 || in.defaultTPM != defaultTPM || in.maxKeyTTL != 30*24*time.Hour:
|
||||
t.Errorf("limits %d %d %s", in.defaultRPM, in.defaultTPM, in.maxKeyTTL)
|
||||
case in.evictAfterIdle != time.Minute || in.minLifetime != time.Minute:
|
||||
t.Errorf("turns %s %s", in.evictAfterIdle, in.minLifetime)
|
||||
case in.mirrorPath != "/data/models":
|
||||
t.Errorf("mirror %q", in.mirrorPath)
|
||||
}
|
||||
v := m.View()
|
||||
for _, want := range []string{"--starter qwen3-8b", "--starter-context 8192", "--judge-fail open", "--judge-replicas 3",
|
||||
"--language tr", "--media", "--mirror-dir /data/models", "--evict-after-idle 1m", "--max-key-ttl 720h"} {
|
||||
if !strings.Contains(v, want) {
|
||||
t.Errorf("the command lacks %q:\n%s", want, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStarterModelWantsTheMirror(t *testing.T) {
|
||||
in := &serverInit{}
|
||||
var m tea.Model = newInitModel(in, nil, true)
|
||||
m = toNext(press(toPart(t, m, "Model mirror"), "space"))
|
||||
if m.(initModel).plan || !strings.Contains(m.View(), "turn the mirror on") {
|
||||
t.Errorf("a starter model with no mirror to fetch it into was taken:\n%s", m.View())
|
||||
}
|
||||
}
|
||||
|
||||
func TestALongDialogScrollsWithItsFocus(t *testing.T) {
|
||||
in := &serverInit{}
|
||||
m := newInitModel(in, nil, true)
|
||||
m.choices.height = 24
|
||||
var tm tea.Model = press(toPart(t, m, "Guardrails' judge"), "enter")
|
||||
_, top := tm.(initModel).choices.overlay()
|
||||
if !strings.Contains(top, "↓ more") || strings.Contains(top, "Copyright Violation") {
|
||||
t.Fatalf("a dialog taller than the screen is not cut:\n%s", top)
|
||||
}
|
||||
for range 11 {
|
||||
tm = press(tm, "tab")
|
||||
}
|
||||
if _, bottom := tm.(initModel).choices.overlay(); !strings.Contains(bottom, "Copyright Violation") {
|
||||
t.Errorf("the focus went where it cannot be seen:\n%s", bottom)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInitsLinesAreFiledAndThePasswordKeptInView(t *testing.T) {
|
||||
in := &serverInit{admin: "ops@example.com", pkiDir: "/etc/llmgw/pki"}
|
||||
var m tea.Model = newInitModel(in, nil, false)
|
||||
@@ -111,16 +258,14 @@ func TestInitsLinesAreFiledAndThePasswordKeptInView(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithAMirrorElsewhereTheFormDoesNotOfferOne(t *testing.T) {
|
||||
func TestWithAMirrorElsewhereTheFormSaysSo(t *testing.T) {
|
||||
in := &serverInit{mirrorURL: "http://10.0.0.150:8082"}
|
||||
var m tea.Model = newInitModel(in, nil, true)
|
||||
if v := m.View(); !strings.Contains(v, "at http://10.0.0.150:8082, from --mirror") {
|
||||
if v := m.View(); !strings.Contains(v, "at http://10.0.0.150:8082") {
|
||||
t.Errorf("the mirror in use is not shown:\n%s", v)
|
||||
}
|
||||
// Tab from the addresses goes past the mirror line to PostgreSQL, then
|
||||
// the node.
|
||||
m = send(m, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab}, tea.KeyMsg{Type: tea.KeyTab})
|
||||
if f := m.(initModel).focus; f != fNode {
|
||||
t.Errorf("focus %d, want the node toggle %d", f, fNode)
|
||||
m = toNext(m)
|
||||
if !m.(initModel).plan || in.mirrorURL != "http://10.0.0.150:8082" || !in.noMirror {
|
||||
t.Errorf("mirror %q noMirror %v: %s", in.mirrorURL, in.noMirror, m.(initModel).choices.err)
|
||||
}
|
||||
}
|
||||
@@ -269,15 +269,20 @@ func (m *consoleModel) addModelForm() {
|
||||
}
|
||||
note.WriteString("A ComfyUI one has the nodes install ComfyUI if the cluster has none.\n" +
|
||||
"Or a definition file: the YAML a model takes in server.yaml's models.")
|
||||
var choices []string
|
||||
for _, n := range catalog.PresetNames() {
|
||||
choices = append(choices, presetChoice(m.data.presets, n))
|
||||
}
|
||||
m.openForm(&formDialog{title: "Add a model",
|
||||
note: note.String(),
|
||||
fields: []formField{
|
||||
choiceF("model", append(catalog.PresetNames(), fromFile), catalog.PresetNames()[0]),
|
||||
choiceF("model", append(choices, fromFile), choices[0]),
|
||||
textF("file", "", "for "+fromFile+": like /root/qwen.yaml"),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
if v[0] != fromFile {
|
||||
return m.runCLI("Add "+v[0], "ctl", server, "model", "add", v[0]), nil
|
||||
// The name, without what the choice says beside it.
|
||||
return m.addPreset(strings.Fields(v[0])[0]), nil
|
||||
}
|
||||
if v[1] == "" {
|
||||
return nil, fmt.Errorf("a file, or a model that comes with llmgw")
|
||||
|
||||
@@ -0,0 +1,715 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
|
||||
// A node's overview: what a bare `sudo llmgw` opens on a machine that runs
|
||||
// only llmgw-agent. Everything on it is read here — the agent's own files,
|
||||
// systemd, the cards and the disks — because a node has no admin token, and
|
||||
// the control plane may be the very thing it cannot reach.
|
||||
|
||||
// nodeCmdTimeout bounds every command and check the overview runs: a screen
|
||||
// refreshed every three seconds cannot wait on a tool that hangs.
|
||||
const nodeCmdTimeout = 2 * time.Second
|
||||
|
||||
// nodeControlPlanesFile is where the agent keeps the control planes the
|
||||
// leader last named (controlPlanesFile in the agent package).
|
||||
var nodeControlPlanesFile = "/var/lib/llmgw-agent/control-planes.json"
|
||||
|
||||
// nodeProblemLines is how many of the agent's warnings the overview shows.
|
||||
const nodeProblemLines = 6
|
||||
|
||||
type nodeOverviewMsg nodeOverview
|
||||
|
||||
// nodeOverview is everything the overview shows, read in one go.
|
||||
type nodeOverview struct {
|
||||
at time.Time
|
||||
version string
|
||||
agent agentService
|
||||
planes []planeCheck
|
||||
gpus []nodeGPU
|
||||
// gpuNote says why there are no GPU rows: no card, or no answer.
|
||||
gpuNote string
|
||||
// names is the cards' names by index, which do not change and are asked
|
||||
// for once rather than every refresh.
|
||||
names map[int]string
|
||||
engines []engineRun
|
||||
enginesNote string
|
||||
disks []diskUse
|
||||
problems []string
|
||||
// problemsNote replaces the problems when the journal cannot be read.
|
||||
problemsNote string
|
||||
}
|
||||
|
||||
// agentService is what systemd says of llmgw-agent.
|
||||
type agentService struct {
|
||||
state, sub string
|
||||
since time.Time
|
||||
// note replaces the rest when systemd cannot say.
|
||||
note string
|
||||
}
|
||||
|
||||
// planeCheck is one control plane the agent dials, and whether it answers.
|
||||
type planeCheck struct {
|
||||
addr string
|
||||
// learned is an address the leader named rather than one in agent.yaml.
|
||||
learned bool
|
||||
err string
|
||||
}
|
||||
|
||||
type nodeGPU struct {
|
||||
index int
|
||||
name string
|
||||
// util is -1 where the driver does not report it.
|
||||
util float64
|
||||
memUsedMB, memTotalMB float64
|
||||
tempC float64
|
||||
models []string
|
||||
}
|
||||
|
||||
// engineRun is one engine the supervisor started here, from its state file.
|
||||
type engineRun struct {
|
||||
replica, model, engine string
|
||||
gpus []int
|
||||
port, pid int
|
||||
alive bool
|
||||
started time.Time
|
||||
}
|
||||
|
||||
type diskUse struct {
|
||||
what, path string
|
||||
free, total uint64
|
||||
err string
|
||||
}
|
||||
|
||||
// nodeProbe reads this machine. Every path and tool is a field, so a test
|
||||
// reads a directory of its own instead.
|
||||
type nodeProbe struct {
|
||||
configured []string
|
||||
planesFile string
|
||||
workDir string
|
||||
weightsDir string
|
||||
enginesDir string
|
||||
root bool
|
||||
names map[int]string
|
||||
|
||||
run func(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
dial func(ctx context.Context, addr string) error
|
||||
gpus func(ctx context.Context) []discovery.GPUSensor
|
||||
cards func(ctx context.Context) ([]discovery.GPU, error)
|
||||
alive func(pid int) bool
|
||||
statfs func(path string) (free, total uint64, err error)
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// newNodeProbe reads the machine agent.yaml describes, as the agent would.
|
||||
func newNodeProbe(cfg any, names map[int]string) nodeProbe {
|
||||
c, ok := cfg.(agent.Config)
|
||||
if !ok {
|
||||
c = agent.DefaultConfig()
|
||||
}
|
||||
var configured []string
|
||||
if c.Server != "" {
|
||||
configured = append(configured, c.Server)
|
||||
}
|
||||
configured = append(configured, c.Servers...)
|
||||
// The same defaults the agent applies as it starts.
|
||||
weights := c.WeightsCache
|
||||
if weights == "" && len(c.ModelDirs) > 0 {
|
||||
weights = c.ModelDirs[0]
|
||||
}
|
||||
if weights == "" {
|
||||
weights = agent.DefaultWeightsCache
|
||||
}
|
||||
enginesDir := c.EnginesDir
|
||||
if enginesDir == "" {
|
||||
enginesDir = provision.DefaultRoot
|
||||
}
|
||||
work := c.Superv.WorkDir
|
||||
if work == "" {
|
||||
work = "/var/lib/llmgw-agent"
|
||||
}
|
||||
return nodeProbe{
|
||||
configured: configured, planesFile: nodeControlPlanesFile, workDir: work,
|
||||
weightsDir: weights, enginesDir: enginesDir, root: os.Geteuid() == 0, names: names,
|
||||
run: runLocal, dial: dialTCP, gpus: discovery.SampleGPUs,
|
||||
cards: func(ctx context.Context) ([]discovery.GPU, error) { return discovery.DiscoverGPUsSMI(ctx, "") },
|
||||
alive: pidAlive, statfs: statfs, now: time.Now,
|
||||
}
|
||||
}
|
||||
|
||||
// fetchNodeOverview reads the machine in the background.
|
||||
func (m consoleModel) fetchNodeOverview() tea.Cmd {
|
||||
p := newNodeProbe(m.cfg, m.nodeOv.names)
|
||||
return func() tea.Msg { return nodeOverviewMsg(p.gather(context.Background())) }
|
||||
}
|
||||
|
||||
// nodeTick refreshes the overview while it is on screen, one reading at a
|
||||
// time; the other screens of a node watch nothing.
|
||||
func (m consoleModel) nodeTick() (consoleModel, tea.Cmd) {
|
||||
tick := tea.Tick(3*time.Second, func(time.Time) tea.Msg { return consoleTickMsg{} })
|
||||
if m.menu != cOverview || m.nodeBusy {
|
||||
return m, tick
|
||||
}
|
||||
m.nodeBusy = true
|
||||
return m, tea.Batch(m.fetchNodeOverview(), tick)
|
||||
}
|
||||
|
||||
// gather reads everything at once, each part in its own goroutine, so the
|
||||
// slowest part bounds the whole.
|
||||
func (p nodeProbe) gather(ctx context.Context) nodeOverview {
|
||||
ov := nodeOverview{version: version.Version}
|
||||
var wg sync.WaitGroup
|
||||
do := func(f func()) {
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); f() }()
|
||||
}
|
||||
var engines []engineRun
|
||||
do(func() { ov.agent = p.agentState(ctx) })
|
||||
do(func() { ov.planes = p.checkPlanes(ctx) })
|
||||
do(func() { ov.gpus, ov.names, ov.gpuNote = p.readGPUs(ctx) })
|
||||
do(func() { engines, ov.enginesNote = readShardStates(p.workDir, p.alive) })
|
||||
do(func() { ov.disks = p.readDisks() })
|
||||
do(func() { ov.problems, ov.problemsNote = p.readProblems(ctx) })
|
||||
wg.Wait()
|
||||
ov.engines = engines
|
||||
ov.gpus = modelsOnGPUs(ov.gpus, engines)
|
||||
ov.at = p.now()
|
||||
return ov
|
||||
}
|
||||
|
||||
// --- the agent ---
|
||||
|
||||
func (p nodeProbe) agentState(ctx context.Context) agentService {
|
||||
out, err := p.run(ctx, "systemctl", "show", "llmgw-agent",
|
||||
"-p", "LoadState", "-p", "ActiveState", "-p", "SubState", "-p", "ActiveEnterTimestamp")
|
||||
if err != nil && len(out) == 0 {
|
||||
return agentService{note: "systemctl is not available: " + err.Error()}
|
||||
}
|
||||
return parseSystemctlShow(string(out))
|
||||
}
|
||||
|
||||
// parseSystemctlShow reads `systemctl show -p …`: one key=value a line.
|
||||
func parseSystemctlShow(out string) agentService {
|
||||
kv := map[string]string{}
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if k, v, ok := strings.Cut(strings.TrimSpace(line), "="); ok {
|
||||
kv[k] = v
|
||||
}
|
||||
}
|
||||
if kv["LoadState"] == "not-found" {
|
||||
return agentService{note: "llmgw-agent is not installed as a service here"}
|
||||
}
|
||||
s := agentService{state: kv["ActiveState"], sub: kv["SubState"]}
|
||||
if s.state == "" {
|
||||
s.note = "systemd does not say"
|
||||
}
|
||||
s.since = parseSystemdTime(kv["ActiveEnterTimestamp"])
|
||||
return s
|
||||
}
|
||||
|
||||
// parseSystemdTime reads systemd's "Mon 2026-10-19 08:12:03 CEST", in this
|
||||
// machine's zone; zero when it is empty or n/a.
|
||||
func parseSystemdTime(v string) time.Time {
|
||||
v = strings.TrimSpace(v)
|
||||
if v == "" || v == "n/a" {
|
||||
return time.Time{}
|
||||
}
|
||||
for _, layout := range []string{"Mon 2006-01-02 15:04:05 MST", "Mon 2006-01-02 15:04:05 -0700", "Mon 2006-01-02 15:04:05"} {
|
||||
if t, err := time.ParseInLocation(layout, v, time.Local); err == nil {
|
||||
return t
|
||||
}
|
||||
}
|
||||
return time.Time{}
|
||||
}
|
||||
|
||||
// --- the control planes ---
|
||||
|
||||
// readControlPlanes is every address the agent dials, in its order: the
|
||||
// configured ones, then those the leader named that agent.yaml lacks.
|
||||
func readControlPlanes(file string, configured []string) []planeCheck {
|
||||
var out []planeCheck
|
||||
seen := map[string]bool{}
|
||||
add := func(s string, learned bool) {
|
||||
hp := hostPort(s)
|
||||
if hp == "" || seen[hp] {
|
||||
return
|
||||
}
|
||||
seen[hp] = true
|
||||
out = append(out, planeCheck{addr: hp, learned: learned})
|
||||
}
|
||||
for _, s := range configured {
|
||||
add(s, false)
|
||||
}
|
||||
if b, err := os.ReadFile(file); err == nil {
|
||||
var told []string
|
||||
if json.Unmarshal(b, &told) == nil {
|
||||
for _, s := range told {
|
||||
add(s, true)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// hostPort is the host:port a control plane's address dials, whether it was
|
||||
// written as a URL or not.
|
||||
func hostPort(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
port := ""
|
||||
switch {
|
||||
case strings.HasPrefix(s, "https://"):
|
||||
s, port = strings.TrimPrefix(s, "https://"), "443"
|
||||
case strings.HasPrefix(s, "http://"):
|
||||
s, port = strings.TrimPrefix(s, "http://"), "80"
|
||||
}
|
||||
s, _, _ = strings.Cut(s, "/")
|
||||
if _, _, err := net.SplitHostPort(s); err != nil && port != "" && s != "" {
|
||||
s = net.JoinHostPort(strings.Trim(s, "[]"), port)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func (p nodeProbe) checkPlanes(ctx context.Context) []planeCheck {
|
||||
planes := readControlPlanes(p.planesFile, p.configured)
|
||||
var wg sync.WaitGroup
|
||||
for i := range planes {
|
||||
wg.Add(1)
|
||||
go func(c *planeCheck) {
|
||||
defer wg.Done()
|
||||
if err := p.dial(ctx, c.addr); err != nil {
|
||||
c.err = err.Error()
|
||||
}
|
||||
}(&planes[i])
|
||||
}
|
||||
wg.Wait()
|
||||
return planes
|
||||
}
|
||||
|
||||
func dialTCP(ctx context.Context, addr string) error {
|
||||
d := net.Dialer{Timeout: nodeCmdTimeout}
|
||||
c, err := d.DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return c.Close()
|
||||
}
|
||||
|
||||
// --- the GPUs ---
|
||||
|
||||
// readGPUs samples the cards, naming them from discovery the first time.
|
||||
func (p nodeProbe) readGPUs(ctx context.Context) ([]nodeGPU, map[int]string, string) {
|
||||
ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
|
||||
defer cancel()
|
||||
sensors := p.gpus(ctx)
|
||||
if len(sensors) == 0 {
|
||||
// Without nvidia-smi discovery says nothing and no error: a machine
|
||||
// without a card. With it and no answer, a driver in trouble.
|
||||
if _, err := p.cards(ctx); err != nil {
|
||||
return nil, p.names, "The GPUs do not answer: " + err.Error()
|
||||
}
|
||||
return nil, p.names, "No GPU on this machine."
|
||||
}
|
||||
names := p.names
|
||||
for _, s := range sensors {
|
||||
if names[s.Index] == "" {
|
||||
names = map[int]string{}
|
||||
if cards, err := p.cards(ctx); err == nil {
|
||||
for _, c := range cards {
|
||||
if c.MIG == nil {
|
||||
names[c.Index] = c.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
var out []nodeGPU
|
||||
for _, s := range sensors {
|
||||
g := nodeGPU{index: s.Index, name: names[s.Index], util: s.UtilPercent, tempC: s.TempC,
|
||||
memUsedMB: s.MemoryUsedMB, memTotalMB: s.MemoryTotalMB}
|
||||
if s.MemoryTotalMB == 0 {
|
||||
g.util = -1
|
||||
}
|
||||
out = append(out, g)
|
||||
}
|
||||
return out, names, ""
|
||||
}
|
||||
|
||||
// cardOf is the card a device number is on: itself, or a MIG instance's card.
|
||||
func cardOf(i int) int {
|
||||
if discovery.IsMIGIndex(i) {
|
||||
return (i - discovery.MIGIndex(0, 0)) / 100
|
||||
}
|
||||
return i
|
||||
}
|
||||
|
||||
// modelsOnGPUs names, on each card, the models whose engines hold it.
|
||||
func modelsOnGPUs(gpus []nodeGPU, runs []engineRun) []nodeGPU {
|
||||
for i := range gpus {
|
||||
gpus[i].models = nil
|
||||
for _, r := range runs {
|
||||
for _, d := range r.gpus {
|
||||
if cardOf(d) == gpus[i].index && !slices.Contains(gpus[i].models, r.model) {
|
||||
gpus[i].models = append(gpus[i].models, r.model)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return gpus
|
||||
}
|
||||
|
||||
// --- the engines ---
|
||||
|
||||
// shardStateFile is the part of the supervisor's shard-*.json the overview
|
||||
// reads (shardState in the supervisor package).
|
||||
type shardStateFile struct {
|
||||
ShardID string `json:"shard_id"`
|
||||
ReplicaID string `json:"replica_id"`
|
||||
PID int `json:"pid"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
Spec engines.ShardSpec `json:"spec"`
|
||||
}
|
||||
|
||||
// readShardStates is the engines the supervisor keeps state files for in
|
||||
// its work directory, with whether each process is still there.
|
||||
func readShardStates(dir string, alive func(int) bool) ([]engineRun, string) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
switch {
|
||||
case errors.Is(err, os.ErrPermission):
|
||||
return nil, "The engines' state is root's: run with sudo."
|
||||
case errors.Is(err, os.ErrNotExist):
|
||||
return nil, ""
|
||||
case err != nil:
|
||||
return nil, err.Error()
|
||||
}
|
||||
var runs []engineRun
|
||||
note := ""
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if !strings.HasPrefix(name, "shard-") || !strings.HasSuffix(name, ".json") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, name))
|
||||
if errors.Is(err, os.ErrPermission) {
|
||||
note = "The engines' state is root's: run with sudo."
|
||||
continue
|
||||
}
|
||||
var st shardStateFile
|
||||
if err != nil || json.Unmarshal(b, &st) != nil {
|
||||
continue
|
||||
}
|
||||
replica := st.ReplicaID
|
||||
if replica == "" {
|
||||
replica = st.Spec.ReplicaID
|
||||
}
|
||||
model := st.Spec.Profile.ServedModelName
|
||||
if model == "" {
|
||||
model = modelOfReplica(replica)
|
||||
}
|
||||
runs = append(runs, engineRun{replica: replica, model: model, engine: st.Spec.Profile.Engine,
|
||||
gpus: st.Spec.GPUIndices, port: st.Spec.HTTPPort, pid: st.PID, alive: st.PID > 0 && alive(st.PID),
|
||||
started: st.StartedAt})
|
||||
}
|
||||
sort.Slice(runs, func(i, j int) bool { return runs[i].replica < runs[j].replica })
|
||||
return runs, note
|
||||
}
|
||||
|
||||
// modelOfReplica is the model a replica id names: the control plane calls a
|
||||
// replica the model's name and a suffix.
|
||||
func modelOfReplica(id string) string {
|
||||
if i := strings.LastIndexByte(id, '-'); i > 0 {
|
||||
return id[:i]
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func pidAlive(pid int) bool {
|
||||
err := syscall.Kill(pid, 0)
|
||||
return err == nil || errors.Is(err, syscall.EPERM)
|
||||
}
|
||||
|
||||
// --- the disks ---
|
||||
|
||||
func (p nodeProbe) readDisks() []diskUse {
|
||||
disks := []diskUse{{what: "weights", path: p.weightsDir}, {what: "engines", path: p.enginesDir}}
|
||||
for i := range disks {
|
||||
// In its own goroutine: statfs on a network mount that has gone away
|
||||
// does not return.
|
||||
type res struct {
|
||||
free, total uint64
|
||||
err error
|
||||
}
|
||||
ch := make(chan res, 1)
|
||||
go func(path string) {
|
||||
f, t, err := p.statfs(path)
|
||||
ch <- res{f, t, err}
|
||||
}(disks[i].path)
|
||||
select {
|
||||
case r := <-ch:
|
||||
disks[i].free, disks[i].total = r.free, r.total
|
||||
if r.err != nil {
|
||||
disks[i].err = r.err.Error()
|
||||
if errors.Is(r.err, os.ErrNotExist) {
|
||||
disks[i].err = "not there yet"
|
||||
}
|
||||
}
|
||||
case <-time.After(nodeCmdTimeout):
|
||||
disks[i].err = "does not answer"
|
||||
}
|
||||
}
|
||||
return disks
|
||||
}
|
||||
|
||||
func statfs(path string) (free, total uint64, err error) {
|
||||
var st syscall.Statfs_t
|
||||
if err := syscall.Statfs(path, &st); err != nil {
|
||||
return 0, 0, &os.PathError{Op: "statfs", Path: path, Err: err}
|
||||
}
|
||||
//nolint:unconvert // the field types differ between systems
|
||||
return uint64(st.Bavail) * uint64(st.Bsize), uint64(st.Blocks) * uint64(st.Bsize), nil
|
||||
}
|
||||
|
||||
// --- the problems ---
|
||||
|
||||
func (p nodeProbe) readProblems(ctx context.Context) ([]string, string) {
|
||||
out, err := p.run(ctx, "journalctl", "-u", "llmgw-agent", "-p", "warning", "--since", "-1h",
|
||||
"--no-pager", "-o", "cat", "-n", strconv.Itoa(nodeProblemLines))
|
||||
if err != nil && len(out) == 0 {
|
||||
return nil, "The journal cannot be read: " + err.Error()
|
||||
}
|
||||
lines := parseJournal(string(out), nodeProblemLines)
|
||||
if len(lines) == 0 && !p.root {
|
||||
return nil, "None that this user can see: run with sudo to read the agent's journal."
|
||||
}
|
||||
return lines, ""
|
||||
}
|
||||
|
||||
// parseJournal keeps the last n lines journalctl printed, without its own
|
||||
// remarks.
|
||||
func parseJournal(out string, n int) []string {
|
||||
var lines []string
|
||||
sc := bufio.NewScanner(strings.NewReader(out))
|
||||
sc.Buffer(make([]byte, 64*1024), 1024*1024)
|
||||
for sc.Scan() {
|
||||
l := strings.TrimSpace(sc.Text())
|
||||
if l == "" || strings.HasPrefix(l, "-- ") || strings.HasPrefix(l, "Hint: ") {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, l)
|
||||
}
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return lines
|
||||
}
|
||||
|
||||
// runLocal runs a tool on this machine, bounded by nodeCmdTimeout. A tool
|
||||
// that is not installed is said so, not waited for.
|
||||
func runLocal(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
if _, err := exec.LookPath(name); err != nil {
|
||||
return nil, fmt.Errorf("%s is not installed", name)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, nodeCmdTimeout)
|
||||
defer cancel()
|
||||
//nolint:gosec // fixed tools and arguments, from this file
|
||||
return exec.CommandContext(ctx, name, args...).Output()
|
||||
}
|
||||
|
||||
// --- the panel ---
|
||||
|
||||
func (m consoleModel) nodeOverviewPanel(pw int) (string, string, string) {
|
||||
keys := "↑↓ choose · PgUp/PgDn scroll · Q leave · refreshed every three seconds"
|
||||
if m.nodeOv.at.IsZero() {
|
||||
return "Overview", "\n " + m.spin.View() + " reading this machine…", keys
|
||||
}
|
||||
return "Overview", renderNodeOverview(m.nodeOv, pw, time.Now()), keys
|
||||
}
|
||||
|
||||
// renderNodeOverview draws the overview from what was read.
|
||||
func renderNodeOverview(ov nodeOverview, pw int, now time.Time) string {
|
||||
var b strings.Builder
|
||||
head := func(s string) { b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render(s) + "\n") }
|
||||
kv := func(k, v string) { fmt.Fprintf(&b, " %-14s %s\n", sDim.Render(k), v) }
|
||||
line := func(s string) { b.WriteString(" " + s + "\n") }
|
||||
|
||||
head("Agent")
|
||||
a := ov.agent
|
||||
switch {
|
||||
case a.note != "":
|
||||
kv("llmgw-agent", sWarn.Render(a.note))
|
||||
default:
|
||||
state := a.state
|
||||
if a.sub != "" {
|
||||
state += " (" + a.sub + ")"
|
||||
}
|
||||
if a.state == "active" {
|
||||
state = sOK.Render(state)
|
||||
} else {
|
||||
state = sBad.Render(state)
|
||||
}
|
||||
if !a.since.IsZero() {
|
||||
state += sDim.Render(" since " + a.since.Format("2006-01-02 15:04") + ", " + uptime(now.Sub(a.since)) + " ago")
|
||||
}
|
||||
kv("llmgw-agent", state)
|
||||
}
|
||||
kv("llmgw", ov.version)
|
||||
|
||||
head("Control planes")
|
||||
if len(ov.planes) == 0 {
|
||||
line(sWarn.Render("None configured: agent.yaml names no server."))
|
||||
}
|
||||
for _, c := range ov.planes {
|
||||
state := sOK.Render("reachable")
|
||||
if c.err != "" {
|
||||
state = sBad.Render("✗ ") + truncate(c.err, max(20, pw-30))
|
||||
}
|
||||
if c.learned {
|
||||
state += sDim.Render(" (named by the leader)")
|
||||
}
|
||||
fmt.Fprintf(&b, " %-22s %s\n", c.addr, state)
|
||||
}
|
||||
|
||||
head("GPUs")
|
||||
if len(ov.gpus) == 0 {
|
||||
line(sDim.Render(ov.gpuNote))
|
||||
} else {
|
||||
var rows [][]string
|
||||
for _, g := range ov.gpus {
|
||||
busy, mem, temp := "—", "—", "—"
|
||||
if g.util >= 0 {
|
||||
busy = fmt.Sprintf("%.0f%%", g.util)
|
||||
}
|
||||
if g.memTotalMB > 0 {
|
||||
mem = fmt.Sprintf("%.1f/%.0f GB", g.memUsedMB/1024, g.memTotalMB/1024)
|
||||
}
|
||||
if g.tempC > 0 {
|
||||
temp = fmt.Sprintf("%.0f°C", g.tempC)
|
||||
}
|
||||
models := strings.Join(g.models, ", ")
|
||||
if models == "" {
|
||||
models = sDim.Render("free")
|
||||
}
|
||||
rows = append(rows, []string{strconv.Itoa(g.index), g.name, busy, mem, temp, models})
|
||||
}
|
||||
b.WriteString(indent(table([]string{"GPU", "CARD", "BUSY", "MEMORY", "TEMP", "MODELS"}, rows, -1, pw-2, false), " ") + "\n")
|
||||
}
|
||||
|
||||
head("Engines running here")
|
||||
switch {
|
||||
case len(ov.engines) > 0:
|
||||
var rows [][]string
|
||||
for _, r := range ov.engines {
|
||||
// The process by its pid, marked when it is no longer there.
|
||||
proc := sOK.Render(strconv.Itoa(r.pid))
|
||||
if !r.alive {
|
||||
proc = sBad.Render(strconv.Itoa(r.pid) + " gone")
|
||||
}
|
||||
gpus := strings.Join(discovery.DeviceLabels(r.gpus), ",")
|
||||
if gpus == "" {
|
||||
gpus = sDim.Render("CPU")
|
||||
}
|
||||
up := "—"
|
||||
if !r.started.IsZero() {
|
||||
up = uptime(now.Sub(r.started))
|
||||
}
|
||||
port := "—"
|
||||
if r.port > 0 {
|
||||
port = strconv.Itoa(r.port)
|
||||
}
|
||||
rows = append(rows, []string{r.replica, r.model, r.engine, gpus, port, proc, up})
|
||||
}
|
||||
b.WriteString(indent(table([]string{"REPLICA", "MODEL", "ENGINE", "GPUS", "PORT", "PID", "UP"}, rows, -1, pw-2, false), " ") + "\n")
|
||||
if ov.enginesNote != "" {
|
||||
line(sWarn.Render(ov.enginesNote))
|
||||
}
|
||||
case ov.enginesNote != "":
|
||||
line(sWarn.Render(ov.enginesNote))
|
||||
default:
|
||||
line(sDim.Render("None: the control plane has placed nothing here."))
|
||||
}
|
||||
|
||||
head("Disk")
|
||||
for _, d := range ov.disks {
|
||||
use := sWarn.Render(d.err)
|
||||
if d.err == "" {
|
||||
use = fmt.Sprintf("%s free of %s", diskSize(d.free), diskSize(d.total))
|
||||
if d.total > 0 && d.free*10 < d.total {
|
||||
use = sWarn.Render(use)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, " %-14s %-28s %s\n", sDim.Render(d.what), d.path, use)
|
||||
}
|
||||
|
||||
head("Recent problems")
|
||||
switch {
|
||||
case ov.problemsNote != "":
|
||||
line(sDim.Render(ov.problemsNote))
|
||||
case len(ov.problems) == 0:
|
||||
line(sOK.Render("None in the last hour."))
|
||||
default:
|
||||
for _, l := range ov.problems {
|
||||
l = truncateWidth(l, max(20, pw-4))
|
||||
if strings.Contains(l, "level=ERROR") {
|
||||
l = sBad.Render(l)
|
||||
} else {
|
||||
l = sWarn.Render(l)
|
||||
}
|
||||
line(l)
|
||||
}
|
||||
}
|
||||
b.WriteString("\n " + sDim.Render("read at "+ov.at.Format("15:04:05")) + "\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// uptime is a duration as people say it: 3d4h, 2h15m, 40s.
|
||||
func uptime(d time.Duration) string {
|
||||
d = d.Round(time.Second)
|
||||
switch {
|
||||
case d < 0:
|
||||
return "0s"
|
||||
case d >= 24*time.Hour:
|
||||
return fmt.Sprintf("%dd%dh", d/(24*time.Hour), (d%(24*time.Hour))/time.Hour)
|
||||
case d >= time.Hour:
|
||||
return fmt.Sprintf("%dh%dm", d/time.Hour, (d%time.Hour)/time.Minute)
|
||||
case d >= time.Minute:
|
||||
return fmt.Sprintf("%dm%ds", d/time.Minute, (d%time.Minute)/time.Second)
|
||||
}
|
||||
return d.String()
|
||||
}
|
||||
|
||||
// diskSize is a byte count in the unit disks are sold in.
|
||||
func diskSize(n uint64) string {
|
||||
switch {
|
||||
case n >= 1<<40:
|
||||
return fmt.Sprintf("%.1f TB", float64(n)/(1<<40))
|
||||
case n >= 1<<30:
|
||||
return fmt.Sprintf("%.0f GB", float64(n)/(1<<30))
|
||||
}
|
||||
return fmt.Sprintf("%.0f MB", float64(n)/(1<<20))
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/charmbracelet/x/ansi"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
func writeNodeFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewDialsTheConfiguredControlPlanesThenTheLeadersOwn(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, "control-planes.json")
|
||||
writeNodeFile(t, file, `["https://10.0.0.150:7443","10.0.0.140:7443","https://cp3.example/agent"]`)
|
||||
got := readControlPlanes(file, []string{"10.0.0.140:7443", "http://10.0.0.141"})
|
||||
want := []planeCheck{{addr: "10.0.0.140:7443"}, {addr: "10.0.0.141:80"},
|
||||
{addr: "10.0.0.150:7443", learned: true}, {addr: "cp3.example:443", learned: true}}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %+v, want %+v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("%d: got %+v, want %+v", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
// Without the file, what agent.yaml says.
|
||||
if got := readControlPlanes(filepath.Join(dir, "none.json"), []string{"cp:7443"}); len(got) != 1 || got[0].addr != "cp:7443" {
|
||||
t.Errorf("without the file: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewReadsTheEnginesFromTheSupervisorsStateFiles(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"shard_id":"qwen3-8b-ab12-0","replica_id":"qwen3-8b-ab12",
|
||||
"pid":4242,"started_at":"2026-10-19T08:00:00Z","spec":{"ReplicaID":"qwen3-8b-ab12","GPUIndices":[0,1],"HTTPPort":8101,
|
||||
"Profile":{"Name":"vllm-tp2","Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-bge-m3-7-0.json"), `{"replica_id":"bge-m3-7","pid":99,
|
||||
"spec":{"GPUIndices":[1002],"HTTPPort":8102,"Profile":{"Engine":"llamacpp"}}}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-x.json.tmp"), `{"replica_id":"half-written"}`)
|
||||
writeNodeFile(t, filepath.Join(dir, "control-planes.json"), `[]`)
|
||||
runs, note := readShardStates(dir, func(pid int) bool { return pid == 4242 })
|
||||
if note != "" {
|
||||
t.Fatalf("note %q", note)
|
||||
}
|
||||
if len(runs) != 2 {
|
||||
t.Fatalf("got %+v", runs)
|
||||
}
|
||||
b, q := runs[0], runs[1]
|
||||
if q.replica != "qwen3-8b-ab12" || q.model != "qwen3-8b" || q.engine != "vllm" || q.port != 8101 || !q.alive ||
|
||||
len(q.gpus) != 2 || !q.started.Equal(time.Date(2026, 10, 19, 8, 0, 0, 0, time.UTC)) {
|
||||
t.Errorf("qwen: %+v", q)
|
||||
}
|
||||
// No served name: the model from the replica's id.
|
||||
if b.model != "bge-m3" || b.alive || b.engine != "llamacpp" {
|
||||
t.Errorf("bge: %+v", b)
|
||||
}
|
||||
if runs, note := readShardStates(filepath.Join(dir, "absent"), pidAlive); runs != nil || note != "" {
|
||||
t.Errorf("no work dir: %+v %q", runs, note)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewNamesTheModelOnEachGPU(t *testing.T) {
|
||||
gpus := []nodeGPU{{index: 0}, {index: 1}, {index: 2}}
|
||||
runs := []engineRun{
|
||||
{model: "qwen3-8b", gpus: []int{0, 1}},
|
||||
{model: "bge-m3", gpus: []int{discovery.MIGIndex(1, 2)}},
|
||||
{model: "qwen3-8b", gpus: []int{1}},
|
||||
}
|
||||
got := modelsOnGPUs(gpus, runs)
|
||||
if strings.Join(got[0].models, ",") != "qwen3-8b" || strings.Join(got[1].models, ",") != "qwen3-8b,bge-m3" ||
|
||||
len(got[2].models) != 0 {
|
||||
t.Errorf("got %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewReadsWhatSystemdSaysOfTheAgent(t *testing.T) {
|
||||
s := parseSystemctlShow("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:12:03 UTC\n")
|
||||
if s.state != "active" || s.sub != "running" || s.note != "" || s.since.IsZero() || s.since.Minute() != 12 {
|
||||
t.Errorf("got %+v", s)
|
||||
}
|
||||
if s := parseSystemctlShow("LoadState=not-found\nActiveState=inactive\nActiveEnterTimestamp=\n"); s.note == "" {
|
||||
t.Errorf("a missing service: %+v", s)
|
||||
}
|
||||
if s := parseSystemctlShow("LoadState=loaded\nActiveState=failed\nSubState=failed\nActiveEnterTimestamp=n/a\n"); s.state != "failed" ||
|
||||
!s.since.IsZero() {
|
||||
t.Errorf("a failed service: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewKeepsTheLastWarningsOfTheJournal(t *testing.T) {
|
||||
var b strings.Builder
|
||||
for i := 1; i <= 9; i++ {
|
||||
b.WriteString("level=WARN msg=w" + string(rune('0'+i)) + "\n")
|
||||
}
|
||||
got := parseJournal(b.String(), 6)
|
||||
if len(got) != 6 || got[0] != "level=WARN msg=w4" || got[5] != "level=WARN msg=w9" {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := parseJournal("-- No entries --\n", 6); len(got) != 0 {
|
||||
t.Errorf("no entries: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// fakeProbe is a node with two cards, one engine and a broken second control
|
||||
// plane, and no tool of the machine running the test touched.
|
||||
func fakeProbe(t *testing.T) nodeProbe {
|
||||
dir := t.TempDir()
|
||||
writeNodeFile(t, filepath.Join(dir, "shard-qwen3-8b-ab12-0.json"), `{"replica_id":"qwen3-8b-ab12","pid":4242,
|
||||
"started_at":"2026-10-19T08:00:00Z","spec":{"GPUIndices":[0],"HTTPPort":8101,"Profile":{"Engine":"vllm","ServedModelName":"qwen3-8b"}}}`)
|
||||
return nodeProbe{
|
||||
configured: []string{"10.0.0.140:7443", "10.0.0.150:7443"},
|
||||
planesFile: filepath.Join(dir, "control-planes.json"),
|
||||
workDir: dir, weightsDir: "/var/lib/llmgw-weights", enginesDir: "/opt/llmgw/engines", root: true,
|
||||
run: func(_ context.Context, name string, _ ...string) ([]byte, error) {
|
||||
switch name {
|
||||
case "systemctl":
|
||||
return []byte("LoadState=loaded\nActiveState=active\nSubState=running\nActiveEnterTimestamp=Mon 2026-10-19 08:00:00 UTC\n"), nil
|
||||
case "journalctl":
|
||||
return []byte("level=WARN msg=\"the control plane does not answer\" server=10.0.0.150:7443\n"), nil
|
||||
}
|
||||
return nil, errors.New(name + " is not installed")
|
||||
},
|
||||
dial: func(_ context.Context, addr string) error {
|
||||
if addr == "10.0.0.150:7443" {
|
||||
return errors.New("dial tcp 10.0.0.150:7443: connect: connection refused")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
gpus: func(context.Context) []discovery.GPUSensor {
|
||||
return []discovery.GPUSensor{
|
||||
{Index: 0, TempC: 61, UtilPercent: 87, MemoryUsedMB: 20480, MemoryTotalMB: 24576},
|
||||
{Index: 1, TempC: 35, UtilPercent: 0, MemoryUsedMB: 300, MemoryTotalMB: 24576},
|
||||
}
|
||||
},
|
||||
cards: func(context.Context) ([]discovery.GPU, error) {
|
||||
return []discovery.GPU{{Index: 0, Name: "NVIDIA RTX A5000"}, {Index: 1, Name: "NVIDIA RTX A5000"}}, nil
|
||||
},
|
||||
alive: func(int) bool { return true },
|
||||
statfs: func(string) (uint64, uint64, error) { return 120 << 30, 500 << 30, nil },
|
||||
now: func() time.Time { return time.Date(2026, 10, 19, 11, 12, 0, 0, time.UTC) },
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOverviewShowsWhatRunsOnTheMachine(t *testing.T) {
|
||||
ov := fakeProbe(t).gather(context.Background())
|
||||
out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
|
||||
t.Log("\n" + out)
|
||||
for _, want := range []string{
|
||||
"llmgw-agent", "active (running)", "3h12m ago",
|
||||
"10.0.0.140:7443", "reachable", "connection refused",
|
||||
"NVIDIA RTX A5000", "87%", "20.0/24 GB", "61°C", "qwen3-8b", "free",
|
||||
"qwen3-8b-ab12", "vllm", "8101", "4242", "3h12m",
|
||||
"/var/lib/llmgw-weights", "120 GB free of 500 GB", "/opt/llmgw/engines",
|
||||
"the control plane does not answer",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("the overview lacks %q", want)
|
||||
}
|
||||
}
|
||||
if ov.names[0] != "NVIDIA RTX A5000" {
|
||||
t.Errorf("the names are not kept for the next reading: %v", ov.names)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeWithoutAGPUSaysSoInOneLine(t *testing.T) {
|
||||
p := fakeProbe(t)
|
||||
p.gpus = func(context.Context) []discovery.GPUSensor { return nil }
|
||||
p.cards = func(context.Context) ([]discovery.GPU, error) { return nil, nil }
|
||||
p.run = func(context.Context, string, ...string) ([]byte, error) { return nil, errors.New("not installed") }
|
||||
ov := p.gather(context.Background())
|
||||
out := ansi.Strip(renderNodeOverview(ov, 100, ov.at))
|
||||
if !strings.Contains(out, "No GPU on this machine.") {
|
||||
t.Errorf("no line saying there is no GPU:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "BUSY") {
|
||||
t.Errorf("an empty GPU table:\n%s", out)
|
||||
}
|
||||
// No systemctl or journalctl is said, not a crash.
|
||||
if !strings.Contains(out, "systemctl is not available") || !strings.Contains(out, "The journal cannot be read") {
|
||||
t.Errorf("missing tools are not said:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesConsoleOpensOnItsOverview(t *testing.T) {
|
||||
m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "agent.yaml"))
|
||||
if m.menu != cOverview || !m.entryOK(cOverview) {
|
||||
t.Fatalf("menu %d", m.menu)
|
||||
}
|
||||
_, body, _ := m.panel()
|
||||
if !strings.Contains(body, "reading this machine") {
|
||||
t.Errorf("before the first reading: %q", body)
|
||||
}
|
||||
}
|
||||
@@ -395,6 +395,7 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
|
||||
if l.AllowedModels != nil {
|
||||
models = strings.Join(l.AllowedModels, ",")
|
||||
}
|
||||
tools := strings.Join(l.AllowedTools, ",")
|
||||
m.openForm(&formDialog{title: "Limits on " + who,
|
||||
note: "Each level only narrows what is above it. Empty is no limit here.",
|
||||
fields: []formField{
|
||||
@@ -403,9 +404,11 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
|
||||
textF("budget", budget, "spend per period, in whole units"),
|
||||
choiceF("per", []string{"day", "month"}, period),
|
||||
textF("models", models, "allowed, separated by commas; \"every model\" for no list"),
|
||||
textF("company tools", tools, "MCP tools granted here, as jira.*,db.query; empty grants none here"),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
args := []string{"keys", "limits", "--scope=" + string(scope), "--id=" + id}
|
||||
// Every field the form shows, so an emptied one is cleared.
|
||||
args := []string{"keys", "limits", "--replace", "--scope=" + string(scope), "--id=" + id}
|
||||
for i, flag := range []string{"rpm", "tpm"} {
|
||||
if v[i] != "" {
|
||||
if _, err := strconv.Atoi(v[i]); err != nil {
|
||||
@@ -423,6 +426,9 @@ func (m *consoleModel) limitsForm(scope store.Scope, id, who string) {
|
||||
if v[4] != "every model" {
|
||||
args = append(args, "--models="+strings.ReplaceAll(v[4], " ", ""))
|
||||
}
|
||||
if t := strings.ReplaceAll(v[5], " ", ""); t != "" {
|
||||
args = append(args, "--tools="+t)
|
||||
}
|
||||
return m.runCLI("Limits on "+who, args...), nil
|
||||
}})
|
||||
}
|
||||
@@ -567,13 +573,20 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog {
|
||||
if c.Any {
|
||||
which = "any"
|
||||
}
|
||||
similar := "off"
|
||||
if c.Similar > 0 {
|
||||
similar = strconv.FormatFloat(c.Similar, 'f', -1, 64)
|
||||
}
|
||||
m.openForm(&formDialog{title: "Cache answers · " + r.path,
|
||||
note: "Off by default. On, the same request to the same model within the time below is\n" +
|
||||
"answered again from the gateway's memory, without the model, and costs nothing.\n" +
|
||||
"Reproducible: only temperature 0 or a seed, and embeddings. Any: sampled answers too.",
|
||||
"Reproducible: only temperature 0 or a seed, and embeddings. Any: sampled answers too.\n" +
|
||||
"Similar: a chat question this alike to one answered gets its answer (needs\n" +
|
||||
"router.cache.embedding_model).",
|
||||
fields: []formField{
|
||||
textF("reuse for", ttl, "like 1h; off reuses none"),
|
||||
choiceF("answers", []string{"reproducible", "any"}, which),
|
||||
textF("similar", similar, "like 0.95; off reuses only the same request"),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
args := []string{"keys", "cache", "--project=" + r.id}
|
||||
@@ -587,6 +600,12 @@ func (m consoleModel) orgMenu(r orgRow) *menuDialog {
|
||||
if v[1] == "any" {
|
||||
args = append(args, "--any")
|
||||
}
|
||||
if v[2] != "off" && v[2] != "" && v[2] != "0" {
|
||||
if f, err := strconv.ParseFloat(v[2], 64); err != nil || f <= 0 || f > 1 {
|
||||
return nil, fmt.Errorf("similar is a number like 0.95, or off")
|
||||
}
|
||||
args = append(args, "--similar="+v[2])
|
||||
}
|
||||
return m.runCLI("Cache answers", args...), nil
|
||||
}})
|
||||
return nil
|
||||
@@ -761,5 +780,8 @@ func limitsCell(l store.Limits) string {
|
||||
if l.AllowedModels != nil {
|
||||
p = append(p, fmt.Sprintf("%d models", len(l.AllowedModels)))
|
||||
}
|
||||
if len(l.AllowedTools) > 0 {
|
||||
p = append(p, fmt.Sprintf("%d tool grants", len(l.AllowedTools)))
|
||||
}
|
||||
return strings.Join(p, ", ")
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// presetsBlock is the models that come with llmgw, under the models table:
|
||||
// whether each fits a card here and whether it is in the catalog, and
|
||||
// whether the leader adds them by itself. Empty from a control plane that
|
||||
// cannot say.
|
||||
func presetsBlock(v *adminapi.PresetsView, width int) string {
|
||||
if v == nil || len(v.Presets) == 0 {
|
||||
return ""
|
||||
}
|
||||
var rows [][]string
|
||||
var why []string
|
||||
for _, p := range v.Presets {
|
||||
rows = append(rows, []string{p.Name, p.Makes, orDash(strings.Join(p.FitsClasses, ",")), presetState(p)})
|
||||
// Why the leader leaves it, under the table: a reason is longer than
|
||||
// any column the panel has room for. Automatic adding being off is
|
||||
// said once, below.
|
||||
if p.Blocked != "" && !strings.HasPrefix(p.Blocked, "presets.auto") && !p.Declined {
|
||||
why = append(why, " "+wrap(p.Name+": "+p.Blocked, width-4, " "))
|
||||
}
|
||||
}
|
||||
auto := "Added by themselves where a card fits · P stops that"
|
||||
if !v.Auto {
|
||||
auto = "Not added by themselves · P starts that · A adds one"
|
||||
}
|
||||
out := "\n\n Pictures and 3D models that come with llmgw\n" +
|
||||
table([]string{"PRESET", "MAKES", "FITS", "HERE"}, rows, -1, width, false)
|
||||
if len(why) > 0 {
|
||||
out += "\n" + sDim.Render(strings.Join(why, "\n"))
|
||||
}
|
||||
return out + "\n " + sDim.Render(auto)
|
||||
}
|
||||
|
||||
// presetChoice is how a preset is offered in the add dialog: its name, and
|
||||
// where it stands when that is worth knowing before choosing it.
|
||||
func presetChoice(v *adminapi.PresetsView, name string) string {
|
||||
if v == nil {
|
||||
return name
|
||||
}
|
||||
for _, p := range v.Presets {
|
||||
if p.Name != name {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case p.InCatalog:
|
||||
return name + " (already here)"
|
||||
case presetAddsComfyUI(v, p):
|
||||
return name + " (installs ComfyUI too)"
|
||||
case !p.Fits:
|
||||
return name + " (no card here fits it)"
|
||||
case p.Declined:
|
||||
return name + " (removed before)"
|
||||
}
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
// presetAddsComfyUI is whether adding a preset by hand also sets
|
||||
// engines.comfyui, which has the GPU nodes install it and the control planes
|
||||
// restart on the setting.
|
||||
func presetAddsComfyUI(v *adminapi.PresetsView, p adminapi.PresetView) bool {
|
||||
return v != nil && !p.InCatalog && v.ComfyUI == ""
|
||||
}
|
||||
|
||||
// addPreset adds a preset through the control plane, or, from one from
|
||||
// before it could, the way the release that brought presets did. One whose
|
||||
// adding installs ComfyUI and restarts the control planes is asked about
|
||||
// first.
|
||||
func (m *consoleModel) addPreset(name string) tea.Cmd {
|
||||
if m.data.presets != nil {
|
||||
for _, p := range m.data.presets.Presets {
|
||||
if p.Name == name && presetAddsComfyUI(m.data.presets, p) {
|
||||
m.confirmD = &confirmDialog{title: "Add " + name + " and ComfyUI",
|
||||
text: "The cluster installs no ComfyUI yet. Adding " + name + " sets engines.comfyui:\n" +
|
||||
"every GPU node installs it, about 10 GB each, and the control planes restart\n" +
|
||||
"on the setting, standbys first. Engines keep serving.",
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.postPreset(name) }}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return m.postPreset(name)
|
||||
}
|
||||
|
||||
func (m *consoleModel) postPreset(name string) tea.Cmd {
|
||||
if m.data.presets == nil {
|
||||
return m.runCLI("Add "+name, "ctl", "--server="+m.client.base, "model", "add", name)
|
||||
}
|
||||
return m.call("Add "+name, http.MethodPost, "/api/v1/presets/"+url.PathEscape(name), map[string]string{},
|
||||
func(_ json.RawMessage) string {
|
||||
return name + " is added, and ComfyUI with it where the cluster had none.\n" +
|
||||
"It is placed on the next pass, and loads when first asked for."
|
||||
})
|
||||
}
|
||||
|
||||
// autoPresetsConfirm turns the adding of presets by themselves on or off,
|
||||
// after saying what that does.
|
||||
func (m *consoleModel) autoPresetsConfirm() {
|
||||
on := !m.data.presets.Auto
|
||||
title, text := "Stop adding pictures and 3D models by themselves",
|
||||
"Presets already added stay. A preset is added only when somebody adds it."
|
||||
if on {
|
||||
title, text = "Add pictures and 3D models by themselves",
|
||||
"Where a card fits one and the catalog has nothing that makes what it makes, the leader adds it.\n"+
|
||||
"That also has every GPU node install ComfyUI, about 10 GB each, and the mirror fetch its weights.\n"+
|
||||
"A preset somebody removed is not added again."
|
||||
}
|
||||
m.confirmD = &confirmDialog{title: title, text: text, yes: func(m *consoleModel) tea.Cmd {
|
||||
return m.call(title, http.MethodPut, "/api/v1/presets/auto", adminapi.PresetsAutoRequest{On: on},
|
||||
func(_ json.RawMessage) string {
|
||||
if on {
|
||||
return "Presets are added by themselves where a card fits."
|
||||
}
|
||||
return "Presets are no longer added by themselves."
|
||||
})
|
||||
}}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
func presetsHere() *adminapi.PresetsView {
|
||||
return &adminapi.PresetsView{Auto: true, Presets: []adminapi.PresetView{
|
||||
{Name: "flux-schnell", Makes: "pictures", Card: "cc 7.0+, 24 GB+", Fits: true, FitsClasses: []string{"v100-32g"},
|
||||
Blocked: "the cluster installs no ComfyUI (engines.comfyui); adding it by hand sets that"},
|
||||
{Name: "triposr", Makes: "meshes", Card: "cc 7.0+, 8 GB+", Fits: true, InCatalog: true, AddedBy: "auto"},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestThePresetsAreOnTheModelsScreen(t *testing.T) {
|
||||
m, _ := consoleWith(t, cModels)
|
||||
m.data.presets = presetsHere()
|
||||
v := m.View()
|
||||
for _, want := range []string{"Pictures and 3D models that come with llmgw", "flux-schnell", "v100-32g",
|
||||
"installs no ComfyUI", "added by itself", "P pictures and 3D"} {
|
||||
if !strings.Contains(v, want) {
|
||||
t.Errorf("%q not shown:\n%s", want, v)
|
||||
}
|
||||
}
|
||||
// A control plane that cannot say shows none of it.
|
||||
m.data.presets = nil
|
||||
if v := m.View(); strings.Contains(v, "come with llmgw") || strings.Contains(v, "P pictures") {
|
||||
t.Errorf("presets shown from nothing:\n%s", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPresetThatInstallsComfyUIIsAddedOnlyWhenConfirmed(t *testing.T) {
|
||||
m, s := consoleWith(t, cModels)
|
||||
m.data.presets = presetsHere()
|
||||
var tm tea.Model = m
|
||||
tm = press(tm, "a")
|
||||
f := tm.(consoleModel).formD
|
||||
if f == nil || !strings.Contains(f.fields[0].value(), "installs ComfyUI too") {
|
||||
t.Fatalf("no add dialog saying what flux-schnell brings: %+v", f)
|
||||
}
|
||||
tm = press(tm, "enter", "enter", "enter")
|
||||
if tm.(consoleModel).confirmD == nil || s.last() != "" {
|
||||
t.Fatalf("added without asking: %q", s.last())
|
||||
}
|
||||
tm = pressRun(tm, "y")
|
||||
if got := s.last(); got != "POST /api/v1/presets/flux-schnell {}" {
|
||||
t.Fatalf("sent %q", got)
|
||||
}
|
||||
|
||||
tm = press(tm, "esc", "p") // past what the add said
|
||||
if c := tm.(consoleModel).confirmD; c == nil || !strings.HasPrefix(c.title, "Stop adding") {
|
||||
t.Fatalf("P did not ask to stop: %+v", c)
|
||||
}
|
||||
_ = pressRun(tm, "y")
|
||||
if got := s.last(); got != `PUT /api/v1/presets/auto {"on":false}` {
|
||||
t.Fatalf("sent %q", got)
|
||||
}
|
||||
}
|
||||
+188
-143
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/mail"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -35,7 +34,8 @@ type setupPage int
|
||||
|
||||
const (
|
||||
pWelcome setupPage = iota
|
||||
pAdmin
|
||||
// pChoices is what is installed, part by part, each with its settings.
|
||||
pChoices
|
||||
pMachines
|
||||
// pWait shows the codes for machines joining rather than logged into,
|
||||
// until they have all dialled in.
|
||||
@@ -89,14 +89,21 @@ func (d dbMachine) visible(f int) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// dbConn is a database machine as it is reached, whichever way.
|
||||
// dbConn is a database machine as it is reached, whichever way, or a judge
|
||||
// machine when judge is set.
|
||||
type dbConn struct {
|
||||
remote onboard.Remote
|
||||
addr string
|
||||
ssh *onboard.GoSSH
|
||||
joined *joinpkg.Remote
|
||||
judge bool
|
||||
target string
|
||||
}
|
||||
|
||||
// The machines the page asks for: the store's two, then up to maxJudges for
|
||||
// the guardrails' judges alone.
|
||||
const maxJudges = 2
|
||||
|
||||
func newDBMachine(n int) dbMachine {
|
||||
f := func(ph string, mask bool) textField {
|
||||
t := newTextField()
|
||||
@@ -112,9 +119,31 @@ func newDBMachine(n int) dbMachine {
|
||||
if os.Getenv("SSH_AUTH_SOCK") == "" {
|
||||
m.auth = authKey
|
||||
}
|
||||
// As choosing the way to log in sets them: a key file's path is not a
|
||||
// secret to hide.
|
||||
m.secret.Mask = m.auth == authPassword
|
||||
m.secret.Placeholder = map[int]string{authKey: "/root/.ssh/id_ed25519", authPassword: "the user's ssh password"}[m.auth]
|
||||
return m
|
||||
}
|
||||
|
||||
// newJudgeMachine is the fields of a machine for the judges.
|
||||
func newJudgeMachine(n int) dbMachine {
|
||||
d := newDBMachine(n + 2)
|
||||
d.addr.Placeholder = fmt.Sprintf("10.0.0.%d or judge%d.internal", 20+n, n)
|
||||
return d
|
||||
}
|
||||
|
||||
// shown is how many machines the page asks for.
|
||||
func (m setupModel) shown() int { return 2 + m.judges }
|
||||
|
||||
// machineTitle names machine i: the store's, or the judges'.
|
||||
func machineTitle(i int) string {
|
||||
if i < 2 {
|
||||
return fmt.Sprintf("Database machine %d", i+1)
|
||||
}
|
||||
return fmt.Sprintf("Judge machine %d", i-1)
|
||||
}
|
||||
|
||||
// field is the text field at a machine's position, nil for the selector.
|
||||
func (d *dbMachine) field(i int) *textField {
|
||||
switch i {
|
||||
@@ -150,11 +179,12 @@ type setupModel struct {
|
||||
ips []string
|
||||
choice int // 0 one machine, 1 high availability
|
||||
|
||||
admin, names textField
|
||||
afocus int // admin, names, next
|
||||
choices *installForm
|
||||
|
||||
db [2]dbMachine
|
||||
mfocus int // over both machines' fields, then the check button
|
||||
db [2 + maxJudges]dbMachine
|
||||
// judges is how many judge machines are asked for, none to maxJudges.
|
||||
judges int
|
||||
mfocus int // over the machines' fields, then the judges' choice, then the check button
|
||||
|
||||
conns []dbConn
|
||||
listener *joinpkg.Listener
|
||||
@@ -180,13 +210,17 @@ type setupModel struct {
|
||||
}
|
||||
|
||||
func newSetupModel(host string, ips []string) setupModel {
|
||||
admin := newTextField()
|
||||
admin.Placeholder, admin.CharLimit, admin.Width = "you@example.com", 200, 40
|
||||
names := newTextField()
|
||||
names.Placeholder, names.CharLimit, names.Width = "llm.example.com, 203.0.113.10", 400, 40
|
||||
in := newDefaultServerInit()
|
||||
// The control plane of a replicated setup is not where models run;
|
||||
// nodes are added once it is up. Either can still be chosen.
|
||||
in.noNode, in.noModel = true, true
|
||||
in.standbys = true
|
||||
// Two judges, on two of the three machines' nodes.
|
||||
in.judge.Replicas = 2
|
||||
return setupModel{
|
||||
ref: &progRef{}, host: host, ips: ips, admin: admin, names: names,
|
||||
db: [2]dbMachine{newDBMachine(1), newDBMachine(2)}, spin: newSpinner(), width: 100, height: 30,
|
||||
ref: &progRef{}, host: host, ips: ips, in: &in, choices: newInstallForm(&in, host, ips, true),
|
||||
db: [2 + maxJudges]dbMachine{newDBMachine(1), newDBMachine(2), newJudgeMachine(1), newJudgeMachine(2)},
|
||||
spin: newSpinner(), width: 100, height: 30,
|
||||
log: &[]string{}, logMu: &sync.Mutex{},
|
||||
}
|
||||
}
|
||||
@@ -204,6 +238,7 @@ func (m setupModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
switch msg := msg.(type) {
|
||||
case tea.WindowSizeMsg:
|
||||
m.width, m.height = msg.Width, msg.Height
|
||||
m.choices.height = msg.Height
|
||||
m.run.width = msg.Width - navWidth - 6
|
||||
case spinTickMsg:
|
||||
var cmd tea.Cmd
|
||||
@@ -241,6 +276,11 @@ func (m setupModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
}
|
||||
|
||||
func (m setupModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
// The choices' own dialogs take their keys, Esc among them.
|
||||
if m.page == pChoices && (m.choices.dialog != nil || m.choices.err != "") {
|
||||
m.choices.key(k, m.in)
|
||||
return m, nil
|
||||
}
|
||||
// A dialog open takes the next key, whatever it is, to close it.
|
||||
if m.formErr != "" {
|
||||
m.formErr = ""
|
||||
@@ -272,12 +312,13 @@ func (m setupModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.single = true
|
||||
return m, tea.Quit
|
||||
}
|
||||
m.page = pAdmin
|
||||
m.afocus = 0
|
||||
m.admin.Focus()
|
||||
m.page = pChoices
|
||||
}
|
||||
case pChoices:
|
||||
if m.choices.key(k, m.in) {
|
||||
m.page, m.mfocus = pMachines, 0
|
||||
m.focusMachine()
|
||||
}
|
||||
case pAdmin:
|
||||
return m.adminKey(k)
|
||||
case pMachines:
|
||||
return m.machinesKey(k)
|
||||
case pCheck:
|
||||
@@ -306,51 +347,6 @@ func (m setupModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (m setupModel) adminKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
move := func(d int) {
|
||||
m.afocus = (m.afocus + d + 3) % 3
|
||||
m.admin.Blur()
|
||||
m.names.Blur()
|
||||
switch m.afocus {
|
||||
case 0:
|
||||
m.admin.Focus()
|
||||
case 1:
|
||||
m.names.Focus()
|
||||
}
|
||||
}
|
||||
switch k.Type {
|
||||
case tea.KeyTab, tea.KeyDown:
|
||||
move(1)
|
||||
return m, nil
|
||||
case tea.KeyShiftTab, tea.KeyUp:
|
||||
move(-1)
|
||||
return m, nil
|
||||
case tea.KeyEnter:
|
||||
if m.afocus < 2 {
|
||||
move(1)
|
||||
return m, nil
|
||||
}
|
||||
if _, err := mail.ParseAddress(strings.TrimSpace(m.admin.Value())); err != nil {
|
||||
m.formErr = "the administrator's email is needed, and that is not one"
|
||||
m.afocus = 0
|
||||
move(0)
|
||||
return m, nil
|
||||
}
|
||||
m.formErr = ""
|
||||
m.page, m.mfocus = pMachines, 0
|
||||
m.db[0].addr.Focus()
|
||||
return m, nil
|
||||
}
|
||||
var cmd tea.Cmd
|
||||
switch m.afocus {
|
||||
case 0:
|
||||
m.admin, cmd = m.admin.Update(k)
|
||||
case 1:
|
||||
m.names, cmd = m.names.Update(k)
|
||||
}
|
||||
return m, cmd
|
||||
}
|
||||
|
||||
func (m *setupModel) focusMachine() {
|
||||
for i := range m.db {
|
||||
for f := 0; f < mfCount; f++ {
|
||||
@@ -359,7 +355,7 @@ func (m *setupModel) focusMachine() {
|
||||
}
|
||||
}
|
||||
}
|
||||
if m.mfocus < 2*mfCount {
|
||||
if m.mfocus < m.shown()*mfCount {
|
||||
if t := m.db[m.mfocus/mfCount].field(m.mfocus % mfCount); t != nil {
|
||||
t.Focus()
|
||||
}
|
||||
@@ -367,7 +363,8 @@ func (m *setupModel) focusMachine() {
|
||||
}
|
||||
|
||||
func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
n := 2*mfCount + 1
|
||||
sel := m.shown() * mfCount
|
||||
n := sel + 2
|
||||
switch k.Type {
|
||||
case tea.KeyTab, tea.KeyDown:
|
||||
m.step(1, n)
|
||||
@@ -376,7 +373,7 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.step(-1, n)
|
||||
return m, nil
|
||||
case tea.KeyEnter:
|
||||
if m.mfocus < 2*mfCount {
|
||||
if m.mfocus <= sel {
|
||||
m.step(1, n)
|
||||
return m, nil
|
||||
}
|
||||
@@ -393,7 +390,17 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
cmd := m.startCheck()
|
||||
return m, cmd
|
||||
}
|
||||
if m.mfocus == 2*mfCount {
|
||||
if m.mfocus == sel {
|
||||
switch k.String() {
|
||||
case "left", "h":
|
||||
m.judges = (m.judges + maxJudges) % (maxJudges + 1)
|
||||
case "right", "l", " ":
|
||||
m.judges = (m.judges + 1) % (maxJudges + 1)
|
||||
}
|
||||
m.mfocus = m.shown() * mfCount
|
||||
return m, nil
|
||||
}
|
||||
if m.mfocus > sel {
|
||||
return m, nil
|
||||
}
|
||||
d := &m.db[m.mfocus/mfCount]
|
||||
@@ -423,7 +430,7 @@ func (m setupModel) machinesKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
func (m *setupModel) step(d, n int) {
|
||||
for range n {
|
||||
m.mfocus = (m.mfocus + d + n) % n
|
||||
if m.mfocus == 2*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
|
||||
if m.mfocus >= m.shown()*mfCount || m.db[m.mfocus/mfCount].visible(m.mfocus%mfCount) {
|
||||
break
|
||||
}
|
||||
}
|
||||
@@ -431,7 +438,7 @@ func (m *setupModel) step(d, n int) {
|
||||
}
|
||||
|
||||
func (m setupModel) allJoined() bool {
|
||||
for _, d := range m.db {
|
||||
for _, d := range m.db[:m.shown()] {
|
||||
if d.auth == authJoin && d.joined == nil {
|
||||
return false
|
||||
}
|
||||
@@ -443,7 +450,7 @@ func (m setupModel) allJoined() bool {
|
||||
// codes, and waits for them; it says whether there is anything to wait for.
|
||||
func (m *setupModel) startJoins() (tea.Cmd, bool) {
|
||||
var cmds []tea.Cmd
|
||||
for i := range m.db {
|
||||
for i := range m.db[:m.shown()] {
|
||||
d := &m.db[i]
|
||||
if d.auth != authJoin || d.joined != nil {
|
||||
continue
|
||||
@@ -461,7 +468,11 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
|
||||
m.listener = l
|
||||
}
|
||||
if d.arrive == nil {
|
||||
c, ch := m.listener.Code("database")
|
||||
role := "database"
|
||||
if i >= 2 {
|
||||
role = "judge"
|
||||
}
|
||||
c, ch := m.listener.Code(role)
|
||||
d.code, d.arrive = c.String(), ch
|
||||
}
|
||||
ch, i := d.arrive, i
|
||||
@@ -472,20 +483,20 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
|
||||
|
||||
func (m setupModel) validateMachines() string {
|
||||
seen := map[string]bool{m.thisAddr(): true, m.host: true}
|
||||
for i, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
if d.auth == authJoin {
|
||||
continue
|
||||
}
|
||||
addr := strings.TrimSpace(d.addr.Value())
|
||||
addr, title := strings.TrimSpace(d.addr.Value()), strings.ToLower(machineTitle(i)[:1])+machineTitle(i)[1:]
|
||||
switch {
|
||||
case addr == "":
|
||||
return fmt.Sprintf("database machine %d has no address", i+1)
|
||||
return title + " has no address"
|
||||
case seen[addr]:
|
||||
return fmt.Sprintf("%s is given twice, or is this machine: the three must be different machines", addr)
|
||||
return fmt.Sprintf("%s is given twice, or is this machine: each must be a different machine", addr)
|
||||
case strings.TrimSpace(d.user.Value()) == "":
|
||||
return fmt.Sprintf("database machine %d has no user to log in as", i+1)
|
||||
return title + " has no user to log in as"
|
||||
case d.auth != authAgent && d.secret.Value() == "":
|
||||
return fmt.Sprintf("database machine %d needs its %s", i+1, authNames[d.auth])
|
||||
return fmt.Sprintf("%s needs its %s", title, authNames[d.auth])
|
||||
}
|
||||
seen[addr] = true
|
||||
}
|
||||
@@ -501,15 +512,22 @@ func (m *setupModel) startCheck() tea.Cmd {
|
||||
}
|
||||
}
|
||||
m.conns = nil
|
||||
serves, what := m.in.servingMemoryMB(m.judges > 0)
|
||||
machines := []preflight.Machine{{
|
||||
Name: m.host + " (this machine)", Addr: m.thisAddr(), Local: true, Remote: pgha.Local{},
|
||||
Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
|
||||
Roles: []preflight.Role{preflight.RoleControl, preflight.RoleWitness},
|
||||
ServesMB: serves, Serves: what,
|
||||
}}
|
||||
for _, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
roles, judge := m.dbRoles(), i >= 2
|
||||
if judge {
|
||||
roles = []preflight.Role{preflight.RoleNode}
|
||||
}
|
||||
if d.auth == authJoin {
|
||||
m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined})
|
||||
m.conns = append(m.conns, dbConn{remote: d.joined, addr: d.joined.Addr, joined: d.joined,
|
||||
judge: judge, target: "root@" + d.joined.Addr})
|
||||
machines = append(machines, preflight.Machine{Name: d.joined.Hostname + " (" + d.joined.Addr + ", joined)",
|
||||
Addr: d.joined.Addr, Roles: []preflight.Role{preflight.RoleDatabase}, Remote: d.joined})
|
||||
Addr: d.joined.Addr, Roles: roles, Remote: d.joined})
|
||||
continue
|
||||
}
|
||||
g := &onboard.GoSSH{
|
||||
@@ -522,9 +540,8 @@ func (m *setupModel) startCheck() tea.Cmd {
|
||||
case authPassword:
|
||||
g.Password = d.secret.Value()
|
||||
}
|
||||
m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g})
|
||||
machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr,
|
||||
Roles: []preflight.Role{preflight.RoleDatabase}, Remote: g})
|
||||
m.conns = append(m.conns, dbConn{remote: g, addr: g.Addr, ssh: g, judge: judge, target: g.User + "@" + g.Addr})
|
||||
machines = append(machines, preflight.Machine{Name: g.Addr, Addr: g.Addr, Roles: roles, Remote: g})
|
||||
}
|
||||
m.checking = true
|
||||
m.report = preflight.Report{}
|
||||
@@ -555,10 +572,19 @@ func (m *setupModel) startCheck() tea.Cmd {
|
||||
}
|
||||
}
|
||||
|
||||
// dbRoles are what the other two machines are to be: the store, and a
|
||||
// control plane too when they stand by.
|
||||
func (m setupModel) dbRoles() []preflight.Role {
|
||||
if m.in.standbys {
|
||||
return []preflight.Role{preflight.RoleDatabase, preflight.RoleControl}
|
||||
}
|
||||
return []preflight.Role{preflight.RoleDatabase}
|
||||
}
|
||||
|
||||
// targets are the database machines as user@address, for the command line.
|
||||
func (m setupModel) targets() []string {
|
||||
var out []string
|
||||
for _, d := range m.db {
|
||||
for _, d := range m.db[:2] {
|
||||
if d.joined != nil {
|
||||
out = append(out, "root@"+d.joined.Addr)
|
||||
continue
|
||||
@@ -568,12 +594,15 @@ func (m setupModel) targets() []string {
|
||||
return out
|
||||
}
|
||||
|
||||
func (m setupModel) namesList() []string {
|
||||
var out []string
|
||||
for _, h := range strings.Split(m.names.Value(), ",") {
|
||||
if h = strings.TrimSpace(h); h != "" {
|
||||
out = append(out, h)
|
||||
// judgeTargets are the judge machines as user@address.
|
||||
func (m setupModel) judgeTargets() []judgeMachine {
|
||||
var out []judgeMachine
|
||||
for _, d := range m.db[2:m.shown()] {
|
||||
if d.joined != nil {
|
||||
out = append(out, judgeMachine{target: "root@" + d.joined.Addr})
|
||||
continue
|
||||
}
|
||||
out = append(out, judgeMachine{target: strings.TrimSpace(d.user.Value()) + "@" + strings.TrimSpace(d.addr.Value())})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -581,30 +610,19 @@ func (m setupModel) namesList() []string {
|
||||
// command is what this setup is on the command line, for doing it again
|
||||
// from a script — with keys in place, since a password cannot be given there.
|
||||
func (m setupModel) command() string {
|
||||
parts := []string{"sudo --preserve-env=SSH_AUTH_SOCK llmgw server init",
|
||||
"--admin " + strings.TrimSpace(m.admin.Value())}
|
||||
for _, h := range m.namesList() {
|
||||
parts = append(parts, "--host "+h)
|
||||
}
|
||||
for _, t := range m.targets() {
|
||||
parts = append(parts, "--patroni "+t)
|
||||
}
|
||||
parts = append(parts, "--no-node", "--no-model")
|
||||
return strings.Join(parts, " \\\n ")
|
||||
in := *m.in
|
||||
in.patroni = m.targets()
|
||||
in.judgeMachines = m.judgeTargets()
|
||||
return strings.Join(append([]string{"sudo --preserve-env=SSH_AUTH_SOCK llmgw server init"}, in.initFlags()...), " \\\n ")
|
||||
}
|
||||
|
||||
// startRun installs: sudo granted where it asks for a password, the control
|
||||
// plane here, the store on the other two, then sudo taken back.
|
||||
func (m *setupModel) startRun() tea.Cmd {
|
||||
in := newDefaultServerInit()
|
||||
in.admin = strings.TrimSpace(m.admin.Value())
|
||||
in.hosts = m.namesList()
|
||||
in := m.in
|
||||
in.patroni = m.targets()
|
||||
// The control plane of a replicated setup is not where models run;
|
||||
// nodes are added once it is up.
|
||||
in.noNode, in.noModel = true, true
|
||||
m.in = &in
|
||||
m.run = newInitModel(&in, nil, false)
|
||||
in.judgeMachines = nil
|
||||
m.run = newInitModel(in, nil, false)
|
||||
m.run.running, m.run.width, m.run.spin = true, m.width, m.spin
|
||||
ref, conns, logp, mu, listener := m.ref, m.conns, m.log, m.logMu, m.listener
|
||||
return func() tea.Msg {
|
||||
@@ -624,6 +642,11 @@ func (m *setupModel) startRun() tea.Cmd {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if c.judge {
|
||||
// Added as a node once the control plane is up.
|
||||
in.judgeMachines = append(in.judgeMachines, judgeMachine{target: c.target, remote: c.remote})
|
||||
continue
|
||||
}
|
||||
out, err := c.remote.Run(ctx, "hostname", nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %v", c.addr, err)
|
||||
@@ -659,7 +682,7 @@ var setupSteps = []struct {
|
||||
label string
|
||||
page setupPage
|
||||
}{
|
||||
{"Start", pWelcome}, {"Administrator", pAdmin}, {"Machines", pMachines},
|
||||
{"Start", pWelcome}, {"Choices", pChoices}, {"Machines", pMachines},
|
||||
{"Checks", pCheck}, {"Plan", pPlan}, {"Install", pRun},
|
||||
}
|
||||
|
||||
@@ -692,6 +715,9 @@ func (m setupModel) View() string {
|
||||
if m.formErr != "" {
|
||||
f.DialogTitle, f.Dialog = "Not yet", m.formErr+"\n\n"+sDim.Render("any key to go back")
|
||||
}
|
||||
if m.page == pChoices {
|
||||
f.DialogTitle, f.Dialog = m.choices.overlay()
|
||||
}
|
||||
return f.render()
|
||||
}
|
||||
|
||||
@@ -700,8 +726,12 @@ func (m setupModel) pageView() (string, string, string) {
|
||||
switch m.page {
|
||||
case pWelcome:
|
||||
return "Start", m.welcomeView(), "↑↓ choose · Enter go on · Ctrl+C leave"
|
||||
case pAdmin:
|
||||
return "Administrator", m.adminView(), "Tab/↑↓ move · Enter next · Esc back · Ctrl+C leave"
|
||||
case pChoices:
|
||||
keys := m.choices.keys()
|
||||
if m.choices.dialog == nil && m.choices.err == "" {
|
||||
keys = strings.Replace(keys, " · Ctrl+C", " · Esc back · Ctrl+C", 1)
|
||||
}
|
||||
return "What to install", m.choices.view(m.width - navWidth - 6), keys
|
||||
case pMachines:
|
||||
return "Database machines", m.machinesView(), "Tab/↑↓ move · ←→ how to log in · Enter next · Esc back · PgUp/PgDn scroll"
|
||||
case pWait:
|
||||
@@ -742,24 +772,8 @@ func (m setupModel) welcomeView() string {
|
||||
}
|
||||
opt(0, "One machine", "This machine runs everything: control plane, model mirror,\n and models if it has room.")
|
||||
opt(1, "High availability — three machines",
|
||||
"This one is the control plane; two more keep the store, replicated,\n so losing any one machine loses nothing.")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func (m setupModel) adminView() string {
|
||||
var b strings.Builder
|
||||
label := func(i int, s string) string {
|
||||
if m.afocus == i {
|
||||
return fSelected.Render("› " + s)
|
||||
}
|
||||
return " " + s
|
||||
}
|
||||
b.WriteString("\n")
|
||||
fmt.Fprintf(&b, "%s\n %s\n %s\n\n", label(0, "Administrator's email"), m.admin.View(),
|
||||
sDim.Render("A password is generated and shown once, at the end."))
|
||||
fmt.Fprintf(&b, "%s\n %s\n %s\n\n", label(1, "Other names nodes will reach this machine by"), m.names.View(),
|
||||
sDim.Render("Known: "+strings.Join(append([]string{m.host}, m.ips...), ", ")+". Add a DNS name or a floating address."))
|
||||
b.WriteString(button(m.afocus == 2, "Next") + "\n")
|
||||
"Each of the three can run the gateway; two keep the store, replicated,\n"+
|
||||
" and etcd votes on all three, so losing any one machine loses nothing.")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -775,8 +789,12 @@ func (m setupModel) machinesView() string {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n The two machines that keep the store: Ubuntu 22.04/24.04 or Debian 12.\n" +
|
||||
" PostgreSQL, Patroni and etcd are installed on them.\n\n")
|
||||
for i, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
base := i * mfCount
|
||||
if i == 2 {
|
||||
b.WriteString(" " + sDim.Render("Machines for the guardrails' judges alone: each becomes a node that runs only\n"+
|
||||
" the judges, so a busy control plane cannot leave a request waiting for a judgment.") + "\n\n")
|
||||
}
|
||||
// Padded before it is coloured: the colour's escape codes count
|
||||
// toward a width and threw the focused row out of line.
|
||||
label := func(f int, s string) string {
|
||||
@@ -786,7 +804,7 @@ func (m setupModel) machinesView() string {
|
||||
}
|
||||
return " " + s
|
||||
}
|
||||
fmt.Fprintf(&b, " %s\n", lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1)))
|
||||
fmt.Fprintf(&b, " %s\n", lipgloss.NewStyle().Bold(true).Render(machineTitle(i)))
|
||||
if d.visible(mfAddr) {
|
||||
fmt.Fprintf(&b, " %s %s\n", label(mfAddr, "Address"), d.addr.View())
|
||||
fmt.Fprintf(&b, " %s %s\n", label(mfUser, "User"), d.user.View())
|
||||
@@ -811,7 +829,22 @@ func (m setupModel) machinesView() string {
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
b.WriteString(button(m.mfocus == 2*mfCount, "Check the three machines") + "\n\n")
|
||||
sel := m.shown() * mfCount
|
||||
var opts []string
|
||||
for n, name := range []string{"none", "one", "two"} {
|
||||
if n == m.judges {
|
||||
opts = append(opts, "("+sOK.Render("•")+") "+name)
|
||||
} else {
|
||||
opts = append(opts, "( ) "+sDim.Render(name))
|
||||
}
|
||||
}
|
||||
judgeLabel := fmt.Sprintf(" %-15s", "Judge machines")
|
||||
if m.mfocus == sel {
|
||||
judgeLabel = fSelected.Render(fmt.Sprintf("› %-15s", "Judge machines"))
|
||||
}
|
||||
fmt.Fprintf(&b, " %s %s\n", judgeLabel, strings.Join(opts, " "))
|
||||
b.WriteString(" " + sDim.Render("Without them the judges run on whichever node has room, added later.") + "\n\n")
|
||||
b.WriteString(button(m.mfocus == sel+1, fmt.Sprintf("Check the %d machines", m.shown()+1)) + "\n\n")
|
||||
b.WriteString(" " + sDim.Render("Passwords are kept in memory only, for this setup.") + "\n")
|
||||
return b.String()
|
||||
}
|
||||
@@ -819,11 +852,11 @@ func (m setupModel) machinesView() string {
|
||||
func (m setupModel) waitView() string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "\n On each machine below, as root. It dials this machine on port %d;\n nothing dials in. Each code is for one machine and good for an hour.\n\n", joinpkg.DefaultPort)
|
||||
for i, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
if d.auth != authJoin {
|
||||
continue
|
||||
}
|
||||
title := lipgloss.NewStyle().Bold(true).Render(fmt.Sprintf("Database machine %d", i+1))
|
||||
title := lipgloss.NewStyle().Bold(true).Render(machineTitle(i))
|
||||
if d.joined != nil {
|
||||
fmt.Fprintf(&b, " %s %s joined as %s (%s)\n\n", title, sOK.Render("✓"), d.joined.Hostname, d.joined.Addr)
|
||||
continue
|
||||
@@ -842,7 +875,7 @@ func (m setupModel) checkView() string {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n")
|
||||
if m.checking {
|
||||
b.WriteString(" " + m.spin.View() + " checking the three machines…\n")
|
||||
b.WriteString(" " + m.spin.View() + " checking the machines…\n")
|
||||
return b.String()
|
||||
}
|
||||
switch m.report.Worst() {
|
||||
@@ -851,7 +884,7 @@ func (m setupModel) checkView() string {
|
||||
case preflight.Warn:
|
||||
b.WriteString(" " + sWarn.Render("! It can go ahead; what is marked ! is worth a look.") + "\n")
|
||||
default:
|
||||
b.WriteString(" " + sOK.Render("✓ All three are ready.") + "\n")
|
||||
b.WriteString(" " + sOK.Render("✓ All are ready.") + "\n")
|
||||
}
|
||||
machine := ""
|
||||
for _, f := range m.report.Findings {
|
||||
@@ -876,17 +909,29 @@ func (m setupModel) checkView() string {
|
||||
|
||||
func (m setupModel) planView() string {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n What will be installed:\n\n")
|
||||
fmt.Fprintf(&b, " %s %s\n %s\n", lipgloss.NewStyle().Bold(true).Render(m.host), sDim.Render(m.thisAddr()),
|
||||
"control plane and gateway, model mirror, etcd (the third vote)")
|
||||
fmt.Fprintf(&b, " administrator %s\n", strings.TrimSpace(m.admin.Value()))
|
||||
b.WriteString(m.choices.planView(m.width - navWidth - 6))
|
||||
b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render("The machines") + "\n")
|
||||
here := "control plane, leading at first; model mirror; etcd"
|
||||
there := "PostgreSQL under Patroni, etcd"
|
||||
if m.in.standbys {
|
||||
there = "control plane, standing by; " + there
|
||||
}
|
||||
if m.in.vip != "" {
|
||||
here += "; keepalived"
|
||||
there += "; keepalived"
|
||||
}
|
||||
fmt.Fprintf(&b, " %s %s\n %s\n", lipgloss.NewStyle().Bold(true).Render(m.host), sDim.Render(m.thisAddr()), here)
|
||||
granted := false
|
||||
for _, d := range m.db {
|
||||
for i, d := range m.db[:m.shown()] {
|
||||
addr := strings.TrimSpace(d.addr.Value())
|
||||
if d.joined != nil {
|
||||
addr = d.joined.Hostname + " " + sDim.Render(d.joined.Addr+", joined")
|
||||
}
|
||||
fmt.Fprintf(&b, " %s\n PostgreSQL under Patroni, etcd\n", lipgloss.NewStyle().Bold(true).Render(addr))
|
||||
role := there
|
||||
if i >= 2 {
|
||||
role = "a node for the guardrails' judges alone"
|
||||
}
|
||||
fmt.Fprintf(&b, " %s\n %s\n", lipgloss.NewStyle().Bold(true).Render(addr), role)
|
||||
granted = granted || d.sudo.Value() != ""
|
||||
}
|
||||
b.WriteString("\n The store is replicated from the first to the second; if the primary is lost\n" +
|
||||
@@ -898,7 +943,7 @@ func (m setupModel) planView() string {
|
||||
for _, l := range strings.Split(m.command(), "\n") {
|
||||
b.WriteString(" " + l + "\n")
|
||||
}
|
||||
for _, d := range m.db {
|
||||
for _, d := range m.db[:m.shown()] {
|
||||
if d.joined != nil {
|
||||
b.WriteString(" " + sDim.Render("(a machine that joined with a code needs ssh for that)") + "\n")
|
||||
break
|
||||
|
||||
@@ -26,6 +26,12 @@ func press(m tea.Model, keys ...string) tea.Model {
|
||||
msg = tea.KeyMsg{Type: tea.KeyTab}
|
||||
case "down":
|
||||
msg = tea.KeyMsg{Type: tea.KeyDown}
|
||||
case "up":
|
||||
msg = tea.KeyMsg{Type: tea.KeyUp}
|
||||
case "space":
|
||||
msg = tea.KeyMsg{Type: tea.KeySpace, Runes: []rune(" ")}
|
||||
case "left":
|
||||
msg = tea.KeyMsg{Type: tea.KeyLeft}
|
||||
case "esc":
|
||||
msg = tea.KeyMsg{Type: tea.KeyEsc}
|
||||
case "right":
|
||||
@@ -41,20 +47,28 @@ func press(m tea.Model, keys ...string) tea.Model {
|
||||
func TestTheSetupAsksForWhatHighAvailabilityNeedsAndRefusesWhatWillNotDo(t *testing.T) {
|
||||
m := tea.Model(newSetupModel("ctl", []string{"10.0.0.5"}))
|
||||
m = press(m, "down", "enter")
|
||||
if s := m.(setupModel); s.page != pAdmin {
|
||||
t.Fatalf("high availability did not lead to the administrator: page %d", s.page)
|
||||
if s := m.(setupModel); s.page != pChoices {
|
||||
t.Fatalf("high availability did not lead to the choices: page %d", s.page)
|
||||
}
|
||||
// Three machines, each a control plane, unless told otherwise.
|
||||
if v := m.View(); !strings.Contains(v, "Control plane on all three") || !strings.Contains(v, "two standbys") {
|
||||
t.Errorf("the standbys are not offered, ticked:\n%s", v)
|
||||
}
|
||||
// No email: refused, with a reason.
|
||||
m = press(m, "tab", "tab", "enter")
|
||||
if s := m.(setupModel); s.page != pAdmin || !strings.Contains(s.formErr, "email") {
|
||||
t.Fatalf("went on without an administrator: %+v", s.formErr)
|
||||
m = press(m, "up", "enter")
|
||||
if s := m.(setupModel); s.page != pChoices || !strings.Contains(s.choices.err, "email") {
|
||||
t.Fatalf("went on without an administrator: %+v", s.choices.err)
|
||||
}
|
||||
// The refusal is a dialog; a key closes it, and the cursor is back on
|
||||
// the email.
|
||||
// The refusal is a dialog; a key closes it, and the control plane's
|
||||
// settings open with the cursor on the email.
|
||||
if !strings.Contains(m.View(), "Not yet") {
|
||||
t.Error("the refusal was not shown as a dialog")
|
||||
}
|
||||
m = press(m, "x", "a@example.com", "tab", "tab", "enter")
|
||||
m = press(m, "q")
|
||||
if s := m.(setupModel); s.choices.dialog == nil || s.choices.dialog.Fields[s.choices.dialog.focus] != s.choices.admin {
|
||||
t.Fatal("the email was not given the cursor")
|
||||
}
|
||||
m = press(m, "a@example.com", "esc", "enter")
|
||||
s := m.(setupModel)
|
||||
if s.page != pMachines {
|
||||
t.Fatalf("page %d: %s", s.page, s.formErr)
|
||||
@@ -78,7 +92,7 @@ func TestTheSetupAsksForWhatHighAvailabilityNeedsAndRefusesWhatWillNotDo(t *test
|
||||
t.Errorf("a valid pair refused: %s", err)
|
||||
}
|
||||
cmd := s.command()
|
||||
for _, want := range []string{"--admin a@example.com", "--patroni ubuntu@10.0.0.6", "--patroni ubuntu@10.0.0.7", "--no-node"} {
|
||||
for _, want := range []string{"--admin a@example.com", "--patroni ubuntu@10.0.0.6", "--patroni ubuntu@10.0.0.7", "--no-node", "--standbys"} {
|
||||
if !strings.Contains(cmd, want) {
|
||||
t.Errorf("command lacks %q:\n%s", want, cmd)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,664 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
"go.yaml.in/yaml/v3"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// How people sign in, as the console shows and sets it: single sign-on
|
||||
// through Entra ID or another OpenID provider, an Active Directory checked
|
||||
// over LDAPS, and provisioning by SCIM. The secrets these need are kept in
|
||||
// the store, so every control plane has them without a file put on each,
|
||||
// and a directory is tried before its settings are kept.
|
||||
|
||||
// inChainRule is AD's matching rule for nested group membership.
|
||||
const inChainRule = "1.2.840.113556.1.4.1941"
|
||||
|
||||
type signInState struct {
|
||||
view adminapi.AuthView
|
||||
err error
|
||||
loaded bool
|
||||
// ldap and oidc are the blocks the cluster's settings hold now, for the
|
||||
// forms to start from.
|
||||
ldap config.LDAP
|
||||
oidc config.OIDC
|
||||
}
|
||||
|
||||
type consoleSignInMsg struct {
|
||||
view adminapi.AuthView
|
||||
ldap config.LDAP
|
||||
oidc config.OIDC
|
||||
err error
|
||||
}
|
||||
|
||||
// signInRows are the panel's rows, one per way in.
|
||||
var signInRows = []string{"oidc", "ldap", "scim"}
|
||||
|
||||
func (m consoleModel) fetchSignIn() tea.Cmd {
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
var msg consoleSignInMsg
|
||||
if msg.err = c.get(ctx, "/api/v1/auth", &msg.view); msg.err != nil {
|
||||
return msg
|
||||
}
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err == nil {
|
||||
msg.ldap, msg.oidc = authBlocks(cur.Body)
|
||||
}
|
||||
return msg
|
||||
}
|
||||
}
|
||||
|
||||
// authBlocks are auth.ldap and auth.oidc as a settings document has them.
|
||||
func authBlocks(body string) (config.LDAP, config.OIDC) {
|
||||
var doc struct {
|
||||
Auth struct {
|
||||
LDAP yaml.Node `yaml:"ldap"`
|
||||
OIDC yaml.Node `yaml:"oidc"`
|
||||
} `yaml:"auth"`
|
||||
}
|
||||
var l config.LDAP
|
||||
var o config.OIDC
|
||||
if yaml.Unmarshal([]byte(body), &doc) != nil {
|
||||
return l, o
|
||||
}
|
||||
if doc.Auth.LDAP.Kind != 0 {
|
||||
if b, err := yaml.Marshal(&doc.Auth.LDAP); err == nil {
|
||||
// Read as kept, whether or not it would validate now.
|
||||
l, _ = config.ParseLDAP(b)
|
||||
}
|
||||
}
|
||||
if doc.Auth.OIDC.Kind != 0 {
|
||||
if m, err := yaml.Marshal(&doc.Auth.OIDC); err == nil {
|
||||
if cfg, err := config.FromMap(map[string]any{"auth": map[string]any{"oidc": yamlMap(m)}}); err == nil {
|
||||
o = cfg.Auth.OIDC
|
||||
}
|
||||
}
|
||||
}
|
||||
return l, o
|
||||
}
|
||||
|
||||
func yamlMap(b []byte) map[string]any {
|
||||
var m map[string]any
|
||||
_ = yaml.Unmarshal(b, &m)
|
||||
return m
|
||||
}
|
||||
|
||||
func (m consoleModel) signInPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ row · Enter actions · E Entra ID · D Active Directory · T try the directory · ← back · Q leave"
|
||||
}
|
||||
s := m.signIn
|
||||
var b strings.Builder
|
||||
b.WriteString("\n " + sDim.Render("How people sign in besides a password kept here. E for Entra ID, D for Active Directory.") + "\n\n")
|
||||
if s.err != nil {
|
||||
b.WriteString(" " + sBad.Render(s.err.Error()))
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
if !s.loaded {
|
||||
b.WriteString(" " + sDim.Render("asking the control plane…"))
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
v := s.view
|
||||
local := "passwords kept here; accounts made by an administrator"
|
||||
if v.SignupOpen {
|
||||
local = "passwords kept here; anyone may sign up, then wait for approval"
|
||||
}
|
||||
b.WriteString(" " + sTitle.Render("LOCAL") + " " + sDim.Render(local) + "\n\n")
|
||||
|
||||
var rows [][]string
|
||||
for _, r := range signInRows {
|
||||
switch r {
|
||||
case "oidc":
|
||||
if v.OIDC == nil {
|
||||
rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", "off", "E to set up"})
|
||||
continue
|
||||
}
|
||||
state := "on"
|
||||
if v.OIDC.NoPasswords {
|
||||
state = "on, the only way in"
|
||||
}
|
||||
rows = append(rows, []string{"Single sign-on (Entra ID, OIDC)", state, or(v.OIDC.Label, v.OIDC.Issuer)})
|
||||
case "ldap":
|
||||
if v.LDAP == nil {
|
||||
rows = append(rows, []string{"Active Directory (LDAP)", "off", "D to set up"})
|
||||
continue
|
||||
}
|
||||
state := "on"
|
||||
if v.LDAP.Off != "" {
|
||||
state = "broken here"
|
||||
}
|
||||
rows = append(rows, []string{"Active Directory (LDAP)", state, strings.Join(v.LDAP.URLs, " ")})
|
||||
case "scim":
|
||||
state, detail := "off", "Enter to make a token"
|
||||
if v.SCIM {
|
||||
state, detail = "on", "at /scim/v2"
|
||||
}
|
||||
rows = append(rows, []string{"Provisioning (SCIM)", state, detail})
|
||||
}
|
||||
}
|
||||
b.WriteString(table([]string{"WAY IN", "STATE", ""}, rows, m.row, pw, m.inPanel) + "\n")
|
||||
|
||||
if v.OIDC != nil {
|
||||
b.WriteString("\n " + sTitle.Render("SINGLE SIGN-ON") + "\n")
|
||||
b.WriteString(" issuer " + v.OIDC.Issuer + "\n")
|
||||
b.WriteString(" redirect " + redirectURL(v.BaseURL) + "\n")
|
||||
b.WriteString(" " + sDim.Render(pickText(v.OIDC.CreateAccounts, "accounts made at first sign-in", "only people with an account sign in")) + "\n")
|
||||
}
|
||||
if l := v.LDAP; l != nil {
|
||||
b.WriteString("\n " + sTitle.Render("ACTIVE DIRECTORY") + " " + sDim.Render(l.BaseDN) + "\n")
|
||||
if l.Off != "" {
|
||||
b.WriteString(" " + sBad.Render("off on this control plane: "+l.Off) + "\n")
|
||||
}
|
||||
switch ls := l.LastSync; {
|
||||
case ls == nil:
|
||||
b.WriteString(" " + sDim.Render("not read yet: the leader reads it every sync_interval") + "\n")
|
||||
case ls.Error != "":
|
||||
b.WriteString(" " + sBad.Render("last read "+agoText(ls.At)+": "+ls.Error) + "\n")
|
||||
default:
|
||||
b.WriteString(fmt.Sprintf(" last read %s: %d people there, %d accounts kept in step, %d suspended, %d reinstated\n",
|
||||
agoText(ls.At), ls.People, ls.Accounts, ls.Suspended, ls.Reinstated))
|
||||
}
|
||||
}
|
||||
return "Sign-in", b.String(), keys
|
||||
}
|
||||
|
||||
func agoText(t time.Time) string { return time.Since(t).Round(time.Second).String() + " ago" }
|
||||
|
||||
func pickText(b bool, yes, no string) string {
|
||||
if b {
|
||||
return yes
|
||||
}
|
||||
return no
|
||||
}
|
||||
|
||||
// redirectURL is what an identity provider sends people back to.
|
||||
func redirectURL(base string) string {
|
||||
if base == "" {
|
||||
return "https://<this gateway>/login/oidc/callback"
|
||||
}
|
||||
return strings.TrimRight(base, "/") + "/login/oidc/callback"
|
||||
}
|
||||
|
||||
// signInMenu is what can be done to one way in.
|
||||
func (m consoleModel) signInMenu(row string) *menuDialog {
|
||||
v := m.signIn.view
|
||||
switch row {
|
||||
case "oidc":
|
||||
d := &menuDialog{title: "Single sign-on", items: []menuItem{
|
||||
{"Set up Entra ID…", func(m *consoleModel) tea.Cmd { m.entraForm(); return nil }},
|
||||
}}
|
||||
if v.OIDC != nil {
|
||||
d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("oidc", "Single sign-on", "People sign in with a password kept here again; accounts made through the provider have none.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
case "ldap":
|
||||
d := &menuDialog{title: "Active Directory", items: []menuItem{
|
||||
{"Set up…", func(m *consoleModel) tea.Cmd { m.directoryForm(); return nil }},
|
||||
}}
|
||||
if v.LDAP != nil {
|
||||
d.items = append(d.items,
|
||||
menuItem{"Try it", func(m *consoleModel) tea.Cmd { return m.tryDirectory() }},
|
||||
menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("ldap", "Active Directory", "Its people can no longer sign in; their accounts stay, and are no longer kept in step.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
default:
|
||||
d := &menuDialog{title: "Provisioning (SCIM)", items: []menuItem{
|
||||
{"Make a new token", func(m *consoleModel) tea.Cmd { return m.scimToken() }},
|
||||
}}
|
||||
if v.SCIM {
|
||||
d.items = append(d.items, menuItem{"Turn off…", func(m *consoleModel) tea.Cmd {
|
||||
m.signInOffConfirm("scim", "Provisioning", "The provider can no longer make, change or suspend accounts here.")
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return d
|
||||
}
|
||||
}
|
||||
|
||||
// signInRowKey is the way in the cursor is on.
|
||||
func (m consoleModel) signInRowKey() string {
|
||||
if m.row >= 0 && m.row < len(signInRows) {
|
||||
return signInRows[m.row]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// directoryForm sets up auth.ldap, starting from what is kept.
|
||||
func (m *consoleModel) directoryForm() {
|
||||
l := m.signIn.ldap
|
||||
allowed := ""
|
||||
if f := l.UserFilter; strings.HasPrefix(f, "(memberOf:"+inChainRule+":=") {
|
||||
allowed = strings.TrimSuffix(strings.TrimPrefix(f, "(memberOf:"+inChainRule+":="), ")")
|
||||
} else {
|
||||
allowed = f
|
||||
}
|
||||
pwHint := "the service account's password"
|
||||
if l.BindPasswordFile != "" {
|
||||
pwHint = "empty keeps the one kept"
|
||||
}
|
||||
caHint := "a PEM file on this machine; empty uses the system's"
|
||||
if l.CA != "" || l.CAFile != "" {
|
||||
caHint = "empty keeps the one kept; - for the system's"
|
||||
}
|
||||
create := "yes"
|
||||
if l.Enabled() && !l.CreateAccounts {
|
||||
create = "no"
|
||||
}
|
||||
var teams []string
|
||||
for g, place := range l.Memberships {
|
||||
teams = append(teams, g+"="+place)
|
||||
}
|
||||
pw := textF("password", "", pwHint)
|
||||
pw.text.Mask = true
|
||||
m.openForm(&formDialog{title: "Active Directory",
|
||||
note: "People sign in with their AD name and password, checked over LDAPS. Tried before it is kept.",
|
||||
fields: []formField{
|
||||
textF("servers", strings.Join(l.URLs, " "), "ldaps://dc1.corp.example ldaps://dc2.corp.example"),
|
||||
textF("people under", l.BaseDN, "OU=Staff,DC=corp,DC=example"),
|
||||
textF("service account", l.BindDN, "CN=svc-llmgw,OU=Service,DC=corp,DC=example"),
|
||||
pw,
|
||||
textF("CA certificate", "", caHint),
|
||||
textF("who may sign in", allowed, "a group's DN, nested groups count; empty is everyone under it"),
|
||||
textF("administrators", strings.Join(l.Roles["admin"], ", "), "group names or DNs"),
|
||||
textF("operators", strings.Join(l.Roles["operator"], ", "), "group names or DNs"),
|
||||
textF("teams", strings.Join(teams, "; "), "ml-team=team:Acme/ml; sales=org:Sales"),
|
||||
textF("label", l.Label, "the form says: Sign in with your LABEL account"),
|
||||
choiceF("make accounts", []string{"yes", "no"}, create),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
next, err := directoryFromForm(l, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.saveDirectory(next, v[3]), nil
|
||||
}})
|
||||
}
|
||||
|
||||
// directoryFromForm is the auth.ldap block the form's values make, over
|
||||
// what was kept.
|
||||
func directoryFromForm(kept config.LDAP, v []string) (config.LDAP, error) {
|
||||
l := kept
|
||||
l.URLs = strings.Fields(strings.ReplaceAll(v[0], ",", " "))
|
||||
l.BaseDN, l.BindDN = v[1], v[2]
|
||||
if v[3] != "" || l.BindPasswordFile == "" {
|
||||
l.BindPasswordFile = adminapi.AuthSecretRef("ldap-bind")
|
||||
}
|
||||
switch ca := v[4]; ca {
|
||||
case "":
|
||||
case "-":
|
||||
l.CA, l.CAFile = "", ""
|
||||
default:
|
||||
b, err := os.ReadFile(ca) //nolint:gosec // a file the operator names
|
||||
if err != nil {
|
||||
return l, fmt.Errorf("CA certificate: %w", err)
|
||||
}
|
||||
l.CA, l.CAFile = string(b), ""
|
||||
}
|
||||
switch g := v[5]; {
|
||||
case g == "":
|
||||
l.UserFilter = ""
|
||||
case strings.HasPrefix(g, "("):
|
||||
l.UserFilter = g
|
||||
case strings.Contains(g, "="):
|
||||
l.UserFilter = "(memberOf:" + inChainRule + ":=" + g + ")"
|
||||
default:
|
||||
return l, errors.New("who may sign in: a group's DN (CN=…,OU=…,DC=…), or an LDAP filter in parentheses")
|
||||
}
|
||||
l.Roles = map[string][]string{}
|
||||
for role, i := range map[string]int{"admin": 6, "operator": 7} {
|
||||
for _, g := range strings.Split(v[i], ",") {
|
||||
if g = strings.TrimSpace(g); g != "" {
|
||||
l.Roles[role] = append(l.Roles[role], g)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(l.Roles) == 0 {
|
||||
l.Roles = nil
|
||||
}
|
||||
l.Memberships = nil
|
||||
for _, pair := range strings.Split(v[8], ";") {
|
||||
if pair = strings.TrimSpace(pair); pair == "" {
|
||||
continue
|
||||
}
|
||||
g, place, ok := strings.Cut(pair, "=")
|
||||
if !ok {
|
||||
return l, fmt.Errorf("teams: %q is not GROUP=team:ORG/TEAM", pair)
|
||||
}
|
||||
if l.Memberships == nil {
|
||||
l.Memberships = map[string]string{}
|
||||
}
|
||||
l.Memberships[strings.TrimSpace(g)] = strings.TrimSpace(place)
|
||||
}
|
||||
l.Label, l.CreateAccounts = v[9], v[10] == "yes"
|
||||
return l, l.Validate()
|
||||
}
|
||||
|
||||
// ldapYAML is a block as the cluster's settings write it.
|
||||
func ldapYAML(l config.LDAP) string {
|
||||
m := map[string]any{"urls": l.URLs, "bind_dn": l.BindDN, "bind_password_file": l.BindPasswordFile,
|
||||
"base_dn": l.BaseDN, "create_accounts": l.CreateAccounts}
|
||||
put := func(k, v string) {
|
||||
if v != "" {
|
||||
m[k] = v
|
||||
}
|
||||
}
|
||||
put("ca", l.CA)
|
||||
put("ca_file", l.CAFile)
|
||||
put("user_filter", l.UserFilter)
|
||||
put("group_base_dn", l.GroupBaseDN)
|
||||
put("label", l.Label)
|
||||
if l.StartTLS {
|
||||
m["start_tls"] = true
|
||||
}
|
||||
if len(l.Roles) > 0 {
|
||||
m["roles"] = l.Roles
|
||||
}
|
||||
if len(l.Memberships) > 0 {
|
||||
m["memberships"] = l.Memberships
|
||||
}
|
||||
if l.SyncInterval != 0 {
|
||||
m["sync_interval"] = l.SyncInterval.String()
|
||||
}
|
||||
if l.Timeout != 0 {
|
||||
m["timeout"] = l.Timeout.String()
|
||||
}
|
||||
b, _ := yaml.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// saveDirectory tries the block, and only if every step passes keeps the
|
||||
// password and puts the block in the cluster's settings.
|
||||
func (m *consoleModel) saveDirectory(l config.LDAP, password string) tea.Cmd {
|
||||
m.busy = "Active Directory"
|
||||
c, block := m.client, ldapYAML(l)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
var tv adminapi.LDAPTestView
|
||||
if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test",
|
||||
adminapi.LDAPTestRequest{Settings: block, Password: password}, &tv); err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
report := checksText(tv.Checks)
|
||||
if !tv.OK {
|
||||
return actionDoneMsg{title: "Active Directory", err: errors.New("not kept: the directory did not pass\n\n" + report)}
|
||||
}
|
||||
if password != "" {
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/ldap-bind", adminapi.SecretRequest{Value: password}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.ldap from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "ldap"}, block)
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Active Directory", err: err}
|
||||
}
|
||||
return actionDoneMsg{title: "Active Directory", text: report + "\n\n" + said}
|
||||
}
|
||||
}
|
||||
|
||||
// tryDirectory tries the block in force.
|
||||
func (m *consoleModel) tryDirectory() tea.Cmd {
|
||||
m.busy = "Trying the directory"
|
||||
c, block := m.client, ldapYAML(m.signIn.ldap)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
var tv adminapi.LDAPTestView
|
||||
if err := c.do(ctx, http.MethodPost, "/api/v1/auth/ldap/test", adminapi.LDAPTestRequest{Settings: block}, &tv); err != nil {
|
||||
return actionDoneMsg{title: "The directory", err: err}
|
||||
}
|
||||
if !tv.OK {
|
||||
return actionDoneMsg{title: "The directory", err: errors.New("it did not pass\n\n" + checksText(tv.Checks))}
|
||||
}
|
||||
return actionDoneMsg{title: "The directory", text: checksText(tv.Checks)}
|
||||
}
|
||||
}
|
||||
|
||||
func checksText(checks []adminapi.CheckView) string {
|
||||
var b strings.Builder
|
||||
for i, c := range checks {
|
||||
if i > 0 {
|
||||
b.WriteString("\n")
|
||||
}
|
||||
mark := sOK.Render("✓")
|
||||
if !c.OK {
|
||||
mark = sBad.Render("✗")
|
||||
}
|
||||
fmt.Fprintf(&b, "%s %s: %s", mark, c.Step, c.Detail)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// entraForm sets up single sign-on through Entra ID, or another provider
|
||||
// when an issuer address is given in place of the tenant.
|
||||
func (m *consoleModel) entraForm() {
|
||||
o := m.signIn.oidc
|
||||
tenant := strings.TrimSuffix(strings.TrimPrefix(o.Issuer, "https://login.microsoftonline.com/"), "/v2.0")
|
||||
secretHint := "the app registration's client secret"
|
||||
if o.ClientSecretFile != "" {
|
||||
secretHint = "empty keeps the one kept"
|
||||
}
|
||||
create, only := "yes", "no"
|
||||
if o.Enabled() && !o.CreateAccounts {
|
||||
create = "no"
|
||||
}
|
||||
if o.NoPasswords {
|
||||
only = "yes"
|
||||
}
|
||||
secret := textF("client secret", "", secretHint)
|
||||
secret.text.Mask = true
|
||||
m.openForm(&formDialog{title: "Single sign-on: Entra ID",
|
||||
note: "Register an app in Entra with the redirect address " + redirectURL(m.signIn.view.BaseURL) +
|
||||
", and a groups claim of the groups assigned to it.",
|
||||
fields: []formField{
|
||||
textF("tenant", tenant, "the directory (tenant) ID; or another provider's issuer URL"),
|
||||
textF("client ID", o.ClientID, "the application (client) ID"),
|
||||
secret,
|
||||
textF("administrators", strings.Join(o.Roles["admin"], ", "), "group object IDs, as Entra sends them"),
|
||||
textF("operators", strings.Join(o.Roles["operator"], ", "), "group object IDs"),
|
||||
textF("label", or(o.Label, "Microsoft"), "the button says: Sign in with LABEL"),
|
||||
choiceF("make accounts", []string{"yes", "no"}, create),
|
||||
choiceF("only this way", []string{"no", "yes"}, only),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
next, err := oidcFromForm(o, v)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.saveOIDC(next, v[2]), nil
|
||||
}})
|
||||
}
|
||||
|
||||
func oidcFromForm(kept config.OIDC, v []string) (config.OIDC, error) {
|
||||
o := kept
|
||||
switch t := strings.TrimSpace(v[0]); {
|
||||
case t == "":
|
||||
return o, errors.New("tenant: the directory (tenant) ID, or an issuer URL")
|
||||
case strings.Contains(t, "://"):
|
||||
o.Issuer = t
|
||||
default:
|
||||
o.Issuer = "https://login.microsoftonline.com/" + t + "/v2.0"
|
||||
}
|
||||
if o.ClientID = v[1]; o.ClientID == "" {
|
||||
return o, errors.New("client ID: the application (client) ID")
|
||||
}
|
||||
if v[2] != "" || o.ClientSecretFile == "" {
|
||||
if v[2] == "" {
|
||||
return o, errors.New("client secret: the app registration's secret")
|
||||
}
|
||||
o.ClientSecretFile = adminapi.AuthSecretRef("oidc-client-secret")
|
||||
}
|
||||
o.Roles = map[string][]string{}
|
||||
for role, i := range map[string]int{"admin": 3, "operator": 4} {
|
||||
for _, g := range strings.Split(v[i], ",") {
|
||||
if g = strings.TrimSpace(g); g != "" {
|
||||
o.Roles[role] = append(o.Roles[role], g)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(o.Roles) == 0 {
|
||||
o.Roles = nil
|
||||
}
|
||||
o.Label, o.CreateAccounts, o.NoPasswords = v[5], v[6] == "yes", v[7] == "yes"
|
||||
return o, o.Validate()
|
||||
}
|
||||
|
||||
func oidcYAML(o config.OIDC) string {
|
||||
m := map[string]any{"issuer": o.Issuer, "client_id": o.ClientID, "client_secret_file": o.ClientSecretFile,
|
||||
"create_accounts": o.CreateAccounts}
|
||||
if o.Label != "" {
|
||||
m["label"] = o.Label
|
||||
}
|
||||
if o.NoPasswords {
|
||||
m["no_passwords"] = true
|
||||
}
|
||||
if o.RedirectURL != "" {
|
||||
m["redirect_url"] = o.RedirectURL
|
||||
}
|
||||
if len(o.Scopes) > 0 {
|
||||
m["scopes"] = o.Scopes
|
||||
}
|
||||
if o.GroupsClaim != "" {
|
||||
m["groups_claim"] = o.GroupsClaim
|
||||
}
|
||||
if len(o.Roles) > 0 {
|
||||
m["roles"] = o.Roles
|
||||
}
|
||||
if len(o.Memberships) > 0 {
|
||||
m["memberships"] = o.Memberships
|
||||
}
|
||||
b, _ := yaml.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func (m *consoleModel) saveOIDC(o config.OIDC, secret string) tea.Cmd {
|
||||
m.busy = "Single sign-on"
|
||||
c, block, redirect := m.client, oidcYAML(o), redirectURL(m.signIn.view.BaseURL)
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if secret != "" {
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/oidc-client-secret", adminapi.SecretRequest{Value: secret}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Single sign-on", err: err}
|
||||
}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.oidc from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "oidc"}, block)
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Single sign-on", err: err}
|
||||
}
|
||||
return actionDoneMsg{title: "Single sign-on", text: said + "\n\nIn the app registration the redirect address is " + redirect +
|
||||
". For people in more than 200 groups, emit the groups assigned to the application."}
|
||||
}
|
||||
}
|
||||
|
||||
// scimToken makes a new provisioning token, keeps it, and shows it once.
|
||||
func (m *consoleModel) scimToken() tea.Cmd {
|
||||
m.busy = "Provisioning"
|
||||
c, base := m.client, m.signIn.view.BaseURL
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
token := hex.EncodeToString(b)
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/auth/secrets/scim-token", adminapi.SecretRequest{Value: token}, nil); err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
said, err := changeClusterSettings(ctx, c, "auth.scim from the console", func(doc *settings.Doc) error {
|
||||
return doc.SetYAML([]string{"auth", "scim", "token_file"}, adminapi.AuthSecretRef("scim-token"))
|
||||
})
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: "Provisioning", err: err}
|
||||
}
|
||||
url := "https://<this gateway>/scim/v2"
|
||||
if base != "" {
|
||||
url = strings.TrimRight(base, "/") + "/scim/v2"
|
||||
}
|
||||
return actionDoneMsg{title: "Provisioning", text: "Give the provider the tenant URL " + url + " and this token, shown only now:\n\n " +
|
||||
token + "\n\n" + said}
|
||||
}
|
||||
}
|
||||
|
||||
// signInOffConfirm turns one way in off, after asking.
|
||||
func (m *consoleModel) signInOffConfirm(key, title, text string) {
|
||||
m.confirmD = &confirmDialog{title: "Turn off " + strings.ToLower(title), text: text,
|
||||
yes: func(m *consoleModel) tea.Cmd {
|
||||
m.busy = title
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
said, err := changeClusterSettings(ctx, c, "auth."+key+" off from the console", func(doc *settings.Doc) error {
|
||||
doc.Unset([]string{"auth", key})
|
||||
return nil
|
||||
})
|
||||
return actionDoneMsg{title: title, text: said, err: err}
|
||||
}
|
||||
}}
|
||||
}
|
||||
|
||||
// changeClusterSettings is changeSettings for the console: the newest
|
||||
// settings read, changed, and put in force, said in a sentence.
|
||||
func changeClusterSettings(ctx context.Context, c *ctlClient, note string, change func(doc *settings.Doc) error) (string, error) {
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return "", err
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := change(doc); err != nil {
|
||||
return "", err
|
||||
}
|
||||
body, err := doc.Bytes()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
before, _ := config.Parse([]byte(cur.Body))
|
||||
after, err := config.Parse(body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
changed := config.Changed(before, after)
|
||||
if len(changed) == 0 {
|
||||
return "Nothing changed.", nil
|
||||
}
|
||||
var v adminapi.SettingsView
|
||||
if err := c.do(ctx, http.MethodPut, "/api/v1/settings",
|
||||
adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note}, &v); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprintf("Settings version %d. Every control plane starts again on it within seconds, standbys first; "+
|
||||
"engines keep serving.", v.Version), nil
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// adForm is the directory form's values, in its order.
|
||||
func adForm(password, ca, allowed, teams string) []string {
|
||||
return []string{"ldaps://dc1.corp.example, ldaps://dc2.corp.example", "OU=Staff,DC=corp,DC=example",
|
||||
"CN=svc,OU=Service,DC=corp,DC=example", password, ca, allowed, "llmgw-admins", "", teams, "Acme", "yes"}
|
||||
}
|
||||
|
||||
func TestTheDirectoryFormMakesTheBlockTheSettingsKeep(t *testing.T) {
|
||||
pem := "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n"
|
||||
caFile := filepath.Join(t.TempDir(), "ca.pem")
|
||||
if err := os.WriteFile(caFile, []byte(pem), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, err := directoryFromForm(config.LDAP{}, adForm("pw", caFile, "CN=llmgw-users,OU=Groups,DC=corp,DC=example",
|
||||
"ml-team=team:Acme/ml; sales=org:Sales"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The group, nested ones counting; the CA written into the settings;
|
||||
// the password named in the store, not a file on one machine.
|
||||
if l.UserFilter != "(memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)" ||
|
||||
l.CA != pem || l.BindPasswordFile != "store:auth/ldap-bind" || len(l.URLs) != 2 ||
|
||||
l.Memberships["sales"] != "org:Sales" || l.Roles["admin"][0] != "llmgw-admins" || l.Roles["operator"] != nil {
|
||||
t.Errorf("%+v", l)
|
||||
}
|
||||
// What the settings keep reads back the same.
|
||||
back, err := config.ParseLDAP([]byte(ldapYAML(l)))
|
||||
if err != nil || back.UserFilter != l.UserFilter || back.CA != pem || back.Memberships["ml-team"] != "team:Acme/ml" {
|
||||
t.Errorf("read back %+v %v", back, err)
|
||||
}
|
||||
|
||||
// Changed again with nothing typed for the password or the CA: both kept.
|
||||
again, err := directoryFromForm(l, adForm("", "", "(objectClass=user)", ""))
|
||||
if err != nil || again.CA != pem || again.BindPasswordFile != l.BindPasswordFile || again.UserFilter != "(objectClass=user)" ||
|
||||
again.Memberships != nil {
|
||||
t.Errorf("kept: %+v %v", again, err)
|
||||
}
|
||||
// A group by its bare name cannot be a filter, and a plaintext server is refused.
|
||||
if _, err := directoryFromForm(config.LDAP{}, adForm("pw", "", "llmgw-users", "")); err == nil {
|
||||
t.Error("a bare group name was taken")
|
||||
}
|
||||
plain := adForm("pw", "", "", "")
|
||||
plain[0] = "ldap://dc1"
|
||||
if _, err := directoryFromForm(config.LDAP{}, plain); err == nil || !strings.Contains(err.Error(), "plaintext") {
|
||||
t.Errorf("plaintext: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheEntraFormMakesTheIssuerFromTheTenant(t *testing.T) {
|
||||
o, err := oidcFromForm(config.OIDC{}, []string{"0f2c-tenant", "app-id", "s3cret", "g-admins", "", "Microsoft", "yes", "no"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if o.Issuer != "https://login.microsoftonline.com/0f2c-tenant/v2.0" || o.ClientSecretFile != "store:auth/oidc-client-secret" ||
|
||||
o.Roles["admin"][0] != "g-admins" || !o.CreateAccounts || o.NoPasswords {
|
||||
t.Errorf("%+v", o)
|
||||
}
|
||||
// Another provider by its issuer; the secret kept when none is typed.
|
||||
k, err := oidcFromForm(o, []string{"https://sso.example/realms/acme", "llmgw", "", "", "", "Acme SSO", "no", "yes"})
|
||||
if err != nil || k.Issuer != "https://sso.example/realms/acme" || k.ClientSecretFile != o.ClientSecretFile || !k.NoPasswords {
|
||||
t.Errorf("%+v %v", k, err)
|
||||
}
|
||||
if _, err := oidcFromForm(config.OIDC{}, []string{"t", "app", "", "", "", "", "yes", "no"}); err == nil {
|
||||
t.Error("a first setup without a secret was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSignInPanelSaysWhatIsOnAndHowTheDirectoryLastWent(t *testing.T) {
|
||||
m := consoleModel{menu: cSignIn, loaded: true, width: 120, height: 40}
|
||||
m.signIn = signInState{loaded: true, view: adminapi.AuthView{
|
||||
BaseURL: "https://llm.example",
|
||||
OIDC: &adminapi.OIDCView{Issuer: "https://login.microsoftonline.com/t/v2.0", Label: "Microsoft", CreateAccounts: true},
|
||||
LDAP: &adminapi.LDAPView{URLs: []string{"ldaps://dc1"}, BaseDN: "OU=Staff,DC=corp",
|
||||
LastSync: &adminapi.DirectorySyncView{At: time.Now().Add(-time.Minute), People: 40, Accounts: 12, Suspended: 1}},
|
||||
}}
|
||||
_, body, _ := m.signInPanel(110)
|
||||
for _, want := range []string{"Single sign-on (Entra ID, OIDC)", "Microsoft", "https://llm.example/login/oidc/callback",
|
||||
"Active Directory (LDAP)", "ldaps://dc1", "40 people there, 12 accounts kept in step, 1 suspended", "Provisioning (SCIM)"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the panel lacks %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
m.signIn.view.LDAP.Off = "auth.ldap.ca: holds no certificate"
|
||||
if _, body, _ := m.signInPanel(110); !strings.Contains(body, "broken here") || !strings.Contains(body, "holds no certificate") {
|
||||
t.Errorf("a directory off here is not said:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuthBlocksAreReadFromTheSettingsDocument(t *testing.T) {
|
||||
body := "auth:\n ldap:\n urls: [ldaps://dc1]\n base_dn: DC=corp\n bind_dn: CN=svc\n bind_password_file: store:auth/ldap-bind\n" +
|
||||
" sync_interval: 1m\n oidc:\n issuer: https://sso.example\n client_id: llmgw\n"
|
||||
l, o := authBlocks(body)
|
||||
if l.BaseDN != "DC=corp" || l.SyncInterval != time.Minute || o.ClientID != "llmgw" {
|
||||
t.Errorf("%+v %+v", l, o)
|
||||
}
|
||||
}
|
||||
@@ -107,7 +107,7 @@ var Routes = []Route{
|
||||
{Method: "GET", Path: "/v1/memory/{key}", Summary: "A note", Response: MemoryNote{}},
|
||||
{Method: "GET", Path: "/v1/memory", Summary: "The project's notes", Response: MemoryList{}},
|
||||
{Method: "DELETE", Path: "/v1/memory/{key}", Summary: "Forget a note"},
|
||||
{Method: "POST", Path: "/mcp", Summary: "The gateway's MCP server: the memory notes as tools",
|
||||
{Method: "POST", Path: "/mcp", Summary: "The gateway's MCP server: the memory notes, and pictures, 3D models, vision and workflows, as tools",
|
||||
Request: MCPRequest{}, Response: anyJSON},
|
||||
{Method: "GET", Path: "/me", Summary: "The caller: its key, its place in the tree, its limits and what it has used",
|
||||
Response: Me{}},
|
||||
|
||||
@@ -222,6 +222,10 @@ func fillString(s string, p Params, missing *[]string) any {
|
||||
if f, err := strconv.ParseFloat(def, 64); err == nil {
|
||||
return f, true
|
||||
}
|
||||
// A BOOLEAN input refuses the word: {{albedo_match|false}}.
|
||||
if b, err := strconv.ParseBool(def); err == nil && (def == "true" || def == "false") {
|
||||
return b, true
|
||||
}
|
||||
return def, true
|
||||
}
|
||||
*missing = append(*missing, name)
|
||||
|
||||
@@ -252,3 +252,19 @@ func TestAPictureLosesTheWorkflowComfyUIWroteIntoIt(t *testing.T) {
|
||||
t.Error("a truncated PNG was changed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestATrueOrFalseDefaultIsABoolean(t *testing.T) {
|
||||
// A BOOLEAN input refuses the word "false": sf3d's albedo_match.
|
||||
tpl, err := Load((`{"5":{"class_type":"LLMGWSaveMesh","inputs":{"albedo_match":"{{albedo_match|false}}","count":"{{count|2}}","s":"{{s|x}}"}}}`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
in := g["5"].(map[string]any)["inputs"].(map[string]any)
|
||||
if in["albedo_match"] != false || in["count"] != 2.0 || in["s"] != "x" {
|
||||
t.Errorf("inputs %#v", in)
|
||||
}
|
||||
}
|
||||
+132
-5
@@ -20,6 +20,7 @@ import (
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
@@ -112,7 +113,15 @@ func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
return err
|
||||
}
|
||||
models := filepath.Join(state, "models")
|
||||
if err := LinkWeights(cfg.Weights, models); err != nil {
|
||||
shared, err := parseShared(os.Getenv(SharedModelsEnv))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
all := []string{cfg.Weights}
|
||||
for _, m := range shared {
|
||||
all = append(all, m.Path)
|
||||
}
|
||||
if err := LinkAllWeights(all, models); err != nil {
|
||||
return err
|
||||
}
|
||||
if repo := os.Getenv(HFRepoEnv); repo != "" {
|
||||
@@ -120,6 +129,20 @@ func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
return fmt.Errorf("showing the weights as %s: %w", repo, err)
|
||||
}
|
||||
}
|
||||
for _, m := range shared {
|
||||
if m.HFRepo == "" {
|
||||
continue
|
||||
}
|
||||
if err := LinkHFCache(m.Path, m.HFRepo, hfHubDir()); err != nil {
|
||||
return fmt.Errorf("showing %s's weights as %s: %w", m.Name, m.HFRepo, err)
|
||||
}
|
||||
}
|
||||
names := []string{cfg.Model}
|
||||
for _, m := range shared {
|
||||
if !slices.Contains(names, m.Name) {
|
||||
names = append(names, m.Name)
|
||||
}
|
||||
}
|
||||
extra := filepath.Join(state, "extra_model_paths.yaml")
|
||||
if err := os.WriteFile(extra, []byte(extraModelPaths(models)), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return err
|
||||
@@ -177,7 +200,7 @@ func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
go book.watch(ctx, upstream, clientID, log)
|
||||
srv := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: newGate(upstream, cfg.Key, cfg.Model, book, clientID),
|
||||
Handler: newGate(upstream, cfg.Key, names, book, clientID),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
served := make(chan error, 1)
|
||||
@@ -218,12 +241,12 @@ func shutdown(srv *http.Server) {
|
||||
|
||||
// NewGate is the gate's handler, in front of the ComfyUI at upstream.
|
||||
func NewGate(upstream *url.URL, key, model string) http.Handler {
|
||||
return newGate(upstream, key, model, newProgressBook(), "")
|
||||
return newGate(upstream, key, []string{model}, newProgressBook(), "")
|
||||
}
|
||||
|
||||
// newGate is the gate with the book of its workflows' progress, kept under
|
||||
// clientID; an empty one leaves the workflows' own client ids alone.
|
||||
func newGate(upstream *url.URL, key, model string, book *progressBook, clientID string) http.Handler {
|
||||
func newGate(upstream *url.URL, key string, names []string, book *progressBook, clientID string) http.Handler {
|
||||
proxy := httputil.NewSingleHostReverseProxy(upstream)
|
||||
base := proxy.Director
|
||||
proxy.Director = func(r *http.Request) {
|
||||
@@ -269,7 +292,11 @@ func newGate(upstream *url.URL, key, model string, book *progressBook, clientID
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprintf(w, `{"object":"list","data":[{"id":%q,"object":"model","owned_by":"comfyui"}]}`, model)
|
||||
data := make([]map[string]string, 0, len(names))
|
||||
for _, n := range names {
|
||||
data = append(data, map[string]string{"id": n, "object": "model", "owned_by": "comfyui"})
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data})
|
||||
return
|
||||
}
|
||||
if id, ok := strings.CutPrefix(r.URL.Path, "/progress/"); ok && r.Method == http.MethodGet {
|
||||
@@ -306,6 +333,106 @@ func FileName(path string) string {
|
||||
return cachedName.ReplaceAllString(filepath.Base(path), "")
|
||||
}
|
||||
|
||||
// SharedModelsEnv is where the agent hands a shared replica's gate its
|
||||
// members: name, the path their weights were fetched to, and the repository
|
||||
// a node loads them by name from.
|
||||
const SharedModelsEnv = "LLMGW_COMFYUI_MODELS"
|
||||
|
||||
// sharedModel is one member of a shared replica.
|
||||
type sharedModel struct {
|
||||
Name string `json:"name"`
|
||||
Path string `json:"path"`
|
||||
HFRepo string `json:"hf_repo"`
|
||||
}
|
||||
|
||||
func parseShared(raw string) ([]sharedModel, error) {
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
var ms []sharedModel
|
||||
if err := json.Unmarshal([]byte(raw), &ms); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", SharedModelsEnv, err)
|
||||
}
|
||||
for _, m := range ms {
|
||||
if m.Name == "" || m.Path == "" {
|
||||
return nil, fmt.Errorf("%s: a member with no name or no fetched path", SharedModelsEnv)
|
||||
}
|
||||
}
|
||||
return ms, nil
|
||||
}
|
||||
|
||||
// LinkAllWeights shows ComfyUI every model's weights at once, for a replica
|
||||
// that answers for several: links made before are taken away first, then
|
||||
// each file of each is linked by its name. Two weights files of one name
|
||||
// are refused — ComfyUI would load one for both; anything else of one name
|
||||
// (a README, a licence) the first keeps.
|
||||
func LinkAllWeights(weights []string, dir string) error {
|
||||
if len(weights) == 1 {
|
||||
return LinkWeights(weights[0], dir)
|
||||
}
|
||||
if err := LinkWeights(weights[0], dir); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, w := range weights[1:] {
|
||||
files, err := weightFiles(w)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, f := range files {
|
||||
link := filepath.Join(dir, FileName(f))
|
||||
if cur, err := os.Readlink(link); err == nil {
|
||||
if cur == f {
|
||||
continue
|
||||
}
|
||||
if isWeightsFile(f) {
|
||||
return fmt.Errorf("two models' weights are both named %s (%s and %s): give one comfyui.checkpoint of another name",
|
||||
FileName(f), cur, f)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := os.Symlink(f, link); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// weightFiles are the files a weights source is: itself, or each of its
|
||||
// directory's.
|
||||
func weightFiles(weights string) ([]string, error) {
|
||||
weights, err := filepath.Abs(weights)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
info, err := os.Stat(weights)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the weights: %w", err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return []string{weights}, nil
|
||||
}
|
||||
entries, err := os.ReadDir(weights)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []string
|
||||
for _, e := range entries {
|
||||
if e.Type().IsRegular() && !strings.HasPrefix(e.Name(), ".") {
|
||||
out = append(out, filepath.Join(weights, e.Name()))
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func isWeightsFile(path string) bool {
|
||||
switch strings.ToLower(filepath.Ext(path)) {
|
||||
case ".safetensors", ".ckpt", ".pt", ".pth", ".bin", ".gguf", ".sft":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// LinkWeights shows ComfyUI the weights: a link in dir to the file, or to
|
||||
// every file of the directory. Links made before are taken away first, so
|
||||
// a replica started again on other weights sees only those.
|
||||
|
||||
@@ -261,3 +261,41 @@ func TestLLMGWsOwnNodesArePutInEveryReplica(t *testing.T) {
|
||||
t.Fatalf("%v %.80s", err, b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASharedReplicaSeesEveryMembersWeights(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
write := func(p string) string {
|
||||
full := filepath.Join(root, p)
|
||||
_ = os.MkdirAll(filepath.Dir(full), 0o755)
|
||||
if err := os.WriteFile(full, []byte("w"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return full
|
||||
}
|
||||
flux := write("flux1-schnell-fp8.safetensors")
|
||||
write("sf3d/model.safetensors")
|
||||
write("sf3d/README.md")
|
||||
write("tripo/model.ckpt")
|
||||
write("tripo/README.md")
|
||||
dir := filepath.Join(root, "models")
|
||||
if err := LinkAllWeights([]string{flux, filepath.Join(root, "sf3d"), filepath.Join(root, "tripo")}, dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{"flux1-schnell-fp8.safetensors", "model.safetensors", "model.ckpt", "README.md"} {
|
||||
if _, err := os.Lstat(filepath.Join(dir, want)); err != nil {
|
||||
t.Errorf("%s not linked: %v", want, err)
|
||||
}
|
||||
}
|
||||
// Two weights of one name: ComfyUI would load one for both.
|
||||
write("other/model.safetensors")
|
||||
err := LinkAllWeights([]string{filepath.Join(root, "sf3d"), filepath.Join(root, "other")}, dir)
|
||||
if err == nil || !strings.Contains(err.Error(), "both named model.safetensors") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
if ms, err := parseShared(`[{"name":"sf3d","path":"/x","hf_repo":"stabilityai/stable-fast-3d"}]`); err != nil || ms[0].HFRepo == "" {
|
||||
t.Errorf("parse %+v %v", ms, err)
|
||||
}
|
||||
if _, err := parseShared(`[{"name":"sf3d"}]`); err == nil {
|
||||
t.Error("a member with no path was taken")
|
||||
}
|
||||
}
|
||||
@@ -94,22 +94,60 @@ class LLMGWPrepareForMesh:
|
||||
return (both[..., :3].contiguous(), both[..., 3].contiguous())
|
||||
|
||||
|
||||
def _to_linear(x):
|
||||
return np.where(x <= 0.04045, x / 12.92, ((x + 0.055) / 1.055) ** 2.4)
|
||||
|
||||
|
||||
def _to_srgb(x):
|
||||
return np.where(x <= 0.0031308, x * 12.92, 1.055 * np.power(np.clip(x, 0, None), 1 / 2.4) - 0.055)
|
||||
|
||||
|
||||
def _match_albedo(tex, reference, mask):
|
||||
"""The texture's brightness brought to the picture's: Stable Fast 3D
|
||||
takes the light out of what it sees, and a dark object it reads as a
|
||||
light one dimly lit — a basalt of 0.17 came back 0.67. Scaled in linear
|
||||
light by the ratio of the medians over the object, so its marks keep
|
||||
their contrast; texels nothing maps to (black) are left as they are."""
|
||||
t = np.asarray(tex.convert("RGB")).astype(np.float64) / 255.0
|
||||
used = t.max(-1) > 2 / 255
|
||||
ref = reference.cpu().numpy().astype(np.float64)
|
||||
obj = mask.cpu().numpy() > 0.5 if mask is not None else np.ones(ref.shape[:2], bool)
|
||||
if used.sum() < 16 or obj.sum() < 16:
|
||||
return tex
|
||||
lum = lambda a: a @ np.array([0.2126, 0.7152, 0.0722])
|
||||
want = np.median(lum(_to_linear(ref[obj])))
|
||||
have = np.median(lum(_to_linear(t[used])))
|
||||
if have <= 0:
|
||||
return tex
|
||||
k = float(np.clip(want / have, 0.05, 20.0))
|
||||
out = t.copy()
|
||||
out[used] = np.clip(_to_srgb(_to_linear(t[used]) * k), 0, 1)
|
||||
img = Image.fromarray((out * 255 + 0.5).astype(np.uint8))
|
||||
img.format = tex.format
|
||||
return img
|
||||
|
||||
|
||||
class LLMGWSaveMesh:
|
||||
"""Writes each mesh a 3D node made to output/ as a GLB and names the
|
||||
files, which is how the gate finds a result. A model's own save node
|
||||
may return the mesh as base64 (Stable Fast 3D's does) or as an OBJ
|
||||
without its texture, and the gate reads neither."""
|
||||
without its texture, and the gate reads neither.
|
||||
|
||||
With albedo_match and the picture the mesh was made from (and its
|
||||
mask), the mesh's base colour is brought to the picture's brightness."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"mesh": ("MESH",), "filename_prefix": ("STRING", {"default": "mesh"})}}
|
||||
return {"required": {"mesh": ("MESH",), "filename_prefix": ("STRING", {"default": "mesh"})},
|
||||
"optional": {"albedo_match": ("BOOLEAN", {"default": False}),
|
||||
"reference": ("IMAGE",), "reference_mask": ("MASK",)}}
|
||||
|
||||
RETURN_TYPES = ()
|
||||
OUTPUT_NODE = True
|
||||
FUNCTION = "save"
|
||||
CATEGORY = "llmgw"
|
||||
|
||||
def save(self, mesh, filename_prefix="mesh"):
|
||||
def save(self, mesh, filename_prefix="mesh", albedo_match=False, reference=None, reference_mask=None):
|
||||
import os
|
||||
import folder_paths
|
||||
import trimesh
|
||||
@@ -119,6 +157,9 @@ class LLMGWSaveMesh:
|
||||
filename_prefix, folder_paths.get_output_directory())
|
||||
saved = []
|
||||
for i, m in enumerate(mesh if isinstance(mesh, (list, tuple)) else [mesh]):
|
||||
if albedo_match and reference is not None:
|
||||
_match_scene(m, reference[min(i, reference.shape[0] - 1)],
|
||||
None if reference_mask is None else reference_mask[min(i, reference_mask.shape[0] - 1)])
|
||||
file = f"{name.replace('%batch_num%', str(i))}_{counter + i:05}_.glb"
|
||||
scene = m if isinstance(m, trimesh.Scene) else trimesh.Scene(m)
|
||||
with open(os.path.join(folder, file), "wb") as f:
|
||||
@@ -127,6 +168,16 @@ class LLMGWSaveMesh:
|
||||
return {"ui": {"mesh": saved}}
|
||||
|
||||
|
||||
def _match_scene(m, reference, mask):
|
||||
"""_match_albedo on every base colour texture of a mesh or scene."""
|
||||
geoms = m.geometry.values() if hasattr(m, "geometry") else [m]
|
||||
for g in geoms:
|
||||
mat = getattr(getattr(g, "visual", None), "material", None)
|
||||
tex = getattr(mat, "baseColorTexture", None)
|
||||
if tex is not None:
|
||||
mat.baseColorTexture = _match_albedo(tex, reference, mask)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LLMGWPrepareForMesh": LLMGWPrepareForMesh, "LLMGWSaveMesh": LLMGWSaveMesh}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LLMGWPrepareForMesh": "Prepare a picture for a 3D model (llmgw)",
|
||||
"LLMGWSaveMesh": "Save a mesh as GLB (llmgw)"}
|
||||
@@ -60,7 +60,7 @@ func TestTheGateHearsTheWorkflowsItLetsThrough(t *testing.T) {
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
book := newProgressBook()
|
||||
gate := httptest.NewServer(newGate(u, "", "m", book, "llmgw-gate-1"))
|
||||
gate := httptest.NewServer(newGate(u, "", []string{"m"}, book, "llmgw-gate-1"))
|
||||
defer gate.Close()
|
||||
resp, err := http.Post(gate.URL+"/prompt", "application/json", strings.NewReader(`{"prompt":{},"client_id":"router"}`))
|
||||
if err != nil {
|
||||
|
||||
@@ -158,6 +158,13 @@ type Server struct {
|
||||
// Firewall turns llmgw's firewall on every machine on or off.
|
||||
Firewall FirewallSettings `koanf:"firewall"`
|
||||
|
||||
// Presets is how the models that come with llmgw reach the catalog.
|
||||
Presets PresetSettings `koanf:"presets"`
|
||||
|
||||
// OnDemand is how models that load when asked for take turns on the
|
||||
// cards.
|
||||
OnDemand OnDemandSettings `koanf:"on_demand"`
|
||||
|
||||
// Agents are the nodes' agent settings: agent.yaml's keys but those an
|
||||
// agent needs to start, for every node (default) and for one
|
||||
// (nodes.<id>). Sent to the agents as they change; an agent whose
|
||||
@@ -435,6 +442,9 @@ type Auth struct {
|
||||
|
||||
// OIDC is sign-in through an identity provider.
|
||||
OIDC OIDC `koanf:"oidc"`
|
||||
// LDAP is sign-in against a directory, Active Directory above all, and
|
||||
// the accounts kept in step with it.
|
||||
LDAP LDAP `koanf:"ldap"`
|
||||
// SCIM is account provisioning by the identity provider.
|
||||
SCIM SCIM `koanf:"scim"`
|
||||
|
||||
@@ -493,6 +503,10 @@ type Personal struct {
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// CachedPer1k is the cost of a prompt token the engine or provider read
|
||||
// from its cache (the answer's cached_tokens, counted within the
|
||||
// prompt's). Zero charges them at prompt_per_1k.
|
||||
CachedPer1k int64 `koanf:"cached_per_1k"`
|
||||
// PerImage is the cost of one picture, for a model that makes them
|
||||
// (/v1/images/generations), on top of its prompt's tokens.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
@@ -543,7 +557,7 @@ type Control struct {
|
||||
// Only meaningful with a shared store, where a second control plane can
|
||||
// see the claim. Long enough to survive a slow query or a brief network
|
||||
// stall, short enough that a standby takes over before anyone has finished
|
||||
// reading the alert. Zero takes the default.
|
||||
// reading the alert. Zero takes the default, 10s.
|
||||
LeaseTTL time.Duration `koanf:"lease_ttl"`
|
||||
}
|
||||
|
||||
@@ -784,6 +798,11 @@ type Cache struct {
|
||||
// MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
// unless set.
|
||||
MaxEntryBytes int64 `koanf:"max_entry_bytes"`
|
||||
// EmbeddingModel embeds the questions of projects that reuse answers to
|
||||
// similar ones (llmgw keys cache --similar): an embedding model in the
|
||||
// catalog. Unset, those projects reuse only the same request's answer.
|
||||
// The gateway asks it on its own account, as it asks a judge.
|
||||
EmbeddingModel string `koanf:"embedding_model"`
|
||||
}
|
||||
|
||||
// Default returns the configuration used when nothing is specified.
|
||||
@@ -795,6 +814,7 @@ func Default() Server {
|
||||
MaxConnections: 10_000,
|
||||
Mirror: Mirror{UpdateCheck: 6 * time.Hour},
|
||||
Firewall: FirewallSettings{Enabled: true},
|
||||
Presets: PresetSettings{Auto: true},
|
||||
Control: Control{
|
||||
Interval: 5 * time.Second,
|
||||
BaseBackoff: 10 * time.Second,
|
||||
@@ -960,6 +980,12 @@ func (s Server) Validate() error {
|
||||
if err := s.Auth.OIDC.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.Auth.LDAP.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if s.Auth.LDAP.Enabled() && !s.Auth.Enabled {
|
||||
return fmt.Errorf("auth.ldap needs auth.enabled: it signs people in to the accounts the store keeps")
|
||||
}
|
||||
if err := s.AgentTLS.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/knadh/koanf/parsers/yaml"
|
||||
"github.com/knadh/koanf/v2"
|
||||
)
|
||||
|
||||
// LDAP is sign-in against a directory — Active Directory above all — with
|
||||
// the directory's own passwords, and the accounts kept in step with it.
|
||||
//
|
||||
// auth:
|
||||
// ldap:
|
||||
// urls: [ldaps://dc1.corp.example.com, ldaps://dc2.corp.example.com]
|
||||
// bind_dn: CN=llmgw,OU=Service Accounts,DC=corp,DC=example,DC=com
|
||||
// bind_password_file: /etc/llmgw/secrets/ldap-bind
|
||||
// base_dn: OU=Staff,DC=corp,DC=example,DC=com
|
||||
// create_accounts: true
|
||||
// roles: {admin: [llmgw-admins]}
|
||||
// memberships: {ml-team: "team:Acme/ml"}
|
||||
//
|
||||
// A person signs in on the portal's form with their sAMAccountName, their
|
||||
// userPrincipalName or their mail, and their directory password. The
|
||||
// gateway finds them with the bind account, checks the password by binding
|
||||
// as them, and reads their groups, nested ones included. Every sync_interval
|
||||
// the leader reads the directory again: an account disabled or gone there
|
||||
// is suspended here, its sessions ended and its API keys refused.
|
||||
type LDAP struct {
|
||||
// URLs are the directory's servers, tried in turn: ldaps:// (636), or
|
||||
// ldap:// (389) with StartTLS.
|
||||
URLs []string `koanf:"urls"`
|
||||
// StartTLS upgrades an ldap:// connection before anything is sent. A
|
||||
// password never crosses a connection without TLS: ldap:// without it
|
||||
// is refused.
|
||||
StartTLS bool `koanf:"start_tls"`
|
||||
// CAFile is the authority the servers' certificates are checked against,
|
||||
// when it is not the system's: an AD Certificate Services root.
|
||||
CAFile string `koanf:"ca_file"`
|
||||
// CA is the same authority written out, PEM: kept in the cluster's
|
||||
// settings, so no file has to be put on every control plane. Used in
|
||||
// place of CAFile when both are set.
|
||||
CA string `koanf:"ca"`
|
||||
// BindDN and the file holding its password: a service account that may
|
||||
// read users and groups. Nothing is written to the directory.
|
||||
BindDN string `koanf:"bind_dn"`
|
||||
BindPasswordFile string `koanf:"bind_password_file"`
|
||||
// BaseDN is where people are searched for, subtree included.
|
||||
BaseDN string `koanf:"base_dn"`
|
||||
// GroupBaseDN is where their groups are; empty is BaseDN's domain root.
|
||||
GroupBaseDN string `koanf:"group_base_dn"`
|
||||
// UserFilter narrows who may sign in, ANDed with the login match: a
|
||||
// group (memberOf=...) or an OU's people only. Empty is every enabled
|
||||
// user object.
|
||||
UserFilter string `koanf:"user_filter"`
|
||||
// Label is what the form says to sign in with: "Sign in with your
|
||||
// <label> account". Empty is "Active Directory".
|
||||
Label string `koanf:"label"`
|
||||
// CreateAccounts makes an account, active, for a person the directory
|
||||
// vouches for and the gateway does not know. Off, only existing accounts
|
||||
// sign in this way.
|
||||
CreateAccounts bool `koanf:"create_accounts"`
|
||||
// Roles maps a staff role (viewer, operator, billing, admin) to the
|
||||
// groups that give it, by name (CN) or whole DN. Set at every sign-in
|
||||
// and sync: someone taken out of the group loses the role.
|
||||
Roles map[string][]string `koanf:"roles"`
|
||||
// Memberships maps a group, by CN or DN, to a place in the tree —
|
||||
// org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
|
||||
// and kept by the sync, which also takes away a place whose group the
|
||||
// person has left.
|
||||
Memberships map[string]string `koanf:"memberships"`
|
||||
// SyncInterval is how often the leader reads the directory. Zero is
|
||||
// 15m; negative turns the sync off, leaving only sign-in.
|
||||
SyncInterval time.Duration `koanf:"sync_interval"`
|
||||
// Timeout bounds each round trip to a server. Zero is 10s.
|
||||
Timeout time.Duration `koanf:"timeout"`
|
||||
}
|
||||
|
||||
// Enabled reports whether directory sign-in is configured.
|
||||
func (l LDAP) Enabled() bool { return len(l.URLs) > 0 }
|
||||
|
||||
// Validate says what in the configuration cannot work.
|
||||
func (l LDAP) Validate() error {
|
||||
if !l.Enabled() {
|
||||
return nil
|
||||
}
|
||||
for _, raw := range l.URLs {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil || u.Host == "" {
|
||||
return fmt.Errorf("auth.ldap.urls: %q is not an ldap:// or ldaps:// address", raw)
|
||||
}
|
||||
switch u.Scheme {
|
||||
case "ldaps":
|
||||
case "ldap":
|
||||
if !l.StartTLS {
|
||||
return fmt.Errorf("auth.ldap.urls: %q is plaintext: use ldaps://, or set start_tls; passwords would cross it", raw)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("auth.ldap.urls: %q is not an ldap:// or ldaps:// address", raw)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case strings.TrimSpace(l.BindDN) == "" || l.BindPasswordFile == "":
|
||||
return errors.New("auth.ldap: bind_dn and bind_password_file are needed, for the account that finds people")
|
||||
case l.CA != "" && !strings.Contains(l.CA, "-----BEGIN CERTIFICATE-----"):
|
||||
return errors.New("auth.ldap.ca: not a PEM certificate")
|
||||
case strings.TrimSpace(l.BaseDN) == "":
|
||||
return errors.New("auth.ldap: base_dn is needed, where people are searched for")
|
||||
case l.UserFilter != "" && !(strings.HasPrefix(l.UserFilter, "(") && strings.HasSuffix(l.UserFilter, ")")):
|
||||
return fmt.Errorf("auth.ldap.user_filter: %q is not an LDAP filter in parentheses", l.UserFilter)
|
||||
}
|
||||
for role := range l.Roles {
|
||||
if !validStaffRole(role) {
|
||||
return fmt.Errorf("auth.ldap.roles: %q is not viewer, operator, billing or admin", role)
|
||||
}
|
||||
}
|
||||
for g, place := range l.Memberships {
|
||||
if !validPlace(place) {
|
||||
return fmt.Errorf("auth.ldap.memberships[%s]: %q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", g, place)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Interval is SyncInterval with its default; zero means off.
|
||||
func (l LDAP) Interval() time.Duration {
|
||||
switch {
|
||||
case l.SyncInterval < 0:
|
||||
return 0
|
||||
case l.SyncInterval == 0:
|
||||
return 15 * time.Minute
|
||||
}
|
||||
return l.SyncInterval
|
||||
}
|
||||
|
||||
// RoundTrip is Timeout with its default.
|
||||
func (l LDAP) RoundTrip() time.Duration {
|
||||
if l.Timeout <= 0 {
|
||||
return 10 * time.Second
|
||||
}
|
||||
return l.Timeout
|
||||
}
|
||||
|
||||
// ParseLDAP is an auth.ldap block, as YAML, read on its own as the cluster's
|
||||
// settings would read it, and checked.
|
||||
func ParseLDAP(block []byte) (LDAP, error) {
|
||||
var l LDAP
|
||||
k := koanf.New(".")
|
||||
if err := k.Load(bytesProvider(block), yaml.Parser()); err != nil {
|
||||
return l, fmt.Errorf("auth.ldap: %w", err)
|
||||
}
|
||||
if err := k.UnmarshalWithConf("", &l, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
|
||||
return l, fmt.Errorf("auth.ldap: %w", err)
|
||||
}
|
||||
return l, l.Validate()
|
||||
}
|
||||
+18
-5
@@ -72,21 +72,34 @@ func (o OIDC) Validate() error {
|
||||
return errors.New("auth.oidc.client_id is required")
|
||||
}
|
||||
for role := range o.Roles {
|
||||
switch role {
|
||||
case "viewer", "operator", "billing", "admin":
|
||||
default:
|
||||
if !validStaffRole(role) {
|
||||
return fmt.Errorf("auth.oidc.roles: %q is not viewer, operator, billing or admin", role)
|
||||
}
|
||||
}
|
||||
for g, where := range o.Memberships {
|
||||
kind, path, ok := strings.Cut(where, ":")
|
||||
if !ok || path == "" || (kind != "org" && kind != "team" && kind != "project") {
|
||||
if !validPlace(where) {
|
||||
return fmt.Errorf("auth.oidc.memberships.%s is %q: org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", g, where)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// validStaffRole is a role groups may give: viewer, operator, billing, admin.
|
||||
func validStaffRole(role string) bool {
|
||||
switch role {
|
||||
case "viewer", "operator", "billing", "admin":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// validPlace is a place in the tree a group may give: org:NAME,
|
||||
// team:ORG/TEAM or project:ORG/TEAM/PROJECT.
|
||||
func validPlace(where string) bool {
|
||||
kind, path, ok := strings.Cut(where, ":")
|
||||
return ok && path != "" && (kind == "org" || kind == "team" || kind == "project")
|
||||
}
|
||||
|
||||
// SCIM lets an identity provider provision accounts (/scim/v2): made,
|
||||
// suspended and reinstated as they are there, with groups mapped by
|
||||
// auth.oidc's roles and memberships.
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
package config
|
||||
|
||||
import "time"
|
||||
|
||||
// OnDemandSettings is how models that load when asked for take turns on the
|
||||
// cards: on a site with fewer cards than such models, the wait between one
|
||||
// and the next.
|
||||
type OnDemandSettings struct {
|
||||
// EvictAfterIdle is how long an on-demand replica must go unasked-for
|
||||
// before a model that wants its card may have it. Zero keeps 2m.
|
||||
EvictAfterIdle time.Duration `koanf:"evict_after_idle"`
|
||||
// MinLifetime is how long a replica that has just come up is safe from
|
||||
// that, so two models do not take turns paying for loads. Zero keeps
|
||||
// 5m. On one card shared by a picture model and a 3D model, 1m each
|
||||
// brings the wait between a picture and its mesh to about a minute.
|
||||
// Both are in force on the leader within seconds.
|
||||
MinLifetime time.Duration `koanf:"min_lifetime"`
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package config
|
||||
|
||||
// PresetSettings is how the models that come with llmgw reach the catalog.
|
||||
type PresetSettings struct {
|
||||
// Auto has the leading control plane add a picture model (flux-schnell)
|
||||
// and a 3D model (triposr) by itself, on demand, once a node has a card
|
||||
// that fits one and the catalog has no model that makes the same kind of
|
||||
// thing — so a new cluster's portal has its pictures and 3D pages
|
||||
// without anybody writing a definition. Only where the cluster already
|
||||
// installs ComfyUI (engines.comfyui). A preset somebody removed is not
|
||||
// added again by itself. In force on the leader within a minute.
|
||||
Auto bool `koanf:"auto"`
|
||||
}
|
||||
@@ -76,16 +76,22 @@ type API struct {
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
settings *Settings
|
||||
auth *AuthAdmin
|
||||
mcp MCPStore
|
||||
// restartAgents tells every connected agent to start again.
|
||||
restartAgents func(reason string) []string
|
||||
// revoke records a removed node, so its certificate is refused.
|
||||
revoke func(ctx context.Context, nodeID string, at time.Time) error
|
||||
|
||||
models Models
|
||||
capacity Capacity
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
models Models
|
||||
// presets keeps what became of each preset, presetCfg reads the
|
||||
// settings in force; nil without a store.
|
||||
presets PresetStore
|
||||
presetCfg func() PresetConfig
|
||||
capacity Capacity
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
}
|
||||
|
||||
// New builds the API over a controller.
|
||||
@@ -125,6 +131,8 @@ func (a *API) Register(mux Mux) {
|
||||
a.registerTrain(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
a.registerAuth(mux)
|
||||
a.registerMCP(mux)
|
||||
mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
|
||||
mux.HandleFunc("GET /api/v1/audit", a.auditEvents)
|
||||
mux.HandleFunc("GET /api/v1/usage/export", a.usageExport)
|
||||
@@ -166,6 +174,7 @@ func (a *API) Register(mux Mux) {
|
||||
}
|
||||
writeJSON(w, http.StatusOK, t)
|
||||
})
|
||||
a.registerPresets(mux)
|
||||
mux.HandleFunc("GET /api/v1/models", a.listModels)
|
||||
mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
|
||||
// Listing is a read and deleting is a separate verb, so a mistyped
|
||||
@@ -1229,6 +1238,8 @@ func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
// A preset removed is one the leader must not put back by itself.
|
||||
a.presetDeclined(r.Context(), name, actorOf(r))
|
||||
writeJSON(w, http.StatusOK, Removed{Removed: name})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AuthAdmin is what the sign-in endpoints need: how people sign in now, a
|
||||
// directory's settings tried before they are kept, and where the secrets
|
||||
// those settings name are kept for every control plane.
|
||||
type AuthAdmin struct {
|
||||
Status func() AuthView
|
||||
// TestLDAP tries an auth.ldap block, as YAML, binding with password
|
||||
// when one is given rather than the one the block names.
|
||||
TestLDAP func(ctx context.Context, settings, password string) ([]CheckView, error)
|
||||
Secrets interface {
|
||||
SetClusterSecret(ctx context.Context, name, value string) error
|
||||
}
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// SetAuth enables the sign-in endpoints.
|
||||
func (a *API) SetAuth(x *AuthAdmin) { a.auth = x }
|
||||
|
||||
// AuthView is how people sign in on this gateway.
|
||||
type AuthView struct {
|
||||
SignupOpen bool `json:"signup_open"`
|
||||
// OIDC is single sign-on through an identity provider: Entra ID,
|
||||
// Keycloak, Okta.
|
||||
OIDC *OIDCView `json:"oidc,omitempty"`
|
||||
// LDAP is sign-in checked by a directory: Active Directory.
|
||||
LDAP *LDAPView `json:"ldap,omitempty"`
|
||||
// SCIM is whether a provider may keep accounts in step.
|
||||
SCIM bool `json:"scim"`
|
||||
// BaseURL is the portal's address, when the settings say it: what a
|
||||
// provider's redirect address is made from.
|
||||
BaseURL string `json:"base_url,omitempty"`
|
||||
}
|
||||
|
||||
// OIDCView is auth.oidc as it is in force.
|
||||
type OIDCView struct {
|
||||
Issuer string `json:"issuer"`
|
||||
ClientID string `json:"client_id"`
|
||||
Label string `json:"label,omitempty"`
|
||||
CreateAccounts bool `json:"create_accounts"`
|
||||
NoPasswords bool `json:"no_passwords"`
|
||||
}
|
||||
|
||||
// LDAPView is auth.ldap as it is in force, and how its last sync went.
|
||||
type LDAPView struct {
|
||||
URLs []string `json:"urls"`
|
||||
BaseDN string `json:"base_dn"`
|
||||
BindDN string `json:"bind_dn"`
|
||||
Label string `json:"label,omitempty"`
|
||||
// Off is why this control plane could not use the settings, when it
|
||||
// could not; local accounts still sign in.
|
||||
Off string `json:"off,omitempty"`
|
||||
LastSync *DirectorySyncView `json:"last_sync,omitempty"`
|
||||
}
|
||||
|
||||
// DirectorySyncView is the last time the leader read the directory.
|
||||
type DirectorySyncView struct {
|
||||
At time.Time `json:"at"`
|
||||
People int `json:"people"`
|
||||
Accounts int `json:"accounts"`
|
||||
Suspended int `json:"suspended"`
|
||||
Reinstated int `json:"reinstated"`
|
||||
RolesChanged int `json:"roles_changed"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// CheckView is one step of trying a directory.
|
||||
type CheckView struct {
|
||||
Step string `json:"step"`
|
||||
OK bool `json:"ok"`
|
||||
Detail string `json:"detail"`
|
||||
}
|
||||
|
||||
// LDAPTestRequest is an auth.ldap block to try, as YAML, and the service
|
||||
// account's password when it is not kept yet.
|
||||
type LDAPTestRequest struct {
|
||||
Settings string `json:"settings"`
|
||||
Password string `json:"password,omitempty"`
|
||||
}
|
||||
|
||||
// LDAPTestView is how trying the directory went.
|
||||
type LDAPTestView struct {
|
||||
OK bool `json:"ok"`
|
||||
Checks []CheckView `json:"checks"`
|
||||
}
|
||||
|
||||
// SecretRequest is a secret for the store to keep.
|
||||
type SecretRequest struct {
|
||||
Value string `json:"value"`
|
||||
}
|
||||
|
||||
// authSecrets are the secrets an operator may give through the API, and
|
||||
// the reference the settings then name each by.
|
||||
var authSecrets = map[string]bool{"ldap-bind": true, "oidc-client-secret": true, "scim-token": true}
|
||||
|
||||
// AuthSecretRef is the reference auth's settings name a kept secret by.
|
||||
func AuthSecretRef(name string) string { return "store:auth/" + name }
|
||||
|
||||
func (a *API) registerAuth(mux Mux) {
|
||||
need := func(w http.ResponseWriter) bool {
|
||||
if a.auth == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no accounts")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
mux.HandleFunc("GET /api/v1/auth", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.auth.Status())
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/auth/ldap/test", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
var req LDAPTestRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
checks, err := a.auth.TestLDAP(r.Context(), req.Settings, req.Password)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
v := LDAPTestView{OK: len(checks) > 0, Checks: checks}
|
||||
for _, c := range checks {
|
||||
v.OK = v.OK && c.OK
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/auth/secrets/{name}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !need(w) {
|
||||
return
|
||||
}
|
||||
name := r.PathValue("name")
|
||||
if !authSecrets[name] {
|
||||
writeError(w, http.StatusNotFound, "no such secret: ldap-bind, oidc-client-secret or scim-token")
|
||||
return
|
||||
}
|
||||
var req SecretRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.Value) == "" {
|
||||
writeError(w, http.StatusBadRequest, "an empty secret")
|
||||
return
|
||||
}
|
||||
if err := a.auth.Secrets.SetClusterSecret(r.Context(), "auth/"+name, req.Value); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
if a.auth.Log != nil {
|
||||
// What changed and who changed it; never the value.
|
||||
a.auth.Log.Info("a sign-in secret was set", "secret", name, "by", actorOf(r))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"ref": AuthSecretRef(name)})
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type memSecrets map[string]string
|
||||
|
||||
func (m memSecrets) SetClusterSecret(_ context.Context, name, value string) error {
|
||||
m[name] = value
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestSignInIsShownTriedAndItsSecretsKept(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
kept := memSecrets{}
|
||||
var tried string
|
||||
a.SetAuth(&AuthAdmin{
|
||||
Status: func() AuthView { return AuthView{LDAP: &LDAPView{URLs: []string{"ldaps://dc1"}}} },
|
||||
TestLDAP: func(_ context.Context, settings, password string) ([]CheckView, error) {
|
||||
tried = settings + "|" + password
|
||||
return []CheckView{{Step: "ldaps://dc1", OK: true}, {Step: "people", OK: false, Detail: "no one"}}, nil
|
||||
},
|
||||
Secrets: kept,
|
||||
})
|
||||
var v AuthView
|
||||
if code := call(t, a, "GET", "/api/v1/auth", "", &v); code != 200 || v.LDAP == nil {
|
||||
t.Fatalf("status: %d %+v", code, v)
|
||||
}
|
||||
// One step failing fails the whole.
|
||||
var tv LDAPTestView
|
||||
if code := call(t, a, "POST", "/api/v1/auth/ldap/test", `{"settings":"urls: [ldaps://dc1]","password":"pw"}`, &tv); code != 200 ||
|
||||
tv.OK || len(tv.Checks) != 2 || tried != "urls: [ldaps://dc1]|pw" {
|
||||
t.Errorf("test: %d %+v %q", code, tv, tried)
|
||||
}
|
||||
var ref map[string]string
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":"s3cret"}`, &ref); code != 200 ||
|
||||
kept["auth/ldap-bind"] != "s3cret" || ref["ref"] != "store:auth/ldap-bind" {
|
||||
t.Errorf("secret: %d %v %v", code, kept, ref)
|
||||
}
|
||||
// Only the sign-in secrets, and never an empty one.
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/engine-key", `{"value":"x"}`, nil); code != 404 {
|
||||
t.Errorf("another secret: %d", code)
|
||||
}
|
||||
if code := call(t, a, "PUT", "/api/v1/auth/secrets/ldap-bind", `{"value":" "}`, nil); code != 400 {
|
||||
t.Errorf("an empty secret: %d", code)
|
||||
}
|
||||
if strings.Contains(kept["auth/ldap-bind"], " ") {
|
||||
t.Error("an empty secret replaced the kept one")
|
||||
}
|
||||
}
|
||||
@@ -209,6 +209,7 @@ type GuardAttachmentRequest struct {
|
||||
func (a *API) registerGuard(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/guard/judges", a.listJudges)
|
||||
mux.HandleFunc("POST /api/v1/guard/judges/ask", a.askJudge)
|
||||
mux.HandleFunc("POST /api/v1/guard/judges/check", a.checkJudge)
|
||||
mux.HandleFunc("GET /api/v1/guard", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.guard == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
@@ -30,6 +31,9 @@ type JudgeView struct {
|
||||
Judge bool `json:"judge"`
|
||||
Ready int `json:"ready_replicas"`
|
||||
Wanted int `json:"wanted_replicas"`
|
||||
// BudgetMs is how long a request waits for it: the shortest of its
|
||||
// policies' latency budgets.
|
||||
BudgetMs int64 `json:"budget_ms,omitempty"`
|
||||
// LastError is its last failure to give a verdict, if any.
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
LastErrorAt *time.Time `json:"last_error_at,omitempty"`
|
||||
@@ -65,6 +69,9 @@ func (a *API) Judges() []JudgeView {
|
||||
if !slices.Contains(v.Policies, name) {
|
||||
v.Policies = append(v.Policies, name)
|
||||
}
|
||||
if b := p.Judge.Budget().Milliseconds(); v.BudgetMs == 0 || b < v.BudgetMs {
|
||||
v.BudgetMs = b
|
||||
}
|
||||
if p.Fail == "closed" && p.Mode != "shadow" && !slices.Contains(v.Closed, name) {
|
||||
v.Closed = append(v.Closed, name)
|
||||
}
|
||||
@@ -198,3 +205,148 @@ func (a *API) askJudge(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// JudgeCheckRequest asks how a judge keeps up: Burst judgments at once after
|
||||
// three in turn; zero is two for each ready replica.
|
||||
type JudgeCheckRequest struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
Burst int `json:"burst,omitempty"`
|
||||
}
|
||||
|
||||
// JudgeCheckView is how long a judgment took, alone and among others, set
|
||||
// against the time a request waits for one.
|
||||
type JudgeCheckView struct {
|
||||
Model string `json:"model"`
|
||||
BudgetMs int64 `json:"budget_ms"`
|
||||
Ready int `json:"ready_replicas"`
|
||||
// TookMs are three judgments of the same text, one after another.
|
||||
TookMs []int64 `json:"took_ms"`
|
||||
SlowestMs int64 `json:"slowest_ms"`
|
||||
// Burst judgments at once: the slowest of them, how many gave no
|
||||
// verdict, and how many a second the judges read.
|
||||
Burst int `json:"burst"`
|
||||
BurstSlowestMs int64 `json:"burst_slowest_ms"`
|
||||
BurstFailed int `json:"burst_failed"`
|
||||
PerSecond float64 `json:"judgments_per_second"`
|
||||
// Verdict is ok, tight (past half the budget alone, or past it among
|
||||
// others) or slow (past it alone); Advice what to do when not ok.
|
||||
Verdict string `json:"verdict"`
|
||||
Advice []string `json:"advice,omitempty"`
|
||||
}
|
||||
|
||||
// judgeCheckText is a prompt of about 280 tokens, the length the judges
|
||||
// were measured at: ordinary work, so the verdict is safe and the time is
|
||||
// what a request pays.
|
||||
const judgeCheckText = "Our support team has been asked to write a short guide for new customers who are setting " +
|
||||
"up the gateway for the first time. The guide should explain how to create an API key in the portal, how to " +
|
||||
"choose a model from the list the portal shows, and how to send a first request with curl or with the OpenAI " +
|
||||
"Python client by changing only the base URL and the key. It should also say what the usage page shows, how " +
|
||||
"budgets and rate limits are set by an administrator for each team, and what a person should do when a " +
|
||||
"request is refused because a limit was reached. Please keep the tone friendly and plain, use short " +
|
||||
"paragraphs, and add a small table that lists the three most common error codes with what each means and " +
|
||||
"what to try next. The audience is technical but new to the product, so avoid internal names, and end with " +
|
||||
"a short checklist the reader can follow on their first day. Write it in English, about four hundred words."
|
||||
|
||||
func (a *API) checkJudge(w http.ResponseWriter, r *http.Request) {
|
||||
if a.guard == nil || a.guard.Ask == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no guard policies: define them under guard in the cluster's settings (llmgw ctl config edit)")
|
||||
return
|
||||
}
|
||||
var req JudgeCheckRequest
|
||||
_ = json.NewDecoder(r.Body).Decode(&req)
|
||||
var jv *JudgeView
|
||||
for _, j := range a.Judges() {
|
||||
if req.Model == "" || j.Model == req.Model {
|
||||
j := j
|
||||
jv = &j
|
||||
break
|
||||
}
|
||||
}
|
||||
if jv == nil {
|
||||
writeError(w, http.StatusBadRequest, "no policy asks a judge of that name")
|
||||
return
|
||||
}
|
||||
if jv.Ready == 0 {
|
||||
writeError(w, http.StatusConflict, "the judge "+jv.Model+" has no replica ready: "+jv.Problem)
|
||||
return
|
||||
}
|
||||
v := JudgeCheckView{Model: jv.Model, BudgetMs: jv.BudgetMs, Ready: jv.Ready, Burst: req.Burst}
|
||||
if v.Burst <= 0 {
|
||||
v.Burst = 2 * jv.Ready
|
||||
}
|
||||
v.Burst = min(v.Burst, 32)
|
||||
var mu sync.Mutex
|
||||
start := time.Now()
|
||||
// Each text begins differently: the engine keeps what it read before,
|
||||
// and the same text again is answered from that in a tenth of the time
|
||||
// a request's own prompt takes.
|
||||
n := 0
|
||||
text := func() string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
n++
|
||||
return fmt.Sprintf("Request %d of a check begun at %s.\n", n, start.Format(time.RFC3339Nano)) + judgeCheckText
|
||||
}
|
||||
for range 3 {
|
||||
ans, err := a.guard.Ask(r.Context(), jv.Model, text(), "")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, fmt.Sprintf("the judge %s gave no verdict: %v", jv.Model, err))
|
||||
return
|
||||
}
|
||||
v.TookMs = append(v.TookMs, ans.Took.Milliseconds())
|
||||
v.SlowestMs = max(v.SlowestMs, ans.Took.Milliseconds())
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
start = time.Now()
|
||||
for range v.Burst {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
ans, err := a.guard.Ask(r.Context(), jv.Model, text(), "")
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if err != nil {
|
||||
v.BurstFailed++
|
||||
return
|
||||
}
|
||||
v.BurstSlowestMs = max(v.BurstSlowestMs, ans.Took.Milliseconds())
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if wall := time.Since(start).Seconds(); wall > 0 {
|
||||
v.PerSecond = float64(v.Burst-v.BurstFailed) / wall
|
||||
}
|
||||
v.Verdict, v.Advice = judgeVerdict(v, *jv)
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// judgeVerdict reads a check: a judgment alone past the budget refuses every
|
||||
// request under fail: closed; past half of it, or past it among others, a
|
||||
// busy moment does.
|
||||
func judgeVerdict(v JudgeCheckView, jv JudgeView) (string, []string) {
|
||||
verdict := "ok"
|
||||
switch {
|
||||
case v.BudgetMs > 0 && v.SlowestMs > v.BudgetMs:
|
||||
verdict = "slow"
|
||||
case v.BudgetMs > 0 && (v.SlowestMs*2 > v.BudgetMs || v.BurstSlowestMs > v.BudgetMs || v.BurstFailed > 0):
|
||||
verdict = "tight"
|
||||
}
|
||||
if verdict == "ok" {
|
||||
return verdict, nil
|
||||
}
|
||||
refused := "served unjudged"
|
||||
if len(jv.Closed) > 0 {
|
||||
refused = "refused"
|
||||
}
|
||||
advice := []string{
|
||||
fmt.Sprintf("a judgment past %d ms is %s: each request is judged once for its prompt and again every 600 characters of its answer", v.BudgetMs, refused),
|
||||
"more cores for each judge: llmgw ctl model resources " + v.Model + " --cores 4, then llmgw ctl restart --model " + v.Model,
|
||||
"more judges, on machines of their own: llmgw ctl scale " + v.Model + " N, or judge machines at install",
|
||||
"a judge on a GPU: llmgw ctl guard judge use qwen3guard-4b " + v.Model,
|
||||
"a longer wait: guard.policies.<policy>.judge.latency_budget in llmgw ctl config edit",
|
||||
}
|
||||
if len(jv.Closed) > 0 {
|
||||
advice = append(advice, "or fail: open, which serves what the judge did not read, flagged")
|
||||
}
|
||||
return verdict, advice
|
||||
}
|
||||
Loaded 100 of 317 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user