Compare commits
104
Commits
No files matched your search
@@ -34,7 +34,11 @@ jobs:
|
|||||||
run: go vet ./...
|
run: go vet ./...
|
||||||
|
|
||||||
- name: Test with the race detector
|
- name: Test with the race detector
|
||||||
run: go test -race ./...
|
# Under the job's limit, so a package that hangs panics with the
|
||||||
|
# test it was in and where every goroutine waited, rather than the
|
||||||
|
# job ending at its deadline with nothing said: from 2026-10-15 every
|
||||||
|
# run did, printing nothing after internal/release.
|
||||||
|
run: go test -race -timeout 12m ./...
|
||||||
|
|
||||||
- name: Build
|
- name: Build
|
||||||
run: go build ./...
|
run: go build ./...
|
||||||
|
|||||||
@@ -9,3 +9,5 @@ coverage.out
|
|||||||
/terraform-provider-llmgw/terraform-provider-llmgw
|
/terraform-provider-llmgw/terraform-provider-llmgw
|
||||||
/terraform-provider-llmgw/examples/.terraform*
|
/terraform-provider-llmgw/examples/.terraform*
|
||||||
/terraform-provider-llmgw/examples/terraform.tfstate*
|
/terraform-provider-llmgw/examples/terraform.tfstate*
|
||||||
|
# What a development server run from the checkout writes beside it.
|
||||||
|
/secrets/
|
||||||
+425
-4
@@ -15,6 +15,61 @@ Conventions this codebase keeps, which every item here must keep too:
|
|||||||
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
|
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
|
||||||
cut and nothing is deployed to the PoC cluster without being told to.
|
cut and nothing is deployed to the PoC cluster without being told to.
|
||||||
|
|
||||||
|
## Open, in order (2026-10-20)
|
||||||
|
|
||||||
|
What is left, most needed first. The detail is in the item each names.
|
||||||
|
|
||||||
|
**Now: what goes wrong where it is installed.**
|
||||||
|
1. ~~**J8**~~: done 2026-10-20. `ctl guard judge check`, and init times the judge. Left of **J3**: its load test
|
||||||
|
against judges on the control plane, which was stopped half way. Judges on a 4-core node of their own refused
|
||||||
|
0/60 at 2 at once, 13/60 at 4 and 42/60 at 10.
|
||||||
|
2. ~~**I2 P2**~~: done 2026-10-20: the guardrails both ways, a rate per caller and server, a breaker. A portal page
|
||||||
|
for the servers is left, with P3 (stdio, per-person OAuth).
|
||||||
|
3. **K**: fixes from the fabric test the operator runs with docs/large-models.md.
|
||||||
|
|
||||||
|
**Soon: the customer's experience, and HA.**
|
||||||
|
4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
|
||||||
|
forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
|
||||||
|
5. ~~**J4**~~: done 2026-10-29. The done-criterion held on the PoC: with mirror 146 stopped, `ctl model serve` of a
|
||||||
|
new model was fetched by the second mirror (VM 167, 10.0.0.152), placed and answered; a node without a file
|
||||||
|
146's address names was given 152's; what was added meanwhile reached 146 when it came back; a removal was
|
||||||
|
carried both ways. Found and fixed there: add-peer on a mirror machine of its own, files without checksums never
|
||||||
|
copied, and `mirror remove` neither recorded nor seen by the running mirror.
|
||||||
|
6. **F5**: large imports limited by the embedding engine's slots. (M)
|
||||||
|
|
||||||
|
**From Kong** (section L): L1–L5 and L9 done (v0.15.0); left of L4 is routing by cost between outside
|
||||||
|
providers, of L5 cache-write and audio prices. L6 and L7 are llmgw-80's; L8 (A2A) when asked.
|
||||||
|
|
||||||
|
**Models we can trust** (section M, 2026-10-25): M1 done (pickles refused; the explorer lists only what this
|
||||||
|
cluster's hardware holds, with CPU and GPU tok/s). Next M2 (every download pinned and checked against Hugging
|
||||||
|
Face), then M5 (engines without the internet), M3, M4, M6.
|
||||||
|
|
||||||
|
**Governance.**
|
||||||
|
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||||
|
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||||
|
|
||||||
|
**When there is room.**
|
||||||
|
9. ~~**I1**~~: done 2026-10-25 (in memory, not pgvector); left: the 40-rewordings run on the PoC.
|
||||||
|
10. **H1**: tried on the PoC.
|
||||||
|
11. **C4**: Parquet, and a scheduled push for usage export.
|
||||||
|
|
||||||
|
**When needed.**
|
||||||
|
12. **H5**: licences in video and audio. 13. **H3**: replicas shared, the catalog side. 14. **J7**: nginx in front
|
||||||
|
of several control planes. 15. **J6**: rate limits per process (a note).
|
||||||
|
|
||||||
|
**Done since 2026-10-16:**
|
||||||
|
- J1, J5, I3, I4, J2.
|
||||||
|
- Active Directory and Entra ID, tried against a Samba DC, with the console's Sign-in screen.
|
||||||
|
- J3's code.
|
||||||
|
- K's code and docs/large-models.md.
|
||||||
|
- I2 P1 and the cluster's own read-only MCP server (llmgw:admin).
|
||||||
|
- The node console's Overview; a node's page showing its version and the models on each GPU.
|
||||||
|
|
||||||
|
**Not code, the operator's to decide or do:**
|
||||||
|
- Where Alertmanager sends (its receiver is "none"): with a judge down, nobody hears.
|
||||||
|
- The fabric test for K.
|
||||||
|
- The PoC runs a dev build of main (v0.15.0-7-ga583359) on its control planes and nodes, the mirror v0.15.0.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## A. Security — do first
|
## A. Security — do first
|
||||||
@@ -1426,13 +1481,379 @@ console show and add them.
|
|||||||
`on_demand.evict_after_idle` / `min_lifetime` as live settings; an
|
`on_demand.evict_after_idle` / `min_lifetime` as live settings; an
|
||||||
upgrade that leaves nodes behind says which; `ctl model serve` serves an
|
upgrade that leaves nodes behind says which; `ctl model serve` serves an
|
||||||
embedding model for embedding; refreshes wait while a form is filled in.
|
embedding model for embedding; refreshes wait while a form is filled in.
|
||||||
- Open: llmgw's `/mcp` with picture, 3D, VLM and workflow tools, so a
|
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
|
||||||
client needs no server of its own (asked by the game team); the shared
|
so a client needs no server of its own (asked by the game team); the
|
||||||
ComfyUI replica (H3's placement side).
|
shared ComfyUI replica (H3's placement side).
|
||||||
|
- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
|
||||||
|
is and its settings; three machines each run the control plane, with a
|
||||||
|
floating address (VRRP id from the address); a lost store primary costs no
|
||||||
|
request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
|
||||||
|
ttl 20, a standby leaves the lease alone a round after the store returns);
|
||||||
|
a new leader waits for its judge within the budget; a judge is asked again
|
||||||
|
on another machine at half its budget; init sizes the judge (2 GB, no
|
||||||
|
prompt cache, two cores, one judge on up to four cores on one machine).
|
||||||
|
|
||||||
|
## I. What other gateways do that llmgw does not (written 2026-10-16)
|
||||||
|
|
||||||
|
From a comparison with Bifrost (maximhq/bifrost, Apache 2.0, Go), with
|
||||||
|
LiteLLM and Portkey in mind. Bifrost routes to 23+ cloud providers and runs
|
||||||
|
no model; llmgw runs the models, so most of Bifrost is either in llmgw
|
||||||
|
already (keys, budgets, fallbacks, guardrails, tracing, SSO, HA — the last
|
||||||
|
three paid in Bifrost) or does not apply. These four are worth taking for
|
||||||
|
a site that runs its own models. Left until a customer asks: native
|
||||||
|
Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
|
||||||
|
routing by what a request holds. Not taken: a realtime (WebSocket audio)
|
||||||
|
API, prompt management, an `npx` install.
|
||||||
|
|
||||||
|
### I1. Answers to similar questions, not only the same one (M) — done 2026-10-25
|
||||||
|
`keys cache --similar`, `router.cache.embedding_model`, `internal/router/similar.go`. Changed
|
||||||
|
from the plan: the vectors are in the gateway's memory beside the exact answers, not a pgvector
|
||||||
|
table. The exact cache keeps answer text out of the store on purpose (data.md), and a table of
|
||||||
|
answers would undo that; it also works on SQLite. The usage row marks a similar hit as a hit
|
||||||
|
(no column of its own). Open: the 40-rewordings and eval-suite runs on the PoC.
|
||||||
|
The original item follows.
|
||||||
|
The answer cache (`internal/router/cache.go`, C17) answers a request whose
|
||||||
|
body is the same byte for byte. A help desk asked "how do I reset my
|
||||||
|
password" in forty wordings asks the model forty times.
|
||||||
|
- Per project, off by default, beside the exact cache: `llmgw keys cache
|
||||||
|
--project P --similar 0.95` (the cosine similarity a question must reach;
|
||||||
|
higher is stricter). The exact cache is asked first.
|
||||||
|
- The question embedded by a model the operator names
|
||||||
|
(`router.cache.embedding_model`, an embedding model in the catalog), into a
|
||||||
|
pgvector table per project with the answer, the model and the TTL; the
|
||||||
|
store's halfvec and text index work from F applies. SQLite stores keep the
|
||||||
|
exact cache only.
|
||||||
|
- What counts as the question: the last user turn, with the system prompt
|
||||||
|
and the earlier turns hashed and required to match exactly — a similar
|
||||||
|
question in another conversation is not the same question. Tools,
|
||||||
|
`response_format` and the sampling settings also exact.
|
||||||
|
- Said: `x-llmgw-cache: similar` with the similarity, and the usage row
|
||||||
|
says so; guardrails judge the request before a hit, as for the exact
|
||||||
|
cache. A project with `no_store` gets neither.
|
||||||
|
- Done when: forty rewordings of one question on the PoC cost one model call
|
||||||
|
and 39 lookups, and an eval suite run with the cache on scores within a
|
||||||
|
point of the run with it off.
|
||||||
|
|
||||||
|
### I2. MCP gateway: the customer's tool servers behind llmgw (L)
|
||||||
|
P1 done 2026-10-20: HTTP servers registered (`ctl mcp`, `/api/v1/mcp/servers`,
|
||||||
|
store `mcp_servers`), tools as `server.tool` in `/mcp`, `allowed_tools`
|
||||||
|
grants in the tree (deny by default), credentials in the store or by
|
||||||
|
reference, the caller named in `X-Llmgw-*` headers, time and size bounds,
|
||||||
|
progress passed on, audit (Source `mcp`) and metrics. Left: P2, the
|
||||||
|
guardrails both ways, a per-tool rate limit, a breaker, and a portal page;
|
||||||
|
P3, stdio servers and per-person OAuth. The "done when" with Claude Code
|
||||||
|
against a real server is still to try.
|
||||||
|
`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
|
||||||
|
the file server, a database — each an MCP server, each today configured
|
||||||
|
in every client with its own credentials and nobody seeing what was called.
|
||||||
|
- Servers registered once, by an administrator: `llmgw ctl mcp add jira
|
||||||
|
--url https://… --header-file /etc/llmgw/secrets/jira` (HTTP transport;
|
||||||
|
stdio servers run on the control plane under their own user, like an
|
||||||
|
engine). Their credentials stay on the control plane, never in a client.
|
||||||
|
- `/mcp` lists every server's tools, prefixed by its name (`jira.search`),
|
||||||
|
filtered by what the key may use: an allowlist per org, team, project or
|
||||||
|
key, by server or by tool, in the tenancy tree as model allowlists are.
|
||||||
|
- Each call goes to the audit log — who, which tool, the arguments' size,
|
||||||
|
the outcome and how long — and through the guardrails both ways (a tool's
|
||||||
|
answer is prompt input). A per-tool rate limit.
|
||||||
|
- Later, if it earns it: Bifrost's "code mode" (the tools as one typed API
|
||||||
|
the model writes code against), which cuts the tokens of a long tool list.
|
||||||
|
- Done when: Claude Code given only llmgw's `/mcp` reaches a test MCP server
|
||||||
|
through it, a key without the grant does not see the server's tools, and
|
||||||
|
every call is in `ctl audit`.
|
||||||
|
|
||||||
|
### I3. What llmgw adds to a request, measured (S) — done 2026-10-16
|
||||||
|
`llmgw ctl bench overhead` (`internal/cli/bench_overhead.go`): an instant
|
||||||
|
engine and the gateway in one process, four modes (direct, bare, auth, auth
|
||||||
|
and guard patterns) at rising rates. On an Apple M5 the gateway added
|
||||||
|
0.05–0.42 ms at p50 and at most 0.49 ms at p99, in the README.
|
||||||
|
`TestTheGatewaysOverheadIsMeasured` runs a second of it in CI with no bound
|
||||||
|
asserted. Left: the CPU figure is rough with everything in one process; a
|
||||||
|
gateway in a process of its own would measure it properly. The original
|
||||||
|
item follows.
|
||||||
|
|
||||||
|
Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
|
||||||
|
disagree with it). llmgw has none, so any question about its overhead is
|
||||||
|
answered with a guess.
|
||||||
|
- `llmgw ctl bench --overhead`: the same load through the gateway and straight
|
||||||
|
at a `sim-engine` (`internal/cli/simengine.go`) that answers at once, at
|
||||||
|
rising request rates, with auth, accounting and the guardrails' pattern
|
||||||
|
checks on and off. Reports the added latency at p50/p99 and the gateway's
|
||||||
|
CPU per thousand requests.
|
||||||
|
- On the PoC the CPU is oversubscribed (pve1, 28 threads for ~125 vCPUs):
|
||||||
|
report it as a ratio and record the host's load beside it, and repeat on
|
||||||
|
an idle machine before a figure goes into the README.
|
||||||
|
- Done when: the README states the overhead, with how it was measured, and
|
||||||
|
CI runs the short version so a change that doubles it is seen.
|
||||||
|
|
||||||
|
### I4. Several keys for one outside provider (S) — done 2026-10-17
|
||||||
|
`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
|
||||||
|
each in `control.externalEndpoints`, 401 retried on another key and
|
||||||
|
`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
|
||||||
|
still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
|
||||||
|
original item follows.
|
||||||
|
|
||||||
|
An external model has one `api_key_file` (`internal/control/control.go`,
|
||||||
|
where external endpoints are made). A provider's limit is per key; one key
|
||||||
|
answering 429 makes the whole fallback answer 429.
|
||||||
|
- `api_key_files: [a, b, c]`, each with an optional `weight`: one endpoint
|
||||||
|
per key (`external:<model>#<n>`), so the router's scoring, retries and
|
||||||
|
breakers work per key as they already do per replica. A key answering 429
|
||||||
|
or 401 is passed over while its breaker is open; a 401 also notifies
|
||||||
|
(`external_key_refused`).
|
||||||
|
- The usage row says which key answered (by its index, never the key).
|
||||||
|
- Done when: with two keys and one returning 429, every request is answered
|
||||||
|
by the other, and the console shows the one breaker open.
|
||||||
|
|
||||||
|
## J. Found testing HA and single-machine installs (written 2026-10-16)
|
||||||
|
|
||||||
|
Three VMs installed as HA from the setup screens, a fourth as a judge node,
|
||||||
|
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
|
||||||
|
|
||||||
|
### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
|
||||||
|
Not a wait for ever, as first written: on the 4-core VM the judge's restart
|
||||||
|
was *refused* ("fits on no node even with this replica stopped") because the
|
||||||
|
starter, restarted at the same time to reserve fewer cores, still held its
|
||||||
|
old ones. Once refused, nothing tried again, so the judges kept their old
|
||||||
|
figures. Two fixes in `internal/control/reconciler`:
|
||||||
|
- a restart whose replacement fits once other restarts under way have gone
|
||||||
|
(queued, retiring or draining) waits for them, up to DrainGrace, instead
|
||||||
|
of being refused (`fitsAfterRestarts`, restart.go);
|
||||||
|
- a drained replica counts a retiring one that still answers as serving, so
|
||||||
|
one restart after another no longer leaves the first's drained replica,
|
||||||
|
and the room it holds, waiting out its grace (reconciler.go).
|
||||||
|
`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
|
||||||
|
|
||||||
|
### J2. A judge the leader adds is sized as init's is (S)
|
||||||
|
When a policy names a judge the catalog lacks, the leader serves it itself
|
||||||
|
(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
|
||||||
|
`judges_known.go`, through modelserve). That judge reserves 3 GB (the
|
||||||
|
weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
|
||||||
|
with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
|
||||||
|
(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
|
||||||
|
its cores and runs with `--cache-ram 0`.
|
||||||
|
- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
|
||||||
|
request the same memory and cores rule as `cpuPlan`.
|
||||||
|
- Done when: a judge removed from the catalog comes back with the same
|
||||||
|
figures init would have given it.
|
||||||
|
|
||||||
|
### J3. HA setup puts the judges on nodes of their own (S)
|
||||||
|
Under load on a 4-core control plane that was also a node, 38 of 60
|
||||||
|
requests were refused for want of a judgment. With the judges on a node of
|
||||||
|
their own (4 cores, 32 GB, two cores each) the same test refused 2.
|
||||||
|
- The HA setup offers a judge node or two: machines given like the store's,
|
||||||
|
added as nodes with a label (`llmgw.role: judge`), and the judge's
|
||||||
|
profile selects that label.
|
||||||
|
- Done when: the setup installs the judges on the machines given for them,
|
||||||
|
and the load test above refuses no more than it did by hand.
|
||||||
|
|
||||||
|
### J4. The model mirror on more than one machine (M) — done 2026-10-29, tried on the PoC
|
||||||
|
In an HA install the mirror stays on the first machine. While it is down,
|
||||||
|
models the nodes already have keep serving and new ones wait for it.
|
||||||
|
- A second mirror kept in step (each machine of the store, or one more), the
|
||||||
|
control planes and nodes given both, the first that answers used.
|
||||||
|
- Done when: with the first machine powered off, a model added then is
|
||||||
|
fetched and served.
|
||||||
|
|
||||||
|
### J5. An upgrade corrects an earlier install's judge (S) — done 2026-10-16
|
||||||
|
Built in the judge keeper (`internal/cli/judgekeeper.go`, `refit`), on the
|
||||||
|
leader: a judge whose entry is exactly what init before v0.14.7 wrote is
|
||||||
|
given 2 GB, `--cache-ram 0` and cores by init's rule (`refitCores`: up to
|
||||||
|
four for one judge, two each for more), through `modelsvc.SetProfiles`,
|
||||||
|
and its ready replicas are restarted (after J1). Its replica count is left
|
||||||
|
as it is. One that differs is left alone and said once in the log. The
|
||||||
|
original item follows.
|
||||||
|
|
||||||
|
v0.14.6 and earlier installed two judges reserving no cores or one, with
|
||||||
|
2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
|
||||||
|
missed its 5 s budget and every request was refused. v0.14.7's notes say
|
||||||
|
how to fix it by hand (`ctl model resources`, then `ctl restart`).
|
||||||
|
- On upgrade, or on `server init` run again, a judge whose profile is still
|
||||||
|
exactly what an earlier init wrote is moved to today's figures and
|
||||||
|
restarted (after J1). One changed by hand is left alone and named.
|
||||||
|
- Done when: a v0.14.6 single-machine install upgraded to it judges within
|
||||||
|
budget without anything typed.
|
||||||
|
|
||||||
|
### J6. Rate limits per process (note)
|
||||||
|
Rate limits are counted in each control plane's memory. A failover starts
|
||||||
|
every key's minute afresh. Documented in install.md; worth a shared count in
|
||||||
|
the store only if a customer's limits must hold across a failover.
|
||||||
|
|
||||||
|
### J7. nginx in front of several control planes (S)
|
||||||
|
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
|
||||||
|
nginx's open-source build has no active health checks, so its `upstream`
|
||||||
|
block would send to standbys and to a lost leader alike.
|
||||||
|
- nginx.conf: the floating address as its one upstream, and why; or the
|
||||||
|
`max_fails` passive approach with what it costs.
|
||||||
|
|
||||||
|
### J8. A judge budget that fits the machine (S)
|
||||||
|
A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
|
||||||
|
default: two cores are close to it, and a busy machine passes it.
|
||||||
|
- At install, after the judge is up, time three judgments. If the slowest is
|
||||||
|
more than half the budget, say so in the plan's closing text with the
|
||||||
|
choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
|
||||||
|
open`). The budget itself is not changed without being asked.
|
||||||
|
- Done when: an install on a machine too slow for its judge says so before
|
||||||
|
its first refused request.
|
||||||
|
|
||||||
|
## L. Taken from Kong's AI Gateway (written 2026-10-25)
|
||||||
|
|
||||||
|
From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
|
||||||
|
3.9, and only six of its AI plugins are open (proxy, prompt template,
|
||||||
|
decorator, regex prompt guard, request and response transformer); the rest
|
||||||
|
is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
|
||||||
|
plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
|
||||||
|
prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
|
||||||
|
unshown), prompt templates and decorators (no one has asked), Konnect's
|
||||||
|
entity model. MCP items are llmgw-80's (I2); they are listed for it.
|
||||||
|
|
||||||
|
### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
|
||||||
|
`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
|
||||||
|
operation, request and response model, max_tokens, temperature, top_p, the
|
||||||
|
session as conversation id, input/output/cached tokens, the engine as
|
||||||
|
provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
|
||||||
|
Prometheus already counted tokens by direction, cached apart. Left: the MCP
|
||||||
|
calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
|
||||||
|
|
||||||
|
### L2. Redacted personal data put back in the answer (S) — done 2026-10-25
|
||||||
|
`restore: true` (`guard.Rules.Restore`), `Verdict.Restore`, `internal/router/restore.go`
|
||||||
|
ahead of the output filter. The original item follows.
|
||||||
|
The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
|
||||||
|
`placeholders`), the same value always the same placeholder. A model that
|
||||||
|
answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
|
||||||
|
- Per policy, `restore: true`: the placeholders of one request kept in memory
|
||||||
|
for its life, and swapped back in the answer, streamed too (a placeholder
|
||||||
|
split across two chunks held until it is whole).
|
||||||
|
- Only into the answer to the request that held them; never stored, never in
|
||||||
|
usage rows or traces.
|
||||||
|
- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
|
||||||
|
policy reaches the model as placeholders and the client reads the real name
|
||||||
|
and address back.
|
||||||
|
|
||||||
|
### L3. A guardrail that is someone else's service (S) — done 2026-10-25
|
||||||
|
`guard.Service` (`internal/guard/service.go`), policy `services`, asked
|
||||||
|
before the judge on input and once on the whole answer. The original item follows.
|
||||||
|
Customers may be required to use one: Azure Content Safety, Lakera, NeMo
|
||||||
|
Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
|
||||||
|
templated HTTP call with an expression for "block".
|
||||||
|
- A detector kind `http`: URL, headers from secret files, a body template
|
||||||
|
with the text, a JSON path that says blocked and one that says why;
|
||||||
|
timeout and `fail` as the judge's. Checked on input, output or both.
|
||||||
|
- Done when: a policy with an `http` detector against a test service blocks
|
||||||
|
what it says to, and a service that does not answer is treated as `fail`
|
||||||
|
says.
|
||||||
|
|
||||||
|
### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
|
||||||
|
`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
|
||||||
|
what `fallbacks` already does (local, then an outside model when no room). Left:
|
||||||
|
routing by cost between several outside providers. The original item follows.
|
||||||
|
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
|
||||||
|
and KV cache. Kong also balances by measured latency, by tokens or cost
|
||||||
|
used, and in priority tiers.
|
||||||
|
- A `latency` scorer: a moving average of time to first token per endpoint.
|
||||||
|
- Fallbacks in tiers: local replicas first, then an outside provider, with
|
||||||
|
the tier said in the usage row.
|
||||||
|
- Done when: with two replicas of one model, one on a slower card, `latency`
|
||||||
|
sends most of the traffic to the faster and the slower still gets some.
|
||||||
|
|
||||||
|
### L5. Prices by kind and by cache (S–M) — done 2026-10-25
|
||||||
|
`cached_per_1k`, charged on the answer's `cached_tokens`. A budget in money was already
|
||||||
|
there (`keys limits --budget`). Not done: cache-write and audio/video token prices —
|
||||||
|
no engine llmgw runs reports them, and no provider it reaches is billed by them yet.
|
||||||
|
The original item follows.
|
||||||
|
`catalog` prices are per million input/output tokens and per picture/mesh.
|
||||||
|
Kong prices cached reads and cache writes apart, and audio, image and video
|
||||||
|
tokens apart, and limits by cost.
|
||||||
|
- Price fields for cached input and by modality; the usage row's cost from
|
||||||
|
them; a budget limit in money as well as tokens (`keys limits --cost`).
|
||||||
|
- Done when: a request with 90% cached prompt costs what the cached rate says.
|
||||||
|
|
||||||
|
### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
|
||||||
|
Kong turns OpenAPI operations into MCP tools. A company with an internal
|
||||||
|
REST API and no MCP server could offer it to agents through `/mcp`:
|
||||||
|
`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
|
||||||
|
operation a tool, under the same grants, audit, guardrails and rate as I2.
|
||||||
|
|
||||||
|
### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
|
||||||
|
Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
|
||||||
|
resource metadata, tokens checked by JWKS against the company's identity
|
||||||
|
provider), and a token exchanged for a short-lived one per upstream server
|
||||||
|
rather than one long-lived secret.
|
||||||
|
|
||||||
|
### L8. A2A (S, when asked)
|
||||||
|
Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
|
||||||
|
llmgw's, calls audited and traced. The standard is young; wait for a
|
||||||
|
customer.
|
||||||
|
|
||||||
|
### L9. llmgw behind Kong, written down (S) — done 2026-10-25
|
||||||
|
docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on;
|
||||||
|
Kong 3.9 counts no completion tokens of a stream. The original item follows.
|
||||||
|
A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
|
||||||
|
`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
|
||||||
|
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
|
||||||
|
container, and what Kong's token accounting sees.
|
||||||
|
|
||||||
|
## M. Models from Hugging Face we can trust (written 2026-10-25)
|
||||||
|
|
||||||
|
The user asked how we know a model fetched from Hugging Face is safe. Two
|
||||||
|
questions: whether its files can run code on our machines, which we can
|
||||||
|
close, and whether the model itself was trained to misbehave, which no scan
|
||||||
|
proves — only who published it, an eval before it serves, and the
|
||||||
|
guardrails narrow that. What existed: `--trust-remote-code` refused, no
|
||||||
|
repository Python fetched, ComfyUI nodes pinned to a commit, engines
|
||||||
|
hash-pinned and run as `llmgw-engine`, mirror versions pinned to a commit
|
||||||
|
and never overwritten. What did not: everything below.
|
||||||
|
|
||||||
|
### M1. Weights in a pickle format refused (S) — done 2026-10-25
|
||||||
|
`.bin`, `.pt`, `.pth`, `.ckpt`, `.pkl` run code as they are loaded, which is
|
||||||
|
how the malicious models found on Hugging Face work. The node refuses such a
|
||||||
|
source, or a mirror set holding one, before fetching anything; a model being
|
||||||
|
added or given such a source is refused; vLLM and SGLang downloading a model
|
||||||
|
by its id are given `--load-format safetensors`. The explorer hides a repository with neither GGUF nor safetensors
|
||||||
|
(under "runs here", ticked by default) and does not offer a repository's
|
||||||
|
pickle files. `allow_pickle: true` on a model lets it through, on purpose. On the
|
||||||
|
PoC, `triposr` (`hf:stabilityai/TripoSR/model.ckpt`) is the one model that
|
||||||
|
needs it.
|
||||||
|
|
||||||
|
### M2. Every download pinned and checked against Hugging Face (M)
|
||||||
|
- `hf:org/repo/file` follows `main` and is checked by nothing; a bare
|
||||||
|
`org/repo` given to vLLM is downloaded by vLLM, of whatever `main` is.
|
||||||
|
Both: resolve to the commit when the model is added, keep it in the
|
||||||
|
catalog's origin, and fetch that commit only (`--revision` for vLLM).
|
||||||
|
- The sha256 Hugging Face publishes for each LFS file compared on every
|
||||||
|
path, and a file that differs removed: `mirror add`, the manager, the
|
||||||
|
node's `hf:` and http sources, mirror set files with no sha recorded.
|
||||||
|
- Done when: a file changed on Hugging Face after a model was added is
|
||||||
|
refused by the node with both checksums named.
|
||||||
|
|
||||||
|
### M3. Hugging Face's own scan (S)
|
||||||
|
Hugging Face scans every file (pickle imports, ClamAV, partners' scanners)
|
||||||
|
and marks a file unsafe. Ask for it with the repository listing; refuse an
|
||||||
|
unsafe file, show the result on the mirror page and in the catalog's origin.
|
||||||
|
The API field has to be confirmed first.
|
||||||
|
|
||||||
|
### M4. Publishers we trust, and the rest asked about (S)
|
||||||
|
An allow-list of organisations in the cluster's settings (Qwen, meta-llama,
|
||||||
|
mistralai, google, microsoft, …). A model from anyone else is added only by
|
||||||
|
an admin who says so, and who said it is kept in the origin. The PoC already
|
||||||
|
serves four community models whose training nobody here checked.
|
||||||
|
|
||||||
|
### M5. Engines without the internet (M)
|
||||||
|
The firewall is inbound only, so an engine that was made to run code can
|
||||||
|
send what it reads anywhere. With weights from the mirror the engines need
|
||||||
|
no egress: an nft output rule for the engine user (allow the mirror, the
|
||||||
|
cluster, loopback), systemd hardening of the agent's engines
|
||||||
|
(`NoNewPrivileges`, `ProtectSystem`, private /tmp). Agent itself still root.
|
||||||
|
|
||||||
|
### M6. An eval before a new model serves (S)
|
||||||
|
`eval` and `canary` exist; make a new model from an untrusted publisher
|
||||||
|
(M4) serve only after its eval suite passes, so a model trained to answer
|
||||||
|
wrongly on a trigger has at least been asked the suite.
|
||||||
|
|
||||||
## D. Facts worth knowing before starting
|
## D. Facts worth knowing before starting
|
||||||
|
|
||||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||||
|
`/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
|
||||||
Metrics on `:9091` with `/prometheus/targets` beside them.
|
Metrics on `:9091` with `/prometheus/targets` beside them.
|
||||||
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
|
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
|
||||||
the PoC runs PostgreSQL under Patroni.
|
the PoC runs PostgreSQL under Patroni.
|
||||||
|
|||||||
+669
-2
@@ -18,16 +18,638 @@ is only released once its **exit criterion** is met, not when its code lands.
|
|||||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||||
|
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||||
|
|
||||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||||
APIs change between minors without a deprecation period.
|
APIs change between minors without a deprecation period.
|
||||||
|
|
||||||
## Unreleased
|
## v0.15.3 — 2026-10-29
|
||||||
|
|
||||||
|
A model stopped and kept, or removed with its files; the second mirror fixed where it was tried.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- No migrations. The control planes carry stop, start and purge; the nodes need nothing new.
|
||||||
|
- The mirrors carry the fixes: on each mirror's machine, `sudo llmgw upgrade v0.15.3 --service llmgw-mirror`.
|
||||||
|
A mirror with a peer then works out the checksums its older files lack, once, which reads every such file.
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
- **A model stopped and kept, and a model removed with its files.** `llmgw ctl model stop|start MODEL`: none of its
|
||||||
|
replicas run, it stays in the catalog with its files, and start runs it again scaled as it was; requests are told an
|
||||||
|
administrator stopped it. `llmgw ctl model remove MODEL --purge` (`DELETE /api/v1/models/{name}?purge=true`) also
|
||||||
|
deletes every version of it on the mirror, a second mirror following, and the nodes' copies once its replicas have
|
||||||
|
stopped; what another model uses stays. Both are in the console's model menu.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **`llmgw mirror add-peer` on a mirror with a machine of its own.** It set the second mirror up and then
|
||||||
|
failed for want of the control plane's server.yaml; now it prints the `llmgw ctl config set mirror.peers`
|
||||||
|
to run on a control plane.
|
||||||
|
- **Models fetched before checksums were kept reach the second mirror.** A mirror kept in step copies only what it
|
||||||
|
can check, and those files had no sha256; a mirror with peers now works theirs out and records them, once.
|
||||||
|
- **A file removed with `llmgw mirror remove` is gone from the mirror's answers too.** The running mirror kept
|
||||||
|
reporting a file it had fetched as present for a week after it was removed, so a control plane could place a
|
||||||
|
model on a file that was not there; and the removal was not recorded, so a mirror kept in step copied it back.
|
||||||
|
|
||||||
|
## v0.15.2 — 2026-10-29
|
||||||
|
|
||||||
|
A served model moved to more cards, or fewer, while it is served.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- No migrations, and nothing changes until the command is used. The control
|
||||||
|
planes carry it; the nodes need nothing new.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **A served model moved to more cards, or fewer.** `llmgw ctl model gpus
|
||||||
|
MODEL N [--profile P] [--in-place]`, `POST /api/v1/models/{name}/gpus`, and
|
||||||
|
GPUs… in the console's model menu. How many cards of one node each replica
|
||||||
|
uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against
|
||||||
|
the model's KV heads, and the layers split for llama.cpp. Refused for a
|
||||||
|
count no node has, or whose cards would not hold the model and its
|
||||||
|
context, with the figures; its replicas are then started again on it, one
|
||||||
|
by one, beside the old — or, with `--in-place` when there is no room
|
||||||
|
beside them, each stopped first.
|
||||||
|
|
||||||
|
## v0.15.1 — 2026-10-29
|
||||||
|
|
||||||
|
A 125B model on one card, a second mirror, and weights that run code
|
||||||
|
refused. Strata runs as an engine: a mixture-of-experts model with its
|
||||||
|
experts in RAM, served from one 24 GB card. A second model mirror keeps in
|
||||||
|
step with the first, so an HA install has no single machine left. Pickled
|
||||||
|
weights are refused unless a model says otherwise. A served model's context
|
||||||
|
can be changed in place. The explorer says how fast a model would answer
|
||||||
|
here before it is fetched, and a card in the wrong PCIe slot is said so.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- No migrations. Upgrade the control planes before the nodes, for the
|
||||||
|
pickle refusal below.
|
||||||
|
- A model whose weights are in a pickle format (`.bin`, `.pt`, `.pth`,
|
||||||
|
`.ckpt`, `.pkl`) stops being placed: its node refuses the source. Give it
|
||||||
|
its safetensors or GGUF, or add it again with `allow_pickle: true`
|
||||||
|
(`llmgw ctl model add -f`) if its files were checked. Look at the
|
||||||
|
models' sources before upgrading the nodes: one ending in
|
||||||
|
`.ckpt` or `.bin`, or a mirror version holding `pytorch_model.bin`. Upgrade the control planes first: a node on
|
||||||
|
this version refuses those weights until its control plane can say
|
||||||
|
`allow_pickle`.
|
||||||
|
- Strata is opt-in: nothing installs it until `engines` or an
|
||||||
|
`engine_overrides` entry names a version (`0.2.7`).
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Strata as an engine: a 125B model of experts on one card.** Strata
|
||||||
|
(github.com/Niko1221/Strata) keeps a model's experts in RAM and the ones
|
||||||
|
used most in VRAM. On an RTX 6000 24 GB with 251 GB of RAM it serves
|
||||||
|
Qwen3.8-Flash-Next 125B (IQ2_XS) at 70–87 tokens a second to one client;
|
||||||
|
more clients share about that, its slots buying fairness rather than
|
||||||
|
throughput on one card. It runs a few families only — Qwen3.8-Flash-Next
|
||||||
|
and its variants, Qwen3.6-35B-A3B, DeepSeek V4.1-Flash — each named in a
|
||||||
|
profile's args with `--family` and `--size`
|
||||||
|
(`--family qwen --size IQ2_XS --context 32768`). The source is the
|
||||||
|
directory of the model's GGUF files, or the repository, which Strata then
|
||||||
|
downloads itself, pinned. The agent installs it on an NVIDIA node of
|
||||||
|
x86-64: its source pinned by digest, in a Python environment, and its
|
||||||
|
release's engine checked against the digest GitHub publishes — the CUDA 13
|
||||||
|
build for Turing and newer on driver 580, the CUDA 12.9 build for Pascal
|
||||||
|
to Ada, a V100 included, on driver 575. A replica prepares its model the
|
||||||
|
first time (minutes; its draft layer is a 5 GB download) and loads in
|
||||||
|
about a minute and a half after. One card of one node per replica, for
|
||||||
|
now; not in an offline bundle, its packages coming from PyPI.
|
||||||
|
|
||||||
|
- **A served model's context, changed.** `llmgw ctl model context MODEL
|
||||||
|
TOKENS [--slots N]`, `POST /api/v1/models/{name}/context`, and Context
|
||||||
|
length… in the console's model menu. The catalog, the engine's arguments and
|
||||||
|
the memory change together, checked against what the model was trained for
|
||||||
|
and what its cards hold; its replicas are then started again one by one. The
|
||||||
|
models API shows `context_length` and `max_output`.
|
||||||
|
|
||||||
|
- **A second model mirror, kept in step.** In a three-machine install the setup screens put one on a database machine
|
||||||
|
(`--second-mirror 1|2`). An install with one mirror gets one with `sudo llmgw mirror add-peer user@host`.
|
||||||
|
- **In step:** each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and
|
||||||
|
removes what the other removed, even if it was down at the time (`llmgw mirror serve --peer`).
|
||||||
|
- **Control planes:** they name both (`mirror.peers`) and use whichever answers.
|
||||||
|
- **Nodes:** while one mirror is down, a node is given a model's file on the other.
|
||||||
|
|
||||||
|
This takes away the last single point of an HA install: with the first machine off, models are still added and
|
||||||
|
fetched.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **An engine that computes in a process of its own group was not its
|
||||||
|
replica's.** A GPU replica read as running on the CPU, and the memory it
|
||||||
|
held on its card as somebody else's, which placement then kept free of
|
||||||
|
other work. A process the replica started, at any remove, is now the
|
||||||
|
replica's. And what a replica computes on, once sent, was never sent
|
||||||
|
again: a wrong reading kept from before an agent's restart stayed wrong.
|
||||||
|
It is sent again when it changes, and read again when a GPU replica was
|
||||||
|
kept as the CPU.
|
||||||
|
|
||||||
|
- **VS Code's sign-in lost its place through the sign-in page**, so the
|
||||||
|
prompt to confirm came every time; a link to the same gateway by plain
|
||||||
|
http could take an https setting down to http; and a new build of the
|
||||||
|
extension was not installed over an old one (the command now says
|
||||||
|
`--force`).
|
||||||
|
- **The install guide's recordings were not checked for secrets**, only
|
||||||
|
the page; every frame is now.
|
||||||
|
- **MCP server fields could not be emptied**, an address could carry a
|
||||||
|
password, the cluster's own server could not be changed from the
|
||||||
|
console, and a grant to a key that does not exist was said done.
|
||||||
|
- **A guardrail service two policies named was asked as the first said**,
|
||||||
|
not the stricter; a `null` answer read as safe; and with `restore` the
|
||||||
|
request log kept the answer with the redacted values back in it.
|
||||||
|
- **What an engine downloads by a Hugging Face id was not held to the
|
||||||
|
pickle refusal** — an adapter by its id, a TensorRT-LLM model's; a model
|
||||||
|
with either now needs `allow_pickle`.
|
||||||
|
- **`server init` run again on a machine already set up was refused its
|
||||||
|
memory**, as on an 8 GB install fixing its certificate with `--host`, and
|
||||||
|
judges that would not be installed (no mirror) were counted.
|
||||||
|
- **Removing an MCP server left its sealed credential in the store**, and
|
||||||
|
replacing a credential under another header kept the old one sent. A
|
||||||
|
server's secrets now go when it no longer reads them.
|
||||||
|
- **A cloned repository could point the VS Code extension at another
|
||||||
|
gateway.** `llmgw.gatewayUrl` is a machine setting now, never taken from a
|
||||||
|
workspace.
|
||||||
|
- **A node's memory bandwidth could be measured in its cache** on a busy
|
||||||
|
node, and was 0 after a reconnect until the next heartbeat. The buffer is
|
||||||
|
at least eight times the processor's last cache.
|
||||||
|
- **A project kept in the cluster could have its questions embedded outside
|
||||||
|
it.** The similar-question cache now asks the embedding model as the
|
||||||
|
question's project would, external fallbacks refused, and skips a model
|
||||||
|
that is itself outside. Its comparing no longer holds the cache's lock,
|
||||||
|
which every project's lookups waited on.
|
||||||
|
- **Restored values were left as placeholders in streamed tool calls**, in
|
||||||
|
completions and in reasoning, and a value with a quote in a whole answer's
|
||||||
|
tool-call arguments broke their JSON.
|
||||||
|
- **An outside guardrail service read what `redact` had taken out**, and a
|
||||||
|
streamed answer was sent to it twice. The docs now say a service stops a
|
||||||
|
streamed answer only under `stream: buffer_all`.
|
||||||
|
- **A replica found slow once could stay unused.** Its time to first token
|
||||||
|
is forgotten after five minutes without an answer, so it is tried again.
|
||||||
|
- **Weights that run code got past the refusal in four ways.** A mirror
|
||||||
|
version already on a node was used without being looked at again; a
|
||||||
|
`?query` or `#fragment` on an `hf:` source hid the file's name; `.pt2` was
|
||||||
|
not on the list; and a profile could write `--load_format`,
|
||||||
|
`--trust_remote_code` or an abbreviation such as `--trust-remote`, which
|
||||||
|
vLLM and argparse read as the flag, or give SGLang `--load-format`. All
|
||||||
|
refused now. TripoSR's preset says `allow_pickle`, without which it could
|
||||||
|
not be added, and a shared ComfyUI environment takes its first member's.
|
||||||
|
- **A temperature written as a string was refused.** Since the GenAI traces
|
||||||
|
read it, `"temperature": "0.7"` got a 400 before reaching the engine; it
|
||||||
|
is read loosely again, for the trace only.
|
||||||
|
- **A reused answer could carry another caller's personal data.** With
|
||||||
|
`restore: true` and the answer cache on, two callers' requests that
|
||||||
|
redacted to the same text shared one answer — with the first caller's
|
||||||
|
address put back in it. Such requests are now never answered from the
|
||||||
|
cache nor kept in it.
|
||||||
|
- **Granting a server's tools could take others away; revoking could close
|
||||||
|
a level to every tool.** A level with no tool list sees what the levels
|
||||||
|
above grant; giving it its first list made it see only that list. And a
|
||||||
|
level left with an empty list was closed to every tool, which the screens
|
||||||
|
did not show. Now a first list starts from the nearest list above, taking
|
||||||
|
a server away leaves the rest or goes back to no list, the screens say
|
||||||
|
when a level above keeps granted tools out of reach and which levels are
|
||||||
|
closed, and the console reads and writes the level at once rather than
|
||||||
|
from what it loaded.
|
||||||
|
- **Similar questions about different pictures got each other's answer.**
|
||||||
|
A question with a picture, a sound or a file in it is reused only when it
|
||||||
|
is the same request.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **A card in the wrong PCIe slot is said so.** Each agent reads how every
|
||||||
|
card is linked — the fastest generation card and slot allow together, and
|
||||||
|
the lanes in use — against the card's own most, from the kernel (the
|
||||||
|
driver's figure is what card and slot allow together); a card on fewer lanes or
|
||||||
|
an older generation is a node warning, with what it carries and what it
|
||||||
|
could, on the dashboard, the console and in `llmgw doctor`.
|
||||||
|
- **And on a web page, `/admin/mcp`.** For administrators: the same list,
|
||||||
|
adding a server with its credential sealed, testing it and listing its
|
||||||
|
tools, granting them in the tree beside that level's other grants and
|
||||||
|
limits, revoking, changing, replacing the credential, turning it off,
|
||||||
|
removing; and the line that points an agent at this gateway's `/mcp`.
|
||||||
|
In English and Turkish; every change in the audit log, the secret not.
|
||||||
|
- **The company's MCP servers in the console.** A screen of its own: each
|
||||||
|
server with its state, who has its tools, its rate, the names of the
|
||||||
|
credentials it is sent and its address; add one with a bearer token or
|
||||||
|
a header kept sealed, test it and list its tools, grant them to a team,
|
||||||
|
project, organization, person or key beside that level's other grants,
|
||||||
|
revoke, change, replace the credential, turn off, remove.
|
||||||
|
- **And on the cards, as this cluster would lay it out.** The explorer's
|
||||||
|
**GPU tok/s** and a repository's files give one reply's speed for each
|
||||||
|
layout the planner offers here — one card, tensor parallel across a
|
||||||
|
node's cards, llama.cpp's layer split, a pipeline across nodes on a
|
||||||
|
fabric — the fastest of them, with the layout in the tooltip. From each
|
||||||
|
card's memory bandwidth by its model name, NVLink or PCIe between a
|
||||||
|
node's cards, and InfiniBand or RoCE between nodes; a cluster of mixed
|
||||||
|
classes is judged class by class. A model of experts reads 1.6 times its
|
||||||
|
active share, as Qwen3-Coder-30B-A3B did on cuda01's RTX 6000 (144
|
||||||
|
tok/s; qwen3-vl-8b Q8: 62).
|
||||||
|
- **How fast a model would answer on a CPU, before it is fetched.** Each
|
||||||
|
agent measures how fast its node's memory reads when it starts (`Host`
|
||||||
|
gets `mem_bandwidth_gbs`, the node's view `mem_bandwidth_gbs`); the
|
||||||
|
explorer's **CPU tok/s** column and a repository's GGUF files say about
|
||||||
|
how many tokens a second one reply gets on the fastest node that holds
|
||||||
|
the model — the bandwidth over the bytes a token reads, its active part
|
||||||
|
for a model of experts. Calibrated on the PoC against llama.cpp's own
|
||||||
|
timings: within 15% from 1.5B to 14B.
|
||||||
|
- **The explorer lists what this cluster can run.** Judged by the nodes
|
||||||
|
as they are now — every card's memory and compute capability, every
|
||||||
|
node's RAM — through the same planner as serving a model, rather than a
|
||||||
|
size picked from a list. A model too big for any node, in FP8 without a
|
||||||
|
card of 8.9, in MLX, or in neither GGUF nor safetensors, is left out and
|
||||||
|
counted; the page says what it judged by. With cards, safetensors repositories are
|
||||||
|
listed beside GGUF. On a repository's page a file too big for the
|
||||||
|
cluster is shown with what it needs and no Fetch button.
|
||||||
|
- **Weights that run code are refused.** Python pickles — `.bin`, `.pt`,
|
||||||
|
`.pth`, `.ckpt` — run whatever they hold when loaded, which is how the
|
||||||
|
malicious models on Hugging Face work. The node refuses such a source,
|
||||||
|
or a mirror version with one in it, before fetching it; adding such a
|
||||||
|
model is refused; vLLM and SGLang downloading a model by its id read its
|
||||||
|
safetensors only; the mirror page marks the file. `allow_pickle: true`
|
||||||
|
on a model lets it through on purpose. The mirror's explorer no longer
|
||||||
|
lists a repository with nothing a node loads (no GGUF, no safetensors)
|
||||||
|
while "runs here" is ticked, and a repository's pickle files are not
|
||||||
|
offered, only counted. BACKLOG M lists what is next:
|
||||||
|
checksums against Hugging Face everywhere, its own scan, trusted
|
||||||
|
publishers, engines without the internet.
|
||||||
|
|
||||||
|
## v0.15.0 — 2026-10-25
|
||||||
|
|
||||||
|
The company's tools and editors through the gateway, and guardrails that
|
||||||
|
can lean on others. Its own MCP servers sit behind llmgw's `/mcp` under the
|
||||||
|
guardrails, a rate and a breaker. VS Code gets the gateway's models from an
|
||||||
|
extension the gateway serves. A guardrail policy can ask an outside
|
||||||
|
service, and can put redacted values back in the answer. Chat goes to the
|
||||||
|
replica answering soonest. A similar question can get the answer already
|
||||||
|
given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
|
||||||
|
GenAI conventions, and Kong is written down from trying it. Also an install
|
||||||
|
guide with every screen, at `/install`.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- Migrations 0049 (`mcp_servers`), 0050 (their rate and breaker) and 0051
|
||||||
|
(`projects.cache_similar`), applied when the control plane starts.
|
||||||
|
- `llmgw server init` now refuses a machine that would serve models without
|
||||||
|
the memory for them: the starter model and the judges need 12 GB on one
|
||||||
|
machine, where 8 GB was said before. `--no-model`, `--no-guard-model` or
|
||||||
|
`--no-node` make it fit. A running cluster is not checked again.
|
||||||
|
- Traces carry the GenAI conventions' `gen_ai.*` attributes beside the
|
||||||
|
`llmgw.*` ones, which stay; docs/install.md, "Tracing", lists them.
|
||||||
|
- Nothing else changes until it is turned on: `services` and `restore` in a
|
||||||
|
guardrail policy, `keys cache --similar` with
|
||||||
|
`router.cache.embedding_model`, `cached_per_1k` in a price.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Answers to similar questions.** `llmgw keys cache --similar 0.95` also
|
||||||
|
gives a chat question the answer already given to one like it in the same
|
||||||
|
conversation, compared by `router.cache.embedding_model`, as
|
||||||
|
`x-llmgw-cache: similar` with the similarity. Held in memory beside the
|
||||||
|
exact answers; nothing written to the store but the setting.
|
||||||
|
- **A cached prompt costs the cached rate.** `cached_per_1k` in a price
|
||||||
|
charges the prompt tokens an engine or provider read from its cache (the
|
||||||
|
answer's `cached_tokens`) at their own rate; without it they cost what any
|
||||||
|
prompt token does, as before.
|
||||||
|
- **Chat goes to the replica answering soonest.** The router keeps each
|
||||||
|
replica's time to first token, smoothed over its last answers, and the
|
||||||
|
`chat` profile scores by it beside the queue and the KV cache: of two
|
||||||
|
replicas of one model, the one on the slower card or the busier host gets
|
||||||
|
less, and a replica not yet measured is tried.
|
||||||
|
- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9
|
||||||
|
in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw
|
||||||
|
key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
|
||||||
|
no completion tokens of a streamed answer, so bill from llmgw's usage.
|
||||||
|
- **A guardrail that is someone else's service.** A policy's `services`
|
||||||
|
asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
|
||||||
|
HTTP: a body template, keys from files, dotted paths to where the answer
|
||||||
|
says unsafe and why, on input, output or both, block or flag, and the
|
||||||
|
policy's `fail` when it does not answer.
|
||||||
|
- **Redacted values put back in the answer.** `restore: true` on a guardrail
|
||||||
|
policy's input: the model reads `[EMAIL_1]`, the client reads its own
|
||||||
|
address in the reply, streamed or not. Only the request's own values, held
|
||||||
|
in memory for its life; the output rules read the answer as the client
|
||||||
|
will.
|
||||||
|
- **Traces in the OpenTelemetry GenAI conventions.** An inference request
|
||||||
|
and each attempt at an engine carry `gen_ai.operation.name`, the requested
|
||||||
|
and the answering model, `max_tokens`, `temperature` and `top_p` when set,
|
||||||
|
the session as `gen_ai.conversation.id`, and the input, output and cached
|
||||||
|
tokens, and an attempt the engine that answered as `gen_ai.provider.name`.
|
||||||
|
Tools that read the conventions show llmgw's calls as model calls with no
|
||||||
|
dashboard of their own. No prompt or answer text is ever in a span.
|
||||||
|
- **llmgw for VS Code.** An extension of llmgw's own, served by every gateway
|
||||||
|
at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from
|
||||||
|
the binary itself: plain JavaScript, no dependencies, no build, and so no
|
||||||
|
marketplace needed at a site with no internet. It offers the gateway's chat
|
||||||
|
models to VS Code's language-model API — the chat's model picker, agent
|
||||||
|
mode, and every extension that asks VS Code for a model — with the sizes
|
||||||
|
and the tool and vision capabilities the catalog says, streamed, with tool
|
||||||
|
calls and their results both ways. **VS Code** is a tool on the account
|
||||||
|
page's **Connect your editor**: a key, the one-line install, and an **Open
|
||||||
|
in VS Code** link that hands the key to the extension, which keeps it in
|
||||||
|
VS Code's secret storage. **llmgw: Sign In** opens that page with a state
|
||||||
|
of its own; a link without it is confirmed first, the gateway named.
|
||||||
|
- **An install guide with every screen, at `/install`.** From three bare
|
||||||
|
Ubuntu machines to a first answer: what each machine needs and how to
|
||||||
|
check it, the release fetched and checked, the setup screens one by one,
|
||||||
|
the system's state, and a first request with a key from the account page.
|
||||||
|
The screens are a real three-machine install, recorded and kept as
|
||||||
|
terminal frames — sharp at any size and a few kilobytes — with each
|
||||||
|
session played back on request; the administrator's password and every
|
||||||
|
key were masked before it was kept. `/about` says more of what llmgw does
|
||||||
|
(documents, guardrails, sign-in, high availability, pictures and 3D), and
|
||||||
|
its install steps are the ones of today, leading to the guide.
|
||||||
|
- **The company's MCP tools under the guardrails, a rate and a breaker.**
|
||||||
|
- **The answer:** the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
|
||||||
|
ticket is withheld as a pasted one would be, and personal data is redacted.
|
||||||
|
- **The arguments:** they are read before they leave, so a secret pasted into one does not reach the company system.
|
||||||
|
- **Rate:** `ctl mcp add --rate N` allows each caller N calls a minute to a server.
|
||||||
|
- **Breaker:** five failures in a row take a server out for a minute, its tools off the list.
|
||||||
|
|
||||||
|
- **How fast the judge is, measured.** `llmgw ctl guard judge check` (and `POST /api/v1/guard/judges/check`) works
|
||||||
|
out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
|
||||||
|
and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
|
||||||
|
and what to do: more cores, more judges, a GPU judge, a longer wait, or `fail: open`. `server init` runs it once the
|
||||||
|
judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.
|
||||||
|
|
||||||
|
- **The company's own MCP servers through llmgw's `/mcp`.** `llmgw ctl mcp add jira --url … --secret-file
|
||||||
|
Authorization=FILE` registers one; `/mcp` then lists its tools as `jira.search` beside llmgw's own. Only keys granted
|
||||||
|
them see them: `llmgw keys limits --tools 'jira.*'`, patterns at any level of the tenancy tree. None granted anywhere
|
||||||
|
is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
|
||||||
|
server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
|
||||||
|
on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
|
||||||
|
admin API has `/api/v1/mcp/servers`.
|
||||||
|
- **The cluster as a read-only MCP server.** Registered as `llmgw ctl mcp add llmgw --url llmgw:admin` and granted
|
||||||
|
as `llmgw.*`, its tools say what `llmgw ctl` does:
|
||||||
|
- `cluster_status`, `list_nodes`, `list_models`, `list_replicas`, `explain_placement`, `node_logs`;
|
||||||
|
- nothing that changes the cluster.
|
||||||
|
|
||||||
|
It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
|
||||||
|
The admin API serves the same at `POST /api/v1/mcp`. `ctl mcp add --bearer-file` keeps a token, sent as
|
||||||
|
`Authorization: Bearer`.
|
||||||
|
|
||||||
|
- **Large models across cards and nodes.** For a model of experts on more than one card, the plan now adds vLLM's
|
||||||
|
`--enable-expert-parallel`. It reads the expert count from `config.json` under any of its names, and sizes a model
|
||||||
|
that also reads pictures by its `text_config`. Several nodes are offered in any number, not only powers of two. Each
|
||||||
|
way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
|
||||||
|
shows in `ctl model plan`, in the console's Serve a model, and in a serve job's first step. The serve form checks that
|
||||||
|
a longer context still fits. `ctl model serve --env KEY=VALUE` puts engine variables such as `NCCL_DEBUG=INFO` on
|
||||||
|
every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.
|
||||||
|
|
||||||
|
- **Machines of the judges' own in a three-machine install.** The setup screen asks for up to two judge machines after
|
||||||
|
the store's two; `server init` takes `--judge-node user@host` for each. Each is added as a node labelled
|
||||||
|
`llmgw.role: judge`, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
|
||||||
|
control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
|
||||||
|
of their own, 2 were. A node with an `llmgw.role` label takes only profiles whose node selector asks for that role,
|
||||||
|
so no other model is placed on the judges' machine.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A machine too small for the models it would serve is said so first.**
|
||||||
|
The control plane that is also a node, serving the starter model and two
|
||||||
|
judges, needs about 10.5 GB. On 8 GB the install finished every step and
|
||||||
|
then had no room for the starter. Now the setup's checks fail its memory
|
||||||
|
with what it would serve, and `server init` refuses before it installs
|
||||||
|
anything, naming `--no-model`, `--no-guard-model` and `--no-node`.
|
||||||
|
|
||||||
|
- **Turkish pages are Turkish to the end.**
|
||||||
|
- **Errors:** those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
|
||||||
|
catalog", the settings changed meanwhile, and others) are now in the page's language.
|
||||||
|
- **Percent:** a percentage is written the language's way: `%93` in Turkish, `93%` in English.
|
||||||
|
- **Footer:** the time at the foot of a page is in the language's date form.
|
||||||
|
- **Stale forms:** they are refused with a sentence in that language rather than "bad request".
|
||||||
|
|
||||||
|
- **`llmgw keys limits` no longer clears the limits it was not given.** Setting one limit at a scope wrote the whole
|
||||||
|
row, so `--tools 'jira.*'` alone took away that scope's rate, budget and model list. Only what is given changes now.
|
||||||
|
`--replace` writes every limit, as the console's limits form does.
|
||||||
|
|
||||||
|
- **The plan refused tensor parallelism past a model's KV heads.** Qwen3-235B, with four KV heads, had no way to run
|
||||||
|
on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
|
||||||
|
offered, as is one that divides them.
|
||||||
|
- **A big model on a slow mirror was failed while it downloaded.** The startup timeout counted the download, so
|
||||||
|
hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
|
||||||
|
The timeout now runs from when a node's download last moved.
|
||||||
|
|
||||||
|
- **The judge init installs is no longer replaced by the leader's own.** Just after the control plane started, the
|
||||||
|
leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
|
||||||
|
a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
|
||||||
|
after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.
|
||||||
|
- **A judge the leader serves by itself is sized as init's is.** When a guard policy names a judge the catalog lacks,
|
||||||
|
the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
|
||||||
|
3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and `fail: closed` refused the
|
||||||
|
request. It now gets what `server init` gives its judges: 2 GB, two cores each (fewer on a small node), and no
|
||||||
|
prompt cache. Every judge served through `modelserve` runs with `--cache-ram 0`.
|
||||||
|
|
||||||
|
## v0.14.9 — 2026-10-19
|
||||||
|
|
||||||
|
Sign-in set up from the console, with Active Directory tried before it is
|
||||||
|
kept. A node's console now shows what the machine is doing, and a node's web
|
||||||
|
page shows what it runs. Also fixes found trying Active Directory against a
|
||||||
|
Samba domain controller on the test cluster. Sign-in through it, nested
|
||||||
|
groups and a disabled account suspended by the sync all worked there.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- No migration.
|
||||||
|
- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
|
||||||
|
block written into `server.yaml` on a cluster with a store was never
|
||||||
|
read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
|
||||||
|
again from the console's Sign-in screen.
|
||||||
|
- A Samba DC's own autogenerated certificate is refused (a negative serial
|
||||||
|
and no SAN). Give it one from a CA.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
|
||||||
|
Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
|
||||||
|
tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
|
||||||
|
reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
|
||||||
|
password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
|
||||||
|
domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
|
||||||
|
has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
|
||||||
|
- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
|
||||||
|
and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
|
||||||
|
GPUs.
|
||||||
|
- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
|
||||||
|
version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
|
||||||
|
here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
|
||||||
|
itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
|
||||||
|
machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
|
||||||
|
Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
|
||||||
|
- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
|
||||||
|
ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
|
||||||
|
control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
|
||||||
|
- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
|
||||||
|
cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
|
||||||
|
domain controller needs.
|
||||||
|
|
||||||
|
## v0.14.8 — 2026-10-18
|
||||||
|
|
||||||
|
Sign-in with an on-premises Active Directory, and Entra ID made safe for
|
||||||
|
people in many groups. Also several keys for one outside provider, and a
|
||||||
|
measure of what the gateway adds to a request.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- Migration 0048 adds the `user_directory` table, applied when the control
|
||||||
|
plane starts.
|
||||||
|
- Nothing changes until `auth.ldap` is set; docs/install.md, "Active
|
||||||
|
Directory", has an example. It has been tested against a fake directory,
|
||||||
|
not yet against a real domain controller.
|
||||||
|
- A judge installed by an earlier `server init` is refitted by the leader
|
||||||
|
by itself (below); the steps given for v0.14.7 are no longer needed.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Sign-in with Active Directory** (`auth.ldap`). People sign in on the gateway's form with their AD username or mail
|
||||||
|
and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give
|
||||||
|
staff roles and places in the tree, as with OIDC. Accounts are tied to the `objectGUID` and made at first sign-in
|
||||||
|
when `create_accounts` is on. Every 15 minutes the leader reads the directory. People disabled or deleted there are
|
||||||
|
suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend
|
||||||
|
more than half the accounts is refused. See docs/install.md, "Active Directory".
|
||||||
|
|
||||||
|
- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
|
||||||
|
place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
|
||||||
|
passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
|
||||||
|
`external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
|
||||||
|
|
||||||
|
- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
|
||||||
|
once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
|
||||||
|
engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
|
||||||
|
It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the
|
||||||
|
gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Entra ID's groups past 200 no longer take a role away.** For someone in more than 200 groups Entra sends none in
|
||||||
|
the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next
|
||||||
|
sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app
|
||||||
|
registration. docs/install.md has an Entra ID section.
|
||||||
|
- **An account made by SCIM or OIDC could not be reinstated.** Setting it active was refused as "never approved".
|
||||||
|
- **An upgrade refits the judge an earlier `server init` installed.** Upgrading to v0.14.7 left such a judge as it was:
|
||||||
|
no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was
|
||||||
|
fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no
|
||||||
|
prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's
|
||||||
|
replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone,
|
||||||
|
and the log says so once.
|
||||||
|
- **A restart is no longer refused for room another restart is giving back.** With every core of a node reserved, two
|
||||||
|
models restarted together, one to need more and one less (as after `ctl model resources` on a judge and a starter),
|
||||||
|
had the first refused: "fits on no node even with this replica stopped", though the second was about to give back
|
||||||
|
its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained
|
||||||
|
replica also counts a replica being restarted that still answers as serving, so replicas restarted one after
|
||||||
|
another no longer wait out the five-minute grace each.
|
||||||
|
|
||||||
|
## v0.14.7 — 2026-10-16
|
||||||
|
|
||||||
|
The judge installed by `server init` sized for what it does. On a machine
|
||||||
|
of fewer than eight cores v0.14.6 left every judgment past its budget, so
|
||||||
|
`fail: closed` refused every request. A single-machine install of v0.14.7
|
||||||
|
on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- No migration.
|
||||||
|
- **Upgrading does not change a judge already in the catalog.** v0.14.6
|
||||||
|
and earlier installed two judges, reserving no cores or one, and 2 or
|
||||||
|
4 GB. To give them what they need:
|
||||||
|
`llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2`, then
|
||||||
|
`llmgw ctl restart --model qwen3guard-0.6b`. The two then need four cores
|
||||||
|
on a single machine; below that, a longer `latency_budget` or
|
||||||
|
`fail: open`.
|
||||||
|
- On a node whose cores are all reserved, `ctl restart --model` cannot
|
||||||
|
start the new replica beside the old one and waits. Stop the old engine
|
||||||
|
process, and the replica comes back with the new figures.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Both judges fit an 8 GB machine again.** v0.14.6 had `server init` reserve 4 GB a judge replica. That figure came
|
||||||
|
from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default,
|
||||||
|
which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted
|
||||||
|
nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with `--cache-ram 0`, uses about 1.6 GB and
|
||||||
|
stays there, and reserves 2.
|
||||||
|
- **One machine runs one judge, on up to four cores.** A second replica on the same machine shares its cores and is
|
||||||
|
lost with it, so it bought nothing but half the speed of each. `server init` on a single machine now serves one judge
|
||||||
|
on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two
|
||||||
|
on two cores each, as in HA setups. `--judge-replicas` and the setup's Copies still choose.
|
||||||
|
- **A judge gets the two cores it needs.** v0.14.6 gave each judge replica one core on machines under eight. A judgment
|
||||||
|
of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core
|
||||||
|
machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two
|
||||||
|
cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or
|
||||||
|
shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise
|
||||||
|
`latency_budget`, where requests must not wait.
|
||||||
|
|
||||||
|
## v0.14.6 — 2026-10-16
|
||||||
|
|
||||||
|
Setup that says what each part of an install is and lets every one be
|
||||||
|
chosen and set; three machines that each run the gateway, so any one can
|
||||||
|
be lost; a cluster that loses its store's primary without losing a
|
||||||
|
request; one ComfyUI for an environment's models; pictures, 3D and
|
||||||
|
workflows as MCP tools; alerts asked for at setup; extended thinking.
|
||||||
|
|
||||||
|
**Upgrading.**
|
||||||
|
- No migration.
|
||||||
|
- **Control planes**: with `control.lease_ttl` unset the lease is now 10 s
|
||||||
|
(was 15). Upgrade the standbys first and the leader last; a mix of
|
||||||
|
versions meanwhile is fine.
|
||||||
|
- **The store** (PostgreSQL): each member is dialled for 3 s before the
|
||||||
|
next is tried. A store so far away that connecting takes longer needs
|
||||||
|
`connect_timeout=10` in its address.
|
||||||
|
- **Nodes** notice a control plane that died without closing its
|
||||||
|
connection in 8 s (was 15) once they run this release.
|
||||||
|
- **Judges installed by an earlier `server init`** keep reserving no cores
|
||||||
|
and 2 GB. To give them their own:
|
||||||
|
`llmgw ctl model resources qwen3guard-0.6b --memory 4 --cores 2`, then
|
||||||
|
`llmgw ctl restart --model qwen3guard-0.6b`.
|
||||||
|
- **Patroni** replaces a lost primary within 20 s on a new cluster. On an
|
||||||
|
existing one, run `server init --patroni ...` again from the machine with
|
||||||
|
the authority; it changes only that.
|
||||||
|
- **A floating address** keeps VRRP router id 51 until `server add --vip`
|
||||||
|
runs again, which moves it to one of its own; run it for every standby.
|
||||||
|
- A request whose judge is not there yet waits up to the policy's
|
||||||
|
`latency_budget` for one before `fail: closed` refuses it.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Setup lets every part be chosen, each with its settings, and says what each is.** `sudo llmgw` on a new machine lists
|
||||||
|
the parts of an install: the control plane, the store, the model mirror, this machine as a node, the starter model,
|
||||||
|
pictures and 3D, the guardrails' judge, the firewall, notifications, limits and keys, on-demand turns, an offline
|
||||||
|
bundle. Beside each is what it will be or what is missing without it. Space turns a part on or off, and Enter opens
|
||||||
|
its settings in a dialog that explains it at more length. The settings are: which starter model (six Qwen sizes, or
|
||||||
|
any GGUF on Hugging Face) and its context; the mirror's directory or another machine's mirror; the judge's
|
||||||
|
categories, copies, wait and what happens when it does not answer; the portal's language; the sign-up form; the
|
||||||
|
default limits; the longest a key lives; the on-demand waits. A plan page then lists what is installed and what is
|
||||||
|
left out, with the same install as a command. Every setting is a `server init` flag too (`--starter`,
|
||||||
|
`--judge-fail`, `--media`, `--max-key-ttl`, ...).
|
||||||
|
- **Three machines are high availability for the gateway, not only the store.** With three machines, setup makes the two
|
||||||
|
database machines control planes too, standing by while the first leads, with an optional floating address
|
||||||
|
(keepalived) that follows whichever leads. Any one machine can now be lost. By command: `server init --patroni ...
|
||||||
|
--standbys --vip 10.0.0.160/24`. The mirror's management API answers the three control planes, so a standby that
|
||||||
|
comes to lead can still fetch models.
|
||||||
|
|
||||||
|
- **Pictures uploaded to work from**: `/v1/files` takes `purpose=vision` or
|
||||||
|
`user_data` besides `batch`, kept as long as job results, so a picture is
|
||||||
|
sent once and named by its `file-…` id in the MCP tools and a workflow's
|
||||||
|
inputs.
|
||||||
|
- **One ComfyUI for an environment's models**: ComfyUI models in one
|
||||||
|
`comfyui.environment` (two or more) share one replica — the catalog
|
||||||
|
makes it a hidden entry, `comfyui-env-<name>`, on the first member's
|
||||||
|
profiles with the most memory any member needs, carrying every member's
|
||||||
|
weights and custom nodes. Asking for any member starts it; it answers for
|
||||||
|
each with that member's workflow and checkpoint, and the members get no
|
||||||
|
replicas of their own. A picture model and a 3D model on one card stop
|
||||||
|
handing it to each other with a cold start each way.
|
||||||
|
- **Pictures, 3D models, vision and workflows as MCP tools** at `/mcp`,
|
||||||
|
beside the memory notes, each offered only where the gateway has a model
|
||||||
|
for it: `generate_image` (with `concept`, `side` and `texture` styles),
|
||||||
|
`generate_mesh` (from a picture or a prompt), `review_images` (a vision
|
||||||
|
model, to a JSON schema), `run_workflow`, `job_status` and
|
||||||
|
`gateway_status`. Each makes its request under the caller's key, so
|
||||||
|
allowlists, limits, licences and budgets apply and it is charged like
|
||||||
|
any request; what it makes is kept as the caller's file and answered by
|
||||||
|
id and download URL. A client that takes a stream gets
|
||||||
|
`notifications/progress` while a mesh is made and waits up to ten
|
||||||
|
minutes; one that does not, 50 seconds, then the job id.
|
||||||
- **Setup asks where alerts go**: `llmgw server init --webhook URL`
|
- **Setup asks where alerts go**: `llmgw server init --webhook URL`
|
||||||
(repeatable) and a **Notification webhooks** field on both setup screens.
|
(repeatable) and a **Notification webhooks** field on both setup screens.
|
||||||
Slack's, Mattermost's and Rocket.Chat's addresses are sent Slack's text,
|
Slack's, Mattermost's and Rocket.Chat's addresses are sent Slack's text,
|
||||||
@@ -48,6 +670,51 @@ APIs change between minors without a deprecation period.
|
|||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
- **The judges and the starter model divide a machine's CPUs instead of each taking all of them.** `server init` gave the
|
||||||
|
judge and the starter no cores of their own, so every llama.cpp engine ran on every core: two judges and the starter
|
||||||
|
on a 4-core control plane competed, and under load 38 of 60 requests were refused for want of a judgment. Moved to a
|
||||||
|
node of their own with 2 cores each, the same test refused 2. When this machine is the node, init now gives each judge
|
||||||
|
replica 1 core (2 from 8 cores) and the starter what is left (at most 8); judges bound for other nodes get 2 each. A
|
||||||
|
judge also reserves 4 GB, not 2: it uses about 3.2.
|
||||||
|
|
||||||
|
- **A lost store machine stops costing requests most of a minute.** The store's connections try each member for 3 seconds,
|
||||||
|
not until the kernel gives up, and a connection to a machine that is gone is closed within about 10 seconds (TCP
|
||||||
|
keepalive, and `TCP_USER_TIMEOUT` on Linux). Patroni replaces a lost primary within 20 seconds (`ttl` 20, `loop_wait`
|
||||||
|
5, `retry_timeout` 5). Running `server init --patroni` again applies this to an existing cluster. Measured on three
|
||||||
|
VMs with the primary's power cut: about 45 seconds of hanging requests before.
|
||||||
|
- **Leadership stays put while the store's primary is replaced.** Replacing a lost primary takes longer than the
|
||||||
|
leadership lease lives, so the lease had lapsed for everyone by the time the store answered again. Whoever asked first
|
||||||
|
took it, as often a standby as the leader. A standby that saw the store out of reach now leaves the lease alone for one
|
||||||
|
round after it returns, so the leader renews it, and a leader that is truly gone is replaced a round later. With that,
|
||||||
|
the default lease is 10 seconds (was 15). A node notices a control plane that died without closing its connection in
|
||||||
|
8 seconds (was 15), so it is on its way to the new leader as that one takes over.
|
||||||
|
- **A judge replica that hangs no longer costs the request.** All the tries of a judge request shared the policy's
|
||||||
|
budget, so a replica that hung used it all and the try on another replica had none left: under `fail: closed` the
|
||||||
|
request was refused while a judge stood idle. When the first replica has not answered by half the budget, the same
|
||||||
|
question now goes to a replica on another machine too, and the first answer is taken. The first replica is not cut
|
||||||
|
short, which on a slow machine would refuse answers it was about to give. Replicas on one machine whose CPUs are taken
|
||||||
|
are slow together, so asking both only doubles the work there: measured that way, as many requests failed as before.
|
||||||
|
On such a machine the remedy is room, not a second question: judges on more than one node, more cores, or the judge
|
||||||
|
on a GPU.
|
||||||
|
- **A new leader waits for the judge instead of refusing at once.** The moment a control plane takes the lead it knows no
|
||||||
|
judge yet. When every judge fails at once, they are asked again every 250 ms while the policy's `latency_budget` lasts,
|
||||||
|
so `fail: closed` no longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A
|
||||||
|
judge that is slow is not asked again, and an alarm is raised once per request, not once per try.
|
||||||
|
|
||||||
|
- Node discovery for Prometheus listed only the control plane asked and the
|
||||||
|
store, so a machine that is both was a control plane to one and a store
|
||||||
|
member to the other — scraped twice — and a standby asked alone lost every
|
||||||
|
node. Every control plane lists all of them alike; deploy/monitoring's
|
||||||
|
prometheus.yml says to scrape and ask each.
|
||||||
|
- A model whose weights and context pool did not fit its card was added
|
||||||
|
and then failed to make its context on every start, taking the card from
|
||||||
|
others each time (Apertus 70B with a 65536-token pool, 8B with 131072, on
|
||||||
|
24 GB). `ctl model serve` shrinks the pool to what the card holds, saying
|
||||||
|
so, and refuses weights that do not fit at all; the portal's serve form
|
||||||
|
refuses such a pool and says the largest that fits.
|
||||||
|
- Removing weights from the mirror that a catalog model still loads is
|
||||||
|
refused, naming the model: removed from the mirror alone, the model stayed
|
||||||
|
in the catalog and its replicas went on loading the node's copy.
|
||||||
- Jobs ran one at a time, so a 3D job waiting for the card held up another
|
- Jobs ran one at a time, so a 3D job waiting for the card held up another
|
||||||
model's job that could have run. They run side by side, one per model at
|
model's job that could have run. They run side by side, one per model at
|
||||||
a time and eight in all, on the leader as before.
|
a time and eight in all, on the leader as before.
|
||||||
|
|||||||
@@ -40,7 +40,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
|||||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves) |
|
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||||
|
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||||
|
|
||||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||||
@@ -50,6 +51,17 @@ budgets and rate limits decide who may use any of it.
|
|||||||
|
|
||||||
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
|
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
|
||||||
|
|
||||||
|
- **Setup that explains, and high availability that is whole** (v0.14.6).
|
||||||
|
`sudo llmgw` lists the parts of an install, each with what it is for and
|
||||||
|
what is missing without it, and every one can be turned off or set: the
|
||||||
|
starter model and its context, the mirror's directory, the judge's
|
||||||
|
categories and what it does when it does not answer, limits, the
|
||||||
|
portal's language, pictures and 3D. With three machines each runs the
|
||||||
|
control plane, with a floating address on the leader, and any one can
|
||||||
|
be lost: on three VMs, cutting the store primary's power lost no
|
||||||
|
request. One ComfyUI serves an environment's models, pictures, 3D and
|
||||||
|
workflows are MCP tools, and `/v1/messages` gives a model's reasoning as
|
||||||
|
thinking blocks.
|
||||||
- **Picture and 3D models that add themselves** (v0.14.5). Where a node's
|
- **Picture and 3D models that add themselves** (v0.14.5). Where a node's
|
||||||
card fits flux-schnell or triposr and the catalog makes nothing of that
|
card fits flux-schnell or triposr and the catalog makes nothing of that
|
||||||
kind, the leader adds it on demand, so a new cluster's portal has its
|
kind, the leader adds it on demand, so a new cluster's portal has its
|
||||||
@@ -172,6 +184,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
|||||||
cluster that exists
|
cluster that exists
|
||||||
- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
|
- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
|
||||||
Kubernetes with the Helm chart, and why GPU nodes stay outside it
|
Kubernetes with the Helm chart, and why GPU nodes stay outside it
|
||||||
|
- [docs/kong.md](docs/kong.md) — llmgw behind Kong Gateway, or Kong behind
|
||||||
|
llmgw: the configuration, what Kong counts, and which to choose
|
||||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||||
when a replica spanning nodes will not run
|
when a replica spanning nodes will not run
|
||||||
@@ -246,6 +260,33 @@ Add `--json` to any command for machine-readable output.
|
|||||||
says nothing about which of six constraints bit, so it prints every candidate
|
says nothing about which of six constraints bit, so it prints every candidate
|
||||||
considered and the reason it was rejected.
|
considered and the reason it was rejected.
|
||||||
|
|
||||||
|
## What llmgw adds to a request
|
||||||
|
|
||||||
|
Measured with `llmgw ctl bench overhead`. It starts an engine that answers
|
||||||
|
at once and a gateway in front of it, in one process on loopback. It sends
|
||||||
|
the same load straight to the engine and through the gateway, and reports
|
||||||
|
what the gateway added. Run on an Apple M5 (10 CPUs, load about 2) on
|
||||||
|
2026-10-16, 10 s a step, 72,000 requests with none failed:
|
||||||
|
|
||||||
|
| Through the gateway | added at p50 | added at p99 |
|
||||||
|
|---|---|---|
|
||||||
|
| bare | 0.06–0.24 ms | 0.10–0.33 ms |
|
||||||
|
| with authentication and usage accounting | 0.05–0.32 ms | 0.09–0.37 ms |
|
||||||
|
| with the guardrails' pattern checks as well | 0.10–0.42 ms | 0.15–0.49 ms |
|
||||||
|
|
||||||
|
The range is across 100 to 2,000 requests a second. The overhead was
|
||||||
|
*lowest* at 2,000, most likely because a busier processor keeps its cores
|
||||||
|
awake. The guardrails' judge model is not in these figures: a judgment
|
||||||
|
takes seconds on CPUs (see [install.md](docs/install.md)) and is a model's
|
||||||
|
time, not the gateway's.
|
||||||
|
|
||||||
|
The command also prints the gateway's CPU per thousand requests. Gateway,
|
||||||
|
engine and load share one process there, so that figure is rough (0.04 to
|
||||||
|
0.65 s per thousand in the run above); the latency is the one to quote.
|
||||||
|
Figures from a VM on a busy host move with the host: run it where you will
|
||||||
|
deploy, and keep the load it prints beside the result. CI runs a one-second
|
||||||
|
version on every change (`TestTheGatewaysOverheadIsMeasured`).
|
||||||
|
|
||||||
## Running the router
|
## Running the router
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -273,8 +314,9 @@ within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
|||||||
admission off.
|
admission off.
|
||||||
|
|
||||||
Routing profiles select how endpoints are scored: `chat` favours session
|
Routing profiles select how endpoints are scored: `chat` favours session
|
||||||
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
affinity and the replica that has been answering soonest (its time to first
|
||||||
cache. Pick one per model in the config, or per request with the
|
token, smoothed), `batch` favours the shortest queue, `long-context` favours
|
||||||
|
free KV cache. Pick one per model in the config, or per request with the
|
||||||
`x-llmgw-profile` header.
|
`x-llmgw-profile` header.
|
||||||
|
|
||||||
## sim-engine
|
## sim-engine
|
||||||
|
|||||||
@@ -1206,9 +1206,12 @@ type Host struct {
|
|||||||
CpuFlags []string `protobuf:"bytes,5,rep,name=cpu_flags,json=cpuFlags,proto3" json:"cpu_flags,omitempty"`
|
CpuFlags []string `protobuf:"bytes,5,rep,name=cpu_flags,json=cpuFlags,proto3" json:"cpu_flags,omitempty"`
|
||||||
// nvidia_driver is the driver version, empty when there is none. A card
|
// nvidia_driver is the driver version, empty when there is none. A card
|
||||||
// without a driver is not a GPU anything can use.
|
// without a driver is not a GPU anything can use.
|
||||||
NvidiaDriver string `protobuf:"bytes,6,opt,name=nvidia_driver,json=nvidiaDriver,proto3" json:"nvidia_driver,omitempty"`
|
NvidiaDriver string `protobuf:"bytes,6,opt,name=nvidia_driver,json=nvidiaDriver,proto3" json:"nvidia_driver,omitempty"`
|
||||||
unknownFields protoimpl.UnknownFields
|
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
|
||||||
sizeCache protoimpl.SizeCache
|
// the agent started: what decides a model's speed on the CPU. 0 unknown.
|
||||||
|
MemBandwidthGbs float64 `protobuf:"fixed64,7,opt,name=mem_bandwidth_gbs,json=memBandwidthGbs,proto3" json:"mem_bandwidth_gbs,omitempty"`
|
||||||
|
unknownFields protoimpl.UnknownFields
|
||||||
|
sizeCache protoimpl.SizeCache
|
||||||
}
|
}
|
||||||
|
|
||||||
func (x *Host) Reset() {
|
func (x *Host) Reset() {
|
||||||
@@ -1283,6 +1286,13 @@ func (x *Host) GetNvidiaDriver() string {
|
|||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (x *Host) GetMemBandwidthGbs() float64 {
|
||||||
|
if x != nil {
|
||||||
|
return x.MemBandwidthGbs
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
// Storage is one filesystem a node keeps weights on.
|
// Storage is one filesystem a node keeps weights on.
|
||||||
//
|
//
|
||||||
// Per filesystem rather than per configured directory: free space belongs to
|
// Per filesystem rather than per configured directory: free space belongs to
|
||||||
@@ -1579,7 +1589,14 @@ type GPU struct {
|
|||||||
// is then 1000 + 100 × card + device, uuid the instance's (MIG-…), and
|
// is then 1000 + 100 × card + device, uuid the instance's (MIG-…), and
|
||||||
// memory the instance's share; utilization_percent stays zero, because no
|
// memory the instance's share; utilization_percent stays zero, because no
|
||||||
// driver measures it per instance.
|
// driver measures it per instance.
|
||||||
Mig *MIGDevice `protobuf:"bytes,22,opt,name=mig,proto3" json:"mig,omitempty"`
|
Mig *MIGDevice `protobuf:"bytes,22,opt,name=mig,proto3" json:"mig,omitempty"`
|
||||||
|
// How the card is linked to its host: the fastest generation card and
|
||||||
|
// slot allow together, the card's own fastest, and the lanes in use and
|
||||||
|
// the card's most. Zero when the driver did not say.
|
||||||
|
PcieGen int32 `protobuf:"varint,23,opt,name=pcie_gen,json=pcieGen,proto3" json:"pcie_gen,omitempty"`
|
||||||
|
PcieGpuGen int32 `protobuf:"varint,24,opt,name=pcie_gpu_gen,json=pcieGpuGen,proto3" json:"pcie_gpu_gen,omitempty"`
|
||||||
|
PcieWidth int32 `protobuf:"varint,25,opt,name=pcie_width,json=pcieWidth,proto3" json:"pcie_width,omitempty"`
|
||||||
|
PcieWidthMax int32 `protobuf:"varint,26,opt,name=pcie_width_max,json=pcieWidthMax,proto3" json:"pcie_width_max,omitempty"`
|
||||||
unknownFields protoimpl.UnknownFields
|
unknownFields protoimpl.UnknownFields
|
||||||
sizeCache protoimpl.SizeCache
|
sizeCache protoimpl.SizeCache
|
||||||
}
|
}
|
||||||
@@ -1768,6 +1785,34 @@ func (x *GPU) GetMig() *MIGDevice {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (x *GPU) GetPcieGen() int32 {
|
||||||
|
if x != nil {
|
||||||
|
return x.PcieGen
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *GPU) GetPcieGpuGen() int32 {
|
||||||
|
if x != nil {
|
||||||
|
return x.PcieGpuGen
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *GPU) GetPcieWidth() int32 {
|
||||||
|
if x != nil {
|
||||||
|
return x.PcieWidth
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func (x *GPU) GetPcieWidthMax() int32 {
|
||||||
|
if x != nil {
|
||||||
|
return x.PcieWidthMax
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
// MIGDevice is which card a MIG instance is carved from, and how.
|
// MIGDevice is which card a MIG instance is carved from, and how.
|
||||||
type MIGDevice struct {
|
type MIGDevice struct {
|
||||||
state protoimpl.MessageState `protogen:"open.v1"`
|
state protoimpl.MessageState `protogen:"open.v1"`
|
||||||
@@ -2943,8 +2988,12 @@ type Heartbeat struct {
|
|||||||
// is not handed out. Empty from an agent that does not measure it, or from
|
// is not handed out. Empty from an agent that does not measure it, or from
|
||||||
// one that could not this time.
|
// one that could not this time.
|
||||||
GpuForeignMemoryMb []int32 `protobuf:"varint,19,rep,packed,name=gpu_foreign_memory_mb,json=gpuForeignMemoryMb,proto3" json:"gpu_foreign_memory_mb,omitempty"`
|
GpuForeignMemoryMb []int32 `protobuf:"varint,19,rep,packed,name=gpu_foreign_memory_mb,json=gpuForeignMemoryMb,proto3" json:"gpu_foreign_memory_mb,omitempty"`
|
||||||
unknownFields protoimpl.UnknownFields
|
// mem_bandwidth_gbs is how fast the node's memory reads, in GB/s, once
|
||||||
sizeCache protoimpl.SizeCache
|
// the agent has measured it — in the background after it starts, so a
|
||||||
|
// registration usually has none yet. 0 means not measured.
|
||||||
|
MemBandwidthGbs float64 `protobuf:"fixed64,20,opt,name=mem_bandwidth_gbs,json=memBandwidthGbs,proto3" json:"mem_bandwidth_gbs,omitempty"`
|
||||||
|
unknownFields protoimpl.UnknownFields
|
||||||
|
sizeCache protoimpl.SizeCache
|
||||||
}
|
}
|
||||||
|
|
||||||
func (x *Heartbeat) Reset() {
|
func (x *Heartbeat) Reset() {
|
||||||
@@ -3110,6 +3159,13 @@ func (x *Heartbeat) GetGpuForeignMemoryMb() []int32 {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (x *Heartbeat) GetMemBandwidthGbs() float64 {
|
||||||
|
if x != nil {
|
||||||
|
return x.MemBandwidthGbs
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
// Provision is the software a node should have: which engines, at which
|
// Provision is the software a node should have: which engines, at which
|
||||||
// versions. Nothing more.
|
// versions. Nothing more.
|
||||||
//
|
//
|
||||||
@@ -3790,7 +3846,10 @@ type EngineProfile struct {
|
|||||||
Adapters []*LoraAdapter `protobuf:"bytes,12,rep,name=adapters,proto3" json:"adapters,omitempty"`
|
Adapters []*LoraAdapter `protobuf:"bytes,12,rep,name=adapters,proto3" json:"adapters,omitempty"`
|
||||||
// kv_events has a vLLM engine publish its KV cache events to the agent, for
|
// kv_events has a vLLM engine publish its KV cache events to the agent, for
|
||||||
// the router's precise index.
|
// the router's precise index.
|
||||||
KvEvents bool `protobuf:"varint,13,opt,name=kv_events,json=kvEvents,proto3" json:"kv_events,omitempty"`
|
KvEvents bool `protobuf:"varint,13,opt,name=kv_events,json=kvEvents,proto3" json:"kv_events,omitempty"`
|
||||||
|
// allow_pickle lets the node fetch weights in a pickle format, which run
|
||||||
|
// code when loaded: the catalog's allow_pickle, set on purpose.
|
||||||
|
AllowPickle bool `protobuf:"varint,14,opt,name=allow_pickle,json=allowPickle,proto3" json:"allow_pickle,omitempty"`
|
||||||
unknownFields protoimpl.UnknownFields
|
unknownFields protoimpl.UnknownFields
|
||||||
sizeCache protoimpl.SizeCache
|
sizeCache protoimpl.SizeCache
|
||||||
}
|
}
|
||||||
@@ -3916,6 +3975,13 @@ func (x *EngineProfile) GetKvEvents() bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (x *EngineProfile) GetAllowPickle() bool {
|
||||||
|
if x != nil {
|
||||||
|
return x.AllowPickle
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// RemoveWeights deletes cached model files a node no longer needs.
|
// RemoveWeights deletes cached model files a node no longer needs.
|
||||||
//
|
//
|
||||||
// The paths are computed by the control plane from the catalog, but the agent
|
// The paths are computed by the control plane from the catalog, but the agent
|
||||||
@@ -5180,7 +5246,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1aA\n" +
|
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1aA\n" +
|
||||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xb1\x01\n" +
|
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xdd\x01\n" +
|
||||||
"\x04Host\x12\x13\n" +
|
"\x04Host\x12\x13\n" +
|
||||||
"\x05os_id\x18\x01 \x01(\tR\x04osId\x12\x1d\n" +
|
"\x05os_id\x18\x01 \x01(\tR\x04osId\x12\x1d\n" +
|
||||||
"\n" +
|
"\n" +
|
||||||
@@ -5189,7 +5255,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
"osCodename\x12\x12\n" +
|
"osCodename\x12\x12\n" +
|
||||||
"\x04arch\x18\x04 \x01(\tR\x04arch\x12\x1b\n" +
|
"\x04arch\x18\x04 \x01(\tR\x04arch\x12\x1b\n" +
|
||||||
"\tcpu_flags\x18\x05 \x03(\tR\bcpuFlags\x12#\n" +
|
"\tcpu_flags\x18\x05 \x03(\tR\bcpuFlags\x12#\n" +
|
||||||
"\rnvidia_driver\x18\x06 \x01(\tR\fnvidiaDriver\"u\n" +
|
"\rnvidia_driver\x18\x06 \x01(\tR\fnvidiaDriver\x12*\n" +
|
||||||
|
"\x11mem_bandwidth_gbs\x18\a \x01(\x01R\x0fmemBandwidthGbs\"u\n" +
|
||||||
"\aStorage\x12\x12\n" +
|
"\aStorage\x12\x12\n" +
|
||||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
|
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
|
||||||
"\n" +
|
"\n" +
|
||||||
@@ -5219,7 +5286,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
"leaderPort\x12\x1d\n" +
|
"leaderPort\x12\x1d\n" +
|
||||||
"\n" +
|
"\n" +
|
||||||
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12+\n" +
|
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12+\n" +
|
||||||
"\acompute\x18\x10 \x01(\v2\x11.llmgw.v1.ComputeR\acompute\"\xca\x06\n" +
|
"\acompute\x18\x10 \x01(\v2\x11.llmgw.v1.ComputeR\acompute\"\xcc\a\n" +
|
||||||
"\x03GPU\x12\x14\n" +
|
"\x03GPU\x12\x14\n" +
|
||||||
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x12\n" +
|
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x12\n" +
|
||||||
"\x04uuid\x18\x02 \x01(\tR\x04uuid\x12\x12\n" +
|
"\x04uuid\x18\x02 \x01(\tR\x04uuid\x12\x12\n" +
|
||||||
@@ -5245,7 +5312,13 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
"\fcluster_uuid\x18\x13 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
"\fcluster_uuid\x18\x13 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
||||||
"\tclique_id\x18\x14 \x01(\tR\bcliqueId\x12)\n" +
|
"\tclique_id\x18\x14 \x01(\tR\bcliqueId\x12)\n" +
|
||||||
"\x10fabric_bandwidth\x18\x15 \x01(\tR\x0ffabricBandwidth\x12%\n" +
|
"\x10fabric_bandwidth\x18\x15 \x01(\tR\x0ffabricBandwidth\x12%\n" +
|
||||||
"\x03mig\x18\x16 \x01(\v2\x13.llmgw.v1.MIGDeviceR\x03mig\x1a;\n" +
|
"\x03mig\x18\x16 \x01(\v2\x13.llmgw.v1.MIGDeviceR\x03mig\x12\x19\n" +
|
||||||
|
"\bpcie_gen\x18\x17 \x01(\x05R\apcieGen\x12 \n" +
|
||||||
|
"\fpcie_gpu_gen\x18\x18 \x01(\x05R\n" +
|
||||||
|
"pcieGpuGen\x12\x1d\n" +
|
||||||
|
"\n" +
|
||||||
|
"pcie_width\x18\x19 \x01(\x05R\tpcieWidth\x12$\n" +
|
||||||
|
"\x0epcie_width_max\x18\x1a \x01(\x05R\fpcieWidthMax\x1a;\n" +
|
||||||
"\rNicPathsEntry\x12\x10\n" +
|
"\rNicPathsEntry\x12\x10\n" +
|
||||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x84\x02\n" +
|
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x84\x02\n" +
|
||||||
@@ -5361,7 +5434,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
|
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
|
||||||
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
||||||
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
|
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
|
||||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xa9\a\n" +
|
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xd5\a\n" +
|
||||||
"\tHeartbeat\x12\x17\n" +
|
"\tHeartbeat\x12\x17\n" +
|
||||||
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
|
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
|
||||||
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
|
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
|
||||||
@@ -5383,7 +5456,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
"gpu_fabric\x18\x10 \x03(\v2\x18.llmgw.v1.GPUFabricStateR\tgpuFabric\x12\"\n" +
|
"gpu_fabric\x18\x10 \x03(\v2\x18.llmgw.v1.GPUFabricStateR\tgpuFabric\x12\"\n" +
|
||||||
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
|
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
|
||||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x121\n" +
|
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x121\n" +
|
||||||
"\x15gpu_foreign_memory_mb\x18\x13 \x03(\x05R\x12gpuForeignMemoryMb\x1aA\n" +
|
"\x15gpu_foreign_memory_mb\x18\x13 \x03(\x05R\x12gpuForeignMemoryMb\x12*\n" +
|
||||||
|
"\x11mem_bandwidth_gbs\x18\x14 \x01(\x01R\x0fmemBandwidthGbs\x1aA\n" +
|
||||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||||
@@ -5462,7 +5536,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
|
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
|
||||||
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" +
|
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" +
|
||||||
"\vcpu_threads\x18\x11 \x01(\x05R\n" +
|
"\vcpu_threads\x18\x11 \x01(\x05R\n" +
|
||||||
"cpuThreads\"\xcd\x03\n" +
|
"cpuThreads\"\xf0\x03\n" +
|
||||||
"\rEngineProfile\x12\x12\n" +
|
"\rEngineProfile\x12\x12\n" +
|
||||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||||
"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
|
"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
|
||||||
@@ -5478,7 +5552,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
|||||||
" \x01(\tR\tmodelPath\x12*\n" +
|
" \x01(\tR\tmodelPath\x12*\n" +
|
||||||
"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x121\n" +
|
"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x121\n" +
|
||||||
"\badapters\x18\f \x03(\v2\x15.llmgw.v1.LoraAdapterR\badapters\x12\x1b\n" +
|
"\badapters\x18\f \x03(\v2\x15.llmgw.v1.LoraAdapterR\badapters\x12\x1b\n" +
|
||||||
"\tkv_events\x18\r \x01(\bR\bkvEvents\x1a6\n" +
|
"\tkv_events\x18\r \x01(\bR\bkvEvents\x12!\n" +
|
||||||
|
"\fallow_pickle\x18\x0e \x01(\bR\vallowPickle\x1a6\n" +
|
||||||
"\bEnvEntry\x12\x10\n" +
|
"\bEnvEntry\x12\x10\n" +
|
||||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"%\n" +
|
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"%\n" +
|
||||||
|
|||||||
@@ -238,6 +238,9 @@ message Host {
|
|||||||
// nvidia_driver is the driver version, empty when there is none. A card
|
// nvidia_driver is the driver version, empty when there is none. A card
|
||||||
// without a driver is not a GPU anything can use.
|
// without a driver is not a GPU anything can use.
|
||||||
string nvidia_driver = 6;
|
string nvidia_driver = 6;
|
||||||
|
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
|
||||||
|
// the agent started: what decides a model's speed on the CPU. 0 unknown.
|
||||||
|
double mem_bandwidth_gbs = 7;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Storage is one filesystem a node keeps weights on.
|
// Storage is one filesystem a node keeps weights on.
|
||||||
@@ -334,6 +337,13 @@ message GPU {
|
|||||||
// memory the instance's share; utilization_percent stays zero, because no
|
// memory the instance's share; utilization_percent stays zero, because no
|
||||||
// driver measures it per instance.
|
// driver measures it per instance.
|
||||||
MIGDevice mig = 22;
|
MIGDevice mig = 22;
|
||||||
|
// How the card is linked to its host: the fastest generation card and
|
||||||
|
// slot allow together, the card's own fastest, and the lanes in use and
|
||||||
|
// the card's most. Zero when the driver did not say.
|
||||||
|
int32 pcie_gen = 23;
|
||||||
|
int32 pcie_gpu_gen = 24;
|
||||||
|
int32 pcie_width = 25;
|
||||||
|
int32 pcie_width_max = 26;
|
||||||
}
|
}
|
||||||
|
|
||||||
// MIGDevice is which card a MIG instance is carved from, and how.
|
// MIGDevice is which card a MIG instance is carved from, and how.
|
||||||
@@ -588,6 +598,11 @@ message Heartbeat {
|
|||||||
// is not handed out. Empty from an agent that does not measure it, or from
|
// is not handed out. Empty from an agent that does not measure it, or from
|
||||||
// one that could not this time.
|
// one that could not this time.
|
||||||
repeated int32 gpu_foreign_memory_mb = 19;
|
repeated int32 gpu_foreign_memory_mb = 19;
|
||||||
|
|
||||||
|
// mem_bandwidth_gbs is how fast the node's memory reads, in GB/s, once
|
||||||
|
// the agent has measured it — in the background after it starts, so a
|
||||||
|
// registration usually has none yet. 0 means not measured.
|
||||||
|
double mem_bandwidth_gbs = 20;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Provision is the software a node should have: which engines, at which
|
// Provision is the software a node should have: which engines, at which
|
||||||
@@ -749,6 +764,9 @@ message EngineProfile {
|
|||||||
// kv_events has a vLLM engine publish its KV cache events to the agent, for
|
// kv_events has a vLLM engine publish its KV cache events to the agent, for
|
||||||
// the router's precise index.
|
// the router's precise index.
|
||||||
bool kv_events = 13;
|
bool kv_events = 13;
|
||||||
|
// allow_pickle lets the node fetch weights in a pickle format, which run
|
||||||
|
// code when loaded: the catalog's allow_pickle, set on purpose.
|
||||||
|
bool allow_pickle = 14;
|
||||||
}
|
}
|
||||||
|
|
||||||
// RemoveWeights deletes cached model files a node no longer needs.
|
// RemoveWeights deletes cached model files a node no longer needs.
|
||||||
|
|||||||
+37
-12
@@ -1,4 +1,4 @@
|
|||||||
# Two control planes
|
# More than one control plane
|
||||||
|
|
||||||
llmgw handles the hard half of this itself: only one control plane places work,
|
llmgw handles the hard half of this itself: only one control plane places work,
|
||||||
and agents find whichever one is leading. What it does not do is move client
|
and agents find whichever one is leading. What it does not do is move client
|
||||||
@@ -37,7 +37,7 @@ nothing in front of the two had moved.
|
|||||||
So something has to. The options, in the order most people should consider
|
So something has to. The options, in the order most people should consider
|
||||||
them:
|
them:
|
||||||
|
|
||||||
**A load balancer you already run.** Health-check `/ready` on each control
|
**A load balancer you already run.** Health-check `/leader` on each control
|
||||||
plane and send traffic to whichever answers. This is the right answer if you
|
plane and send traffic to whichever answers. This is the right answer if you
|
||||||
have one, and it needs nothing else from this directory.
|
have one, and it needs nothing else from this directory.
|
||||||
|
|
||||||
@@ -50,25 +50,50 @@ in whatever your resolvers actually honour rather than what you set, which on
|
|||||||
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
||||||
one an agent is driving.
|
one an agent is driving.
|
||||||
|
|
||||||
## Check `/ready`, not `/health`
|
## Check `/leader`, not `/health`
|
||||||
|
|
||||||
This is the one thing to get right, and the obvious choice is the wrong one.
|
This is the one thing to get right, and the obvious choice is the wrong one.
|
||||||
|
|
||||||
`/health` says the process is up. It says nothing about whether it can serve,
|
`/health` says the process is up. It says nothing about whether it leads,
|
||||||
and a standby is exactly that case: alive, answering, and holding no endpoints
|
and a standby is exactly that case: alive, answering, and holding no endpoints
|
||||||
at all — its reconciler placed nothing and it has no agent streams to adopt
|
at all — its reconciler placed nothing and it has no agent streams to adopt
|
||||||
from. A load balancer checking `/health` would send it half the traffic, and
|
from. A standby passes client requests on to the leader, so traffic sent to
|
||||||
half the traffic would be a 503.
|
one still works — until the leader is the machine that was lost, and there
|
||||||
|
is nobody to pass them to.
|
||||||
|
|
||||||
|
`/leader` is 200 on the control plane that leads and 503 on a standby. It is
|
||||||
|
what keepalived follows for the floating address, and what a load balancer
|
||||||
|
should send traffic by: to the one machine that answers 200.
|
||||||
|
|
||||||
|
`/ready` is narrower: 200 only when this process has somewhere to send a
|
||||||
|
request, so 503 on a standby and also on a leader with no model running yet.
|
||||||
|
Checking it takes the portal and the admin pages away with the models — on a
|
||||||
|
new cluster with no node yet, nobody can sign in to add one. Use it to stop
|
||||||
|
sending inference only, if your load balancer can tell the paths apart.
|
||||||
|
|
||||||
`/ready` says this process has somewhere to send a request. It is 503 on a
|
|
||||||
standby, 503 on a leader whose replicas have all failed, and 200 when there is
|
|
||||||
something to serve — which is the question a load balancer is actually asking.
|
|
||||||
Both are open, because a load balancer has no key.
|
Both are open, because a load balancer has no key.
|
||||||
|
|
||||||
During a failover neither passes for about the length of the lease handover:
|
During a failover nothing passes for about the length of the lease handover:
|
||||||
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
||||||
is not. That gap is real and there is no way to have it both ways — traffic
|
is not. A proxy that waits that long for a healthy machine rather than
|
||||||
cannot go to a control plane that has nothing to route to.
|
failing at once (Caddy's `lb_try_duration`) hides it from clients.
|
||||||
|
|
||||||
|
**Caddy**, in front of three control planes — the end of
|
||||||
|
[deploy/proxy/Caddyfile](../proxy/Caddyfile) has it with the rest of the
|
||||||
|
settings:
|
||||||
|
|
||||||
|
```
|
||||||
|
reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
|
||||||
|
lb_policy first
|
||||||
|
health_uri /leader
|
||||||
|
health_interval 2s
|
||||||
|
lb_try_duration 15s
|
||||||
|
flush_interval -1
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
With a floating address the address already follows the leader: give Caddy
|
||||||
|
that one address instead.
|
||||||
|
|
||||||
Not the admin port, for a different reason: it serves pprof, and pprof can
|
Not the admin port, for a different reason: it serves pprof, and pprof can
|
||||||
dump the process heap, where API keys sit in plaintext.
|
dump the process heap, where API keys sit in plaintext.
|
||||||
|
|||||||
@@ -41,7 +41,9 @@ vrrp_instance llmgw {
|
|||||||
# MASTER on the machine you would rather serve; BACKUP on the other.
|
# MASTER on the machine you would rather serve; BACKUP on the other.
|
||||||
state MASTER
|
state MASTER
|
||||||
interface eth0
|
interface eth0
|
||||||
# Any number both machines agree on and nothing else on the segment uses.
|
# Any number both machines agree on and nothing else on the segment uses:
|
||||||
|
# another cluster's keepalived with the same one reads as an invalid
|
||||||
|
# password. `llmgw server add --vip` derives one from the address.
|
||||||
virtual_router_id 51
|
virtual_router_id 51
|
||||||
# The higher one wins while both are healthy. The weight above is enough
|
# The higher one wins while both are healthy. The weight above is enough
|
||||||
# to invert this when the preferred machine's gateway is not answering.
|
# to invert this when the preferred machine's gateway is not answering.
|
||||||
|
|||||||
@@ -21,11 +21,18 @@ scrape_configs:
|
|||||||
# The control plane's dedicated metrics port. Not its admin listener: that one
|
# The control plane's dedicated metrics port. Not its admin listener: that one
|
||||||
# is loopback because it also serves pprof, and pprof can dump the heap where
|
# is loopback because it also serves pprof, and pprof can dump the heap where
|
||||||
# API keys sit in plaintext.
|
# API keys sit in plaintext.
|
||||||
|
#
|
||||||
|
# Every control plane, one line each: the leader moves on restarts and the
|
||||||
|
# router's metrics are the leading one's, so scraping one alone leaves gaps
|
||||||
|
# — and alerts on them — whenever another leads.
|
||||||
- job_name: llmgw
|
- job_name: llmgw
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: ['10.0.0.140:9091']
|
- targets: ['10.0.0.140:9091']
|
||||||
labels:
|
labels:
|
||||||
instance: control-plane
|
instance: control-plane-140
|
||||||
|
# - targets: ['10.0.0.150:9091']
|
||||||
|
# labels:
|
||||||
|
# instance: control-plane-150
|
||||||
|
|
||||||
# Every machine of the cluster, for its node exporter: network and disk
|
# Every machine of the cluster, for its node exporter: network and disk
|
||||||
# traffic, filesystems, CPU time. The control plane lists them — nodes,
|
# traffic, filesystems, CPU time. The control plane lists them — nodes,
|
||||||
@@ -33,10 +40,15 @@ scrape_configs:
|
|||||||
# nothing edited here, and each carries its llmgw id as the node label.
|
# nothing edited here, and each carries its llmgw id as the node label.
|
||||||
# `llmgw node add` installs the exporter; on a machine added before that,
|
# `llmgw node add` installs the exporter; on a machine added before that,
|
||||||
# apt-get install prometheus-node-exporter.
|
# apt-get install prometheus-node-exporter.
|
||||||
|
# Asked of every control plane too: a standby lists only the control
|
||||||
|
# planes and the store, and they label every machine alike, so the same
|
||||||
|
# target from two of them is scraped once.
|
||||||
- job_name: nodes
|
- job_name: nodes
|
||||||
http_sd_configs:
|
http_sd_configs:
|
||||||
- url: http://10.0.0.140:9091/prometheus/targets
|
- url: http://10.0.0.140:9091/prometheus/targets
|
||||||
refresh_interval: 1m
|
refresh_interval: 1m
|
||||||
|
# - url: http://10.0.0.150:9091/prometheus/targets
|
||||||
|
# refresh_interval: 1m
|
||||||
|
|
||||||
- job_name: prometheus
|
- job_name: prometheus
|
||||||
static_configs:
|
static_configs:
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ llm.example.com {
|
|||||||
max_size 32MB
|
max_size 32MB
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# One control plane. With more than one, see the end of this file.
|
||||||
reverse_proxy 127.0.0.1:8080 {
|
reverse_proxy 127.0.0.1:8080 {
|
||||||
# Tokens are written the moment they are generated. Without this a
|
# Tokens are written the moment they are generated. Without this a
|
||||||
# streamed reply can arrive in bursts, and a client measuring time to
|
# streamed reply can arrive in bursts, and a client measuring time to
|
||||||
@@ -39,3 +40,36 @@ llm.example.com {
|
|||||||
# Access log to stderr, which under systemd is the journal.
|
# Access log to stderr, which under systemd is the journal.
|
||||||
log
|
log
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# More than one control plane (server init --standbys, server add): one of
|
||||||
|
# them leads, the others stand by. Name them all and send to the one that
|
||||||
|
# leads — /leader is 200 there and 503 on a standby. Not /ready: that is 503
|
||||||
|
# on a leader with no model running yet, and would take the portal away with
|
||||||
|
# it. A standby would pass a request on to the leader anyway, but not while
|
||||||
|
# there is none, and the hop is for nothing.
|
||||||
|
#
|
||||||
|
# With a floating address (--vip) the address already follows the leader:
|
||||||
|
# give reverse_proxy that one address instead, and keep the rest as above.
|
||||||
|
#
|
||||||
|
# llm.example.com {
|
||||||
|
# request_body {
|
||||||
|
# max_size 32MB
|
||||||
|
# }
|
||||||
|
# reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
|
||||||
|
# # The first that is healthy: the leader, whichever it is.
|
||||||
|
# lb_policy first
|
||||||
|
# health_uri /leader
|
||||||
|
# health_interval 2s
|
||||||
|
# health_timeout 2s
|
||||||
|
# # While leadership moves — a few seconds when a leader is lost —
|
||||||
|
# # none is healthy; a request waits that long for one rather than
|
||||||
|
# # failing at once.
|
||||||
|
# lb_try_duration 15s
|
||||||
|
# lb_try_interval 250ms
|
||||||
|
# flush_interval -1
|
||||||
|
# transport http {
|
||||||
|
# response_header_timeout 15m
|
||||||
|
# }
|
||||||
|
# }
|
||||||
|
# log
|
||||||
|
# }
|
||||||
+1171
-1
File diff suppressed because it is too large.
Load diff
@@ -82,7 +82,7 @@
|
|||||||
"/mcp": {
|
"/mcp": {
|
||||||
"post": {
|
"post": {
|
||||||
"operationId": "postMcp",
|
"operationId": "postMcp",
|
||||||
"summary": "The gateway's MCP server: the memory notes as tools",
|
"summary": "The gateway's MCP server: the memory notes, and pictures, 3D models, vision and workflows, as tools",
|
||||||
"tags": [
|
"tags": [
|
||||||
"mcp"
|
"mcp"
|
||||||
],
|
],
|
||||||
|
|||||||
@@ -534,6 +534,60 @@ curl https://llm.example.com/mcp \
|
|||||||
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
|
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Pictures, 3D models, vision and workflows.** Where the gateway has the
|
||||||
|
models for them, the same server offers more tools — only those it has a
|
||||||
|
model for:
|
||||||
|
|
||||||
|
| Tool | Does |
|
||||||
|
|---|---|
|
||||||
|
| `generate_image` | A picture from a prompt. `style` adds what a picture for a mesh wants: `concept` (one object, three-quarter view, grey background), `side` (an exact side view), `texture` (a seamless tile) or `none`. Answers the PNG's `file_id`, `url` and `seed`, with a 512 px preview to look at. |
|
||||||
|
| `generate_mesh` | A GLB from a picture — `image` is a `file-…` id (a `generate_image` result, say), a data URL or base64 — or from a `prompt`, for which a concept picture is made first. `seed`, `resolution`, `texture_resolution`, and any other input the model's workflow declares in `inputs`. Answers the GLB's `file_id`, `url` and licence. |
|
||||||
|
| `review_images` | Asks a vision model about one to four pictures (each shrunk to 1280 px on its long side); `json_schema` makes it answer that JSON. |
|
||||||
|
| `run_workflow` | Runs a ComfyUI workflow from the inputs its catalog entry declares, and answers its result file. |
|
||||||
|
| `job_status` | Where a mesh or a workflow run stands, and its file when done; `wait_seconds` waits for it. |
|
||||||
|
| `gateway_status` | The picture, 3D, vision and workflow models, whether each is loaded on a GPU now, a workflow's inputs, and your last jobs. |
|
||||||
|
|
||||||
|
Each tool makes its request of the gateway under your key, so your
|
||||||
|
allowlist, limits, left-out licences and budget apply, and it is charged
|
||||||
|
and listed like any other request. What it makes is kept as your file for
|
||||||
|
as long as job results are: fetch it from `url` with the same key (`curl -H
|
||||||
|
"Authorization: Bearer $LLMGW_API_KEY" -o rock.glb <url>`). A tool server
|
||||||
|
over HTTP cannot write to your disk, so an agent that needs the file
|
||||||
|
locally — to import it in Blender — fetches it, or goes through a small
|
||||||
|
local bridge that does.
|
||||||
|
|
||||||
|
A picture can be uploaded once and named by its id after: `POST /v1/files`
|
||||||
|
with `purpose=vision` (or `user_data`) keeps it, for as long as job results
|
||||||
|
are kept, as a file of yours that `generate_mesh`, `review_images` and a
|
||||||
|
workflow's file inputs take as `file-…`. A call is at most 64 MB.
|
||||||
|
|
||||||
|
A model that is not loaded is waited for: a picture or a look at one whose
|
||||||
|
model is starting, or waiting for another to give up the GPU, waits as a
|
||||||
|
mesh does, saying so, rather than failing with "model loading".
|
||||||
|
|
||||||
|
A mesh takes from half a minute to a few minutes. A client that takes the
|
||||||
|
answer as a stream (`Accept: text/event-stream`, as Claude Code does) is
|
||||||
|
answered that way: `notifications/progress` for the call's
|
||||||
|
`progressToken` while it runs — queued, waiting for the GPU, the node and
|
||||||
|
its percent — then the result, and the call waits up to ten minutes. One
|
||||||
|
answered in one piece waits 50 seconds; a job not done by then is answered
|
||||||
|
with its `job_id`, to ask after with `job_status`. `wait_seconds` (up to
|
||||||
|
900) sets either.
|
||||||
|
|
||||||
|
```
|
||||||
|
claude mcp add --transport http llmgw https://llm.example.com/mcp \
|
||||||
|
--header "Authorization: Bearer $LLMGW_API_KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Your company's tools.** Where an administrator has registered the
|
||||||
|
company's own MCP servers (Jira, a file server, a database) and your key is
|
||||||
|
granted their tools, the same `/mcp` lists them too, each as `server.tool`
|
||||||
|
(`jira.search`). Give your agent only llmgw's `/mcp` and your key: the
|
||||||
|
servers' own credentials stay with llmgw. A tool your key is not granted is
|
||||||
|
not listed, and calling it by name is refused. Each call is in the audit
|
||||||
|
log under your key: which tool, the size of what was sent, how it ended and
|
||||||
|
how long it took, never what it said.
|
||||||
|
|
||||||
### Licences you leave out
|
### Licences you leave out
|
||||||
|
|
||||||
A model's licence is in `/v1/models` (`license`: its `id`, `name`, whether
|
A model's licence is in `/v1/models` (`license`: its `id`, `name`, whether
|
||||||
|
|||||||
+56
-4
@@ -2,8 +2,8 @@
|
|||||||
|
|
||||||
`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
|
`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
|
||||||
with nothing installed it opens the setup screens; on a control plane, the
|
with nothing installed it opens the setup screens; on a control plane, the
|
||||||
console described here; on a node, the node's own console, which edits
|
console described here; on a node, the node's own console, which shows
|
||||||
`agent.yaml`. Everything on these screens can also be done by command — each
|
what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
|
||||||
screen says which — so a script never needs them and a person never needs
|
screen says which — so a script never needs them and a person never needs
|
||||||
to remember the commands.
|
to remember the commands.
|
||||||
|
|
||||||
@@ -173,6 +173,23 @@ search…** as an administrator, or **Delete…** it. People
|
|||||||
fill collections from their own machines with `llmgw-kb`
|
fill collections from their own machines with `llmgw-kb`
|
||||||
([knowledge.md](knowledge.md)). Off on a SQLite store, and says why.
|
([knowledge.md](knowledge.md)). Off on a SQLite store, and says why.
|
||||||
|
|
||||||
|
**MCP servers** lists the company's own MCP servers that agents reach through
|
||||||
|
the gateway's `/mcp`, their tools named `server.tool`: whether each is on,
|
||||||
|
who has been granted its tools, its rate per caller, the names of the
|
||||||
|
credentials it is sent (never their values) and its address. **A** adds one:
|
||||||
|
its name, which is its tools' prefix, its address, and a bearer token or
|
||||||
|
another header, kept sealed in the store. On a row: **Test, and list its
|
||||||
|
tools**, with its credential, as a call would; **Grant its tools to…** a
|
||||||
|
team, project, organization, person or key — all of them, `jira.*`, or some
|
||||||
|
by name — beside whatever else that level is granted, and saying when a
|
||||||
|
level above keeps them out of reach; **Revoke** a grant;
|
||||||
|
**Change…** its address, timeout or rate; **Replace its credential…**;
|
||||||
|
**Turn off** (its tools are listed nowhere and every call refused) or on;
|
||||||
|
**Remove…**. A server's tools are seen nowhere until granted. The screen
|
||||||
|
also says how a person points an agent at the gateway, with a key of their
|
||||||
|
own from `/profile/keys`. `llmgw ctl mcp` does the same from a shell, and
|
||||||
|
the web page **MCP servers** (`/admin/mcp`, administrators only) in a browser.
|
||||||
|
|
||||||
**Mirror** is what the model mirror holds, one row for each version of each
|
**Mirror** is what the model mirror holds, one row for each version of each
|
||||||
model — a file, or the set of files a model of several is: shards, a vision
|
model — a file, or the set of files a model of several is: shards, a vision
|
||||||
projector, a config and tokenizer — with how far its download has got and
|
projector, a config and tokenizer — with how far its download has got and
|
||||||
@@ -246,6 +263,28 @@ long a key made there may live, and is the expiry of one made without any
|
|||||||
Limits are requests and tokens a minute, a budget per day or month, and the
|
Limits are requests and tokens a minute, a budget per day or month, and the
|
||||||
models allowed. Each level only narrows what is above it.
|
models allowed. Each level only narrows what is above it.
|
||||||
|
|
||||||
|
**Sign-in** — how people get in besides a password kept here: single
|
||||||
|
sign-on, Active Directory and provisioning, each on or off, with the
|
||||||
|
directory's last sync (how many people, accounts kept in step, suspended,
|
||||||
|
reinstated).
|
||||||
|
- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
|
||||||
|
and the groups for administrators and operators. A provider other than
|
||||||
|
Entra takes its issuer URL in place of the tenant. The form shows the
|
||||||
|
redirect address to register.
|
||||||
|
- **D** sets up Active Directory: the servers, where people are, the
|
||||||
|
service account and its password, the domain's CA certificate as a PEM
|
||||||
|
file on this machine, the group whose members may sign in, and groups
|
||||||
|
for roles and teams.
|
||||||
|
- Before it is kept, the directory is tried from the control plane: each
|
||||||
|
server reached and bound, how many people may sign in, how many are in
|
||||||
|
each group. If any step fails, nothing is kept and the steps are shown.
|
||||||
|
**T** tries the settings in force.
|
||||||
|
- On the provisioning row, Enter makes a SCIM token, shown once.
|
||||||
|
- The password, the client secret and the token are kept in the store,
|
||||||
|
sealed by the master key when one is set. The settings name them as
|
||||||
|
`store:auth/…`, so each control plane has them with no file put on it.
|
||||||
|
The CA is written into the settings themselves.
|
||||||
|
|
||||||
**Guardrails…** on a row lists the guardrail policies server.yaml defines.
|
**Guardrails…** on a row lists the guardrail policies server.yaml defines.
|
||||||
Each policy is marked in one of three ways:
|
Each policy is marked in one of three ways:
|
||||||
|
|
||||||
@@ -335,5 +374,18 @@ to restart it.
|
|||||||
|
|
||||||
## The node's console
|
## The node's console
|
||||||
|
|
||||||
On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
|
On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
|
||||||
manual. The cluster is watched and changed from a control plane.
|
on the machine itself, so it works without an admin token and while the control
|
||||||
|
plane is out of reach. Refreshed every three seconds, it shows
|
||||||
|
|
||||||
|
- llmgw-agent's state and since when, and the version of llmgw;
|
||||||
|
- each control plane the agent dials, from `agent.yaml` and the list the
|
||||||
|
leader last sent, and whether it can be reached;
|
||||||
|
- each GPU: how busy, its memory, its heat, and the models on it;
|
||||||
|
- the engines running here, from the agent's own records: replica, model,
|
||||||
|
engine, GPUs, port, whether the process is still there, and for how long;
|
||||||
|
- the free space where weights and engines are kept;
|
||||||
|
- the agent's warnings and errors of the last hour.
|
||||||
|
|
||||||
|
Below it, `agent.yaml` in the same settings view, and the manual. The cluster
|
||||||
|
is watched and changed from a control plane.
|
||||||
+3
-1
@@ -72,7 +72,9 @@ each answer that may be reused, as the client got it, and a hash of the
|
|||||||
request it answers, in the gateway process's memory — never in the store,
|
request it answers, in the gateway process's memory — never in the store,
|
||||||
never on disk — for the project's TTL or until pushed out by newer ones
|
never on disk — for the project's TTL or until pushed out by newer ones
|
||||||
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
||||||
policy. Forgetting a person does not reach into it; the answers age out by
|
policy. A project that reuses answers to similar questions (`--similar`)
|
||||||
|
also has each question's last user message embedded, and the vector kept
|
||||||
|
beside its answer, in the same memory and for as long. Forgetting a person does not reach into it; the answers age out by
|
||||||
the TTL. The store keeps only the setting, and a usage record's mark that it
|
the TTL. The store keeps only the setting, and a usage record's mark that it
|
||||||
was answered from the cache.
|
was answered from the cache.
|
||||||
|
|
||||||
|
|||||||
@@ -91,6 +91,21 @@ guard:
|
|||||||
value is the same placeholder throughout a request).
|
value is the same placeholder throughout a request).
|
||||||
- **`block`** stops it.
|
- **`block`** stops it.
|
||||||
|
|
||||||
|
**Putting them back.** With `restore: true` in a policy's `input`, the
|
||||||
|
values a request had redacted are put back in its answer: the model reads
|
||||||
|
"write to [PERSON_1] at [EMAIL_1]", the client reads the name and address
|
||||||
|
it sent. Streamed answers too — a placeholder split across two events is
|
||||||
|
held until it is whole. Only that request's own values, only into its
|
||||||
|
answer; the map is kept in memory for the request and nowhere else. The
|
||||||
|
output rules then read the answer as the client will, so a policy that
|
||||||
|
redacts personal data in answers still does.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
input:
|
||||||
|
pii: {email: redact, phone: redact}
|
||||||
|
restore: true
|
||||||
|
```
|
||||||
|
|
||||||
**Shapes.**
|
**Shapes.**
|
||||||
|
|
||||||
- `system_prompt` is put first.
|
- `system_prompt` is put first.
|
||||||
@@ -258,6 +273,47 @@ themselves.
|
|||||||
context rather than overflowing it and going unjudged.
|
context rather than overflowing it and going unjudged.
|
||||||
- **Cache:** the same text is not asked about twice within five minutes.
|
- **Cache:** the same text is not asked about twice within five minutes.
|
||||||
|
|
||||||
|
## An outside service
|
||||||
|
|
||||||
|
A site required to use a particular guardrail — Azure Content Safety,
|
||||||
|
Lakera Guard, NeMo Guardrails, a classifier of its own — names it in a
|
||||||
|
policy's `services`. It is asked over HTTP beside the patterns and before
|
||||||
|
the judge; its "unsafe" blocks or flags like theirs, and a service that does
|
||||||
|
not answer is what the policy's `fail` says.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
policies:
|
||||||
|
strict:
|
||||||
|
fail: closed
|
||||||
|
services:
|
||||||
|
- name: lakera
|
||||||
|
url: https://api.lakera.ai/v2/guard
|
||||||
|
header_files: {Authorization: /etc/llmgw/secrets/lakera} # "Bearer …", read each call
|
||||||
|
body: '{"messages": [{"role": "user", "content": {{json .Text}}}]}'
|
||||||
|
blocked: flagged # where the answer says unsafe
|
||||||
|
reason: breakdown.0.detector_type
|
||||||
|
on: both # input, output or both
|
||||||
|
action: block # or flag
|
||||||
|
timeout: 2s
|
||||||
|
```
|
||||||
|
|
||||||
|
`body` is a template given `.Text` (`{{json .Text}}` writes it as a JSON
|
||||||
|
string; empty sends `{"input": …}`). `blocked` and `reason` are dotted paths
|
||||||
|
into the JSON answer, numbers for list items; true, a number above zero, a
|
||||||
|
non-empty list or a string other than "false" is unsafe. On input the
|
||||||
|
service reads the latest user turn as the model will, with what `redact`
|
||||||
|
took out still out; on output the whole answer, once, at its end, as the
|
||||||
|
client reads it — with `restore`, the values put back. Being at the end, a
|
||||||
|
service can stop a streamed answer only where it is held back
|
||||||
|
(`stream: buffer_all`): released a sentence at a time, the client has read
|
||||||
|
it by then, and what the service finds ends the stream with
|
||||||
|
`content_filter`, flags it and is recorded. The rule recorded is
|
||||||
|
`service:<name>:<reason>`. The service is told the text only, never who
|
||||||
|
sent it. Under `fail: open`, the default, a service that cannot be asked
|
||||||
|
lets the text through — and a caller can make it fail, sending what it
|
||||||
|
refuses to read; give `fail: closed` where the service must decide. Of an
|
||||||
|
answer asked for with `n` above 1, the service reads the first.
|
||||||
|
|
||||||
## Where a policy applies
|
## Where a policy applies
|
||||||
|
|
||||||
`guard.default` applies everywhere. Others are attached to a part of the
|
`guard.default` applies everywhere. Others are attached to a part of the
|
||||||
|
|||||||
+520
-28
@@ -89,13 +89,36 @@ CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
|
|||||||
|
|
||||||
**The short way is the screens.** On the machine that will be the control
|
**The short way is the screens.** On the machine that will be the control
|
||||||
plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
|
plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
|
||||||
whether this is one machine or three for high availability; for three, it asks
|
whether this is one machine or three for high availability, then lists what
|
||||||
for the other two — address, user, and an ssh key, the ssh agent, or a
|
an install is made of, each part with what it is for and what is missing
|
||||||
password, and the sudo password if sudo asks for one; or, for a machine
|
without it:
|
||||||
nothing may log into, a join code to run there with `sudo llmgw join <code>`,
|
|
||||||
which dials out to this one on port 7444 — checks all three before
|
| Part | | Its settings (Enter) |
|
||||||
installing anything (system, room, clock, ports, internet, whether they reach
|
|---|---|---|
|
||||||
one another), shows the plan, and installs. Passwords are held in memory for
|
| Control plane | always | the administrator, other names for the certificate, the portal's language, the sign-up form |
|
||||||
|
| Store | always | SQLite or PostgreSQL on this machine; with three machines, PostgreSQL under Patroni on the other two |
|
||||||
|
| Control plane on all three | three machines, on | a floating address for whichever leads |
|
||||||
|
| Model mirror | on | here, and in which directory, or on another machine |
|
||||||
|
| This machine as a node | on for one machine | |
|
||||||
|
| Starter model | on for one machine | which: Qwen2.5 0.5B, 1.5B or 3B, Qwen3 4B, 8B or 14B, or any GGUF on Hugging Face; its context |
|
||||||
|
| Pictures and 3D | off | |
|
||||||
|
| Guardrails' judge | on | refuse or serve flagged when it does not answer; copies; how long it is waited for; what it refuses |
|
||||||
|
| Firewall | on | |
|
||||||
|
| Notifications | off | the webhooks |
|
||||||
|
| Limits and keys | always | requests and tokens a minute; the longest a key lives |
|
||||||
|
| Taking turns on a card | always | how long an on-demand model goes unused, and is safe after loading |
|
||||||
|
| Offline bundle | off | the bundle and its sha256 |
|
||||||
|
|
||||||
|
Space turns a part on or off; Enter opens its settings, which say more. What
|
||||||
|
cannot go together is said before anything is done — a starter model with no
|
||||||
|
mirror to fetch it into — and closing the message opens the setting that is
|
||||||
|
wrong. For three machines it then asks for the other two — address, user, and
|
||||||
|
an ssh key, the ssh agent, or a password, and the sudo password if sudo asks
|
||||||
|
for one; or, for a machine nothing may log into, a join code to run there
|
||||||
|
with `sudo llmgw join <code>`, which dials out to this one on port 7444 —
|
||||||
|
checks all three before installing anything (system, room, clock, ports,
|
||||||
|
internet, whether they reach one another). Last it shows the plan, part by
|
||||||
|
part, what is left out and what that costs, and installs. Passwords are held in memory for
|
||||||
the run only; a sudo password lets the user sudo without one until the install
|
the run only; a sudo password lets the user sudo without one until the install
|
||||||
finishes, and is then taken back. At the end it shows the administrator's
|
finishes, and is then taken back. At the end it shows the administrator's
|
||||||
password and the command that does the same from a script.
|
password and the command that does the same from a script.
|
||||||
@@ -129,15 +152,24 @@ That is the whole step. In order, it:
|
|||||||
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
||||||
engine
|
engine
|
||||||
9. closes llmgw's internal ports to everything but the cluster's networks
|
9. closes llmgw's internal ports to everything but the cluster's networks
|
||||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
|
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
|
||||||
replicas on the CPUs of nodes with room — this one, or the first that
|
CPUs: one replica on up to four cores when this machine is the only
|
||||||
join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||||
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||||
first thing anyone opens `/chat` to find is a model that answers. A
|
first thing anyone opens `/chat` to find is a model that answers. A
|
||||||
machine without room for it is told so within two minutes, and why.
|
machine without room for it is told so within two minutes, and why.
|
||||||
|
|
||||||
`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9,
|
`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9,
|
||||||
`--no-guard-model` 10 and `--no-model` 11. An NVIDIA card with no working
|
`--no-guard-model` 10 and `--no-model` 11. Every setting the screens ask for
|
||||||
|
is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
|
||||||
|
(or `hf:<owner>/<repo>/<file>.gguf`) and `--starter-context`, `--mirror-dir`,
|
||||||
|
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
|
||||||
|
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
|
||||||
|
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
|
||||||
|
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
|
||||||
|
go into `server.yaml` as init writes it, and become the cluster's first
|
||||||
|
settings when it starts; a `server.yaml` already there is never overwritten,
|
||||||
|
so run again they change nothing — the settings page does. An NVIDIA card with no working
|
||||||
driver is reported rather than silently ignored; install the driver and run
|
driver is reported rather than silently ignored; install the driver and run
|
||||||
init again to serve on it.
|
init again to serve on it.
|
||||||
|
|
||||||
@@ -155,15 +187,43 @@ demonstration one:
|
|||||||
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
||||||
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
||||||
|
|
||||||
The judge runs as two replicas, each on a CPU — about half a second for a
|
The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
|
||||||
short text on any CPU with AVX2 — and reads a streamed answer every 600
|
cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
|
||||||
characters. The policy fails **closed**: a request the judge cannot read
|
streamed answer every 600 characters. The policy fails **closed**: a request
|
||||||
within 5 seconds is refused with 503 rather than served unjudged, and
|
the judge cannot read within 5 seconds is refused with 503 rather than
|
||||||
`judge_unavailable` goes to the notification webhooks. Two replicas are what
|
served unjudged, and `judge_unavailable` goes to the notification webhooks.
|
||||||
make that bearable; on one small machine, or a VM without AVX2 (see
|
So a judge gets cores of its own: on one machine one replica on up to four,
|
||||||
|
and on a larger cluster two replicas of two cores each, spread over its nodes
|
||||||
|
so that losing one leaves the other. On a machine of fewer than four cores,
|
||||||
|
a busy one, or a VM without AVX2 (see
|
||||||
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
||||||
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
|
first thing to fail. The choices there are a longer `latency_budget`,
|
||||||
`--no-guard-model` is the choice there. Defamation, specialized advice,
|
`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
|
||||||
|
or `--no-guard-model`.
|
||||||
|
|
||||||
|
init times the judge once it is up. If a judgment takes more than half the
|
||||||
|
wait, or more than the wait when several come at once, it says so before the
|
||||||
|
next steps, with what to do. The wait itself is not changed without being
|
||||||
|
asked. `llmgw ctl guard judge check` does the same at any time:
|
||||||
|
|
||||||
|
```
|
||||||
|
qwen3guard-0.6b, 2 replica(s) ready, a request waits 5.0 s for it
|
||||||
|
alone: 2.4, 2.3, 2.3 s
|
||||||
|
4 at once: slowest 6.1 s; about 0.7 judgments a second, so about 0.3 requests a second with short answers
|
||||||
|
tight: a busy moment passes the budget
|
||||||
|
```
|
||||||
|
|
||||||
|
In a three-machine install the judges are better on machines of their own.
|
||||||
|
On a 4-core control plane that was also a node, 38 of 60 requests under load
|
||||||
|
were refused for want of a judgment. With the judges on a node of their own
|
||||||
|
(4 cores, two judges of two cores each), 2 were refused. The setup screen asks
|
||||||
|
for up to two **judge machines** after the store's two, or give
|
||||||
|
`--judge-node user@host` to `server init`, once for each. Each is added as a
|
||||||
|
node labelled `llmgw.role: judge`, and the judge's profile selects that
|
||||||
|
label, so the judges run only there and nothing else does. Four cores and
|
||||||
|
8 GB are enough for two judges.
|
||||||
|
|
||||||
|
Defamation, specialized advice,
|
||||||
intellectual property and elections are left out of what it refuses: they
|
intellectual property and elections are left out of what it refuses: they
|
||||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||||
|
|
||||||
@@ -331,7 +391,11 @@ by itself. Three things are worth knowing before you edit it:
|
|||||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||||
budgets mean anything. Without a price every request costs zero and a budget
|
budgets mean anything. Without a price every request costs zero and a budget
|
||||||
never binds; without the defaults, a project created without limits is
|
never binds; without the defaults, a project created without limits is
|
||||||
metered by nothing at all.
|
metered by nothing at all. A price may set `cached_per_1k` beside
|
||||||
|
`prompt_per_1k`: the prompt tokens the engine or provider read from its
|
||||||
|
cache (the answer's `cached_tokens`) are charged at it, the rest at the
|
||||||
|
prompt rate. A provider that bills cached reads at a tenth is
|
||||||
|
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
|
||||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||||
is empty. Without it, a file with no models is refused as having nothing to
|
is empty. Without it, a file with no models is refused as having nothing to
|
||||||
serve.
|
serve.
|
||||||
@@ -812,6 +876,27 @@ Nothing to do here if you built the mirror in step 3 — the agent fetches what
|
|||||||
model's `source` names, and that source is your mirror. This section is for the
|
model's `source` names, and that source is your mirror. This section is for the
|
||||||
arrangements where the files are already on the machine.
|
arrangements where the files are already on the machine.
|
||||||
|
|
||||||
|
**Weights in a pickle format are refused.** `.bin`, `.pt`, `.pth`, `.ckpt` and
|
||||||
|
`.pkl` are Python pickles, and loading one runs whatever it holds, as the
|
||||||
|
engine's user, on the node — which is how the malicious models found on Hugging
|
||||||
|
Face work. A model whose source is one, or a mirror version holding one, is
|
||||||
|
refused when it is added and by the node before it fetches anything; vLLM and
|
||||||
|
SGLang given a model by its Hugging Face id read only its safetensors. Take the
|
||||||
|
repository's safetensors or GGUF. A model that has no other form, and whose
|
||||||
|
files were checked, can say so:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: triposr
|
||||||
|
source: http://mirror:8081/triposr/<rev>/model.ckpt
|
||||||
|
allow_pickle: true
|
||||||
|
```
|
||||||
|
|
||||||
|
A local path is judged by its name as any source is; a local directory is
|
||||||
|
not looked into. What an engine downloads itself by a Hugging Face id, past
|
||||||
|
the node — an adapter given by its id, a TensorRT-LLM model's — is refused
|
||||||
|
when the model is added, unless it says `allow_pickle`: mirror its
|
||||||
|
safetensors and give that instead.
|
||||||
|
|
||||||
Put them where the node can read them. `scp`, a shared filesystem, or the
|
Put them where the node can read them. `scp`, a shared filesystem, or the
|
||||||
engine's own downloader are all valid, and which you want depends on how many
|
engine's own downloader are all valid, and which you want depends on how many
|
||||||
nodes you have.
|
nodes you have.
|
||||||
@@ -970,6 +1055,27 @@ A node with no location is placed as before, as a rack of its own.
|
|||||||
See [facility-aware-placement.md](facility-aware-placement.md) for power caps
|
See [facility-aware-placement.md](facility-aware-placement.md) for power caps
|
||||||
and holds.
|
and holds.
|
||||||
|
|
||||||
|
### A card in the wrong slot
|
||||||
|
|
||||||
|
An x16 card in a slot wired x8 or x4, behind a riser, or in a slot of an
|
||||||
|
older PCIe generation than the card's, links slower than it was built to and
|
||||||
|
works otherwise. A model on the card still decodes at the card's own memory
|
||||||
|
speed, but it loads slower, a replica split over several cards by PCIe is
|
||||||
|
slower, and so is anything that spills to the host. Each card's agent reads
|
||||||
|
how its link stands — the fastest generation card and slot allow together,
|
||||||
|
and the lanes in use — against what the card could do, and the node says so
|
||||||
|
among its warnings (the dashboard, the console's Nodes, `llmgw doctor`):
|
||||||
|
|
||||||
|
```
|
||||||
|
GPU 1 (NVIDIA H100 80GB HBM3) linked at x8 of its x16 lanes: 31.5 GB/s where it
|
||||||
|
could have 63.0. Check the slot (wired fewer lanes, or an older generation), a
|
||||||
|
riser, or the BIOS's PCIe setting
|
||||||
|
```
|
||||||
|
|
||||||
|
The generation a card reports now drops while it idles, to save power; what is
|
||||||
|
compared is the fastest it may reach, which does not. `lspci -vv -s BUS-ID`
|
||||||
|
shows the same: `LnkCap` is the card's, `LnkSta` the link's.
|
||||||
|
|
||||||
## 6. Setting the node's CPU model, if it is a VM
|
## 6. Setting the node's CPU model, if it is a VM
|
||||||
|
|
||||||
Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
|
Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
|
||||||
@@ -1287,6 +1393,26 @@ catalog, and a key rotated in the file is used within seconds. A model whose key
|
|||||||
file cannot be read is not offered. Everything else applies as to any model:
|
file cannot be read is not offered. Everything else applies as to any model:
|
||||||
prices and budgets, rate limits, usage records, and key allowlists.
|
prices and budgets, rate limits, usage records, and key allowlists.
|
||||||
|
|
||||||
|
A provider's limits are per key. With several keys, one answering 429 need not
|
||||||
|
hold up the model: give them all, each with an optional weight (its share
|
||||||
|
against the others'), in place of `api_key_file`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
external:
|
||||||
|
url: https://api.openai.com
|
||||||
|
model: gpt-4o-mini
|
||||||
|
api_key_files:
|
||||||
|
- {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
|
||||||
|
- {file: /etc/llmgw/secrets/openai-2.key}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
|
||||||
|
answers 429 is passed over while its breaker is open, and the request goes to
|
||||||
|
another. One that answers 401 (revoked, expired, mistyped) is passed over too,
|
||||||
|
and `external_key_refused` goes to the notification webhooks, naming the
|
||||||
|
endpoint and never the key. The usage records say which key answered by the
|
||||||
|
same name. A key whose file cannot be read is left out, and the others stay.
|
||||||
|
|
||||||
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
||||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||||
by name. The cluster page and `ctl models` mark it `external`.
|
by name. The cluster page and `ctl models` mark it `external`.
|
||||||
@@ -1503,6 +1629,79 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
|
|||||||
fallback (the catalog in the file is checked before the peers are read), and
|
fallback (the catalog in the file is checked before the peers are read), and
|
||||||
a peer behind a private CA needs that CA in the system's trust store.
|
a peer behind a private CA needs that CA in the system's trust store.
|
||||||
|
|
||||||
|
### Stopping a model, and removing one with its files
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw ctl model stop qwen-7b # none of its replicas run; it is kept
|
||||||
|
llmgw ctl model start qwen-7b # runs again, scaled as it was
|
||||||
|
llmgw ctl model remove qwen-7b # leaves the catalog; its files stay
|
||||||
|
llmgw ctl model remove qwen-7b --purge # its files on the mirror and the nodes too
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Stop** keeps the model in the catalog with its files and what it was
|
||||||
|
scaled to; a request for it is told an administrator stopped it. `ctl
|
||||||
|
models` shows it as stopped. A `ctl scale` while it is stopped starts it at
|
||||||
|
that number.
|
||||||
|
- **Remove** leaves its files on the mirror and the nodes, for adding it
|
||||||
|
again without fetching it. `ctl weights` lists such files on the nodes.
|
||||||
|
- **Remove --purge** deletes every version of it on the mirror (a second
|
||||||
|
mirror follows), its adapters and the weights its last update replaced,
|
||||||
|
and the nodes' copies once its replicas have stopped. What another model
|
||||||
|
uses — a canary's weights, a shared adapter — stays. It cannot be undone:
|
||||||
|
adding it again fetches it again.
|
||||||
|
|
||||||
|
The console has the same in a model's menu: Stop… or Start, Remove…, and
|
||||||
|
Remove with its files….
|
||||||
|
|
||||||
|
### Changing a served model's context
|
||||||
|
|
||||||
|
How much one request may use is set when a model is served. To change it
|
||||||
|
afterwards:
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw ctl model context qwen-7b 32768 # tokens a request
|
||||||
|
llmgw ctl model context qwen-7b 32768 --slots 4 # llama.cpp: four at once
|
||||||
|
```
|
||||||
|
|
||||||
|
The catalog, the engine's arguments (`--ctx-size` and `--parallel` for
|
||||||
|
llama.cpp, `--max-model-len` for vLLM, `--context-length` for SGLang) and
|
||||||
|
the memory its KV cache takes change together. A context longer than the
|
||||||
|
model was trained for is refused, and so is one its cards cannot hold — with
|
||||||
|
the figures, so you know how far to go. Its replicas are then started again on
|
||||||
|
it, one by one, a new one ready before the old one stops. The console has the
|
||||||
|
same under a model's menu, Context length…. A model declared in the cluster's
|
||||||
|
settings is changed there (`llmgw ctl config edit`).
|
||||||
|
|
||||||
|
### Moving a served model to more cards
|
||||||
|
|
||||||
|
How many cards of one node each replica uses can be changed while the model
|
||||||
|
is served:
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw ctl model gpus qwen3-coder-30b 2 # two cards a replica
|
||||||
|
llmgw ctl model gpus llama-70b 4 --profile h100 # the model has several profiles on cards
|
||||||
|
llmgw ctl model gpus qwen3-coder-30b 2 --in-place # no room beside the running replica
|
||||||
|
```
|
||||||
|
|
||||||
|
vLLM, SGLang and TensorRT-LLM spread the model by tensor parallel (`tp`),
|
||||||
|
which needs the count to divide the model's KV heads or be divided by them:
|
||||||
|
a model with 8 splits 2, 4, 8 or 16 ways, not 3 or 12. llama.cpp has no
|
||||||
|
tensor parallel; it splits the layers over the cards (`gpus_per_node`), and
|
||||||
|
a `--tensor-split` written for the old cards is dropped. ComfyUI and Strata
|
||||||
|
run on one card. A count no node has is refused, and so is one whose cards
|
||||||
|
would not hold the weights and the context — with the figures; fewer cards
|
||||||
|
with a long context may need `llmgw ctl model context` first.
|
||||||
|
|
||||||
|
Its replicas are then started again on it, each new one ready beside the old
|
||||||
|
before the old stops. That needs the cards free beside it: on a node of two
|
||||||
|
cards with the model on one, two do not fit until it is gone, and the command
|
||||||
|
says so. `--in-place` stops each replica first and starts its replacement on
|
||||||
|
the cards it held and the free ones, and the model does not answer while the
|
||||||
|
new one loads — a model with several replicas keeps answering on the others.
|
||||||
|
A replica spread over several nodes is laid out by hand, with a fabric
|
||||||
|
(`nodes`, `gpus_per_node` in the profile). The console has the same under a
|
||||||
|
model's menu, GPUs….
|
||||||
|
|
||||||
### Trying new weights on some of the traffic first
|
### Trying new weights on some of the traffic first
|
||||||
|
|
||||||
When a model's repository publishes new weights, `llmgw ctl model update`
|
When a model's repository publishes new weights, `llmgw ctl model update`
|
||||||
@@ -1877,6 +2076,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
|
|||||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||||
time went to the download, the launch or the engine loading.
|
time went to the download, the launch or the engine loading.
|
||||||
|
|
||||||
|
An inference request and each `upstream` attempt also carry the
|
||||||
|
[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
|
||||||
|
so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
|
||||||
|
llmgw's calls as model calls without a dashboard made for llmgw:
|
||||||
|
`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
|
||||||
|
`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
|
||||||
|
when the client set them, `gen_ai.conversation.id` (the session),
|
||||||
|
`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
|
||||||
|
`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
|
||||||
|
engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
|
||||||
|
what was answered are never in a span.
|
||||||
|
|
||||||
A node follows the control plane's decision: a start the control plane kept is
|
A node follows the control plane's decision: a start the control plane kept is
|
||||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||||
@@ -2041,8 +2252,10 @@ llmgw keys cache --project Acme/ml/train --off
|
|||||||
- **Policies still apply:** the limits are checked and the guardrails judge
|
- **Policies still apply:** the limits are checked and the guardrails judge
|
||||||
the request before the cache is looked in, and the output policy judges a
|
the request before the cache is looked in, and the output policy judges a
|
||||||
reused answer as it would a new one. A request continuing a stored
|
reused answer as it would a new one. A request continuing a stored
|
||||||
conversation, or under a `no_store` policy, is never answered from the
|
conversation, under a `no_store` policy, or whose redacted values are put
|
||||||
cache nor kept in it.
|
back in its answer (`restore: true`) is never answered from the cache nor
|
||||||
|
kept in it: redacted, two callers' requests read the same, and the answer
|
||||||
|
has the first caller's values in it.
|
||||||
- **What a hit costs:** nothing. It is recorded as a request — status 200,
|
- **What a hit costs:** nothing. It is recorded as a request — status 200,
|
||||||
the model, the key, the rate limit's count — with no tokens and no cost,
|
the model, the key, the rate limit's count — with no tokens and no cost,
|
||||||
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
|
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
|
||||||
@@ -2062,8 +2275,46 @@ router:
|
|||||||
max_entry_bytes: 2097152 # 2 MB
|
max_entry_bytes: 2097152 # 2 MB
|
||||||
```
|
```
|
||||||
|
|
||||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
### Similar questions
|
||||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
|
||||||
|
A help desk asked "how do I reset my password" in forty wordings asks the
|
||||||
|
model forty times under the exact cache. `--similar` also gives a chat
|
||||||
|
question the answer to one like it:
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
|
||||||
|
```
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
router:
|
||||||
|
cache:
|
||||||
|
embedding_model: bge-m3 # an embedding model in the catalog
|
||||||
|
```
|
||||||
|
|
||||||
|
- **What is compared:** the last user message, embedded by
|
||||||
|
`router.cache.embedding_model`, against the questions already answered
|
||||||
|
in the same conversation — the same system prompt, the same earlier turns,
|
||||||
|
tools, `response_format` and sampling settings, the same model and
|
||||||
|
project. A like question in another conversation, after a tool's result,
|
||||||
|
or with a picture, a sound or a file in it, is not taken for this one. The exact cache is asked first, and an
|
||||||
|
exact hit embeds nothing.
|
||||||
|
- **How alike:** the cosine similarity of the two embeddings, at least the
|
||||||
|
project's `--similar`. 0.95 is strict; below 0.9 different questions start
|
||||||
|
to meet. A question that differs only in a number or a name ("2+2" and
|
||||||
|
"2+3") can be closer than that, so a project whose questions turn on
|
||||||
|
figures should not use it. A question over 2,000 characters is reused only
|
||||||
|
when it is the same.
|
||||||
|
- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
|
||||||
|
and `Age`. Recorded and charged as a hit. The same rules as a hit decide
|
||||||
|
which answers are reused and kept; guardrails judge the request before.
|
||||||
|
- **The embedding** is asked of the model by the gateway on its own account,
|
||||||
|
as a judge is, within 2 seconds; every miss waits for it. Without the
|
||||||
|
model, or when it does not answer, only the same request is reused. The
|
||||||
|
vectors are kept in memory with their answers and counted in
|
||||||
|
`router.cache.max_bytes`.
|
||||||
|
|
||||||
|
`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
|
||||||
|
and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||||
console has it as **Cache answers…** on a project.
|
console has it as **Cache answers…** on a project.
|
||||||
|
|
||||||
## Batches
|
## Batches
|
||||||
@@ -2172,6 +2423,7 @@ triposr`. What that adds is this entry, and its workflow is built in:
|
|||||||
```yaml
|
```yaml
|
||||||
- name: triposr
|
- name: triposr
|
||||||
source: hf:stabilityai/TripoSR/model.ckpt
|
source: hf:stabilityai/TripoSR/model.ckpt
|
||||||
|
allow_pickle: true # published as this checkpoint only
|
||||||
comfyui:
|
comfyui:
|
||||||
workflow: triposr # built in; a model of your own names a JSON file on every control plane
|
workflow: triposr # built in; a model of your own names a JSON file on every control plane
|
||||||
output: meshes
|
output: meshes
|
||||||
@@ -2319,6 +2571,62 @@ Without authentication, `threads.store` and `threads.retain_hours` in
|
|||||||
`server.yaml` keep both conversations and notes in one shared pool. That is for
|
`server.yaml` keep both conversations and notes in one shared pool. That is for
|
||||||
one trusted machine only; see [development.md](development.md).
|
one trusted machine only; see [development.md](development.md).
|
||||||
|
|
||||||
|
## Three machines, each a control plane
|
||||||
|
|
||||||
|
The screens' three machines are high availability for the gateway as well as
|
||||||
|
the store: the two database machines are made control planes too, standing
|
||||||
|
by while the first leads — what `server add`, below, does for one machine.
|
||||||
|
By command, `--standbys` with `--patroni`, and `--vip` for the floating
|
||||||
|
address:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --admin you@example.com \
|
||||||
|
--patroni ubuntu@10.0.0.6 --patroni ubuntu@10.0.0.7 \
|
||||||
|
--standbys --vip 10.0.0.160/24 --no-node --no-model
|
||||||
|
```
|
||||||
|
|
||||||
|
| Machine | Runs |
|
||||||
|
|---|---|
|
||||||
|
| the first | control plane, leading at first; model mirror; etcd |
|
||||||
|
| the other two | control plane, standing by; PostgreSQL under Patroni; etcd |
|
||||||
|
|
||||||
|
etcd votes on all three and PostgreSQL is on two, so any one machine can be
|
||||||
|
lost: a standby leads within seconds, Patroni promotes the standby database
|
||||||
|
if it was the primary, and the floating address follows whichever leads. The
|
||||||
|
mirror's management API answers the three control planes' addresses, not
|
||||||
|
only its own machine's, so a standby that comes to lead can still fetch and
|
||||||
|
add models.
|
||||||
|
|
||||||
|
**A second mirror.** The mirror is on the first machine, and without a
|
||||||
|
second one, models the nodes already have keep serving while that machine
|
||||||
|
is down, and new ones wait for it. With a second, the cluster carries on.
|
||||||
|
- **Setting it up:** the screens offer one on either database machine
|
||||||
|
("A second mirror"); by command, `--second-mirror 1` (or `2`). For an
|
||||||
|
install that has one mirror, run this on the mirror's machine:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo llmgw mirror add-peer ubuntu@10.0.0.7 --control-plane 10.0.0.5 --control-plane 10.0.0.6 --control-plane 10.0.0.7
|
||||||
|
```
|
||||||
|
|
||||||
|
It logs in with root's ssh key on the mirror's machine, so that key must
|
||||||
|
be accepted for the user named on the second machine. When the mirror has
|
||||||
|
a machine of its own, without the control plane, the command cannot change
|
||||||
|
the cluster's settings itself: it prints the `llmgw ctl config set
|
||||||
|
mirror.peers …` to run on a control plane.
|
||||||
|
- **Kept in step:** each mirror reads the other every thirty seconds and
|
||||||
|
copies what it lacks, keeping a copy only if its checksum matches.
|
||||||
|
- A model removed from one is removed from the other, even if that one
|
||||||
|
was down at the time and comes back later.
|
||||||
|
- A model added again after it was removed stands on both.
|
||||||
|
- Neither mirror leads.
|
||||||
|
- **Control planes:** they name both (`mirror.peers` in the cluster's
|
||||||
|
settings) and send the mirror's management calls to the first that
|
||||||
|
answers.
|
||||||
|
- **Nodes:** while the mirror a model's address names is down, a node is
|
||||||
|
given the same file on the other.
|
||||||
|
- **Disk:** the second machine needs room for every model, as the first
|
||||||
|
does.
|
||||||
|
|
||||||
## Two control planes
|
## Two control planes
|
||||||
|
|
||||||
Once the store is replicated (`server init --patroni`, above), one command on
|
Once the store is replicated (`server init --patroni`, above), one command on
|
||||||
@@ -2335,8 +2643,12 @@ and lets it into the store; it starts there as a standby. Every node is told
|
|||||||
of it within half a minute and fails over to it without being touched.
|
of it within half a minute and fails over to it without being touched.
|
||||||
`--vip` keeps a floating address on whichever control plane leads, with
|
`--vip` keeps a floating address on whichever control plane leads, with
|
||||||
keepalived on both: that is the address to give clients, or a proxy in front.
|
keepalived on both: that is the address to give clients, or a proxy in front.
|
||||||
Without it, put a load balancer in front that checks `/ready` — not
|
Without it, put a proxy in front that sends to whichever answers `/leader`
|
||||||
`/health`, which is 200 on a standby that would refuse every request.
|
with 200 — not `/health`, which is 200 on a standby too, and not `/ready`,
|
||||||
|
which is 503 on a leader with no model yet and would take the portal away
|
||||||
|
with it. The end of [deploy/proxy/Caddyfile](../deploy/proxy/Caddyfile) has
|
||||||
|
Caddy doing that for three control planes, and
|
||||||
|
[deploy/ha/](../deploy/ha/README.md#check-leader-not-health) why.
|
||||||
|
|
||||||
- **What a failover costs.** Measured on the reference cluster, stopping the
|
- **What a failover costs.** Measured on the reference cluster, stopping the
|
||||||
leader: the standby led within five seconds, every node had moved to it
|
leader: the standby led within five seconds, every node had moved to it
|
||||||
@@ -2397,6 +2709,89 @@ the same files and keys as `locales/en/`, and its `_meta.*` entries saying how
|
|||||||
it writes numbers and dates. The tests refuse a catalogue that lacks a key a
|
it writes numbers and dates. The tests refuse a catalogue that lacks a key a
|
||||||
page uses.
|
page uses.
|
||||||
|
|
||||||
|
## The company's MCP servers
|
||||||
|
|
||||||
|
llmgw's `/mcp` can serve the company's own MCP servers beside its own tools.
|
||||||
|
Each server, a ticket system, a file server or a database, is registered once.
|
||||||
|
Its credentials stay on the control planes, and people's agents are given only
|
||||||
|
`/mcp` and their llmgw key.
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw ctl mcp add jira --url https://jira.internal/mcp \
|
||||||
|
--description "Acme Jira" --secret-file Authorization=/root/jira-token
|
||||||
|
llmgw ctl mcp test jira # its tools, asked with its credentials
|
||||||
|
llmgw keys limits --scope team --id Acme/support --tools 'jira.*'
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Credentials:**
|
||||||
|
- `--secret-file NAME=FILE` reads a header's value once and keeps it in
|
||||||
|
the store, sealed by the master key, so no file has to be on each
|
||||||
|
control plane.
|
||||||
|
- `--header-ref NAME=REF` reads it at each call from a file, `env:`,
|
||||||
|
`vault:` or `store:` instead.
|
||||||
|
- `--header NAME=VALUE` is for what is not secret.
|
||||||
|
- **Who may call what:** tools are named `server.tool` and granted as
|
||||||
|
patterns (`jira.*`, `db.query`) at any level of the tenancy tree, as
|
||||||
|
models are.
|
||||||
|
- A level that names any narrows what is above it: a tool is seen where
|
||||||
|
every level above that names tools names it too. A team under an
|
||||||
|
organization that names `hr.*` sees no `jira.*` whatever the team is
|
||||||
|
given, until the organization names it as well.
|
||||||
|
- Unlike models, **none granted anywhere is none**: a tool reaches into a
|
||||||
|
company system, so it is granted, never assumed.
|
||||||
|
- `keys limits --tools` writes a level's list whole, every server's
|
||||||
|
together; an empty list (`--tools ''`) closes the level to every tool.
|
||||||
|
The console's **MCP servers** screen and the `/admin/mcp` page grant
|
||||||
|
and take back one server's tools: a level given its first list starts
|
||||||
|
from the nearest list above it, so it keeps what it saw; one left
|
||||||
|
naming none goes back to naming none, or stays closed when a level
|
||||||
|
above names tools, so taking one server away opens it to no other. They
|
||||||
|
say when a level above keeps the granted tools out of reach.
|
||||||
|
- **The server is told who calls:** `X-Llmgw-Key`, and `X-Llmgw-User` and
|
||||||
|
`X-Llmgw-Project` where there are some. It is called with llmgw's
|
||||||
|
credentials, never the caller's key.
|
||||||
|
- **Bounds:** a call has 60 seconds (`--timeout`) and an answer 4 MB. A
|
||||||
|
server's progress is passed on to a client that takes a stream. `--rate N`
|
||||||
|
allows each caller N calls a minute to a server's tools.
|
||||||
|
- **Guardrails, both ways:** the caller's policies read a tool's answer as
|
||||||
|
the prompt it becomes.
|
||||||
|
- An instruction planted in a ticket is caught as a pasted one would be
|
||||||
|
(`injection: block`).
|
||||||
|
- Personal data is redacted or refused as the policy says. A refused
|
||||||
|
answer does not reach the agent.
|
||||||
|
- The arguments are read before they leave, so a secret pasted into one
|
||||||
|
does not reach the company system.
|
||||||
|
- **A server that keeps failing:** five calls failing in a row take a server
|
||||||
|
out for a minute. Its tools leave the list, and a call is answered at once.
|
||||||
|
- **Record:** every call is in `llmgw ctl audit`, with its arguments' size
|
||||||
|
and a digest, never their text. `llmgw_mcp_tool_calls_total` and
|
||||||
|
`llmgw_mcp_tool_call_seconds` count them by server, tool and outcome.
|
||||||
|
- **Changes:** a server added, changed or removed on one control plane is
|
||||||
|
served by every one within half a minute; its tool list is read again
|
||||||
|
every five minutes. `--disabled` keeps a server but serves none of its
|
||||||
|
tools.
|
||||||
|
|
||||||
|
**This cluster's own, read only.** `llmgw ctl mcp add llmgw --url
|
||||||
|
llmgw:admin` registers the cluster itself as a server: tools that say what
|
||||||
|
`llmgw ctl` does, and change nothing. It is answered in the control plane's
|
||||||
|
own process, with no address and no token to give.
|
||||||
|
|
||||||
|
| Tool | Says |
|
||||||
|
|---|---|
|
||||||
|
| `llmgw.cluster_status` | control planes, store, nodes, models against what they asked for, GPUs, what is unhealthy |
|
||||||
|
| `llmgw.list_nodes` | each node's state, GPUs, load, fabrics, agent version and engines |
|
||||||
|
| `llmgw.list_models` | each model's replicas ready against wanted, profiles, why one is parked |
|
||||||
|
| `llmgw.list_replicas` | each replica's shards: node, GPUs, state (`model` for one model's) |
|
||||||
|
| `llmgw.explain_placement` | why a model is not placed, node by node |
|
||||||
|
| `llmgw.node_logs` | a node's last agent log lines, or an engine's (`source`) |
|
||||||
|
|
||||||
|
Grant it as any other: `llmgw keys limits --scope key --id KEY-ID --tools
|
||||||
|
'llmgw.*'`. It is the same as an administrator's view, so grant it to the
|
||||||
|
people who would have `ctl`.
|
||||||
|
|
||||||
|
Servers over HTTP only for now (the streamable HTTP transport). A server
|
||||||
|
that runs as a local process (stdio) is planned.
|
||||||
|
|
||||||
## Single sign-on
|
## Single sign-on
|
||||||
|
|
||||||
People can sign in through your identity provider — Keycloak, Entra ID,
|
People can sign in through your identity provider — Keycloak, Entra ID,
|
||||||
@@ -2404,8 +2799,10 @@ Google, Okta, Authentik, anything that speaks OpenID Connect.
|
|||||||
|
|
||||||
1. Register a client there with the redirect address
|
1. Register a client there with the redirect address
|
||||||
`https://<gateway>/login/oidc/callback`.
|
`https://<gateway>/login/oidc/callback`.
|
||||||
2. Put its secret in a file only root reads.
|
2. Put its secret in a file the llmgw user reads, on every control plane.
|
||||||
3. Add to `server.yaml`:
|
3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
|
||||||
|
cluster with a store, `auth` is the cluster's: the same block in
|
||||||
|
`server.yaml` is not read, and the log says so.
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
auth:
|
auth:
|
||||||
@@ -2454,6 +2851,99 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
|
|||||||
join and leave them. Membership in a group is removed when someone leaves
|
join and leave them. Membership in a group is removed when someone leaves
|
||||||
it.
|
it.
|
||||||
|
|
||||||
|
### Entra ID
|
||||||
|
|
||||||
|
The console sets this up: `sudo llmgw`, People → Sign-in, **E**. What
|
||||||
|
follows is what it writes, and what to do in Entra.
|
||||||
|
|
||||||
|
Entra ID is an OpenID Connect provider like the others, with two things to
|
||||||
|
set in the app registration:
|
||||||
|
|
||||||
|
1. **Redirect URI:** under Authentication, add a Web platform with
|
||||||
|
`https://<gateway>/login/oidc/callback`. The issuer is
|
||||||
|
`https://login.microsoftonline.com/<tenant-id>/v2.0`.
|
||||||
|
2. **Groups in the token:** under Token configuration, add a groups claim.
|
||||||
|
Choose **Groups assigned to the application**, and assign the groups you
|
||||||
|
map in `roles` and `memberships` to the enterprise application. Entra
|
||||||
|
sends groups by object id unless you choose to emit names
|
||||||
|
(`sAMAccountName` for groups synced from AD), so write them the same way
|
||||||
|
in the mapping.
|
||||||
|
|
||||||
|
Entra leaves the groups out of the token for someone in more than 200 of
|
||||||
|
them and sends a pointer to Graph instead. llmgw then keeps that person's
|
||||||
|
role and memberships as they were, rather than reading it as no groups, and
|
||||||
|
logs an error saying so. "Groups assigned to the application" avoids this.
|
||||||
|
So does SCIM below, which Entra also speaks: in the enterprise application
|
||||||
|
under Provisioning, set the tenant URL to `https://<gateway>/scim/v2` and
|
||||||
|
the secret token to the contents of `auth.scim.token_file`.
|
||||||
|
|
||||||
|
## Active Directory
|
||||||
|
|
||||||
|
With an on-premises Active Directory, people sign in on the gateway's own
|
||||||
|
form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
|
||||||
|
password. The gateway checks them with the domain controllers over LDAPS.
|
||||||
|
|
||||||
|
The console sets this up and tries it first: `sudo llmgw`, People →
|
||||||
|
Sign-in, **D**. It keeps the service account's password in the store
|
||||||
|
(`bind_password_file: store:auth/ldap-bind`) and writes the CA into the
|
||||||
|
settings (`ca:`), so nothing has to be copied to each control plane.
|
||||||
|
By hand it is this block in the cluster's settings (`llmgw ctl config
|
||||||
|
edit`), where files it names must then be on every control plane:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
auth:
|
||||||
|
enabled: true
|
||||||
|
ldap:
|
||||||
|
urls: [ldaps://dc1.corp.example, ldaps://dc2.corp.example] # tried in order
|
||||||
|
ca_file: /etc/llmgw/corp-ca.pem # the domain's CA, when not in the system's
|
||||||
|
bind_dn: CN=svc-llmgw,OU=Service,DC=corp,DC=example
|
||||||
|
bind_password_file: /etc/llmgw/secrets/ldap-bind # root-only
|
||||||
|
base_dn: OU=Staff,DC=corp,DC=example # where people are
|
||||||
|
user_filter: (memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)
|
||||||
|
label: Acme # "Sign in with your Acme account"
|
||||||
|
create_accounts: true
|
||||||
|
roles: # by group CN or full DN; nested groups count
|
||||||
|
admin: [llmgw-admins]
|
||||||
|
operator: [llmgw-ops]
|
||||||
|
memberships:
|
||||||
|
ml-engineers: "team:Acme/ml"
|
||||||
|
sync_interval: 15m # negative turns the sync off
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Service account:** an ordinary domain user is enough: it only reads.
|
||||||
|
It finds the person and reads their groups. The person's password is
|
||||||
|
checked by binding as them, and is never stored.
|
||||||
|
- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
|
||||||
|
Plaintext LDAP is refused, since passwords would cross the network.
|
||||||
|
- **Certificate:** the domain controller's certificate must name it in
|
||||||
|
its subject alternative names: a name in the CN only is refused, as are
|
||||||
|
certificates with a negative serial number. Samba's own autogenerated
|
||||||
|
certificate is both, so a Samba DC needs one from a CA (`tls certfile`
|
||||||
|
in smb.conf). A CA file a control plane cannot read turns directory
|
||||||
|
sign-in off on that control plane, said in its log. Local accounts still
|
||||||
|
sign in.
|
||||||
|
- **Who may sign in:** enabled user objects under `base_dn` that match
|
||||||
|
`user_filter`. The example admits members of `llmgw-users`, nested groups
|
||||||
|
included.
|
||||||
|
- **Accounts:** an account is tied to the person's `objectGUID`, so a changed
|
||||||
|
mail or name is still the same account. An account that already exists
|
||||||
|
with the person's mail is taken over. Without `create_accounts`, only
|
||||||
|
people who already have an account can sign in. Local accounts, such as
|
||||||
|
the administrator `server init` made, keep signing in with their own
|
||||||
|
password.
|
||||||
|
- **Sync:** every `sync_interval` the leader reads the directory.
|
||||||
|
- Someone disabled or deleted in AD is suspended and their sessions
|
||||||
|
end.
|
||||||
|
- When they are enabled again they are reinstated. An account an
|
||||||
|
administrator suspended by hand stays suspended.
|
||||||
|
- Roles follow groups, and memberships the directory gave are taken away
|
||||||
|
when the person leaves the group.
|
||||||
|
- A sync that would suspend more than half of the directory's accounts is
|
||||||
|
refused and logged, rather than locking everyone out over a wrong
|
||||||
|
`base_dn` or filter.
|
||||||
|
- **Passwords:** a person's password is changed in AD. The gateway sends no
|
||||||
|
reset link for these accounts.
|
||||||
|
|
||||||
## Secrets in Vault
|
## Secrets in Vault
|
||||||
|
|
||||||
Every setting that names a file holding a secret — the store's `@address`,
|
Every setting that names a file holding a secret — the store's `@address`,
|
||||||
@@ -3025,6 +3515,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
|||||||
|
|
||||||
- `docs/choosing-models.md` — which models to run and how many, if the answer
|
- `docs/choosing-models.md` — which models to run and how many, if the answer
|
||||||
to "we have GPUs, now what" is not already obvious
|
to "we have GPUs, now what" is not already obvious
|
||||||
|
- `docs/large-models.md` — a model on several cards or several nodes: how it
|
||||||
|
is split, what each way needs of a card, and bringing one up on a fabric
|
||||||
- `docs/data.md` — what is stored and what leaves the machine, for the review
|
- `docs/data.md` — what is stored and what leaves the machine, for the review
|
||||||
that happens before this is allowed near anyone's documents
|
that happens before this is allowed near anyone's documents
|
||||||
- `docs/reference.md` — every configuration key, generated from the source, so
|
- `docs/reference.md` — every configuration key, generated from the source, so
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# llmgw and Kong
|
||||||
|
|
||||||
|
For a site that already runs Kong. The two meet over plain HTTP: Kong in
|
||||||
|
front of llmgw, as one more API it governs, or Kong behind llmgw, as an
|
||||||
|
OpenAI-compatible provider. Nothing of either is built into the other.
|
||||||
|
|
||||||
|
Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
|
||||||
|
and later need a Kong licence) in DB-less mode, in front of llmgw.
|
||||||
|
|
||||||
|
## Kong in front of llmgw
|
||||||
|
|
||||||
|
Two ways, both tried.
|
||||||
|
|
||||||
|
**A plain route.** Kong forwards the request as it came. Tried with chat
|
||||||
|
completions, streamed and not; llmgw's other APIs — Anthropic's
|
||||||
|
`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
|
||||||
|
since Kong does not read them. Kong adds its own authentication, rate limits
|
||||||
|
and logging; llmgw still checks the key, the budget and the guardrails.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
_format_version: "3.0"
|
||||||
|
services:
|
||||||
|
- name: llmgw
|
||||||
|
url: https://llm.example.com # llmgw's client port
|
||||||
|
routes:
|
||||||
|
- name: llmgw
|
||||||
|
paths: [/llmgw] # /llmgw/v1/chat/completions → /v1/chat/completions
|
||||||
|
```
|
||||||
|
|
||||||
|
**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
|
||||||
|
route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
- name: llmgw-chat
|
||||||
|
url: https://llm.example.com
|
||||||
|
routes:
|
||||||
|
- name: chat
|
||||||
|
paths: [/chat]
|
||||||
|
plugins:
|
||||||
|
- name: ai-proxy
|
||||||
|
config:
|
||||||
|
route_type: llm/v1/chat
|
||||||
|
logging: {log_statistics: true, log_payloads: false}
|
||||||
|
model:
|
||||||
|
provider: openai
|
||||||
|
name: qwen-coder # a model llmgw serves
|
||||||
|
options:
|
||||||
|
upstream_url: https://llm.example.com/v1/chat/completions
|
||||||
|
```
|
||||||
|
|
||||||
|
**Keys.** Without an `auth` block, `ai-proxy` passes the client's
|
||||||
|
`Authorization` header on: each team keeps its own llmgw key, and llmgw's
|
||||||
|
budgets, limits, guardrails and usage stay per team. With
|
||||||
|
`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
|
||||||
|
sends one key of its own instead, and llmgw sees every caller as that key —
|
||||||
|
then budgets per team are Kong's to keep.
|
||||||
|
|
||||||
|
**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
|
||||||
|
records prompt and completion tokens as llmgw did. For a streamed answer
|
||||||
|
Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
|
||||||
|
even with `stream_options.include_usage` set — the usage chunk reaches it
|
||||||
|
and is not read. llmgw's own accounting counts both; use its usage export
|
||||||
|
(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
|
||||||
|
|
||||||
|
**Timeouts.** A long answer from a model still loading outlasts Kong's
|
||||||
|
60-second defaults: set the service's `read_timeout` to the time a model may
|
||||||
|
take to start, as for nginx in [install.md](install.md).
|
||||||
|
|
||||||
|
## Kong behind llmgw
|
||||||
|
|
||||||
|
Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
|
||||||
|
it reaches any provider: a model whose source is `external`.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: kong-gpt
|
||||||
|
source: external
|
||||||
|
external:
|
||||||
|
url: https://kong.example.com/openai # Kong's route, without /v1
|
||||||
|
model: gpt-4o-mini
|
||||||
|
api_key_file: /etc/llmgw/secrets/kong-consumer.key
|
||||||
|
```
|
||||||
|
|
||||||
|
The key is the one Kong's own authentication asks of a consumer. As a
|
||||||
|
fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
|
||||||
|
room.
|
||||||
|
|
||||||
|
## Which to choose
|
||||||
|
|
||||||
|
Kong in front when the site's API policy says every API goes through Kong:
|
||||||
|
llmgw is then one more upstream, and its per-team accounting still holds
|
||||||
|
when each team keeps its own key. Kong behind when the site already reaches
|
||||||
|
outside providers through Kong and llmgw should too. llmgw does not need
|
||||||
|
Kong for anything it does itself — keys, budgets, guardrails, routing to
|
||||||
|
its own replicas.
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
# Large models: several cards, several nodes
|
||||||
|
|
||||||
|
A model too big for one card runs on several. llmgw chooses how from the
|
||||||
|
model's own figures and the cluster as it is; this page is the rules it
|
||||||
|
follows, the commands, and what to look at when a first one is brought up
|
||||||
|
on new hardware.
|
||||||
|
|
||||||
|
## How a model is split
|
||||||
|
|
||||||
|
In order, the first that fits:
|
||||||
|
|
||||||
|
1. **One card.** Nothing to split.
|
||||||
|
2. **Several cards of one node, tensor parallel.** Every layer is cut
|
||||||
|
across the cards, so they talk for every token: they must be on one
|
||||||
|
NVLink domain. The agent reads `nvidia-smi topo -m` and gives a replica
|
||||||
|
the cards with the most NVLink between them
|
||||||
|
([install.md](install.md#nvlink-if-your-gpus-have-it)).
|
||||||
|
3. **Several nodes, pipeline parallel between them.** Each node holds a
|
||||||
|
run of layers on all its cards (tensor parallel inside it), and only a
|
||||||
|
layer's output crosses to the next node. The nodes must share one
|
||||||
|
InfiniBand or RoCE fabric: a replica never spans two, and never a node
|
||||||
|
with none. Any number of nodes, not only two or four.
|
||||||
|
|
||||||
|
The plan takes the fewest cards, then the fewest nodes, then the better
|
||||||
|
precision up to eight bits.
|
||||||
|
|
||||||
|
**Tensor width and the KV heads.** The cards split the model's KV heads
|
||||||
|
among them, or past their number each card holds a copy of one. So the
|
||||||
|
width divides the KV heads or is a multiple of them: a model with 8 KV
|
||||||
|
heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8.
|
||||||
|
|
||||||
|
**Mixture of experts.** A model of experts (Qwen3-235B-A22B, DeepSeek-V3,
|
||||||
|
Llama 4, Mixtral) on more than one card is served with vLLM's
|
||||||
|
`--enable-expert-parallel`: each card holds whole experts rather than a
|
||||||
|
slice of every one, which is less traffic per token. It is read from
|
||||||
|
`config.json` (`num_experts`, `n_routed_experts`, `num_local_experts`).
|
||||||
|
|
||||||
|
**When it does not fit.** A smaller precision first: an FP8 or AWQ
|
||||||
|
repository of the same model, or a GGUF quantisation on llama.cpp if one
|
||||||
|
node's cards hold it. Then more nodes. `llmgw ctl model plan REPO
|
||||||
|
--precision bf16,fp8` shows each precision's fit from the model's config.
|
||||||
|
|
||||||
|
## What a way needs
|
||||||
|
|
||||||
|
Each way the plan offers says what a replica needs of each card, part by
|
||||||
|
part, and of all its cards together:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768
|
||||||
|
FILE SIZE BEST WAY A CARD OF ALL CARDS NOW
|
||||||
|
original 131 GB 4 × h100-80g 37.4 GB 72 GB 150 GB free
|
||||||
|
|
||||||
|
Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards.
|
||||||
|
The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another).
|
||||||
|
```
|
||||||
|
|
||||||
|
- **weights:** the files, divided over the cards.
|
||||||
|
- **context:** the KV cache for the context planned, divided likewise.
|
||||||
|
- **engine:** the engine's own: 2 GB and a fiftieth of the weights for vLLM.
|
||||||
|
- **usable:** 90% of a card; the rest is the driver's and fragmentation's.
|
||||||
|
|
||||||
|
The console's **Serve a model** shows the same for every way a file runs.
|
||||||
|
A context longer than planned is checked against it before anything is
|
||||||
|
fetched.
|
||||||
|
|
||||||
|
## Serving one
|
||||||
|
|
||||||
|
Without flags the plan's best way is taken:
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768
|
||||||
|
```
|
||||||
|
|
||||||
|
Or a way given by hand, for instance two nodes of eight:
|
||||||
|
|
||||||
|
```
|
||||||
|
llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \
|
||||||
|
--gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO
|
||||||
|
```
|
||||||
|
|
||||||
|
`--gpus` is the cards on each node and `--nodes` the nodes. The pipeline
|
||||||
|
is the nodes and the tensor width the cards of each.
|
||||||
|
|
||||||
|
`--env KEY=VALUE` goes into the engine's environment on every node:
|
||||||
|
`NCCL_DEBUG=INFO` while bringing a cluster up, `NCCL_IB_*` to steer it.
|
||||||
|
Only the variables engines are tuned with are taken. llmgw already sets
|
||||||
|
these:
|
||||||
|
- `NCCL_IB_HCA` to the fabric's adapters;
|
||||||
|
- `NCCL_IB_GID_INDEX` on RoCE;
|
||||||
|
- `NCCL_SOCKET_IFNAME` to the management network;
|
||||||
|
- for an NVLink domain spanning nodes, `NCCL_MNNVL_ENABLE`.
|
||||||
|
|
||||||
|
The job's first step says what it needs of each card. The weights then
|
||||||
|
come to the mirror once and from it to each node. The startup timeout (20
|
||||||
|
minutes unless the profile's `startup_timeout` says otherwise) runs from
|
||||||
|
when a node's download last moved. So a model of hundreds of gigabytes is
|
||||||
|
not failed while it is still arriving, and one whose download stops is
|
||||||
|
failed that long after.
|
||||||
|
|
||||||
|
## Watching it come up
|
||||||
|
|
||||||
|
| What | Where |
|
||||||
|
|---|---|
|
||||||
|
| The replica and its shards, one per node, each with its download | `llmgw ctl replicas` |
|
||||||
|
| Why it is not placed | `llmgw ctl explain-placement MODEL` |
|
||||||
|
| A node's engine output (NCCL's lines with `NCCL_DEBUG=INFO`) | `llmgw ctl logs NODE --source SHARD` |
|
||||||
|
| The bandwidth between two nodes, rail by rail | `llmgw ctl fabric-test NODE NODE` |
|
||||||
|
| Each node's GPUs, NVLink and adapters | `llmgw doctor` on the node; its page under /admin |
|
||||||
|
| Every step of a start, across nodes | the `replica start` trace ([install.md](install.md)) |
|
||||||
|
|
||||||
|
With `NCCL_DEBUG=INFO`, NCCL says which transport it took between nodes:
|
||||||
|
`NET/IB` is the fabric. `NET/Socket` means it fell back to TCP, and the
|
||||||
|
model will run, slowly.
|
||||||
|
|
||||||
|
## Bringing it up on new hardware
|
||||||
|
|
||||||
|
A first run on a fabric, in order. Each step is worth its output if
|
||||||
|
something goes wrong.
|
||||||
|
|
||||||
|
1. **The nodes.** Add each with its fabric:
|
||||||
|
`llmgw node add host --fabric ib-a:mlx5_0,mlx5_1`. Then
|
||||||
|
`llmgw ctl nodes` shows every node ready, on `ib-a`.
|
||||||
|
2. **The fabric.** `llmgw ctl fabric-test node1 node2`: each rail at its
|
||||||
|
rated speed (a 400 Gb/s rail near 45 GB/s).
|
||||||
|
3. **One node, several cards.** A dense 70B at bf16 on four or eight
|
||||||
|
cards of one node, by the plan:
|
||||||
|
`llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors`.
|
||||||
|
Ready, then a question answered.
|
||||||
|
4. **Two nodes.** The same model forced over two:
|
||||||
|
`--gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO`, under
|
||||||
|
another `--name`. Ready, answering, and `NET/IB` in both shards' logs.
|
||||||
|
5. **Experts.** A mixture-of-experts model on one node's eight cards,
|
||||||
|
for instance `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`: its profile
|
||||||
|
carries `--enable-expert-parallel` (`llmgw ctl models`, or its
|
||||||
|
definition from the admin API).
|
||||||
|
6. **A node lost.** With the two-node replica serving, stop one node's
|
||||||
|
agent (`systemctl stop llmgw-agent`): the replica fails and is placed
|
||||||
|
again once the node is back or another has room. Nothing is left
|
||||||
|
holding cards.
|
||||||
|
|
||||||
|
For each, worth keeping:
|
||||||
|
- the time from serve to ready;
|
||||||
|
- tokens a second from `llmgw ctl bench`;
|
||||||
|
- the plan's per-card figure against what `nvidia-smi` shows once loaded;
|
||||||
|
- anything it said that was wrong.
|
||||||
+13
-3
@@ -54,10 +54,20 @@ on the page that creates it and never again.
|
|||||||
|
|
||||||
### Connecting an editor
|
### Connecting an editor
|
||||||
|
|
||||||
The account page's **Connect your editor** makes a key for Continue, Codex or
|
The account page's **Connect your editor** makes a key for VS Code, Continue,
|
||||||
Cline and shows what the tool needs with that key, this gateway's address and a
|
Codex or Cline and shows what the tool needs with that key, this gateway's
|
||||||
model already in it:
|
address and a model already in it:
|
||||||
|
|
||||||
|
- **VS Code** — llmgw's own extension, downloaded from the gateway at
|
||||||
|
`/setup/vscode/llmgw.vsix` with this gateway's address already in it. It
|
||||||
|
offers every chat model the key may use to VS Code itself: the chat's model
|
||||||
|
picker, agent mode, and any extension that asks VS Code for a language model
|
||||||
|
(Cline's "VS Code LM API" provider among them), with no setup of their own.
|
||||||
|
**llmgw: Sign In** in VS Code opens this page with VS Code chosen; the page's
|
||||||
|
**Open in VS Code** link hands the key back, and the extension keeps it in
|
||||||
|
VS Code's secret storage. A link the extension did not ask for is confirmed
|
||||||
|
first, with the gateway's address shown, since it decides where code is sent.
|
||||||
|
**llmgw: Enter API Key** takes a pasted one instead.
|
||||||
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
|
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
|
||||||
so their `config.yaml` is not touched.
|
so their `config.yaml` is not touched.
|
||||||
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
|
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
|
||||||
|
|||||||
+30
-2
@@ -8,7 +8,7 @@ explains every choice.
|
|||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
|
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 12 GB RAM when it is also a node running the starter model and the two judges, which setup checks before it installs anything; four cores or more, or the judges on machines of their own, keep a judgment within its wait. |
|
||||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
||||||
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
||||||
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
||||||
@@ -146,7 +146,35 @@ agent is stopped.
|
|||||||
## 4. Add a model
|
## 4. Add a model
|
||||||
|
|
||||||
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
||||||
Hugging Face, showing only what your nodes can run.
|
Hugging Face, showing only what your nodes can run: judged by the cluster as it
|
||||||
|
is now — the largest card and its compute capability, the most memory on one
|
||||||
|
node — so a model too big for them, in a form those cards lack (FP8 needs 8.9),
|
||||||
|
or in neither GGUF nor safetensors is left out, and the page says how many.
|
||||||
|
Without GPUs it lists GGUF only. Untick **only what these nodes can run** to
|
||||||
|
see everything. On a model's page a file too big for the cluster is shown
|
||||||
|
without a **Fetch** button, with what it would need.
|
||||||
|
|
||||||
|
The **CPU tok/s** column is about how fast one reply would come on the CPU of
|
||||||
|
the fastest node whose memory holds the model at Q4_K_M; a model's page gives
|
||||||
|
it for each GGUF file by its own size. Every node's agent measures how fast its
|
||||||
|
memory reads when it starts (it is in the agent's log as `memory_gbs`), and a
|
||||||
|
token on a CPU reads every weight once: the estimate is that speed over the
|
||||||
|
model's size — for a model of experts named like `30B-A3B`, its active part.
|
||||||
|
On the test cluster it came within 15% of what llama.cpp measured. Several
|
||||||
|
replies at once share the reads and each is slower.
|
||||||
|
|
||||||
|
**GPU tok/s** is the same on the cards, for the layout this cluster would
|
||||||
|
give the model — the fewest cards that hold it, of each class, the fastest of
|
||||||
|
those; its tooltip says which ("2× h100-80g, NVLink, vLLM", "2 nodes × 1×
|
||||||
|
a100-80g over ib0 (InfiniBand 200 Gb/s), vLLM"). A card's memory bandwidth is
|
||||||
|
known from its model name (an H100 SXM and an H100 PCIe differ by half); a card
|
||||||
|
not in the table gets no figure. Tensor parallel across a node's cards divides
|
||||||
|
what each reads, and adds two all-reduces a layer, short over NVLink and
|
||||||
|
longer over PCIe; llama.cpp across cards splits by layer, so the cards take
|
||||||
|
turns and a reply is no faster than on one; a pipeline across nodes takes turns
|
||||||
|
too, and adds a hop over the fabric for each node. One card's figures were
|
||||||
|
measured on the test cluster's RTX 6000; tensor parallel and the fabric are
|
||||||
|
from what NCCL is known to take, not measured here.
|
||||||
|
|
||||||
1. Click a model to read its card.
|
1. Click a model to read its card.
|
||||||
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
||||||
|
|||||||
+340
-1
@@ -37,6 +37,7 @@ rather than repeated.
|
|||||||
| `default_tpm` | `int` | [Auth](#auth) |
|
| `default_tpm` | `int` | [Auth](#auth) |
|
||||||
| `signup_open` | `bool` | [Auth](#auth) |
|
| `signup_open` | `bool` | [Auth](#auth) |
|
||||||
| `oidc` | `OIDC` | [Auth](#auth) |
|
| `oidc` | `OIDC` | [Auth](#auth) |
|
||||||
|
| `ldap` | `LDAP` | [Auth](#auth) |
|
||||||
| `scim` | `SCIM` | [Auth](#auth) |
|
| `scim` | `SCIM` | [Auth](#auth) |
|
||||||
| `max_key_ttl` | `time.Duration` | [Auth](#auth) |
|
| `max_key_ttl` | `time.Duration` | [Auth](#auth) |
|
||||||
| `personal` | `Personal` | [Auth](#auth) |
|
| `personal` | `Personal` | [Auth](#auth) |
|
||||||
@@ -52,6 +53,7 @@ rather than repeated.
|
|||||||
| `retain_days` | `int` | [Batch](#batch) |
|
| `retain_days` | `int` | [Batch](#batch) |
|
||||||
| `max_bytes` | `int64` | [Cache](#cache) |
|
| `max_bytes` | `int64` | [Cache](#cache) |
|
||||||
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
||||||
|
| `embedding_model` | `string` | [Cache](#cache) |
|
||||||
| `model` | `string` | [Canary](#canary) |
|
| `model` | `string` | [Canary](#canary) |
|
||||||
| `percent` | `int` | [Canary](#canary) |
|
| `percent` | `int` | [Canary](#canary) |
|
||||||
| `min_requests` | `int` | [Canary](#canary) |
|
| `min_requests` | `int` | [Canary](#canary) |
|
||||||
@@ -105,10 +107,13 @@ rather than repeated.
|
|||||||
| `url` | `string` | [External](#external) |
|
| `url` | `string` | [External](#external) |
|
||||||
| `model` | `string` | [External](#external) |
|
| `model` | `string` | [External](#external) |
|
||||||
| `api_key_file` | `string` | [External](#external) |
|
| `api_key_file` | `string` | [External](#external) |
|
||||||
|
| `api_key_files` | `[]ExternalKey` | [External](#external) |
|
||||||
| `peer` | `string` | [External](#external) |
|
| `peer` | `string` | [External](#external) |
|
||||||
| `api` | `string` | [External](#external) |
|
| `api` | `string` | [External](#external) |
|
||||||
| `workflow` | `string` | [External](#external) |
|
| `workflow` | `string` | [External](#external) |
|
||||||
| `output` | `string` | [External](#external) |
|
| `output` | `string` | [External](#external) |
|
||||||
|
| `file` | `string` | [ExternalKey](#externalkey) |
|
||||||
|
| `weight` | `float64` | [ExternalKey](#externalkey) |
|
||||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||||
| `margin` | `float64` | [Facility](#facility) |
|
| `margin` | `float64` | [Facility](#facility) |
|
||||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||||
@@ -144,6 +149,21 @@ rather than repeated.
|
|||||||
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
||||||
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
||||||
| `max_blocks` | `int` | [KVIndex](#kvindex) |
|
| `max_blocks` | `int` | [KVIndex](#kvindex) |
|
||||||
|
| `urls` | `[]string` | [LDAP](#ldap) |
|
||||||
|
| `start_tls` | `bool` | [LDAP](#ldap) |
|
||||||
|
| `ca_file` | `string` | [LDAP](#ldap) |
|
||||||
|
| `ca` | `string` | [LDAP](#ldap) |
|
||||||
|
| `bind_dn` | `string` | [LDAP](#ldap) |
|
||||||
|
| `bind_password_file` | `string` | [LDAP](#ldap) |
|
||||||
|
| `base_dn` | `string` | [LDAP](#ldap) |
|
||||||
|
| `group_base_dn` | `string` | [LDAP](#ldap) |
|
||||||
|
| `user_filter` | `string` | [LDAP](#ldap) |
|
||||||
|
| `label` | `string` | [LDAP](#ldap) |
|
||||||
|
| `create_accounts` | `bool` | [LDAP](#ldap) |
|
||||||
|
| `roles` | `map[string][]string` | [LDAP](#ldap) |
|
||||||
|
| `memberships` | `map[string]string` | [LDAP](#ldap) |
|
||||||
|
| `sync_interval` | `time.Duration` | [LDAP](#ldap) |
|
||||||
|
| `timeout` | `time.Duration` | [LDAP](#ldap) |
|
||||||
| `contact` | `string` | [Landing](#landing) |
|
| `contact` | `string` | [Landing](#landing) |
|
||||||
| `notice` | `string` | [Landing](#landing) |
|
| `notice` | `string` | [Landing](#landing) |
|
||||||
| `language` | `string` | [Landing](#landing) |
|
| `language` | `string` | [Landing](#landing) |
|
||||||
@@ -160,6 +180,7 @@ rather than repeated.
|
|||||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||||
| `api` | `string` | [Mirror](#mirror) |
|
| `api` | `string` | [Mirror](#mirror) |
|
||||||
|
| `peers` | `[]string` | [Mirror](#mirror) |
|
||||||
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
|
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
|
||||||
| `registry` | `string` | [Mirror](#mirror) |
|
| `registry` | `string` | [Mirror](#mirror) |
|
||||||
| `timeout` | `time.Duration` | [Mirror](#mirror) |
|
| `timeout` | `time.Duration` | [Mirror](#mirror) |
|
||||||
@@ -168,8 +189,10 @@ rather than repeated.
|
|||||||
| `aliases` | `[]string` | [Model](#model) |
|
| `aliases` | `[]string` | [Model](#model) |
|
||||||
| `description` | `string` | [Model](#model) |
|
| `description` | `string` | [Model](#model) |
|
||||||
| `source` | `string` | [Model](#model) |
|
| `source` | `string` | [Model](#model) |
|
||||||
|
| `allow_pickle` | `bool` | [Model](#model) |
|
||||||
| `profiles` | `[]Profile` | [Model](#model) |
|
| `profiles` | `[]Profile` | [Model](#model) |
|
||||||
| `scaling` | `Scaling` | [Model](#model) |
|
| `scaling` | `Scaling` | [Model](#model) |
|
||||||
|
| `stopped` | `*Scaling` | [Model](#model) |
|
||||||
| `class` | `string` | [Model](#model) |
|
| `class` | `string` | [Model](#model) |
|
||||||
| `owner` | `string` | [Model](#model) |
|
| `owner` | `string` | [Model](#model) |
|
||||||
| `external` | `*External` | [Model](#model) |
|
| `external` | `*External` | [Model](#model) |
|
||||||
@@ -238,6 +261,7 @@ rather than repeated.
|
|||||||
| `require_capabilities` | `bool` | [Policy](#policy) |
|
| `require_capabilities` | `bool` | [Policy](#policy) |
|
||||||
| `validate_structured` | `bool` | [Policy](#policy) |
|
| `validate_structured` | `bool` | [Policy](#policy) |
|
||||||
| `judge` | `*JudgePolicy` | [Policy](#policy) |
|
| `judge` | `*JudgePolicy` | [Policy](#policy) |
|
||||||
|
| `services` | `[]Service` | [Policy](#policy) |
|
||||||
| `stream` | `string` | [Policy](#policy) |
|
| `stream` | `string` | [Policy](#policy) |
|
||||||
| `enabled` | `bool` | [Prefix](#prefix) |
|
| `enabled` | `bool` | [Prefix](#prefix) |
|
||||||
| `max_nodes` | `int` | [Prefix](#prefix) |
|
| `max_nodes` | `int` | [Prefix](#prefix) |
|
||||||
@@ -301,6 +325,7 @@ rather than repeated.
|
|||||||
| `injection` | `string` | [Rules](#rules) |
|
| `injection` | `string` | [Rules](#rules) |
|
||||||
| `deny` | `[]Pattern` | [Rules](#rules) |
|
| `deny` | `[]Pattern` | [Rules](#rules) |
|
||||||
| `max_chars` | `int` | [Rules](#rules) |
|
| `max_chars` | `int` | [Rules](#rules) |
|
||||||
|
| `restore` | `bool` | [Rules](#rules) |
|
||||||
| `token_file` | `string` | [SCIM](#scim) |
|
| `token_file` | `string` | [SCIM](#scim) |
|
||||||
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
||||||
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
||||||
@@ -361,8 +386,20 @@ rather than repeated.
|
|||||||
| `facility` | `Facility` | [Server](#server) |
|
| `facility` | `Facility` | [Server](#server) |
|
||||||
| `federation` | `Federation` | [Server](#server) |
|
| `federation` | `Federation` | [Server](#server) |
|
||||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||||
|
| `name` | `string` | [Service](#service) |
|
||||||
|
| `url` | `string` | [Service](#service) |
|
||||||
|
| `header_files` | `map[string]string` | [Service](#service) |
|
||||||
|
| `body` | `string` | [Service](#service) |
|
||||||
|
| `blocked` | `string` | [Service](#service) |
|
||||||
|
| `reason` | `string` | [Service](#service) |
|
||||||
|
| `on` | `string` | [Service](#service) |
|
||||||
|
| `action` | `string` | [Service](#service) |
|
||||||
|
| `timeout` | `time.Duration` | [Service](#service) |
|
||||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||||
|
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||||
|
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||||
|
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||||
| `store` | `string` | [Threads](#threads) |
|
| `store` | `string` | [Threads](#threads) |
|
||||||
| `retain_hours` | `int` | [Threads](#threads) |
|
| `retain_hours` | `int` | [Threads](#threads) |
|
||||||
@@ -394,6 +431,7 @@ rather than repeated.
|
|||||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||||
|
| `cached_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||||
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
||||||
| `name` | `string` | [admission.Band](#admissionband) |
|
| `name` | `string` | [admission.Band](#admissionband) |
|
||||||
@@ -432,6 +470,7 @@ rather than repeated.
|
|||||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||||
|
| `cached_per_1k` | `int64` | [config.Price](#configprice) |
|
||||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||||
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
||||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||||
@@ -441,6 +480,7 @@ rather than repeated.
|
|||||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||||
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
||||||
|
| `strata` | `StrataConfig` | [engines.Config](#enginesconfig) |
|
||||||
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
||||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||||
@@ -616,6 +656,15 @@ OIDC is sign-in through an identity provider.
|
|||||||
|
|
||||||
Takes a [OIDC](#oidc) block.
|
Takes a [OIDC](#oidc) block.
|
||||||
|
|
||||||
|
### `ldap`
|
||||||
|
|
||||||
|
`LDAP`
|
||||||
|
|
||||||
|
LDAP is sign-in against a directory, Active Directory above all, and
|
||||||
|
the accounts kept in step with it.
|
||||||
|
|
||||||
|
Takes a [LDAP](#ldap) block.
|
||||||
|
|
||||||
### `scim`
|
### `scim`
|
||||||
|
|
||||||
`SCIM`
|
`SCIM`
|
||||||
@@ -728,6 +777,15 @@ Past it, the answers used longest ago go first.
|
|||||||
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||||
unless set.
|
unless set.
|
||||||
|
|
||||||
|
### `embedding_model`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
EmbeddingModel embeds the questions of projects that reuse answers to
|
||||||
|
similar ones (llmgw keys cache --similar): an embedding model in the
|
||||||
|
catalog. Unset, those projects reuse only the same request's answer.
|
||||||
|
The gateway asks it on its own account, as it asks a judge.
|
||||||
|
|
||||||
## Canary
|
## Canary
|
||||||
|
|
||||||
### `model`
|
### `model`
|
||||||
@@ -948,7 +1006,7 @@ work stands without being renewed.
|
|||||||
Only meaningful with a shared store, where a second control plane can
|
Only meaningful with a shared store, where a second control plane can
|
||||||
see the claim. Long enough to survive a slow query or a brief network
|
see the claim. Long enough to survive a slow query or a brief network
|
||||||
stall, short enough that a standby takes over before anyone has finished
|
stall, short enough that a standby takes over before anyone has finished
|
||||||
reading the alert. Zero takes the default.
|
reading the alert. Zero takes the default, 10s.
|
||||||
|
|
||||||
## CustomNode
|
## CustomNode
|
||||||
|
|
||||||
@@ -1103,6 +1161,15 @@ APIKeyFile holds the provider's key, on the control plane. A file
|
|||||||
rather than the key itself, so the key is in neither the catalog, the
|
rather than the key itself, so the key is in neither the catalog, the
|
||||||
store nor the audit log; empty for a provider that asks for none.
|
store nor the audit log; empty for a provider that asks for none.
|
||||||
|
|
||||||
|
### `api_key_files`
|
||||||
|
|
||||||
|
`[]ExternalKey`
|
||||||
|
|
||||||
|
APIKeyFiles are several keys for the provider, in place of
|
||||||
|
APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
|
||||||
|
answers 429 or 401 is passed over while its breaker is open and the
|
||||||
|
request goes to another. A provider's limits are per key.
|
||||||
|
|
||||||
### `peer`
|
### `peer`
|
||||||
|
|
||||||
`string`
|
`string`
|
||||||
@@ -1135,6 +1202,20 @@ plane.
|
|||||||
|
|
||||||
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||||
|
|
||||||
|
## ExternalKey
|
||||||
|
|
||||||
|
### `file`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
File holds the key, on the control plane, as APIKeyFile does.
|
||||||
|
|
||||||
|
### `weight`
|
||||||
|
|
||||||
|
`float64`
|
||||||
|
|
||||||
|
Weight is its share of the requests against the others'; zero is 1.
|
||||||
|
|
||||||
## Facility
|
## Facility
|
||||||
|
|
||||||
### `budgets`
|
### `budgets`
|
||||||
@@ -1401,6 +1482,114 @@ MaxBlocks bounds how many of a prompt's blocks are hashed and looked
|
|||||||
up. Beyond it a longer match still helps, but the difference no longer
|
up. Beyond it a longer match still helps, but the difference no longer
|
||||||
decides anything. Zero takes 4096.
|
decides anything. Zero takes 4096.
|
||||||
|
|
||||||
|
## LDAP
|
||||||
|
|
||||||
|
### `urls`
|
||||||
|
|
||||||
|
`[]string`
|
||||||
|
|
||||||
|
URLs are the directory's servers, tried in turn: ldaps:// (636), or
|
||||||
|
ldap:// (389) with StartTLS.
|
||||||
|
|
||||||
|
### `start_tls`
|
||||||
|
|
||||||
|
`bool`
|
||||||
|
|
||||||
|
StartTLS upgrades an ldap:// connection before anything is sent. A
|
||||||
|
password never crosses a connection without TLS: ldap:// without it
|
||||||
|
is refused.
|
||||||
|
|
||||||
|
### `ca_file`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
CAFile is the authority the servers' certificates are checked against,
|
||||||
|
when it is not the system's: an AD Certificate Services root.
|
||||||
|
|
||||||
|
### `ca`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
CA is the same authority written out, PEM: kept in the cluster's
|
||||||
|
settings, so no file has to be put on every control plane. Used in
|
||||||
|
place of CAFile when both are set.
|
||||||
|
|
||||||
|
### `bind_dn`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
BindDN and the file holding its password: a service account that may
|
||||||
|
read users and groups. Nothing is written to the directory.
|
||||||
|
|
||||||
|
### `bind_password_file`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
### `base_dn`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
BaseDN is where people are searched for, subtree included.
|
||||||
|
|
||||||
|
### `group_base_dn`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
GroupBaseDN is where their groups are; empty is BaseDN's domain root.
|
||||||
|
|
||||||
|
### `user_filter`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
UserFilter narrows who may sign in, ANDed with the login match: a
|
||||||
|
group (memberOf=...) or an OU's people only. Empty is every enabled
|
||||||
|
user object.
|
||||||
|
|
||||||
|
### `label`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Label is what the form says to sign in with: "Sign in with your
|
||||||
|
<label> account". Empty is "Active Directory".
|
||||||
|
|
||||||
|
### `create_accounts`
|
||||||
|
|
||||||
|
`bool`
|
||||||
|
|
||||||
|
CreateAccounts makes an account, active, for a person the directory
|
||||||
|
vouches for and the gateway does not know. Off, only existing accounts
|
||||||
|
sign in this way.
|
||||||
|
|
||||||
|
### `roles`
|
||||||
|
|
||||||
|
`map[string][]string`
|
||||||
|
|
||||||
|
Roles maps a staff role (viewer, operator, billing, admin) to the
|
||||||
|
groups that give it, by name (CN) or whole DN. Set at every sign-in
|
||||||
|
and sync: someone taken out of the group loses the role.
|
||||||
|
|
||||||
|
### `memberships`
|
||||||
|
|
||||||
|
`map[string]string`
|
||||||
|
|
||||||
|
Memberships maps a group, by CN or DN, to a place in the tree —
|
||||||
|
org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
|
||||||
|
and kept by the sync, which also takes away a place whose group the
|
||||||
|
person has left.
|
||||||
|
|
||||||
|
### `sync_interval`
|
||||||
|
|
||||||
|
`time.Duration`
|
||||||
|
|
||||||
|
SyncInterval is how often the leader reads the directory. Zero is
|
||||||
|
15m; negative turns the sync off, leaving only sign-in.
|
||||||
|
|
||||||
|
### `timeout`
|
||||||
|
|
||||||
|
`time.Duration`
|
||||||
|
|
||||||
|
Timeout bounds each round trip to a server. Zero is 10s.
|
||||||
|
|
||||||
## Landing
|
## Landing
|
||||||
|
|
||||||
### `contact`
|
### `contact`
|
||||||
@@ -1539,6 +1728,14 @@ The management port, not the one nodes fetch bytes from. Those are two
|
|||||||
listeners on the mirror precisely so this one can be locked down without
|
listeners on the mirror precisely so this one can be locked down without
|
||||||
re-certifying every agent's fetcher.
|
re-certifying every agent's fetcher.
|
||||||
|
|
||||||
|
### `peers`
|
||||||
|
|
||||||
|
`[]string`
|
||||||
|
|
||||||
|
Peers are the management addresses of mirrors kept in step with this
|
||||||
|
one (llmgw mirror serve --peer): asked when it cannot be reached, and
|
||||||
|
a node given the same file on one of them while it is down.
|
||||||
|
|
||||||
### `tls`
|
### `tls`
|
||||||
|
|
||||||
`tlsconf.Config`
|
`tlsconf.Config`
|
||||||
@@ -1608,6 +1805,16 @@ choice by guesswork.
|
|||||||
|
|
||||||
Source is the weights path or a HuggingFace id.
|
Source is the weights path or a HuggingFace id.
|
||||||
|
|
||||||
|
### `allow_pickle`
|
||||||
|
|
||||||
|
`bool`
|
||||||
|
|
||||||
|
AllowPickle lets the nodes fetch this model's weights, or its
|
||||||
|
adapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code
|
||||||
|
as they are loaded. Off, such a source is refused when the model is
|
||||||
|
added and by the node before it fetches anything. For a model that has
|
||||||
|
no safetensors or GGUF form and whose files were checked.
|
||||||
|
|
||||||
### `profiles`
|
### `profiles`
|
||||||
|
|
||||||
`[]Profile`
|
`[]Profile`
|
||||||
@@ -1618,6 +1825,16 @@ Source is the weights path or a HuggingFace id.
|
|||||||
|
|
||||||
Takes a [Scaling](#scaling) block.
|
Takes a [Scaling](#scaling) block.
|
||||||
|
|
||||||
|
### `stopped`
|
||||||
|
|
||||||
|
`*Scaling`
|
||||||
|
|
||||||
|
Stopped is the scaling a model had when an administrator stopped it:
|
||||||
|
kept in the catalog with none running (Scaling 0/0) until started
|
||||||
|
again with this. Nil for a model not stopped.
|
||||||
|
|
||||||
|
Takes a [Scaling](#scaling) block.
|
||||||
|
|
||||||
### `class`
|
### `class`
|
||||||
|
|
||||||
`string`
|
`string`
|
||||||
@@ -2136,6 +2353,12 @@ Judge has a judge model on the cluster read the text too.
|
|||||||
|
|
||||||
Takes a [JudgePolicy](#judgepolicy) block.
|
Takes a [JudgePolicy](#judgepolicy) block.
|
||||||
|
|
||||||
|
### `services`
|
||||||
|
|
||||||
|
`[]Service`
|
||||||
|
|
||||||
|
Services are outside guardrails asked over HTTP, beside the judge.
|
||||||
|
|
||||||
### `stream`
|
### `stream`
|
||||||
|
|
||||||
`string`
|
`string`
|
||||||
@@ -2610,6 +2833,14 @@ Deny are patterns, matched against the text as a model reads it.
|
|||||||
|
|
||||||
MaxChars refuses text longer than this; zero is no limit.
|
MaxChars refuses text longer than this; zero is no limit.
|
||||||
|
|
||||||
|
### `restore`
|
||||||
|
|
||||||
|
`bool`
|
||||||
|
|
||||||
|
Restore, on input, puts what redact rewrote as placeholders back into
|
||||||
|
the answer: the model reads "[EMAIL_1]", the client reads its own
|
||||||
|
address. Only the values that request sent, only into its answer.
|
||||||
|
|
||||||
## SCIM
|
## SCIM
|
||||||
|
|
||||||
### `token_file`
|
### `token_file`
|
||||||
@@ -3117,6 +3348,67 @@ Endpoints is the phase-1 static replica list. It is only consulted when
|
|||||||
no model declares a profile, which is what makes a router-only smoke test
|
no model declares a profile, which is what makes a router-only smoke test
|
||||||
against sim-engines possible without a control plane.
|
against sim-engines possible without a control plane.
|
||||||
|
|
||||||
|
## Service
|
||||||
|
|
||||||
|
### `name`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Name says which service decided, in the verdict's rule.
|
||||||
|
|
||||||
|
### `url`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
URL is POSTed the body.
|
||||||
|
|
||||||
|
### `header_files`
|
||||||
|
|
||||||
|
`map[string]string`
|
||||||
|
|
||||||
|
HeaderFiles are headers whose values are read from files each time,
|
||||||
|
so a rotated key needs no restart: {Authorization: /etc/…/lakera}.
|
||||||
|
|
||||||
|
### `body`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Body is the request, a template given .Text; {{json .Text}} writes it
|
||||||
|
as a JSON string. Empty is {"input": <text>}.
|
||||||
|
|
||||||
|
### `blocked`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Blocked is where in the answer the service says the text is unsafe:
|
||||||
|
a dotted path, numbers for list items (results.0.flagged). True, a
|
||||||
|
number above zero, a non-empty list or string other than "false" is
|
||||||
|
unsafe.
|
||||||
|
|
||||||
|
### `reason`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Reason is where it says why, for the verdict; optional.
|
||||||
|
|
||||||
|
### `on`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
On: input, output or both. Empty is input.
|
||||||
|
|
||||||
|
### `action`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Action on unsafe: block or flag. Empty is block.
|
||||||
|
|
||||||
|
### `timeout`
|
||||||
|
|
||||||
|
`time.Duration`
|
||||||
|
|
||||||
|
Timeout bounds one call. Empty is 3s; past it the policy's fail says.
|
||||||
|
|
||||||
## ServingSettings
|
## ServingSettings
|
||||||
|
|
||||||
### `stopped`
|
### `stopped`
|
||||||
@@ -3138,6 +3430,27 @@ entries ask. In force on every control plane within seconds.
|
|||||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||||
Empty means whatever started this agent, which is right almost always.
|
Empty means whatever started this agent, which is right almost always.
|
||||||
|
|
||||||
|
## StrataConfig
|
||||||
|
|
||||||
|
### `dir`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Dir is Strata's source directory, where setup.py and serve/ are.
|
||||||
|
|
||||||
|
### `python`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Python is the interpreter of its environment; Dir's .venv when unset.
|
||||||
|
|
||||||
|
### `engine`
|
||||||
|
|
||||||
|
`string`
|
||||||
|
|
||||||
|
Engine is the directory of its prebuilt engine, where the strata
|
||||||
|
binary and the CUDA libraries it ships are; Dir's engine/ when unset.
|
||||||
|
|
||||||
## TRTLLMConfig
|
## TRTLLMConfig
|
||||||
|
|
||||||
### `python`
|
### `python`
|
||||||
@@ -3355,6 +3668,15 @@ is sha256=<hex HMAC of the body>.
|
|||||||
|
|
||||||
`int64`
|
`int64`
|
||||||
|
|
||||||
|
### `cached_per_1k`
|
||||||
|
|
||||||
|
`int64`
|
||||||
|
|
||||||
|
CachedPer1k is what a prompt token costs that the engine or provider
|
||||||
|
read from its cache instead of computing: the part of prompt_tokens
|
||||||
|
the answer counts as cached_tokens. Zero charges those at
|
||||||
|
PromptPer1k, as before there was a rate for them.
|
||||||
|
|
||||||
### `per_image`
|
### `per_image`
|
||||||
|
|
||||||
`int64`
|
`int64`
|
||||||
@@ -3629,6 +3951,14 @@ had rather wait behind people than be turned away.
|
|||||||
|
|
||||||
`int64`
|
`int64`
|
||||||
|
|
||||||
|
### `cached_per_1k`
|
||||||
|
|
||||||
|
`int64`
|
||||||
|
|
||||||
|
CachedPer1k is the cost of a prompt token the engine or provider read
|
||||||
|
from its cache (the answer's cached_tokens, counted within the
|
||||||
|
prompt's). Zero charges them at prompt_per_1k.
|
||||||
|
|
||||||
### `per_image`
|
### `per_image`
|
||||||
|
|
||||||
`int64`
|
`int64`
|
||||||
@@ -3693,6 +4023,15 @@ ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
|||||||
|
|
||||||
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
||||||
|
|
||||||
|
### `strata`
|
||||||
|
|
||||||
|
`StrataConfig`
|
||||||
|
|
||||||
|
Strata serves a few mixture-of-experts models from RAM and one card;
|
||||||
|
offered by a node whose Strata is named here, or that installed it.
|
||||||
|
|
||||||
|
Takes a [StrataConfig](#strataconfig) block.
|
||||||
|
|
||||||
## guard.Config
|
## guard.Config
|
||||||
|
|
||||||
### `enabled`
|
### `enabled`
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ require (
|
|||||||
github.com/charmbracelet/x/ansi v0.11.6
|
github.com/charmbracelet/x/ansi v0.11.6
|
||||||
github.com/coreos/go-oidc/v3 v3.21.0
|
github.com/coreos/go-oidc/v3 v3.21.0
|
||||||
github.com/go-jose/go-jose/v4 v4.1.4
|
github.com/go-jose/go-jose/v4 v4.1.4
|
||||||
|
github.com/go-ldap/ldap/v3 v3.4.15
|
||||||
github.com/go-zeromq/zmq4 v0.17.0
|
github.com/go-zeromq/zmq4 v0.17.0
|
||||||
github.com/jackc/pgx/v5 v5.11.0
|
github.com/jackc/pgx/v5 v5.11.0
|
||||||
github.com/knadh/koanf/parsers/yaml v1.1.1
|
github.com/knadh/koanf/parsers/yaml v1.1.1
|
||||||
@@ -35,6 +36,7 @@ require (
|
|||||||
golang.org/x/net v0.59.0
|
golang.org/x/net v0.59.0
|
||||||
golang.org/x/oauth2 v0.37.0
|
golang.org/x/oauth2 v0.37.0
|
||||||
golang.org/x/sync v0.23.0
|
golang.org/x/sync v0.23.0
|
||||||
|
golang.org/x/sys v0.48.0
|
||||||
golang.org/x/term v0.46.0
|
golang.org/x/term v0.46.0
|
||||||
golang.org/x/text v0.42.0
|
golang.org/x/text v0.42.0
|
||||||
google.golang.org/protobuf v1.36.12
|
google.golang.org/protobuf v1.36.12
|
||||||
@@ -44,6 +46,7 @@ require (
|
|||||||
require (
|
require (
|
||||||
filippo.io/edwards25519 v1.2.0 // indirect
|
filippo.io/edwards25519 v1.2.0 // indirect
|
||||||
filippo.io/hpke v0.4.0 // indirect
|
filippo.io/hpke v0.4.0 // indirect
|
||||||
|
github.com/Azure/go-ntlmssp v0.1.1 // indirect
|
||||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||||
github.com/beorn7/perks v1.0.1 // indirect
|
github.com/beorn7/perks v1.0.1 // indirect
|
||||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||||
@@ -56,6 +59,7 @@ require (
|
|||||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
||||||
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
||||||
|
github.com/go-asn1-ber/asn1-ber v1.5.8 // indirect
|
||||||
github.com/go-logr/logr v1.4.4 // indirect
|
github.com/go-logr/logr v1.4.4 // indirect
|
||||||
github.com/go-logr/stdr v1.2.2 // indirect
|
github.com/go-logr/stdr v1.2.2 // indirect
|
||||||
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
||||||
@@ -88,7 +92,6 @@ require (
|
|||||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
|
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
|
||||||
go.opentelemetry.io/otel/metric v1.46.0 // indirect
|
go.opentelemetry.io/otel/metric v1.46.0 // indirect
|
||||||
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
|
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
|
||||||
golang.org/x/sys v0.48.0 // indirect
|
|
||||||
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
|
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||||
google.golang.org/grpc v1.83.1 // indirect
|
google.golang.org/grpc v1.83.1 // indirect
|
||||||
|
|||||||
@@ -8,8 +8,12 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
|||||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||||
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
||||||
|
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
|
||||||
|
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
|
||||||
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
|
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
|
||||||
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
|
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
|
||||||
|
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktpoUAgOJK3OTFc/xug0PCXYCqU0FgDKI=
|
||||||
|
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
|
||||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
||||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
||||||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||||
@@ -46,8 +50,12 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
|
|||||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
|
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
|
||||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||||
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||||
|
github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
|
||||||
|
github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
|
||||||
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
|
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
|
||||||
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
||||||
|
github.com/go-ldap/ldap/v3 v3.4.15 h1:eRB4f4Ecn1POL1mO86+uUDlecmyWsdiJwh60bAUS4UY=
|
||||||
|
github.com/go-ldap/ldap/v3 v3.4.15/go.mod h1:jeD0c98EdZQCKKKKe4kTWs5aCf6y+j8KuYsYLznZSQk=
|
||||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||||
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
|
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
|
||||||
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||||
@@ -69,6 +77,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
|||||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
|
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
|
||||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
|
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
|
||||||
|
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
|
||||||
|
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||||
@@ -81,6 +91,18 @@ github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
|
|||||||
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||||
|
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
|
||||||
|
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
|
||||||
|
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
|
||||||
|
github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
|
||||||
|
github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
|
||||||
|
github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
|
||||||
|
github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
|
||||||
|
github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
|
||||||
|
github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
|
||||||
|
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
|
||||||
|
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
|
||||||
|
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
|
||||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||||
github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=
|
github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
package account
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeDirectory struct {
|
||||||
|
asked []string
|
||||||
|
u store.User
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeDirectory) SignIn(_ context.Context, login, _ string) (store.User, error) {
|
||||||
|
f.asked = append(f.asked, login)
|
||||||
|
return f.u, f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
// With a directory, a name the store has no password for is the
|
||||||
|
// directory's to check; an account with a password here — the
|
||||||
|
// administrator init made — is still checked here, so a directory that is
|
||||||
|
// down locks nobody out.
|
||||||
|
func TestPasswordsKeptElsewhereAreTheDirectorysToCheck(t *testing.T) {
|
||||||
|
svc, st := newService(t)
|
||||||
|
ctx := t.Context()
|
||||||
|
hash, _ := HashPassword("local-secret")
|
||||||
|
admin := store.User{ID: "usr-admin", Email: "admin@example.com", Name: "a", PasswordHash: hash,
|
||||||
|
State: store.UserActive, CreatedAt: time.Now()}
|
||||||
|
ada := store.User{ID: "usr-ada", Email: "ada@corp.example", Name: "Ada", PasswordHash: "!ldap",
|
||||||
|
State: store.UserActive, CreatedAt: time.Now()}
|
||||||
|
for _, u := range []store.User{admin, ada} {
|
||||||
|
if err := st.CreateUser(ctx, u); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
dir := &fakeDirectory{u: ada}
|
||||||
|
svc.SetDirectory(dir)
|
||||||
|
|
||||||
|
if _, _, err := svc.SignIn(ctx, "admin@example.com", "local-secret"); err != nil || len(dir.asked) != 0 {
|
||||||
|
t.Fatalf("the local administrator: %v, directory asked %v", err, dir.asked)
|
||||||
|
}
|
||||||
|
// A name, not an address, and an account kept by the directory.
|
||||||
|
for _, login := range []string{`CORP\ada`, "ada@corp.example"} {
|
||||||
|
if tok, _, err := svc.SignIn(ctx, login, "ad-secret"); err != nil || tok == "" {
|
||||||
|
t.Errorf("%s: %v", login, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(dir.asked) != 2 || dir.asked[0] != `CORP\ada` {
|
||||||
|
t.Errorf("the directory was asked %v", dir.asked)
|
||||||
|
}
|
||||||
|
dir.err = ErrBadCredentials
|
||||||
|
if _, _, err := svc.SignIn(ctx, "ada", "wrong"); !errors.Is(err, ErrBadCredentials) {
|
||||||
|
t.Errorf("a password the directory refused: %v", err)
|
||||||
|
}
|
||||||
|
// A reset link would give a directory's account a password here.
|
||||||
|
if _, _, err := svc.RequestReset(ctx, st, "ada@corp.example"); !errors.Is(err, ErrNoReset) {
|
||||||
|
t.Errorf("a directory's account was sent a reset link: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -80,8 +80,26 @@ type Service struct {
|
|||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
attempts map[string]*attemptWindow
|
attempts map[string]*attemptWindow
|
||||||
|
|
||||||
|
// dir checks the passwords of accounts this store keeps none for.
|
||||||
|
dir Directory
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Directory checks a password against an outside directory — Active
|
||||||
|
// Directory over LDAP — and answers the account to sign in: made or kept in
|
||||||
|
// step with the directory. Its errors are this package's ErrBadCredentials
|
||||||
|
// and ErrSuspended for the person's own, and anything else for the
|
||||||
|
// directory's.
|
||||||
|
type Directory interface {
|
||||||
|
SignIn(ctx context.Context, login, password string) (store.User, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetDirectory has passwords for accounts with none here, and names that are
|
||||||
|
// not addresses, checked by the directory. An account with a password here
|
||||||
|
// — the administrator init made — is still checked here, so a directory that
|
||||||
|
// is down locks nobody out.
|
||||||
|
func (s *Service) SetDirectory(d Directory) { s.dir = d }
|
||||||
|
|
||||||
type attemptWindow struct {
|
type attemptWindow struct {
|
||||||
count int
|
count int
|
||||||
since time.Time
|
since time.Time
|
||||||
@@ -137,6 +155,7 @@ func (s *Service) SignUp(ctx context.Context, email, name, password string) erro
|
|||||||
// SignIn checks a password and opens a session, returning the session token and
|
// SignIn checks a password and opens a session, returning the session token and
|
||||||
// the CSRF token that must accompany the forms in it.
|
// the CSRF token that must accompany the forms in it.
|
||||||
func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
|
func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
|
||||||
|
login := strings.TrimSpace(email)
|
||||||
email = store.NormalizeEmail(email)
|
email = store.NormalizeEmail(email)
|
||||||
// Counted per address and the client trying it, so somebody guessing
|
// Counted per address and the client trying it, so somebody guessing
|
||||||
// from elsewhere cannot lock the owner out of their own account; and per
|
// from elsewhere cannot lock the owner out of their own account; and per
|
||||||
@@ -151,6 +170,15 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
|
|||||||
}
|
}
|
||||||
|
|
||||||
u, err := s.store.UserByEmail(ctx, email)
|
u, err := s.store.UserByEmail(ctx, email)
|
||||||
|
// No password kept here — the directory's, or nobody's — and a
|
||||||
|
// directory to ask: it decides.
|
||||||
|
if s.dir != nil && (errors.Is(err, store.ErrNotFound) || (err == nil && strings.HasPrefix(u.PasswordHash, "!"))) {
|
||||||
|
if u, err = s.dir.SignIn(ctx, login, password); err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
s.clearAttempts(byClient)
|
||||||
|
return s.open(ctx, u)
|
||||||
|
}
|
||||||
if errors.Is(err, store.ErrNotFound) {
|
if errors.Is(err, store.ErrNotFound) {
|
||||||
// Verify anyway, against a hash that cannot match. Returning early here
|
// Verify anyway, against a hash that cannot match. Returning early here
|
||||||
// would let a stopwatch enumerate every account on the system.
|
// would let a stopwatch enumerate every account on the system.
|
||||||
@@ -180,7 +208,11 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
|
|||||||
case store.UserSuspended:
|
case store.UserSuspended:
|
||||||
return "", "", ErrSuspended
|
return "", "", ErrSuspended
|
||||||
}
|
}
|
||||||
|
return s.open(ctx, u)
|
||||||
|
}
|
||||||
|
|
||||||
|
// open makes a session for a person whose password was right.
|
||||||
|
func (s *Service) open(ctx context.Context, u store.User) (token, csrf string, err error) {
|
||||||
if token, err = store.NewSessionToken(); err != nil {
|
if token, err = store.NewSessionToken(); err != nil {
|
||||||
return "", "", err
|
return "", "", err
|
||||||
}
|
}
|
||||||
@@ -402,7 +434,10 @@ func (s *Service) RequestReset(ctx context.Context, rs ResetStore, email string)
|
|||||||
return "", store.User{}, ErrTooManyTries
|
return "", store.User{}, ErrTooManyTries
|
||||||
}
|
}
|
||||||
u, err := rs.UserByEmail(ctx, email)
|
u, err := rs.UserByEmail(ctx, email)
|
||||||
if err != nil || u.PasswordHash == "" || u.State != store.UserActive {
|
// An account whose password is kept elsewhere — "!oidc", "!scim",
|
||||||
|
// "!ldap" — gets no link: setting one here would give it a second way
|
||||||
|
// in that its directory knows nothing of, and outlives its leaving.
|
||||||
|
if err != nil || u.PasswordHash == "" || strings.HasPrefix(u.PasswordHash, "!") || u.State != store.UserActive {
|
||||||
return "", store.User{}, ErrNoReset
|
return "", store.User{}, ErrNoReset
|
||||||
}
|
}
|
||||||
token, err := rs.CreatePasswordReset(ctx, u.ID, ResetLinkTTL)
|
token, err := rs.CreatePasswordReset(ctx, u.ID, ResetLinkTTL)
|
||||||
|
|||||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
|||||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||||
price = a.pricing.For(u.ServedModel)
|
price = a.pricing.For(u.ServedModel)
|
||||||
}
|
}
|
||||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
|
u.CostMicros = price.Cost(u.PromptTokens, u.CachedTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||||
for _, b := range res.Budgets {
|
for _, b := range res.Budgets {
|
||||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -39,20 +39,37 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
|||||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||||
// exactly the traffic shape a quota exists to bound.
|
// exactly the traffic shape a quota exists to bound.
|
||||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||||
if got := p.Cost(1, 0, 0, 0); got != 1 {
|
if got := p.Cost(1, 0, 0, 0, 0); got != 1 {
|
||||||
t.Errorf("one prompt token cost %d", got)
|
t.Errorf("one prompt token cost %d", got)
|
||||||
}
|
}
|
||||||
if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
|
if got := p.Cost(1000, 0, 1000, 0, 0); got != 4000 {
|
||||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||||
}
|
}
|
||||||
if got := p.Cost(0, 0, 0, 0); got != 0 {
|
if got := p.Cost(0, 0, 0, 0, 0); got != 0 {
|
||||||
t.Errorf("an empty request cost %d", got)
|
t.Errorf("an empty request cost %d", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestACachedPromptCostsTheCachedRate(t *testing.T) {
|
||||||
|
p := Price{PromptPer1k: 1000, CachedPer1k: 100, CompletionPer1k: 3000}
|
||||||
|
// 10,000 prompt tokens of which 9,000 were read from the cache: 1,000 at
|
||||||
|
// the prompt rate and 9,000 at the cached one.
|
||||||
|
if got, want := p.Cost(10000, 9000, 0, 0, 0), int64(1000+900); got != want {
|
||||||
|
t.Errorf("a 90%% cached prompt cost %d, want %d", got, want)
|
||||||
|
}
|
||||||
|
// More cached than prompt is an engine's miscount, not a refund.
|
||||||
|
if got := p.Cost(100, 500, 0, 0, 0); got != 10 {
|
||||||
|
t.Errorf("cached beyond the prompt cost %d, want 10", got)
|
||||||
|
}
|
||||||
|
// No cached rate: cached tokens cost what any prompt token does.
|
||||||
|
if got := (Price{PromptPer1k: 1000}).Cost(10000, 9000, 0, 0, 0); got != 10000 {
|
||||||
|
t.Errorf("without a cached rate a cached prompt cost %d, want 10000", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||||
if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
|
if got := p.Cost(20, 0, 0, 3, 0); got != 20+150000 {
|
||||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||||
}
|
}
|
||||||
if (Price{PerImage: 1}).Zero() {
|
if (Price{PerImage: 1}).Zero() {
|
||||||
|
|||||||
@@ -18,6 +18,11 @@ import (
|
|||||||
type Price struct {
|
type Price struct {
|
||||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||||
|
// CachedPer1k is what a prompt token costs that the engine or provider
|
||||||
|
// read from its cache instead of computing: the part of prompt_tokens
|
||||||
|
// the answer counts as cached_tokens. Zero charges those at
|
||||||
|
// PromptPer1k, as before there was a rate for them.
|
||||||
|
CachedPer1k int64 `koanf:"cached_per_1k"`
|
||||||
// PerImage is what one picture costs, for a model that makes them: a
|
// PerImage is what one picture costs, for a model that makes them: a
|
||||||
// picture's cost is the card's time, which its prompt's few tokens do
|
// picture's cost is the card's time, which its prompt's few tokens do
|
||||||
// not measure.
|
// not measure.
|
||||||
@@ -33,15 +38,24 @@ type Price struct {
|
|||||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||||
// (§7.7).
|
// (§7.7).
|
||||||
func (p Price) Zero() bool {
|
func (p Price) Zero() bool {
|
||||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.CachedPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||||
}
|
}
|
||||||
|
|
||||||
// Cost computes what a request spent, rounding up.
|
// Cost computes what a request spent, rounding up.
|
||||||
//
|
//
|
||||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||||
func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
|
//
|
||||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
// cachedTokens are counted within promptTokens, as OpenAI's usage counts
|
||||||
|
// them, and are charged at the cached rate instead of the prompt rate.
|
||||||
|
func (p Price) Cost(promptTokens, cachedTokens, completionTokens, images, meshes int64) int64 {
|
||||||
|
cachedTokens = min(max(cachedTokens, 0), max(promptTokens, 0))
|
||||||
|
cachedRate := p.CachedPer1k
|
||||||
|
if cachedRate == 0 {
|
||||||
|
cachedRate = p.PromptPer1k
|
||||||
|
}
|
||||||
|
return divUp((promptTokens-cachedTokens)*p.PromptPer1k+cachedTokens*cachedRate, 1000) +
|
||||||
|
divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||||
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+91
-11
@@ -6,11 +6,13 @@ import (
|
|||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"crypto/tls"
|
"crypto/tls"
|
||||||
"encoding/hex"
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"maps"
|
"maps"
|
||||||
|
"math"
|
||||||
"math/rand/v2"
|
"math/rand/v2"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -391,8 +393,11 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
|||||||
_ = a.sup.Stop(ctx, o.ShardID)
|
_ = a.sup.Stop(ctx, o.ShardID)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
// What it computes on, for one started before the agent kept that.
|
// What it computes on, for one started before the agent kept that — or
|
||||||
if a.sup.ComputeOf(o.ShardID) == nil {
|
// kept as the CPU for a GPU placement by an agent that did not follow
|
||||||
|
// an engine into a process group of its own, which the check above has
|
||||||
|
// just found on its GPUs.
|
||||||
|
if c := a.sup.ComputeOf(o.ShardID); c == nil || (len(o.Spec.GPUIndices) > 0 && c.Backend == "CPU") {
|
||||||
if c, ok := a.gpuCompute(ctx, o.Spec.GPUIndices, sh.Status().PID); ok {
|
if c, ok := a.gpuCompute(ctx, o.Spec.GPUIndices, sh.Status().PID); ok {
|
||||||
a.sup.SetCompute(o.ShardID, c)
|
a.sup.SetCompute(o.ShardID, c)
|
||||||
}
|
}
|
||||||
@@ -512,6 +517,7 @@ func (a *Agent) scan(ctx context.Context) error {
|
|||||||
inv, err := discovery.Discover(ctx, opts, a.log)
|
inv, err := discovery.Discover(ctx, opts, a.log)
|
||||||
host := discovery.DiscoverHost()
|
host := discovery.DiscoverHost()
|
||||||
host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
|
host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
|
||||||
|
host.MemBandwidthGBs = a.memBandwidth()
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
a.inv = inv
|
a.inv = inv
|
||||||
a.prov.Host = host
|
a.prov.Host = host
|
||||||
@@ -568,6 +574,9 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
|||||||
reg.Host = &llmgwv1.Host{
|
reg.Host = &llmgwv1.Host{
|
||||||
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
|
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
|
||||||
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
|
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
|
||||||
|
// What is measured by now, not what the startup scan found before
|
||||||
|
// the measurement ended: a reconnect would otherwise say 0.
|
||||||
|
MemBandwidthGbs: a.memBandwidth(),
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := stream.Send(&llmgwv1.AgentMessage{
|
if err := stream.Send(&llmgwv1.AgentMessage{
|
||||||
@@ -665,6 +674,7 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
|||||||
// without anybody restarting anything.
|
// without anybody restarting anything.
|
||||||
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
|
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
|
||||||
hb.CpuCount = int32(runtime.NumCPU())
|
hb.CpuCount = int32(runtime.NumCPU())
|
||||||
|
hb.MemBandwidthGbs = a.memBandwidth()
|
||||||
|
|
||||||
a.reprobeEngines()
|
a.reprobeEngines()
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
@@ -1123,7 +1133,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
|||||||
},
|
},
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath); err != nil {
|
if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath, spec.Profile.AllowPickle); err != nil {
|
||||||
stop()
|
stop()
|
||||||
report(supervisor.Failed, err.Error(), nil)
|
report(supervisor.Failed, err.Error(), nil)
|
||||||
return
|
return
|
||||||
@@ -1131,11 +1141,36 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
|||||||
stop()
|
stop()
|
||||||
spec.Profile.ModelPath = path
|
spec.Profile.ModelPath = path
|
||||||
}
|
}
|
||||||
|
// A shared ComfyUI replica's members' weights, each fetched like the
|
||||||
|
// model's, and handed to the gate where they landed.
|
||||||
|
if raw := spec.Profile.Env[engines.SharedModelsEnv]; raw != "" {
|
||||||
|
var ms []map[string]string
|
||||||
|
if err := json.Unmarshal([]byte(raw), &ms); err != nil {
|
||||||
|
report(supervisor.Failed, "the shared replica's models are not readable: "+err.Error(), nil)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for i, m := range ms {
|
||||||
|
stop := a.watchFetch(m["source"], func(p weights.Progress) {
|
||||||
|
send(&llmgwv1.ShardStatus{State: agentsvc.StateToWire(supervisor.Starting),
|
||||||
|
Fetch: &llmgwv1.FetchProgress{Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal}})
|
||||||
|
})
|
||||||
|
path, err := a.weights().Ensure(ctx, m["source"], m["allow_pickle"] == "true")
|
||||||
|
stop()
|
||||||
|
if err != nil {
|
||||||
|
report(supervisor.Failed, fmt.Sprintf("model %s: %v", m["name"], err), nil)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ms[i]["path"] = path
|
||||||
|
}
|
||||||
|
b, _ := json.Marshal(ms)
|
||||||
|
spec.Profile.Env = maps.Clone(spec.Profile.Env)
|
||||||
|
spec.Profile.Env[engines.SharedModelsEnv] = string(b)
|
||||||
|
}
|
||||||
// Adapters are fetched like the model, and before the engine starts: it
|
// Adapters are fetched like the model, and before the engine starts: it
|
||||||
// loads them at start, and one missing is a replica that answers for the
|
// loads them at start, and one missing is a replica that answers for the
|
||||||
// base model under the adapter's name.
|
// base model under the adapter's name.
|
||||||
for i, ad := range spec.Profile.Adapters {
|
for i, ad := range spec.Profile.Adapters {
|
||||||
path, err := a.weights().Ensure(ctx, ad.Path)
|
path, err := a.weights().Ensure(ctx, ad.Path, spec.Profile.AllowPickle)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil)
|
report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil)
|
||||||
return
|
return
|
||||||
@@ -1391,11 +1426,14 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
|
|||||||
// off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
|
// off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
|
||||||
// in the socket's buffer until the kernel gives up retransmitting, fifteen
|
// in the socket's buffer until the kernel gives up retransmitting, fifteen
|
||||||
// minutes later. Found by switching off a leader: the standby took over in 14
|
// minutes later. Found by switching off a leader: the standby took over in 14
|
||||||
// seconds and the node stayed with the dead one. An HTTP/2 ping after ten
|
// seconds and the node stayed with the dead one. An HTTP/2 ping after five
|
||||||
// quiet seconds, given five to answer, finds it in a quarter of a minute.
|
// quiet seconds, given three to answer, finds it in eight: by the time the
|
||||||
|
// standby leads, as it does within the lease's ten seconds, the node is on
|
||||||
|
// its way to it. At fifteen, with the lease at fifteen too, the new leader
|
||||||
|
// went without its nodes, and so its judges, for half a minute.
|
||||||
const (
|
const (
|
||||||
streamIdle = 10 * time.Second
|
streamIdle = 5 * time.Second
|
||||||
streamPing = 5 * time.Second
|
streamPing = 3 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
// localizeSpec fills in what only this node knows.
|
// localizeSpec fills in what only this node knows.
|
||||||
@@ -1476,18 +1514,25 @@ func ResolveManagement(c Config) (addr, iface string) {
|
|||||||
// shard computes on as soon as that is known: an adopted engine's is found
|
// shard computes on as soon as that is known: an adopted engine's is found
|
||||||
// after registration, which carried what was known then.
|
// after registration, which carried what was known then.
|
||||||
func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMessage) {
|
func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMessage) {
|
||||||
sent := map[string]bool{}
|
// What was sent for each shard, so a compute that changes is sent again:
|
||||||
|
// an adopted shard's kept one is sent at once, and adopting it may then
|
||||||
|
// find it was kept wrong.
|
||||||
|
sent := map[string]string{}
|
||||||
t := time.NewTicker(5 * time.Second)
|
t := time.NewTicker(5 * time.Second)
|
||||||
defer t.Stop()
|
defer t.Stop()
|
||||||
for range 60 {
|
for range 60 {
|
||||||
for _, st := range a.sup.Statuses() {
|
for _, st := range a.sup.Statuses() {
|
||||||
if st.State.Terminal() || sent[st.ID] {
|
if st.State.Terminal() {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
c := a.sup.ComputeOf(st.ID)
|
c := a.sup.ComputeOf(st.ID)
|
||||||
if c == nil {
|
if c == nil {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
said := fmt.Sprint(*c)
|
||||||
|
if sent[st.ID] == said {
|
||||||
|
continue
|
||||||
|
}
|
||||||
sh, err := a.sup.Get(st.ID)
|
sh, err := a.sup.Get(st.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
continue
|
continue
|
||||||
@@ -1498,7 +1543,7 @@ func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMess
|
|||||||
}}}
|
}}}
|
||||||
select {
|
select {
|
||||||
case out <- msg:
|
case out <- msg:
|
||||||
sent[st.ID] = true
|
sent[st.ID] = said
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -1510,3 +1555,38 @@ func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMess
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// memBandwidth is the node's memory read rate in GB/s, 0 until measured.
|
||||||
|
// Measured in the background, the first time it is asked for: it reads half
|
||||||
|
// a gigabyte, which must not hold up an agent reaching its control plane,
|
||||||
|
// and the heartbeat carries it when it is known. Not at once but after a
|
||||||
|
// minute or so, chosen at random, and again every hour, keeping the best:
|
||||||
|
// VMs on one host that all restart together measure each other, and read a
|
||||||
|
// third less than any of them gets alone. Once a process rather than an
|
||||||
|
// agent, so tests running many agents in one read it once.
|
||||||
|
func (a *Agent) memBandwidth() float64 {
|
||||||
|
bwOnce.Do(func() {
|
||||||
|
go func() {
|
||||||
|
time.Sleep(bandwidthDelay())
|
||||||
|
for {
|
||||||
|
if bw := measureBandwidth(); bw > math.Float64frombits(bwBits.Load()) {
|
||||||
|
bwBits.Store(math.Float64bits(bw))
|
||||||
|
a.log.Info("this node's memory", "read_gbs", math.Round(bw*10)/10)
|
||||||
|
}
|
||||||
|
time.Sleep(time.Hour)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
})
|
||||||
|
return math.Float64frombits(bwBits.Load())
|
||||||
|
}
|
||||||
|
|
||||||
|
// bandwidthDelay is how long an agent waits before measuring, so agents
|
||||||
|
// started together measure apart.
|
||||||
|
var bandwidthDelay = func() time.Duration { return 10*time.Second + rand.N(110*time.Second) }
|
||||||
|
|
||||||
|
var (
|
||||||
|
bwOnce sync.Once
|
||||||
|
bwBits atomic.Uint64
|
||||||
|
// measureBandwidth is replaced in tests, which have no use for it.
|
||||||
|
measureBandwidth = discovery.MeasureMemoryBandwidth
|
||||||
|
)
|
||||||
@@ -671,7 +671,7 @@ func TestAShardWithMissingWeightsFailsBeforeLaunching(t *testing.T) {
|
|||||||
// nobody is watching — so it is caught here, where the message can say
|
// nobody is watching — so it is caught here, where the message can say
|
||||||
// what is actually wrong.
|
// what is actually wrong.
|
||||||
a := &Agent{cfg: Config{WeightsCache: t.TempDir()}, log: slog.New(slog.DiscardHandler)}
|
a := &Agent{cfg: Config{WeightsCache: t.TempDir()}, log: slog.New(slog.DiscardHandler)}
|
||||||
_, err := a.weights().Ensure(t.Context(), filepath.Join(t.TempDir(), "not-there.gguf"))
|
_, err := a.weights().Ensure(t.Context(), filepath.Join(t.TempDir(), "not-there.gguf"), false)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("a missing weights file was accepted")
|
t.Fatal("a missing weights file was accepted")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,6 +52,8 @@ type GPU struct {
|
|||||||
// memory the instance's share. Utilisation stays unreported, because no
|
// memory the instance's share. Utilisation stays unreported, because no
|
||||||
// driver measures it per instance.
|
// driver measures it per instance.
|
||||||
MIG *MIGDevice `json:"mig,omitempty"`
|
MIG *MIGDevice `json:"mig,omitempty"`
|
||||||
|
// PCIe is how the card is linked to its host, zero when not read.
|
||||||
|
PCIe PCIeLink `json:"pcie,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// GPUHealth records conditions that make a GPU unsafe to schedule onto.
|
// GPUHealth records conditions that make a GPU unsafe to schedule onto.
|
||||||
|
|||||||
@@ -67,10 +67,23 @@ func DiscoverGPUsNVML(_ context.Context) ([]GPU, error) {
|
|||||||
if major, minor, ret := dev.GetCudaComputeCapability(); ret == nvml.SUCCESS {
|
if major, minor, ret := dev.GetCudaComputeCapability(); ret == nvml.SUCCESS {
|
||||||
g.ComputeCapability = fmt.Sprintf("%d.%d", major, minor)
|
g.ComputeCapability = fmt.Sprintf("%d.%d", major, minor)
|
||||||
}
|
}
|
||||||
|
if v, ret := dev.GetMaxPcieLinkGeneration(); ret == nvml.SUCCESS {
|
||||||
|
g.PCIe.Gen = v
|
||||||
|
}
|
||||||
|
if v, ret := dev.GetGpuMaxPcieLinkGeneration(); ret == nvml.SUCCESS {
|
||||||
|
g.PCIe.GPUGen = v
|
||||||
|
}
|
||||||
|
if v, ret := dev.GetCurrPcieLinkWidth(); ret == nvml.SUCCESS {
|
||||||
|
g.PCIe.Width = v
|
||||||
|
}
|
||||||
|
if v, ret := dev.GetMaxPcieLinkWidth(); ret == nvml.SUCCESS {
|
||||||
|
g.PCIe.WidthMax = v
|
||||||
|
}
|
||||||
if pci, ret := dev.GetPciInfo(); ret == nvml.SUCCESS {
|
if pci, ret := dev.GetPciInfo(); ret == nvml.SUCCESS {
|
||||||
g.PCIBusID = busIDString(pci.BusId[:])
|
g.PCIBusID = busIDString(pci.BusId[:])
|
||||||
byBusID[normalizeBusID(g.PCIBusID)] = i
|
byBusID[normalizeBusID(g.PCIBusID)] = i
|
||||||
}
|
}
|
||||||
|
g.PCIe = withSysfs(g.PCIe, "/sys/bus/pci/devices", g.PCIBusID)
|
||||||
if t, ret := dev.GetTemperature(nvml.TEMPERATURE_GPU); ret == nvml.SUCCESS {
|
if t, ret := dev.GetTemperature(nvml.TEMPERATURE_GPU); ret == nvml.SUCCESS {
|
||||||
g.Health.TemperatureC = int(t)
|
g.Health.TemperatureC = int(t)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -64,6 +64,7 @@ func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
|
|||||||
if err != nil || len(cards) == 0 {
|
if err != nil || len(cards) == 0 {
|
||||||
return cards, err
|
return cards, err
|
||||||
}
|
}
|
||||||
|
readPCIe(ctx, binary, cards)
|
||||||
// The query lists whole cards, MIG mode or not. A card in MIG mode is
|
// The query lists whole cards, MIG mode or not. A card in MIG mode is
|
||||||
// replaced by its instances, which only -L names and only the full
|
// replaced by its instances, which only -L names and only the full
|
||||||
// report sizes; asked only when some card could be in MIG mode, since
|
// report sizes; asked only when some card could be in MIG mode, since
|
||||||
|
|||||||
@@ -31,6 +31,10 @@ type Host struct {
|
|||||||
// MinComputeCapability is the oldest GPU's, 0 when there are none or it
|
// MinComputeCapability is the oldest GPU's, 0 when there are none or it
|
||||||
// is not known. Not read here but from the GPU inventory, by the agent.
|
// is not known. Not read here but from the GPU inventory, by the agent.
|
||||||
MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
|
MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
|
||||||
|
// MemBandwidthGBs is how fast the memory reads, in GB/s, 0 when not
|
||||||
|
// measured: what decides a model's speed on the CPU. Not read here but
|
||||||
|
// measured by the agent when it starts (MeasureMemoryBandwidth).
|
||||||
|
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Family is the packaging family: "deb", "rpm", or "" when neither is known.
|
// Family is the packaging family: "deb", "rpm", or "" when neither is known.
|
||||||
|
|||||||
@@ -0,0 +1,146 @@
|
|||||||
|
package discovery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// How fast this machine reads its memory, which on a CPU is what decides how
|
||||||
|
// fast a model answers: every token of a reply reads all the weights it uses
|
||||||
|
// once, so a node that reads 40 GB/s writes a 4 GB model's tokens about ten
|
||||||
|
// times a second, whatever its core count. Measured once when the agent
|
||||||
|
// starts, in a fraction of a second, by every CPU summing a share of a buffer
|
||||||
|
// too large for any cache — a VM sharing its host's memory with others gets
|
||||||
|
// its share, which is the number that matters for it.
|
||||||
|
|
||||||
|
// bandwidthBuffer is what is read at most; the least is eight times the
|
||||||
|
// processor's last cache, and never under minBuffer. Smaller, it would be
|
||||||
|
// the cache that was measured — some hold a gigabyte — and the figure kept
|
||||||
|
// would be too high.
|
||||||
|
const (
|
||||||
|
bandwidthBuffer = 2 << 30
|
||||||
|
minBuffer = 256 << 20
|
||||||
|
)
|
||||||
|
|
||||||
|
// MeasureMemoryBandwidth is the memory's read rate in GB/s, the best of a
|
||||||
|
// few passes; 0 when the machine has too little free memory to measure.
|
||||||
|
func MeasureMemoryBandwidth() float64 {
|
||||||
|
size := int64(bandwidthBuffer)
|
||||||
|
// Never more than a quarter of what is free: a node already holding
|
||||||
|
// models must not be pushed into swap by a measurement. Without room
|
||||||
|
// for a buffer past the cache, none is measured: the hourly retry
|
||||||
|
// measures it when there is.
|
||||||
|
if avail := memAvailable("/proc/meminfo"); avail > 0 {
|
||||||
|
size = min(size, avail/4)
|
||||||
|
}
|
||||||
|
if size < max(minBuffer, 8*lastCacheBytes("/sys/devices/system/cpu")) {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
buf := make([]uint64, size/8)
|
||||||
|
for i := range buf {
|
||||||
|
buf[i] = uint64(i)
|
||||||
|
}
|
||||||
|
workers := runtime.NumCPU()
|
||||||
|
chunk := (len(buf) + workers - 1) / workers
|
||||||
|
best := 0.0
|
||||||
|
for pass := 0; pass < 4; pass++ {
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
start := time.Now()
|
||||||
|
for w := 0; w < workers; w++ {
|
||||||
|
lo, hi := w*chunk, min((w+1)*chunk, len(buf))
|
||||||
|
if lo >= hi {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
wg.Add(1)
|
||||||
|
go func(part []uint64) {
|
||||||
|
defer wg.Done()
|
||||||
|
sink(sum(part))
|
||||||
|
}(buf[lo:hi])
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if s := time.Since(start).Seconds(); s > 0 {
|
||||||
|
best = max(best, float64(size)/s/1e9)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best
|
||||||
|
}
|
||||||
|
|
||||||
|
// sum reads every word, eight at a time so the loop is not what limits it.
|
||||||
|
func sum(p []uint64) uint64 {
|
||||||
|
var a, b, c, d, e, f, g, h uint64
|
||||||
|
i := 0
|
||||||
|
for ; i+8 <= len(p); i += 8 {
|
||||||
|
a += p[i]
|
||||||
|
b += p[i+1]
|
||||||
|
c += p[i+2]
|
||||||
|
d += p[i+3]
|
||||||
|
e += p[i+4]
|
||||||
|
f += p[i+5]
|
||||||
|
g += p[i+6]
|
||||||
|
h += p[i+7]
|
||||||
|
}
|
||||||
|
for ; i < len(p); i++ {
|
||||||
|
a += p[i]
|
||||||
|
}
|
||||||
|
return a + b + c + d + e + f + g + h
|
||||||
|
}
|
||||||
|
|
||||||
|
// sinkValue keeps the sums alive, so the reads are not optimised away.
|
||||||
|
var sinkValue struct {
|
||||||
|
sync.Mutex
|
||||||
|
v uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
func sink(v uint64) {
|
||||||
|
sinkValue.Lock()
|
||||||
|
sinkValue.v += v
|
||||||
|
sinkValue.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// memAvailable is MemAvailable in bytes, 0 when unknown.
|
||||||
|
func memAvailable(path string) int64 {
|
||||||
|
f, err := os.Open(path) //nolint:gosec // a fixed system path
|
||||||
|
if err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
sc := bufio.NewScanner(f)
|
||||||
|
for sc.Scan() {
|
||||||
|
fields := strings.Fields(sc.Text())
|
||||||
|
if len(fields) >= 2 && fields[0] == "MemAvailable:" {
|
||||||
|
kb, _ := strconv.ParseInt(fields[1], 10, 64)
|
||||||
|
return kb << 10
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastCacheBytes is the largest cache any CPU reports, 0 when unknown.
|
||||||
|
func lastCacheBytes(root string) int64 {
|
||||||
|
paths, _ := filepath.Glob(filepath.Join(root, "cpu*", "cache", "index*", "size"))
|
||||||
|
var most int64
|
||||||
|
for _, p := range paths {
|
||||||
|
b, err := os.ReadFile(p) //nolint:gosec // under /sys
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
v := strings.TrimSpace(string(b))
|
||||||
|
mult := int64(1)
|
||||||
|
switch {
|
||||||
|
case strings.HasSuffix(v, "K"):
|
||||||
|
mult, v = 1<<10, strings.TrimSuffix(v, "K")
|
||||||
|
case strings.HasSuffix(v, "M"):
|
||||||
|
mult, v = 1<<20, strings.TrimSuffix(v, "M")
|
||||||
|
}
|
||||||
|
if n, err := strconv.ParseInt(v, 10, 64); err == nil {
|
||||||
|
most = max(most, n*mult)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return most
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package discovery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMemoryBandwidthIsMeasured(t *testing.T) {
|
||||||
|
if testing.Short() {
|
||||||
|
t.Skip("reads half a gigabyte")
|
||||||
|
}
|
||||||
|
bw := MeasureMemoryBandwidth()
|
||||||
|
// Any machine running this reads more than a gigabyte a second and less
|
||||||
|
// than a few terabytes.
|
||||||
|
if bw < 1 || bw > 5000 {
|
||||||
|
t.Errorf("measured %.1f GB/s", bw)
|
||||||
|
}
|
||||||
|
t.Logf("%.1f GB/s", bw)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheLastCacheIsReadFromSys(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
for path, size := range map[string]string{"cpu0/cache/index0/size": "48K", "cpu0/cache/index3/size": "96M"} {
|
||||||
|
p := filepath.Join(root, path)
|
||||||
|
_ = os.MkdirAll(filepath.Dir(p), 0o755)
|
||||||
|
_ = os.WriteFile(p, []byte(size+"\n"), 0o644)
|
||||||
|
}
|
||||||
|
if got := lastCacheBytes(root); got != 96<<20 {
|
||||||
|
t.Errorf("%d", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
package discovery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/csv"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A card in the wrong slot: an x16 card in a slot wired x8 or x4, behind a
|
||||||
|
// riser, or in a slot of an older PCIe generation than the card's, links
|
||||||
|
// slower than it was built to, and nothing says so. A model on the card
|
||||||
|
// still decodes at the card's own memory speed, but it loads slower, a
|
||||||
|
// tensor-parallel replica over PCIe is slower, and so is anything that
|
||||||
|
// spills to the host. The driver says how the link stands and what the
|
||||||
|
// card could do; the difference is reported where the node's warnings are.
|
||||||
|
|
||||||
|
// PCIeLink is how a card is linked to its host.
|
||||||
|
type PCIeLink struct {
|
||||||
|
// Gen is the fastest generation the card and the slot together allow —
|
||||||
|
// not the current one, which drops while the card idles, to save power.
|
||||||
|
// GPUGen is the card's own fastest, 0 when the driver does not say.
|
||||||
|
Gen int `json:"gen,omitempty"`
|
||||||
|
GPUGen int `json:"gpu_gen,omitempty"`
|
||||||
|
// Width is the lanes in use now, WidthMax the card's most.
|
||||||
|
Width int `json:"width,omitempty"`
|
||||||
|
WidthMax int `json:"width_max,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// laneGBs is one lane's usable rate in GB/s, by generation.
|
||||||
|
var laneGBs = map[int]float64{1: 0.25, 2: 0.5, 3: 0.985, 4: 1.969, 5: 3.938, 6: 7.877}
|
||||||
|
|
||||||
|
// GBs is what a link of gen and width carries, in GB/s each way.
|
||||||
|
func pcieGBs(gen, width int) float64 { return laneGBs[gen] * float64(width) }
|
||||||
|
|
||||||
|
// Problem says how a card's link falls short of what the card could do,
|
||||||
|
// empty when it does not or the driver did not say.
|
||||||
|
func (p PCIeLink) Problem() string {
|
||||||
|
gen := p.Gen
|
||||||
|
if p.GPUGen > 0 && gen == 0 {
|
||||||
|
gen = p.GPUGen
|
||||||
|
}
|
||||||
|
slowGen := p.GPUGen > 0 && p.Gen > 0 && p.Gen < p.GPUGen
|
||||||
|
narrow := p.WidthMax > 0 && p.Width > 0 && p.Width < p.WidthMax
|
||||||
|
if !slowGen && !narrow {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
could := pcieGBs(max(p.GPUGen, gen), p.WidthMax)
|
||||||
|
has := pcieGBs(gen, p.Width)
|
||||||
|
var why []string
|
||||||
|
if narrow {
|
||||||
|
why = append(why, fmt.Sprintf("x%d of its x%d lanes", p.Width, p.WidthMax))
|
||||||
|
}
|
||||||
|
if slowGen {
|
||||||
|
why = append(why, fmt.Sprintf("PCIe Gen%d of its Gen%d", p.Gen, p.GPUGen))
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("linked at %s: %.1f GB/s where it could have %.1f. "+
|
||||||
|
"Check the slot (wired fewer lanes, or an older generation), a riser, or the BIOS's PCIe setting",
|
||||||
|
strings.Join(why, " and "), has, could)
|
||||||
|
}
|
||||||
|
|
||||||
|
// pcieFields are asked apart from the rest, and the card's own fastest
|
||||||
|
// generation apart again: a driver that does not know a field refuses the
|
||||||
|
// whole query, and these must not cost a node its cards.
|
||||||
|
var pcieFields = [][]string{
|
||||||
|
{"index", "pcie.link.gen.max", "pcie.link.width.current", "pcie.link.width.max", "pcie.link.gen.gpumax"},
|
||||||
|
{"index", "pcie.link.gen.max", "pcie.link.width.current", "pcie.link.width.max"},
|
||||||
|
}
|
||||||
|
|
||||||
|
// readPCIe fills each card's link from nvidia-smi, leaving it empty when
|
||||||
|
// the driver says nothing of it.
|
||||||
|
func readPCIe(ctx context.Context, binary string, cards []GPU) {
|
||||||
|
for _, fields := range pcieFields {
|
||||||
|
out, err := exec.CommandContext(ctx, binary, "--query-gpu="+strings.Join(fields, ","),
|
||||||
|
"--format=csv,noheader,nounits").Output()
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
links := parsePCIe(string(out))
|
||||||
|
for i := range cards {
|
||||||
|
if l, ok := links[cards[i].Index]; ok && cards[i].MIG == nil {
|
||||||
|
cards[i].PCIe = l
|
||||||
|
}
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
for i := range cards {
|
||||||
|
if cards[i].MIG == nil {
|
||||||
|
cards[i].PCIe = withSysfs(cards[i].PCIe, "/sys/bus/pci/devices", cards[i].PCIBusID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// withSysfs takes the card's own lane count, and its own fastest
|
||||||
|
// generation, from the kernel: the driver's width.max is what card and slot
|
||||||
|
// allow together, so a x16 card in a slot wired x8 reads x8 of x8 there,
|
||||||
|
// while the device's max_link_width is the card's, 16.
|
||||||
|
func withSysfs(l PCIeLink, root, busID string) PCIeLink {
|
||||||
|
dev := sysfsBusID(busID)
|
||||||
|
if dev == "" {
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
read := func(name string) string {
|
||||||
|
b, err := os.ReadFile(filepath.Join(root, dev, name)) //nolint:gosec // under /sys
|
||||||
|
if err != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(string(b))
|
||||||
|
}
|
||||||
|
if n, err := strconv.Atoi(read("max_link_width")); err == nil && n > 0 {
|
||||||
|
l.WidthMax = n
|
||||||
|
}
|
||||||
|
if n, err := strconv.Atoi(read("current_link_width")); err == nil && n > 0 {
|
||||||
|
l.Width = n
|
||||||
|
}
|
||||||
|
if g := genOfSpeed(read("max_link_speed")); g > 0 && l.GPUGen == 0 {
|
||||||
|
l.GPUGen = g
|
||||||
|
}
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
|
||||||
|
// sysfsBusID is nvidia-smi's 00000000:3B:00.0 as the kernel names it,
|
||||||
|
// 0000:3b:00.0.
|
||||||
|
func sysfsBusID(id string) string {
|
||||||
|
id = strings.ToLower(strings.TrimSpace(id))
|
||||||
|
dom, rest, ok := strings.Cut(id, ":")
|
||||||
|
if !ok || len(dom) < 4 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return dom[len(dom)-4:] + ":" + rest
|
||||||
|
}
|
||||||
|
|
||||||
|
// genOfSpeed is the PCIe generation of a link speed as sysfs writes it.
|
||||||
|
func genOfSpeed(s string) int {
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(s, "2.5 GT/s"):
|
||||||
|
return 1
|
||||||
|
case strings.HasPrefix(s, "5.0 GT/s"), strings.HasPrefix(s, "5 GT/s"):
|
||||||
|
return 2
|
||||||
|
case strings.HasPrefix(s, "8.0 GT/s"), strings.HasPrefix(s, "8 GT/s"):
|
||||||
|
return 3
|
||||||
|
case strings.HasPrefix(s, "16.0 GT/s"), strings.HasPrefix(s, "16 GT/s"):
|
||||||
|
return 4
|
||||||
|
case strings.HasPrefix(s, "32.0 GT/s"), strings.HasPrefix(s, "32 GT/s"):
|
||||||
|
return 5
|
||||||
|
case strings.HasPrefix(s, "64.0 GT/s"), strings.HasPrefix(s, "64 GT/s"):
|
||||||
|
return 6
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func parsePCIe(out string) map[int]PCIeLink {
|
||||||
|
r := csv.NewReader(strings.NewReader(out))
|
||||||
|
r.TrimLeadingSpace = true
|
||||||
|
r.FieldsPerRecord = -1
|
||||||
|
rows, _ := r.ReadAll()
|
||||||
|
num := func(s string) int {
|
||||||
|
n, _ := strconv.Atoi(strings.TrimSpace(s))
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
links := map[int]PCIeLink{}
|
||||||
|
for _, row := range rows {
|
||||||
|
if len(row) < 4 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
idx, err := strconv.Atoi(strings.TrimSpace(row[0]))
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
l := PCIeLink{Gen: num(row[1]), Width: num(row[2]), WidthMax: num(row[3])}
|
||||||
|
if len(row) > 4 {
|
||||||
|
l.GPUGen = num(row[4])
|
||||||
|
}
|
||||||
|
links[idx] = l
|
||||||
|
}
|
||||||
|
return links
|
||||||
|
}
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
package discovery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestACardInTheWrongSlotIsSaidSo(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
link PCIeLink
|
||||||
|
want string // empty: nothing to say
|
||||||
|
}{
|
||||||
|
// The PoC's Quadro RTX 6000, idle: current Gen1, but its fastest is 3,
|
||||||
|
// as the card's is, on all 16 lanes. Nothing wrong.
|
||||||
|
{PCIeLink{Gen: 3, GPUGen: 3, Width: 16, WidthMax: 16}, ""},
|
||||||
|
// An x16 card in a slot wired x8.
|
||||||
|
{PCIeLink{Gen: 4, GPUGen: 4, Width: 8, WidthMax: 16}, "x8 of its x16 lanes: 15.8 GB/s where it could have 31.5"},
|
||||||
|
// A Gen5 card in a Gen4 slot.
|
||||||
|
{PCIeLink{Gen: 4, GPUGen: 5, Width: 16, WidthMax: 16}, "PCIe Gen4 of its Gen5: 31.5 GB/s where it could have 63.0"},
|
||||||
|
// Both, behind a riser.
|
||||||
|
{PCIeLink{Gen: 3, GPUGen: 4, Width: 4, WidthMax: 16}, "x4 of its x16 lanes and PCIe Gen3 of its Gen4"},
|
||||||
|
// A driver that says nothing.
|
||||||
|
{PCIeLink{}, ""},
|
||||||
|
} {
|
||||||
|
got := c.link.Problem()
|
||||||
|
if (c.want == "") != (got == "") || !strings.Contains(got, c.want) {
|
||||||
|
t.Errorf("%+v: %q, want %q", c.link, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheLinkIsReadFromNvidiaSMI(t *testing.T) {
|
||||||
|
links := parsePCIe("0, 3, 16, 16, 3\n1, 4, 8, 16, [N/A]\n")
|
||||||
|
if links[0] != (PCIeLink{Gen: 3, GPUGen: 3, Width: 16, WidthMax: 16}) || links[1] != (PCIeLink{Gen: 4, Width: 8, WidthMax: 16}) {
|
||||||
|
t.Errorf("%+v", links)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheCardsOwnLanesComeFromTheKernel(t *testing.T) {
|
||||||
|
// The driver says x8 of x8 for a x16 card in a slot wired x8; the
|
||||||
|
// device's own max_link_width says 16.
|
||||||
|
root := t.TempDir()
|
||||||
|
dev := filepath.Join(root, "0000:3b:00.0")
|
||||||
|
_ = os.MkdirAll(dev, 0o755)
|
||||||
|
for f, v := range map[string]string{"max_link_width": "16", "current_link_width": "8", "max_link_speed": "16.0 GT/s PCIe"} {
|
||||||
|
_ = os.WriteFile(filepath.Join(dev, f), []byte(v+"\n"), 0o644)
|
||||||
|
}
|
||||||
|
l := withSysfs(PCIeLink{Gen: 4, Width: 8, WidthMax: 8}, root, "00000000:3B:00.0")
|
||||||
|
if l.WidthMax != 16 || l.Width != 8 || l.GPUGen != 4 || !strings.Contains(l.Problem(), "x8 of its x16") {
|
||||||
|
t.Errorf("%+v: %q", l, l.Problem())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -74,6 +74,13 @@ func NewSensorSampler() *SensorSampler {
|
|||||||
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SampleGPUs reads every card's heat, power and how busy it is now, without
|
||||||
|
// a sampler's ten seconds of memory: for a screen that asks every few seconds
|
||||||
|
// and wants what it asked for. Nil on a machine without nvidia-smi or a card.
|
||||||
|
func SampleGPUs(ctx context.Context) []GPUSensor {
|
||||||
|
return (&SensorSampler{smi: runSMI}).gpus(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
||||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|||||||
@@ -129,7 +129,7 @@ func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
|||||||
// so taken before baseEnv drops them. Without the repository the gate
|
// so taken before baseEnv drops them. Without the repository the gate
|
||||||
// links no Hugging Face cache, and sf3d's node went to the network for
|
// links no Hugging Face cache, and sf3d's node went to the network for
|
||||||
// its gated repository and was refused with a 401.
|
// its gated repository and was refused with a 401.
|
||||||
for _, k := range []string{comfyNodesEnv, comfyHFRepoEnv} {
|
for _, k := range []string{comfyNodesEnv, comfyHFRepoEnv, SharedModelsEnv} {
|
||||||
if v := p.Env[k]; v != "" {
|
if v := p.Env[k]; v != "" {
|
||||||
env[k] = v
|
env[k] = v
|
||||||
}
|
}
|
||||||
@@ -144,3 +144,8 @@ const comfyNodesEnv = "LLMGW_COMFYUI_NODES"
|
|||||||
// comfyHFRepoEnv is the repository a custom node loads the weights by name
|
// comfyHFRepoEnv is the repository a custom node loads the weights by name
|
||||||
// from; catalog.HFRepoEnv and comfy.HFRepoEnv by the same name.
|
// from; catalog.HFRepoEnv and comfy.HFRepoEnv by the same name.
|
||||||
const comfyHFRepoEnv = "LLMGW_COMFYUI_HF_REPO"
|
const comfyHFRepoEnv = "LLMGW_COMFYUI_HF_REPO"
|
||||||
|
|
||||||
|
// SharedModelsEnv is a shared replica's members — name, source, and the
|
||||||
|
// path the agent fetched it to — as JSON; catalog.SharedModelsEnv and
|
||||||
|
// comfy.SharedModelsEnv by the same name.
|
||||||
|
const SharedModelsEnv = "LLMGW_COMFYUI_MODELS"
|
||||||
@@ -47,6 +47,9 @@ type Profile struct {
|
|||||||
// KVEvents has the engine publish its KV cache events to the agent,
|
// KVEvents has the engine publish its KV cache events to the agent,
|
||||||
// for the router's precise index. vLLM only.
|
// for the router's precise index. vLLM only.
|
||||||
KVEvents bool `json:"kv_events,omitempty"`
|
KVEvents bool `json:"kv_events,omitempty"`
|
||||||
|
// AllowPickle lets the node fetch weights in a pickle format, which run
|
||||||
|
// code when loaded; the catalog's allow_pickle.
|
||||||
|
AllowPickle bool `json:"allow_pickle,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ShardSpec is one shard of one replica on this node.
|
// ShardSpec is one shard of one replica on this node.
|
||||||
@@ -215,6 +218,7 @@ func NewRegistry(cfg Config) *Registry {
|
|||||||
r.Add(&TRTLLM{cfg: cfg.TRTLLM})
|
r.Add(&TRTLLM{cfg: cfg.TRTLLM})
|
||||||
r.Add(&Sim{cfg: cfg.Sim})
|
r.Add(&Sim{cfg: cfg.Sim})
|
||||||
r.Add(&ComfyUI{cfg: cfg.ComfyUI})
|
r.Add(&ComfyUI{cfg: cfg.ComfyUI})
|
||||||
|
r.Add(&Strata{cfg: cfg.Strata})
|
||||||
return r
|
return r
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -265,6 +269,9 @@ type Config struct {
|
|||||||
Sim SimConfig `koanf:"sim"`
|
Sim SimConfig `koanf:"sim"`
|
||||||
// ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
// ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||||
ComfyUI ComfyUIConfig `koanf:"comfyui"`
|
ComfyUI ComfyUIConfig `koanf:"comfyui"`
|
||||||
|
// Strata serves a few mixture-of-experts models from RAM and one card;
|
||||||
|
// offered by a node whose Strata is named here, or that installed it.
|
||||||
|
Strata StrataConfig `koanf:"strata"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- shared helpers ---
|
// --- shared helpers ---
|
||||||
|
|||||||
@@ -477,14 +477,14 @@ func TestValidateRequiresModelPath(t *testing.T) {
|
|||||||
|
|
||||||
func TestRegistry(t *testing.T) {
|
func TestRegistry(t *testing.T) {
|
||||||
r := NewRegistry(Config{})
|
r := NewRegistry(Config{})
|
||||||
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "strata", "trtllm", "vllm"}) {
|
||||||
t.Errorf("registry contents: %v", got)
|
t.Errorf("registry contents: %v", got)
|
||||||
}
|
}
|
||||||
if _, err := r.Get("vllm"); err != nil {
|
if _, err := r.Get("vllm"); err != nil {
|
||||||
t.Errorf("vllm missing: %v", err)
|
t.Errorf("vllm missing: %v", err)
|
||||||
}
|
}
|
||||||
err := func() error { _, err := r.Get("nope"); return err }()
|
err := func() error { _, err := r.Get("nope"); return err }()
|
||||||
if err == nil || !strings.Contains(err.Error(), "llamacpp, sglang, sim, trtllm, vllm") {
|
if err == nil || !strings.Contains(err.Error(), "llamacpp, sglang, sim, strata, trtllm, vllm") {
|
||||||
t.Errorf("an unknown engine should list what is available: %v", err)
|
t.Errorf("an unknown engine should list what is available: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
package engines
|
||||||
|
|
||||||
|
import (
|
||||||
|
"slices"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAModelDownloadedByTheEngineIsReadAsSafetensorsOnly(t *testing.T) {
|
||||||
|
for path, want := range map[string]bool{
|
||||||
|
"Qwen/Qwen2.5-7B-Instruct": true,
|
||||||
|
"/var/lib/llmgw/weights/abc/": false, "./local": false,
|
||||||
|
"http://m/q/v/": false, "q.gguf": false,
|
||||||
|
} {
|
||||||
|
got := slices.Contains(safetensorsOnly(Profile{ModelPath: path}), "safetensors")
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("%q: safetensors only = %v, want %v", path, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(safetensorsOnly(Profile{ModelPath: "Qwen/Qwen2.5-7B-Instruct", AllowPickle: true})) != 0 {
|
||||||
|
t.Error("allow_pickle still forced safetensors")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAForbiddenFlagIsKnownHoweverItIsSpelled(t *testing.T) {
|
||||||
|
for engine, args := range map[string][]string{
|
||||||
|
"vllm": {"--trust_remote_code"},
|
||||||
|
"sglang": {"--load-format", "pt"},
|
||||||
|
} {
|
||||||
|
if UnsafeArgs(engine, args) == nil {
|
||||||
|
t.Errorf("%s %v was let through", engine, args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, a := range [][]string{{"--load_format", "pt"}, {"--trust-remote"}, {"--load-form=pt"}} {
|
||||||
|
if UnsafeArgs("vllm", a) == nil {
|
||||||
|
t.Errorf("vllm %v was let through", a)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// What is not one of them, nor the start of one, still passes.
|
||||||
|
for _, a := range [][]string{{"--max-model-len", "8192"}, {"--tokenizer-mode", "mistral"}, {"--gpu-memory-utilization=0.9"}} {
|
||||||
|
if err := UnsafeArgs("vllm", a); err != nil {
|
||||||
|
t.Errorf("vllm %v was refused: %v", a, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Short flags are matched as written; llama.cpp takes no abbreviations,
|
||||||
|
// so its --chat-template is not taken for --chat-template-file.
|
||||||
|
for _, a := range [][]string{{"-m", "x"}, {"-hf", "o/r"}, {"-mu", "http://x"}} {
|
||||||
|
if UnsafeArgs("llamacpp", a) == nil {
|
||||||
|
t.Errorf("llamacpp %v was let through", a)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := UnsafeArgs("llamacpp", []string{"--chat-template", "chatml"}); err != nil {
|
||||||
|
t.Errorf("llama.cpp's own --chat-template was refused: %v", err)
|
||||||
|
}
|
||||||
|
// ComfyUI's argparse reads --li as --listen.
|
||||||
|
if UnsafeArgs("comfyui", []string{"--li", "0.0.0.0"}) == nil {
|
||||||
|
t.Error("comfyui --li was let through")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -102,6 +102,7 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
|
|||||||
if p.ServedModelName != "" {
|
if p.ServedModelName != "" {
|
||||||
args = append(args, "--served-model-name", p.ServedModelName)
|
args = append(args, "--served-model-name", p.ServedModelName)
|
||||||
}
|
}
|
||||||
|
args = append(args, safetensorsOnly(p)...)
|
||||||
if s.NNodes > 1 {
|
if s.NNodes > 1 {
|
||||||
args = append(args,
|
args = append(args,
|
||||||
"--nnodes", strconv.Itoa(s.NNodes),
|
"--nnodes", strconv.Itoa(s.NNodes),
|
||||||
|
|||||||
@@ -0,0 +1,144 @@
|
|||||||
|
package engines
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"sync/atomic"
|
||||||
|
)
|
||||||
|
|
||||||
|
// StrataConfig locates a Strata installation.
|
||||||
|
type StrataConfig struct {
|
||||||
|
// Dir is Strata's source directory, where setup.py and serve/ are.
|
||||||
|
Dir string `koanf:"dir"`
|
||||||
|
// Python is the interpreter of its environment; Dir's .venv when unset.
|
||||||
|
Python string `koanf:"python"`
|
||||||
|
// Engine is the directory of its prebuilt engine, where the strata
|
||||||
|
// binary and the CUDA libraries it ships are; Dir's engine/ when unset.
|
||||||
|
Engine string `koanf:"engine"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Strata runs Strata (github.com/Niko1221/Strata): a mixture-of-experts
|
||||||
|
// engine that keeps a model's experts in RAM and the ones used most in VRAM,
|
||||||
|
// so a 125B model serves from one 24 GB card. It runs a few model families
|
||||||
|
// only, each prepared from its GGUF files by Strata's own setup, so the agent
|
||||||
|
// starts llmgw strata-serve, which prepares the model and then becomes
|
||||||
|
// Strata's server.
|
||||||
|
type Strata struct {
|
||||||
|
cfg StrataConfig
|
||||||
|
installed atomic.Pointer[string]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Name identifies the engine in the catalog.
|
||||||
|
func (*Strata) Name() string { return "strata" }
|
||||||
|
|
||||||
|
// HealthPath answers 200 only once the model is loaded.
|
||||||
|
func (*Strata) HealthPath() string { return "/health" }
|
||||||
|
|
||||||
|
// MetricsPath is Strata's Prometheus text, in vLLM's names; its /metrics is
|
||||||
|
// JSON for its own monitor unless asked for this format.
|
||||||
|
func (*Strata) MetricsPath() string { return "/metrics?format=prometheus" }
|
||||||
|
|
||||||
|
// ServesHTTP is true: a Strata replica is one process on one node.
|
||||||
|
func (*Strata) ServesHTTP(ShardSpec) bool { return true }
|
||||||
|
|
||||||
|
// UseInstalled points the engine at the environment the agent installed:
|
||||||
|
// its interpreter, with Strata's code beside its bin in strata/ and the
|
||||||
|
// engine in strata-engine/.
|
||||||
|
func (s *Strata) UseInstalled(python string) { s.installed.Store(&python) }
|
||||||
|
|
||||||
|
// Configured reports whether the configuration names a Strata.
|
||||||
|
func (s *Strata) Configured() bool { return s.cfg.Dir != "" }
|
||||||
|
|
||||||
|
// StrataSourceDir and StrataEngineDir are where an installed environment
|
||||||
|
// keeps Strata's code and its prebuilt engine, beside bin/.
|
||||||
|
const (
|
||||||
|
StrataSourceDir = "strata"
|
||||||
|
StrataEngineDir = "strata-engine"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (s *Strata) env() string {
|
||||||
|
if p := s.installed.Load(); p != nil {
|
||||||
|
return filepath.Dir(filepath.Dir(*p))
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Strata) dir() string {
|
||||||
|
if e := s.env(); e != "" {
|
||||||
|
return filepath.Join(e, StrataSourceDir)
|
||||||
|
}
|
||||||
|
return s.cfg.Dir
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Strata) engineDir() string {
|
||||||
|
if e := s.env(); e != "" {
|
||||||
|
return filepath.Join(e, StrataEngineDir)
|
||||||
|
}
|
||||||
|
if s.cfg.Engine != "" {
|
||||||
|
return s.cfg.Engine
|
||||||
|
}
|
||||||
|
return filepath.Join(s.cfg.Dir, "engine")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Strata) python() string {
|
||||||
|
if p := s.installed.Load(); p != nil {
|
||||||
|
return *p
|
||||||
|
}
|
||||||
|
if s.cfg.Python != "" {
|
||||||
|
return s.cfg.Python
|
||||||
|
}
|
||||||
|
return filepath.Join(s.cfg.Dir, ".venv", "bin", "python")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate checks the profile: one card, on one node. Strata splits a model
|
||||||
|
// over a node's cards by layer, but that is not driven from here yet.
|
||||||
|
func (*Strata) Validate(p Profile, gpus []GPUInfo) error {
|
||||||
|
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||||
|
return fmt.Errorf("profile %q: strata runs on one GPU of one node", p.Name)
|
||||||
|
}
|
||||||
|
if err := requireComputeCapability(p, gpus, 7.0, "strata"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return validateCommon(p, gpus)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Commands is llmgw strata-serve, which prepares the model with Strata's
|
||||||
|
// setup and then execs Strata's server.
|
||||||
|
func (s *Strata) Commands(sh ShardSpec) ([]Command, error) {
|
||||||
|
dir := s.dir()
|
||||||
|
if dir == "" {
|
||||||
|
return nil, fmt.Errorf("strata: no Strata directory is configured (engines.strata.dir)")
|
||||||
|
}
|
||||||
|
bin, err := os.Executable()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
p := sh.Profile
|
||||||
|
args := []string{"strata-serve",
|
||||||
|
"--listen", net.JoinHostPort(sh.Host(), strconv.Itoa(sh.HTTPPort)),
|
||||||
|
"--model", p.ServedModelName,
|
||||||
|
"--weights", p.ModelPath,
|
||||||
|
// Relative to where engines run — their user's home — so they are
|
||||||
|
// theirs to write. The data is shared by the node's replicas: a
|
||||||
|
// prepared model and its draft layer are gigabytes, the same for
|
||||||
|
// every replica of it.
|
||||||
|
"--state", filepath.Join("strata", sh.ReplicaID),
|
||||||
|
"--data", filepath.Join("strata", "data"),
|
||||||
|
"--python", s.python(),
|
||||||
|
"--strata", dir,
|
||||||
|
"--engine", s.engineDir(),
|
||||||
|
"--",
|
||||||
|
}
|
||||||
|
args = append(args, p.Args...)
|
||||||
|
env := baseEnv(sh)
|
||||||
|
if sh.APIKey != "" {
|
||||||
|
// Strata's server reads its key from here, and strata-serve passes
|
||||||
|
// it on in the environment: not on a command line every user on the
|
||||||
|
// node can read.
|
||||||
|
env["STRATA_API_KEY"] = sh.APIKey
|
||||||
|
}
|
||||||
|
return []Command{{Name: "strata", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
package engines
|
||||||
|
|
||||||
|
import (
|
||||||
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestStrataStartsStrataServeWithTheKeyInTheEnvironment(t *testing.T) {
|
||||||
|
s := &Strata{}
|
||||||
|
s.UseInstalled("/opt/llmgw/engines/strata/0.2.7-cuda13/bin/python")
|
||||||
|
spec := ShardSpec{
|
||||||
|
ReplicaID: "r1", HTTPPort: 18090, BindAddr: "10.0.0.147", GPUIndices: []int{0},
|
||||||
|
APIKey: "secret-key",
|
||||||
|
Profile: Profile{Name: "p", Engine: "strata", TP: 1, ModelPath: "/var/lib/llmgw/models/q38",
|
||||||
|
ServedModelName: "qwen3.8-flash-next", Args: []string{"--family", "qwen", "--size", "IQ2_XS"}},
|
||||||
|
}
|
||||||
|
cmds, err := s.Commands(spec)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
c := cmds[0]
|
||||||
|
args := strings.Join(c.Args, " ")
|
||||||
|
env := "/opt/llmgw/engines/strata/0.2.7-cuda13"
|
||||||
|
for _, want := range []string{
|
||||||
|
"strata-serve", "--listen 10.0.0.147:18090", "--model qwen3.8-flash-next",
|
||||||
|
"--weights /var/lib/llmgw/models/q38", "--state " + filepath.Join("strata", "r1"),
|
||||||
|
"--python " + env + "/bin/python", "--strata " + env + "/strata", "--engine " + env + "/strata-engine",
|
||||||
|
"-- --family qwen --size IQ2_XS",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(args, want) {
|
||||||
|
t.Errorf("args lack %q: %s", want, args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.Contains(args, "secret-key") {
|
||||||
|
t.Error("the key is on the command line, which every user on the node can read")
|
||||||
|
}
|
||||||
|
if !slices.Contains(c.Env, "STRATA_API_KEY=secret-key") {
|
||||||
|
t.Error("the key is not in the environment, where Strata's server reads it")
|
||||||
|
}
|
||||||
|
if !slices.Contains(c.Env, "CUDA_VISIBLE_DEVICES=0") {
|
||||||
|
t.Error("the shard is not pinned to its GPU")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStrataRunsOnOneCard(t *testing.T) {
|
||||||
|
s := &Strata{}
|
||||||
|
p := Profile{Name: "p", Engine: "strata", TP: 1, GPUsPerNode: 2, ModelPath: "/m"}
|
||||||
|
if err := s.Validate(p, nil); err == nil {
|
||||||
|
t.Error("a two-card profile was accepted")
|
||||||
|
}
|
||||||
|
p.GPUsPerNode = 1
|
||||||
|
if err := s.Validate(p, []GPUInfo{{Name: "Quadro RTX 6000", ComputeCapability: "7.5"}}); err != nil {
|
||||||
|
t.Errorf("one Turing card was refused: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStrataSetupFlagsThatMoveFilesAreRefused(t *testing.T) {
|
||||||
|
for _, bad := range []string{"--gguf-dir", "--gguf", "--data-dir=/etc", "--model", "--mod", "--feed", "--host", "--gpus"} {
|
||||||
|
if err := UnsafeArgs("strata", []string{bad, "x"}); err == nil {
|
||||||
|
t.Errorf("%s was accepted", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := UnsafeArgs("strata", []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768",
|
||||||
|
"--parallel", "2", "--vram-reserve-mib", "2048", "--vision", "no"}); err != nil {
|
||||||
|
t.Errorf("an ordinary profile was refused: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ package engines
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"os"
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -101,6 +102,13 @@ var flagsNever = map[string]map[string]string{
|
|||||||
"--hf-token": "is a credential", "--alias": "is set by the agent",
|
"--hf-token": "is a credential", "--alias": "is set by the agent",
|
||||||
"--model-draft": "loads a file", "-md": "loads a file",
|
"--model-draft": "loads a file", "-md": "loads a file",
|
||||||
"--prompt-cache": "writes a file", "--logdir": "writes files",
|
"--prompt-cache": "writes a file", "--logdir": "writes files",
|
||||||
|
// The short spellings of the same.
|
||||||
|
"-mm": "loads a file; the projector is fetched with the model", "--mmproj-url": "downloads a file",
|
||||||
|
"-mmu": "downloads a file", "-hfr": "downloads a model", "-hff": "downloads a model",
|
||||||
|
"--hf-repo-draft": "downloads a model", "-hfd": "downloads a model", "-hfrd": "downloads a model",
|
||||||
|
"--hf-repo-v": "downloads a model", "-hfv": "downloads a model", "--hf-file-v": "downloads a model",
|
||||||
|
"-hffv": "downloads a model", "-hft": "is a credential", "-jf": "reads a file", "-a": "is set by the agent",
|
||||||
|
"--model-vocoder": "loads a file", "-mv": "loads a file",
|
||||||
},
|
},
|
||||||
"vllm": {
|
"vllm": {
|
||||||
"--download-dir": "writes files", "--tokenizer": "reads another repository or path",
|
"--download-dir": "writes files", "--tokenizer": "reads another repository or path",
|
||||||
@@ -118,6 +126,7 @@ var flagsNever = map[string]map[string]string{
|
|||||||
"--file-storage-path": "writes files", "--log-requests": "logs prompts",
|
"--file-storage-path": "writes files", "--log-requests": "logs prompts",
|
||||||
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
||||||
"--lora-paths": "loads files", "--download-dir": "writes files",
|
"--lora-paths": "loads files", "--download-dir": "writes files",
|
||||||
|
"--load-format": "changes how weights are loaded, some ways running code",
|
||||||
},
|
},
|
||||||
// ComfyUI's main.py, behind the gate that sets where it listens and
|
// ComfyUI's main.py, behind the gate that sets where it listens and
|
||||||
// where its files are.
|
// where its files are.
|
||||||
@@ -132,6 +141,22 @@ var flagsNever = map[string]map[string]string{
|
|||||||
"--tls-keyfile": "reads a file", "--tls-certfile": "reads a file",
|
"--tls-keyfile": "reads a file", "--tls-certfile": "reads a file",
|
||||||
"--comfy-api-base": "sends work to another service",
|
"--comfy-api-base": "sends work to another service",
|
||||||
},
|
},
|
||||||
|
// Strata's setup.py, which a profile's arguments go to: llmgw
|
||||||
|
// strata-serve sets where the model is, where its files go and where
|
||||||
|
// the server listens, and the engine is the one the agent installed.
|
||||||
|
// Its own --model, the size, is spelt --size in a profile, since
|
||||||
|
// --model is the source everywhere else.
|
||||||
|
"strata": {
|
||||||
|
"--gguf-dir": "is the model's source, set by the agent", "--models-dir": "is set by the agent",
|
||||||
|
"--data-dir": "is set by the agent", "--move-data": "moves files", "--set-default-data-dir": "writes settings",
|
||||||
|
"--feed": "downloads an engine", "--prebuilt": "loads an engine", "--build": "compiles an engine",
|
||||||
|
"--update": "downloads an engine", "--rollback-engine": "changes the engine",
|
||||||
|
"--dsv41-package": "loads a file", "--inspect": "reads another repository or path",
|
||||||
|
"--source": "downloads from elsewhere", "--setup": "is not for a server",
|
||||||
|
"--browser": "opens a browser", "--gpu": "is set by the agent", "--gpus": "is set by the agent",
|
||||||
|
"--gpu-roles": "is set by the agent", "--backend": "is set by the agent",
|
||||||
|
"--experimental-speed-projection": "changes what the model refuses",
|
||||||
|
},
|
||||||
// trtllm-serve spells its flags with underscores, so the shared list's
|
// trtllm-serve spells its flags with underscores, so the shared list's
|
||||||
// hyphenated names do not reach them.
|
// hyphenated names do not reach them.
|
||||||
"trtllm": {
|
"trtllm": {
|
||||||
@@ -163,14 +188,43 @@ var flagsNever = map[string]map[string]string{
|
|||||||
// unsafeFlag is why an engine may not be given a flag, or "".
|
// unsafeFlag is why an engine may not be given a flag, or "".
|
||||||
func unsafeFlag(engine, flag, _ string) string {
|
func unsafeFlag(engine, flag, _ string) string {
|
||||||
name, _, _ := strings.Cut(flag, "=")
|
name, _, _ := strings.Cut(flag, "=")
|
||||||
|
if !strings.HasPrefix(name, "-") {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
// Written as the engine would read it: vLLM takes --load_format for
|
||||||
|
// --load-format, TensorRT-LLM the other way round. Short flags (-m,
|
||||||
|
// -hf) are matched as written.
|
||||||
|
norm := func(f string) string {
|
||||||
|
if strings.HasPrefix(f, "--") {
|
||||||
|
return "--" + strings.ReplaceAll(f[2:], "_", "-")
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
given := norm(name)
|
||||||
|
// argparse also takes any unambiguous beginning of a long flag for the
|
||||||
|
// flag — --trust-remote is --trust-remote-code, --li is ComfyUI's
|
||||||
|
// --listen — so for the engines that parse with it, a beginning of a
|
||||||
|
// refused flag is refused. llama.cpp and TensorRT-LLM take no such
|
||||||
|
// thing, and --chat-template there is a flag of its own.
|
||||||
|
abbreviates := argparseEngines[engine] && strings.HasPrefix(given, "--") && len(given) >= 4
|
||||||
for _, set := range []string{"", engine} {
|
for _, set := range []string{"", engine} {
|
||||||
if why, ok := flagsNever[set][name]; ok && why != "" {
|
for never, why := range flagsNever[set] {
|
||||||
return why
|
if why == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
n := norm(never)
|
||||||
|
if n == given || (abbreviates && strings.HasPrefix(n, "--") && strings.HasPrefix(n, given)) {
|
||||||
|
return why
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// argparseEngines read their flags with Python's argparse, which takes a
|
||||||
|
// long flag's beginning for the flag.
|
||||||
|
var argparseEngines = map[string]bool{"vllm": true, "sglang": true, "comfyui": true, "strata": true}
|
||||||
|
|
||||||
// UnsafeArgs refuses the first of a profile's arguments an engine may not be
|
// UnsafeArgs refuses the first of a profile's arguments an engine may not be
|
||||||
// given, saying why.
|
// given, saying why.
|
||||||
func UnsafeArgs(engine string, args []string) error {
|
func UnsafeArgs(engine string, args []string) error {
|
||||||
@@ -185,3 +239,29 @@ func UnsafeArgs(engine string, args []string) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// safetensorsOnly has vLLM or SGLang read only safetensors from a model it
|
||||||
|
// downloads itself, by its Hugging Face id: left to choose, either takes
|
||||||
|
// the repository's pickle files (pytorch_model.bin) when it has no
|
||||||
|
// safetensors, and loading those runs whatever they hold. A path was
|
||||||
|
// fetched by the agent, which checked it, or put there by the operator.
|
||||||
|
func safetensorsOnly(p Profile) []string {
|
||||||
|
if p.AllowPickle || !hubID(p.ModelPath) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return []string{"--load-format", "safetensors"}
|
||||||
|
}
|
||||||
|
|
||||||
|
// hubID reports whether a model path is a Hugging Face id, org/name, rather
|
||||||
|
// than a file or directory.
|
||||||
|
func hubID(s string) bool {
|
||||||
|
if s == "" || strings.HasPrefix(s, "/") || strings.HasPrefix(s, ".") || strings.Contains(s, "://") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// org/name:Q4_K_M is a GGUF in a repository, which vLLM reads as one.
|
||||||
|
if strings.Count(s, "/") != 1 || strings.Contains(s, ":") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
_, err := os.Stat(s)
|
||||||
|
return err != nil
|
||||||
|
}
|
||||||
@@ -119,6 +119,7 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
|
|||||||
if p.ServedModelName != "" {
|
if p.ServedModelName != "" {
|
||||||
args = append(args, "--served-model-name", p.ServedModelName)
|
args = append(args, "--served-model-name", p.ServedModelName)
|
||||||
}
|
}
|
||||||
|
args = append(args, safetensorsOnly(p)...)
|
||||||
|
|
||||||
env := baseEnv(s)
|
env := baseEnv(s)
|
||||||
|
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ var (
|
|||||||
gpuProcesses = discovery.GPUProcesses
|
gpuProcesses = discovery.GPUProcesses
|
||||||
gpuMemory = discovery.GPUMemoryByProcess
|
gpuMemory = discovery.GPUMemoryByProcess
|
||||||
processGroup = pgidOf
|
processGroup = pgidOf
|
||||||
|
parentOf = ppidOf
|
||||||
// gpuCheckWait is how long a shard has, once it answers, to show up on
|
// gpuCheckWait is how long a shard has, once it answers, to show up on
|
||||||
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
|
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
|
||||||
gpuCheckWait = 15 * time.Second
|
gpuCheckWait = 15 * time.Second
|
||||||
@@ -116,19 +117,44 @@ func (a *Agent) foreignGPUMemory(ctx context.Context, shards []int) (foreign, us
|
|||||||
return foreign, used
|
return foreign, used
|
||||||
}
|
}
|
||||||
|
|
||||||
// holds reports whether any of pids is the shard's process or in its group:
|
// holds reports whether any of pids is the shard's process, in its group,
|
||||||
// an engine that computes in a child process is still the shard.
|
// or a process it started: an engine that computes in a child process is
|
||||||
|
// still the shard, and Strata's server starts its engine in a group of the
|
||||||
|
// engine's own.
|
||||||
func holds(pids []int, pid, group int) bool {
|
func holds(pids []int, pid, group int) bool {
|
||||||
for _, p := range pids {
|
for _, p := range pids {
|
||||||
if p == pid || (group > 0 && processGroup(p) == group) {
|
if p == pid || (group > 0 && processGroup(p) == group) || descends(p, pid) {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// descends reports whether process p was started, at any remove, by pid.
|
||||||
|
func descends(p, pid int) bool {
|
||||||
|
// Bounded: a chain this deep is no engine's, and a /proc read racing an
|
||||||
|
// exit must not loop.
|
||||||
|
for range 16 {
|
||||||
|
p = parentOf(p)
|
||||||
|
switch {
|
||||||
|
case p == pid:
|
||||||
|
return true
|
||||||
|
case p <= 1:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// pgidOf reads a process's group from /proc; zero when it cannot.
|
// pgidOf reads a process's group from /proc; zero when it cannot.
|
||||||
func pgidOf(pid int) int {
|
func pgidOf(pid int) int { return statField(pid, 2) }
|
||||||
|
|
||||||
|
// ppidOf reads a process's parent from /proc; zero when it cannot.
|
||||||
|
func ppidOf(pid int) int { return statField(pid, 1) }
|
||||||
|
|
||||||
|
// statField is one of the numbers after the command name in /proc's stat
|
||||||
|
// line: 1 is the parent, 2 the group.
|
||||||
|
func statField(pid, field int) int {
|
||||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0
|
return 0
|
||||||
@@ -141,10 +167,10 @@ func pgidOf(pid int) int {
|
|||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
f := strings.Fields(s[i+1:])
|
f := strings.Fields(s[i+1:])
|
||||||
if len(f) < 3 {
|
if len(f) <= field {
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
n, _ := strconv.Atoi(f[2])
|
n, _ := strconv.Atoi(f[field])
|
||||||
return n
|
return n
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -123,6 +123,30 @@ func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAnEngineStartedInAGroupOfItsOwnIsStillTheShards(t *testing.T) {
|
||||||
|
// cuda01: Strata's server (259709) starts its engine (261072) in a
|
||||||
|
// group of the engine's own, and the engine holds the card. It read as
|
||||||
|
// a shard on the CPU, and as 22 GB of somebody else's memory.
|
||||||
|
oldMem, oldGroup, oldParent := gpuMemory, processGroup, parentOf
|
||||||
|
t.Cleanup(func() { gpuMemory, processGroup, parentOf = oldMem, oldGroup, oldParent })
|
||||||
|
processGroup = func(pid int) int { return pid }
|
||||||
|
parentOf = func(pid int) int {
|
||||||
|
return map[int]int{261072: 259709, 259709: 258734, 258734: 1, 999: 1}[pid]
|
||||||
|
}
|
||||||
|
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
|
||||||
|
return []discovery.GPUMemory{{Index: 0, UsedMB: 23000,
|
||||||
|
Procs: []discovery.GPUProcess{{PID: 261072, UsedMB: 22226}, {PID: 999, UsedMB: 700}}}}, nil
|
||||||
|
}
|
||||||
|
a := &Agent{log: slog.Default()}
|
||||||
|
if c, ok := a.gpuCompute(context.Background(), []int{0}, 259709); !ok || c.Backend != "CUDA" ||
|
||||||
|
len(c.Devices) != 1 || c.Devices[0] != "GPU 0: 21.7 GB" {
|
||||||
|
t.Errorf("the engine's child on GPU 0: %+v %v", c, ok)
|
||||||
|
}
|
||||||
|
if foreign, _ := a.foreignGPUMemory(context.Background(), []int{259709}); !slices.Equal(foreign, []int32{774}) {
|
||||||
|
t.Errorf("foreign %v; want only what the other process and nobody holds, [774]", foreign)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLlamaCppsOwnWordOnTheCPUIsEnoughToRefuse(t *testing.T) {
|
func TestLlamaCppsOwnWordOnTheCPUIsEnoughToRefuse(t *testing.T) {
|
||||||
eng := fallbackEngine{}
|
eng := fallbackEngine{}
|
||||||
// Without nvidia-smi to ask, the engine's own output says it.
|
// Without nvidia-smi to ask, the engine's own output says it.
|
||||||
|
|||||||
@@ -0,0 +1,15 @@
|
|||||||
|
package agent
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMain(m *testing.M) {
|
||||||
|
// Half a gigabyte read per test binary, and contending with the tests
|
||||||
|
// for the CPUs: these never look at the figure.
|
||||||
|
measureBandwidth = func() float64 { return 0 }
|
||||||
|
bandwidthDelay = func() time.Duration { return 0 }
|
||||||
|
os.Exit(m.Run())
|
||||||
|
}
|
||||||
@@ -174,7 +174,7 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
|||||||
}
|
}
|
||||||
return st
|
return st
|
||||||
}
|
}
|
||||||
if w.Offline != nil && recipe.Repo == "" {
|
if w.Offline != nil && (recipe.Repo == "" || builds[0].Python != "") {
|
||||||
// Said, not attempted: uv would spend minutes failing to reach PyPI,
|
// Said, not attempted: uv would spend minutes failing to reach PyPI,
|
||||||
// and the failure would read like a network fault rather than the
|
// and the failure would read like a network fault rather than the
|
||||||
// design of an offline site.
|
// design of an offline site.
|
||||||
|
|||||||
@@ -109,6 +109,11 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
|||||||
}
|
}
|
||||||
reqs = append(reqs, more...)
|
reqs = append(reqs, more...)
|
||||||
}
|
}
|
||||||
|
if len(b.Assets) > 0 {
|
||||||
|
if err := in.unpackAssets(ctx, r, w, b, tmp, st); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
if b.Wheel != nil {
|
if b.Wheel != nil {
|
||||||
// The engine from its GitHub release, pinned to the digest GitHub
|
// The engine from its GitHub release, pinned to the digest GitHub
|
||||||
// publishes for the file: the same trust as an archive, and uv
|
// publishes for the file: the same trust as an archive, and uv
|
||||||
@@ -372,6 +377,48 @@ func (in *Installer) unpackSource(ctx context.Context, w Want, b Build, env stri
|
|||||||
return requirementsIn(string(f), b.IndexPackages)
|
return requirementsIn(string(f), b.IndexPackages)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// unpackAssets fetches a Python engine's release binaries, each checked
|
||||||
|
// against the digest GitHub publishes for it, and unpacks them into the
|
||||||
|
// environment's AssetDir.
|
||||||
|
func (in *Installer) unpackAssets(ctx context.Context, r Recipe, w Want, b Build, env string, st *Status) error {
|
||||||
|
tag := b.AssetTag
|
||||||
|
if tag == "" {
|
||||||
|
tag = w.Version
|
||||||
|
}
|
||||||
|
in.progress(st, "asking GitHub for the checksums of %s %s", r.Repo, tag)
|
||||||
|
sums, err := in.checksums(ctx, r.Repo, tag)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
dst := filepath.Join(env, b.AssetDir)
|
||||||
|
if err := os.MkdirAll(dst, 0o755); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, asset := range b.Assets {
|
||||||
|
want, ok := sums[asset]
|
||||||
|
if !ok {
|
||||||
|
// Refused rather than installed unchecked: this is a binary that
|
||||||
|
// will run on every node.
|
||||||
|
return fmt.Errorf("%s %s publishes no checksum for %s, so it cannot be verified", r.Repo, tag, asset)
|
||||||
|
}
|
||||||
|
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + r.Repo + "/releases/download/" + tag + "/" + asset
|
||||||
|
archive := filepath.Join(env+".source", asset)
|
||||||
|
if err := os.MkdirAll(filepath.Dir(archive), 0o755); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := in.download(ctx, url, asset, archive, want, st); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
in.progress(st, "unpacking %s", asset)
|
||||||
|
err := untarGz(archive, dst)
|
||||||
|
_ = os.RemoveAll(filepath.Dir(archive))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("unpacking %s: %w", asset, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// requirementsIn reads a requirements file's lines. Options (-r, -e, --index-url
|
// requirementsIn reads a requirements file's lines. Options (-r, -e, --index-url
|
||||||
// and the like) are refused: they would fetch from somewhere other than PyPI,
|
// and the like) are refused: they would fetch from somewhere other than PyPI,
|
||||||
// or install what the lock never saw.
|
// or install what the lock never saw.
|
||||||
|
|||||||
@@ -76,6 +76,11 @@ type Build struct {
|
|||||||
// was published, which PyPI cannot be asked for.
|
// was published, which PyPI cannot be asked for.
|
||||||
Source *Source
|
Source *Source
|
||||||
AsOf time.Time
|
AsOf time.Time
|
||||||
|
// AssetTag and AssetDir, for a Python engine that also ships a binary
|
||||||
|
// in its releases, are the release tag Assets are taken from, checked
|
||||||
|
// against the digests GitHub publishes, and the directory in the
|
||||||
|
// environment they are unpacked into.
|
||||||
|
AssetTag, AssetDir string
|
||||||
// FreeGB is the disk space the installation needs before it starts: a
|
// FreeGB is the disk space the installation needs before it starts: a
|
||||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||||
// leaves the node worse off than one that never began.
|
// leaves the node worse off than one that never began.
|
||||||
@@ -125,6 +130,7 @@ var Recipes = map[string]Recipe{
|
|||||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||||
"comfyui": {Candidates: comfyuiCandidates},
|
"comfyui": {Candidates: comfyuiCandidates},
|
||||||
|
"strata": {Repo: StrataRepo, Candidates: strataCandidates},
|
||||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -172,6 +178,9 @@ func Builds(engine, version, arch string) ([]Build, error) {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
for _, b := range bs {
|
for _, b := range bs {
|
||||||
|
if b.Python != "" {
|
||||||
|
return nil, fmt.Errorf("%s is a Python environment from PyPI, which an offline bundle does not carry", engine)
|
||||||
|
}
|
||||||
if !seen[b.Variant] {
|
if !seen[b.Variant] {
|
||||||
seen[b.Variant] = true
|
seen[b.Variant] = true
|
||||||
out = append(out, b)
|
out = append(out, b)
|
||||||
@@ -672,3 +681,80 @@ print("comfyui version", "` + version + `", "on torch", torch.__version__)`},
|
|||||||
{cu126, gpuNeeds{minDriverForCUDA12, 5.0}, "ComfyUI " + version + " on CUDA 12.6's torch"},
|
{cu126, gpuNeeds{minDriverForCUDA12, 5.0}, "ComfyUI " + version + " on CUDA 12.6's torch"},
|
||||||
}, h)
|
}, h)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- Strata ---
|
||||||
|
|
||||||
|
// StrataRepo is where Strata is published: its source, and in each release
|
||||||
|
// the prebuilt engine for each CUDA.
|
||||||
|
const StrataRepo = "Niko1221/Strata"
|
||||||
|
|
||||||
|
// strataReleases are the Strata versions this agent installs: the source
|
||||||
|
// archive's digest, taken when it was added here and confirmed from a second
|
||||||
|
// machine, when the version was published, and the engine its release
|
||||||
|
// carries, whose files GitHub publishes digests for.
|
||||||
|
var strataReleases = map[string]struct {
|
||||||
|
sha256 string
|
||||||
|
published string
|
||||||
|
engine string
|
||||||
|
}{
|
||||||
|
"0.2.7": {"41ae861478f5e5d9506864f97dfb5dd7cfe6b83193300e2b0d4f04c0380e3af9", "2026-10-27T14:54:42Z", "0.1.52"},
|
||||||
|
}
|
||||||
|
|
||||||
|
// strataCandidates are the Strata builds for a machine: its source, in a
|
||||||
|
// Python environment, with the prebuilt engine for the node's driver and
|
||||||
|
// cards — built for CUDA 13, Turing to Blackwell; or for CUDA 12.9, Pascal
|
||||||
|
// to Ada, which is what a V100, or a driver older than 580, gets.
|
||||||
|
func strataCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||||
|
if _, err := gpuArch("Strata", h); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if h.Arch != "amd64" {
|
||||||
|
return nil, NotHere{"Strata publishes its Linux CUDA engine for x86-64 only"}
|
||||||
|
}
|
||||||
|
rel, ok := strataReleases[version]
|
||||||
|
if !ok {
|
||||||
|
known := make([]string, 0, len(strataReleases))
|
||||||
|
for v := range strataReleases {
|
||||||
|
known = append(known, v)
|
||||||
|
}
|
||||||
|
slices.Sort(known)
|
||||||
|
return nil, fmt.Errorf("this agent knows the source digest of Strata %s, not of %s; "+
|
||||||
|
"install that one by hand and name it in agent.yaml (engines.strata.dir)",
|
||||||
|
strings.Join(known, ", "), version)
|
||||||
|
}
|
||||||
|
published, err := time.Parse(time.RFC3339, rel.published)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
build := func(cuda string) Build {
|
||||||
|
return Build{
|
||||||
|
Variant: cuda,
|
||||||
|
Python: pythonForEngines,
|
||||||
|
Source: &Source{
|
||||||
|
URL: "/" + StrataRepo + "/archive/refs/tags/v" + version + ".tar.gz",
|
||||||
|
SHA256: rel.sha256, Top: "Strata-" + version, Dir: "strata",
|
||||||
|
Requirements: "requirements.txt",
|
||||||
|
},
|
||||||
|
AsOf: published.Add(24 * time.Hour),
|
||||||
|
Assets: []string{"strata-engine-" + rel.engine + "-linux-x86_64-cuda-" + cuda + ".tar.gz"},
|
||||||
|
AssetTag: "v" + version,
|
||||||
|
AssetDir: "strata-engine",
|
||||||
|
// Strata is started as a script, so what the agent is given is
|
||||||
|
// the environment's interpreter; the proof is importing what its
|
||||||
|
// server needs, and finding its engine beside it.
|
||||||
|
Binary: "python",
|
||||||
|
VersionArgs: []string{"-c", `import os, sys, numpy, jinja2, regex
|
||||||
|
e = os.path.dirname(os.path.dirname(os.path.abspath(sys.executable)))
|
||||||
|
for f in ("strata/setup.py", "strata/serve/server.py", "strata-engine/strata"):
|
||||||
|
assert os.path.isfile(os.path.join(e, f)), f + " is missing"
|
||||||
|
print("strata version", "` + version + `", "engine", "` + rel.engine + `")`},
|
||||||
|
FreeGB: 5,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return pick("Strata "+version, variant, []candidate{
|
||||||
|
{build("cuda13"), gpuNeeds{minDriverForCUDA13, 7.5}, "Strata " + version + " built for CUDA 13"},
|
||||||
|
// 12.9 rather than 12: its machine code is for 6.0 to 8.9 and its
|
||||||
|
// PTX was compiled by 12.9, so the driver must be 575.
|
||||||
|
{build("cuda12"), gpuNeeds{575, 6.0}, "Strata " + version + " built for CUDA 12.9"},
|
||||||
|
}, h)
|
||||||
|
}
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
package provision
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// strataRelease is Strata's source archive as GitHub makes one, and its
|
||||||
|
// release's engine for CUDA 13, with the recipe pointed at the archive.
|
||||||
|
func strataRelease(t *testing.T, rel *release) {
|
||||||
|
t.Helper()
|
||||||
|
archive := tarGz(t, map[string]string{
|
||||||
|
"Strata-0.2.7/setup.py": "print('setup')\n",
|
||||||
|
"Strata-0.2.7/serve/server.py": "print('server')\n",
|
||||||
|
"Strata-0.2.7/requirements.txt": "numpy==2.5.3; python_version >= \"3.12\"\njinja2==3.1.6\n",
|
||||||
|
}, nil)
|
||||||
|
rel.assets["v0.2.7.tar.gz"] = archive
|
||||||
|
sum := sha256.Sum256(archive)
|
||||||
|
engine := tarGz(t, map[string]string{"strata": "\x7fELF", "libcudart.so.13": "lib"}, nil)
|
||||||
|
name := "strata-engine-0.1.52-linux-x86_64-cuda-cuda13.tar.gz"
|
||||||
|
rel.assets[name] = engine
|
||||||
|
esum := sha256.Sum256(engine)
|
||||||
|
rel.digests[name] = "sha256:" + hex.EncodeToString(esum[:])
|
||||||
|
|
||||||
|
old := strataReleases
|
||||||
|
strataReleases = map[string]struct {
|
||||||
|
sha256 string
|
||||||
|
published string
|
||||||
|
engine string
|
||||||
|
}{"0.2.7": {hex.EncodeToString(sum[:]), "2026-10-27T14:54:42Z", "0.1.52"}}
|
||||||
|
t.Cleanup(func() { strataReleases = old })
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStrataIsInstalledWithItsEngineBesideItsCode(t *testing.T) {
|
||||||
|
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||||
|
in, rel := pythonInstaller(t, turingNode, sys)
|
||||||
|
strataRelease(t, rel)
|
||||||
|
|
||||||
|
got := in.Converge(t.Context(), []Want{{Engine: "strata", Version: "0.2.7"}})
|
||||||
|
if got[0].State != Ready {
|
||||||
|
t.Fatalf("not ready: %+v", got[0])
|
||||||
|
}
|
||||||
|
dir := filepath.Join(in.Root, "strata", "0.2.7-cuda13")
|
||||||
|
if got[0].Binary != filepath.Join(dir, "bin", "python") {
|
||||||
|
t.Errorf("binary %s, want the environment's interpreter", got[0].Binary)
|
||||||
|
}
|
||||||
|
for _, f := range []string{"strata/setup.py", "strata/serve/server.py", "strata-engine/strata", "strata-engine/libcudart.so.13"} {
|
||||||
|
if _, err := os.Stat(filepath.Join(dir, f)); err != nil {
|
||||||
|
t.Errorf("%s is not in the environment: %v", f, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
all := strings.Join(sys.commands, "\n")
|
||||||
|
for _, want := range []string{"--exclude-newer 2026-10-28T14:54:42Z", "jinja2==3.1.6"} {
|
||||||
|
if !strings.Contains(all, want) {
|
||||||
|
t.Errorf("the installation never used %q:\n%s", want, all)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStrataRefusesAnEngineWhoseDigestDoesNotMatch(t *testing.T) {
|
||||||
|
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||||
|
in, rel := pythonInstaller(t, turingNode, sys)
|
||||||
|
strataRelease(t, rel)
|
||||||
|
rel.digests["strata-engine-0.1.52-linux-x86_64-cuda-cuda13.tar.gz"] = "sha256:" + strings.Repeat("0", 64)
|
||||||
|
|
||||||
|
got := in.Converge(t.Context(), []Want{{Engine: "strata", Version: "0.2.7", Variant: "cuda13"}})
|
||||||
|
if got[0].State != Failed || !strings.Contains(got[0].Detail, "does not match the checksum") {
|
||||||
|
t.Fatalf("an engine that does not match its digest was not refused: %+v", got[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStrataPicksTheEngineForTheCardsAndDriver(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
host discovery.Host
|
||||||
|
want []string
|
||||||
|
err string
|
||||||
|
}{
|
||||||
|
{"turing on 580", turingNode, []string{"cuda13", "cuda12"}, ""},
|
||||||
|
{"volta", discovery.Host{Arch: "amd64", NVIDIADriver: "580.82.07", MinComputeCapability: 7.0}, []string{"cuda12"}, ""},
|
||||||
|
{"driver 575", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 8.6}, []string{"cuda12"}, ""},
|
||||||
|
{"driver 570", discovery.Host{Arch: "amd64", NVIDIADriver: "570.86.10", MinComputeCapability: 8.6}, nil, "575"},
|
||||||
|
{"no gpu", discovery.Host{Arch: "amd64"}, nil, "no"},
|
||||||
|
{"arm", discovery.Host{Arch: "arm64", NVIDIADriver: "580.82.07", MinComputeCapability: 9.0}, nil, "x86-64"},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
bs, err := strataCandidates("0.2.7", "", tc.host)
|
||||||
|
if tc.err != "" {
|
||||||
|
if err == nil || !strings.Contains(err.Error(), tc.err) {
|
||||||
|
t.Fatalf("got %v, %v; want an error saying %q", bs, err, tc.err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var got []string
|
||||||
|
for _, b := range bs {
|
||||||
|
got = append(got, b.Variant)
|
||||||
|
}
|
||||||
|
if strings.Join(got, ",") != strings.Join(tc.want, ",") {
|
||||||
|
t.Errorf("builds %v, want %v", got, tc.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStrataIsNotInAnOfflineBundle(t *testing.T) {
|
||||||
|
if _, err := Builds("strata", "0.2.7", "amd64"); err == nil {
|
||||||
|
t.Fatal("an offline bundle would carry Strata, which needs PyPI")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -90,14 +90,20 @@ var ErrNotLocal = errors.New("the weights are not on this node and the source sa
|
|||||||
// already cached, downloaded, or downloaded by another shard this one waited
|
// already cached, downloaded, or downloaded by another shard this one waited
|
||||||
// for — since that is most of the difference between a replica that starts
|
// for — since that is most of the difference between a replica that starts
|
||||||
// in a minute and one that takes an hour.
|
// in a minute and one that takes an hour.
|
||||||
func (f *Fetcher) Ensure(ctx context.Context, source string) (path string, err error) {
|
//
|
||||||
|
// A source in a pickle format, or a set with such a file in it, is refused
|
||||||
|
// before anything is fetched unless allowPickle; see Pickled.
|
||||||
|
func (f *Fetcher) Ensure(ctx context.Context, source string, allowPickle bool) (path string, err error) {
|
||||||
ctx, span := obs.Tracer(tracerName).Start(ctx, "fetch weights",
|
ctx, span := obs.Tracer(tracerName).Start(ctx, "fetch weights",
|
||||||
trace.WithAttributes(attribute.String("llmgw.weights.source", redact(source))))
|
trace.WithAttributes(attribute.String("llmgw.weights.source", redact(source))))
|
||||||
defer func() {
|
defer func() {
|
||||||
span.SetAttributes(attribute.String("llmgw.weights.path", path))
|
span.SetAttributes(attribute.String("llmgw.weights.path", path))
|
||||||
obs.End(span, err)
|
obs.End(span, err)
|
||||||
}()
|
}()
|
||||||
return f.ensure(ctx, source)
|
if !allowPickle && !IsSet(source) && Pickled(source) {
|
||||||
|
return "", refusePickle(source)
|
||||||
|
}
|
||||||
|
return f.ensure(ctx, source, allowPickle)
|
||||||
}
|
}
|
||||||
|
|
||||||
// tracerName names the spans this package opens.
|
// tracerName names the spans this package opens.
|
||||||
@@ -119,7 +125,7 @@ func redact(source string) string {
|
|||||||
return u.String()
|
return u.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *Fetcher) ensure(ctx context.Context, source string) (string, error) {
|
func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (string, error) {
|
||||||
if source == "" {
|
if source == "" {
|
||||||
return "", fmt.Errorf("no source given")
|
return "", fmt.Errorf("no source given")
|
||||||
}
|
}
|
||||||
@@ -127,7 +133,7 @@ func (f *Fetcher) ensure(ctx context.Context, source string) (string, error) {
|
|||||||
// file, and a set has to be handled before any of it.
|
// file, and a set has to be handled before any of it.
|
||||||
if IsSet(source) {
|
if IsSet(source) {
|
||||||
outcome(ctx, "set")
|
outcome(ctx, "set")
|
||||||
return f.ensureSet(ctx, source)
|
return f.ensureSet(ctx, source, allowPickle)
|
||||||
}
|
}
|
||||||
if !isRemote(source) {
|
if !isRemote(source) {
|
||||||
if !isPath(source) {
|
if !isPath(source) {
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ func TestALocalPathIsUsedNeverCopied(t *testing.T) {
|
|||||||
}
|
}
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
|
|
||||||
got, err := f.Ensure(t.Context(), path)
|
got, err := f.Ensure(t.Context(), path, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -53,7 +53,7 @@ func TestAMissingLocalPathSaysWhatIsWrong(t *testing.T) {
|
|||||||
// The commonest setup mistake, and "no such file" from an engine twenty
|
// The commonest setup mistake, and "no such file" from an engine twenty
|
||||||
// minutes into a load is a poor way to learn it.
|
// minutes into a load is a poor way to learn it.
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
_, err := f.Ensure(t.Context(), filepath.Join(t.TempDir(), "absent.gguf"))
|
_, err := f.Ensure(t.Context(), filepath.Join(t.TempDir(), "absent.gguf"), false)
|
||||||
if !errors.Is(err, ErrNotLocal) {
|
if !errors.Is(err, ErrNotLocal) {
|
||||||
t.Fatalf("err = %v, want ErrNotLocal", err)
|
t.Fatalf("err = %v, want ErrNotLocal", err)
|
||||||
}
|
}
|
||||||
@@ -67,7 +67,7 @@ func TestARemoteFileIsFetchedOnceAndReused(t *testing.T) {
|
|||||||
srv := serveBytes(t, []byte("the weights"), &hits)
|
srv := serveBytes(t, []byte("the weights"), &hits)
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
|
|
||||||
first, err := f.Ensure(t.Context(), srv.URL+"/model.gguf")
|
first, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -76,7 +76,7 @@ func TestARemoteFileIsFetchedOnceAndReused(t *testing.T) {
|
|||||||
t.Fatalf("fetched content = %q, err %v", body, err)
|
t.Fatalf("fetched content = %q, err %v", body, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
second, err := f.Ensure(t.Context(), srv.URL+"/model.gguf")
|
second, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -103,7 +103,7 @@ func TestConcurrentRequestsProduceOneDownload(t *testing.T) {
|
|||||||
wg.Add(1)
|
wg.Add(1)
|
||||||
go func() {
|
go func() {
|
||||||
defer wg.Done()
|
defer wg.Done()
|
||||||
paths[i], errs[i] = f.Ensure(t.Context(), srv.URL+"/model.gguf")
|
paths[i], errs[i] = f.Ensure(t.Context(), srv.URL+"/model.gguf", false)
|
||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
wg.Wait()
|
wg.Wait()
|
||||||
@@ -139,7 +139,7 @@ func TestAPartialDownloadIsNeverVisibleAsTheModel(t *testing.T) {
|
|||||||
|
|
||||||
cache := t.TempDir()
|
cache := t.TempDir()
|
||||||
f := &Fetcher{CacheDir: cache}
|
f := &Fetcher{CacheDir: cache}
|
||||||
if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf"); err == nil {
|
if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false); err == nil {
|
||||||
t.Fatal("a truncated download was reported as success")
|
t.Fatal("a truncated download was reported as success")
|
||||||
}
|
}
|
||||||
entries, _ := os.ReadDir(cache)
|
entries, _ := os.ReadDir(cache)
|
||||||
@@ -169,7 +169,7 @@ func TestAnInterruptedDownloadResumes(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
got, err := f.Ensure(t.Context(), srv.URL+"/model.gguf")
|
got, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -232,7 +232,7 @@ func TestProgressIsReportedWithAPercentage(t *testing.T) {
|
|||||||
CacheDir: t.TempDir(),
|
CacheDir: t.TempDir(),
|
||||||
OnProgress: func(p Progress) { seen = append(seen, p) },
|
OnProgress: func(p Progress) { seen = append(seen, p) },
|
||||||
}
|
}
|
||||||
if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf"); err != nil {
|
if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if len(seen) == 0 {
|
if len(seen) == 0 {
|
||||||
@@ -254,7 +254,7 @@ func TestPercentIsUnknownRatherThanZeroWithoutATotal(t *testing.T) {
|
|||||||
|
|
||||||
func TestNoCacheDirectoryIsAClearRefusal(t *testing.T) {
|
func TestNoCacheDirectoryIsAClearRefusal(t *testing.T) {
|
||||||
f := &Fetcher{}
|
f := &Fetcher{}
|
||||||
_, err := f.Ensure(t.Context(), "https://example.invalid/model.gguf")
|
_, err := f.Ensure(t.Context(), "https://example.invalid/model.gguf", false)
|
||||||
if err == nil || !strings.Contains(err.Error(), "cache directory") {
|
if err == nil || !strings.Contains(err.Error(), "cache directory") {
|
||||||
t.Errorf("err = %v, want a complaint about the cache directory", err)
|
t.Errorf("err = %v, want a complaint about the cache directory", err)
|
||||||
}
|
}
|
||||||
@@ -269,7 +269,7 @@ func TestABareModelIdIsLeftToTheEngine(t *testing.T) {
|
|||||||
"Qwen/Qwen2.5-7B-Instruct",
|
"Qwen/Qwen2.5-7B-Instruct",
|
||||||
"meta-llama/Llama-3.3-70B-Instruct",
|
"meta-llama/Llama-3.3-70B-Instruct",
|
||||||
} {
|
} {
|
||||||
got, err := f.Ensure(t.Context(), id)
|
got, err := f.Ensure(t.Context(), id, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Errorf("%s was refused: %v", id, err)
|
t.Errorf("%s was refused: %v", id, err)
|
||||||
}
|
}
|
||||||
@@ -283,7 +283,7 @@ func TestAnAbsolutePathThatIsMissingIsStillAnError(t *testing.T) {
|
|||||||
// The other half: an operator writing a path means a file, and a missing
|
// The other half: an operator writing a path means a file, and a missing
|
||||||
// one should be said plainly rather than handed to an engine.
|
// one should be said plainly rather than handed to an engine.
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
if _, err := f.Ensure(t.Context(), "/models/definitely-not-here.gguf"); !errors.Is(err, ErrNotLocal) {
|
if _, err := f.Ensure(t.Context(), "/models/definitely-not-here.gguf", false); !errors.Is(err, ErrNotLocal) {
|
||||||
t.Errorf("err = %v, want ErrNotLocal", err)
|
t.Errorf("err = %v, want ErrNotLocal", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
package weights
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"path"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Weights in Python's pickle format are a program as much as a file: loading
|
||||||
|
// one runs whatever it says, as the engine's user, on the node. That is how
|
||||||
|
// the malicious models found on Hugging Face work, and nothing in the file's
|
||||||
|
// name or size says so. safetensors and GGUF are only tensors and are read,
|
||||||
|
// never run, so those are what a node fetches unless the catalog says
|
||||||
|
// otherwise for a model, with allow_pickle, on purpose.
|
||||||
|
|
||||||
|
// pickleExts are the extensions of PyTorch's pickle formats, and of pickle
|
||||||
|
// itself. A .bin beside safetensors is the same model in pickle form.
|
||||||
|
var pickleExts = []string{".bin", ".pt", ".pt2", ".pth", ".ckpt", ".pkl", ".pickle"}
|
||||||
|
|
||||||
|
// Pickled reports whether a file name or URL is in a pickle format.
|
||||||
|
func Pickled(name string) bool {
|
||||||
|
// A query or a fragment is not the file's name, on a URL or an hf:
|
||||||
|
// source alike: model.bin?download=true is still model.bin.
|
||||||
|
if i := strings.IndexAny(name, "?#"); i >= 0 {
|
||||||
|
name = name[:i]
|
||||||
|
}
|
||||||
|
ext := strings.ToLower(path.Ext(name))
|
||||||
|
for _, e := range pickleExts {
|
||||||
|
if ext == e {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// ErrPickle is a source refused for being in a pickle format.
|
||||||
|
var ErrPickle = errors.New("weights in a pickle format run code when they are loaded")
|
||||||
|
|
||||||
|
func refusePickle(name string) error {
|
||||||
|
return fmt.Errorf("%w: %s is refused; use the model's safetensors or GGUF, or set allow_pickle: true "+
|
||||||
|
"on the model if it was checked and has no other form", ErrPickle, redact(name))
|
||||||
|
}
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
package weights
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPickleFormatsAreKnownByTheirNames(t *testing.T) {
|
||||||
|
for name, want := range map[string]bool{
|
||||||
|
"pytorch_model.bin": true, "model.ckpt": true, "x.PT": true, "w.pth": true, "a.pkl": true,
|
||||||
|
"https://h/r/resolve/main/model.ckpt?download=true": true,
|
||||||
|
"hf:o/r/model.bin?download=true": true, "hf:o/r/model.ckpt#x": true, "m.pt2": true,
|
||||||
|
"model.safetensors": false, "m-Q4_K_M.gguf": false, "Qwen/Qwen2.5-7B-Instruct": false,
|
||||||
|
"config.json": false, "": false,
|
||||||
|
} {
|
||||||
|
if got := Pickled(name); got != want {
|
||||||
|
t.Errorf("Pickled(%q) = %v, want %v", name, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAPickleFileIsRefusedBeforeItIsFetched(t *testing.T) {
|
||||||
|
var hits atomic.Int64
|
||||||
|
srv := serveBytes(t, []byte("not really a pickle"), &hits)
|
||||||
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
|
|
||||||
|
_, err := f.Ensure(t.Context(), srv.URL+"/model.ckpt", false)
|
||||||
|
if !errors.Is(err, ErrPickle) {
|
||||||
|
t.Fatalf("a .ckpt was not refused: %v", err)
|
||||||
|
}
|
||||||
|
if hits.Load() != 0 {
|
||||||
|
t.Errorf("the refused file was fetched %d times", hits.Load())
|
||||||
|
}
|
||||||
|
if _, err := f.Ensure(t.Context(), srv.URL+"/model.ckpt", true); err != nil {
|
||||||
|
t.Errorf("allowed, the .ckpt was still refused: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestASetWithAPickleFileIsRefusedWhole(t *testing.T) {
|
||||||
|
f, srv, hits := mirrorOf(t, map[string]string{
|
||||||
|
"pytorch_model.bin": "pickle",
|
||||||
|
"config.json": "{}",
|
||||||
|
})
|
||||||
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); !errors.Is(err, ErrPickle) {
|
||||||
|
t.Fatalf("a set holding pytorch_model.bin was not refused: %v", err)
|
||||||
|
}
|
||||||
|
if *hits != 0 {
|
||||||
|
t.Errorf("%d files of the refused set were fetched", *hits)
|
||||||
|
}
|
||||||
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", true); err != nil {
|
||||||
|
t.Errorf("allowed, the set was still refused: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestASetAlreadyHereIsLookedAtAgain(t *testing.T) {
|
||||||
|
// Fetched once for a model that allowed pickles, or before they were
|
||||||
|
// refused: another asking for it without leave is refused all the same.
|
||||||
|
f, srv, _ := mirrorOf(t, map[string]string{"pytorch_model.bin": "pickle", "config.json": "{}"})
|
||||||
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", true); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); !errors.Is(err, ErrPickle) {
|
||||||
|
t.Errorf("a set already on the node was not looked at: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -46,7 +46,7 @@ type manifest struct {
|
|||||||
// The whole set or nothing. A directory with three of four shards in it loads
|
// The whole set or nothing. A directory with three of four shards in it loads
|
||||||
// as a broken model rather than failing to load, so a partial set is left
|
// as a broken model rather than failing to load, so a partial set is left
|
||||||
// unfinished under its temporary name and the next attempt starts again.
|
// unfinished under its temporary name and the next attempt starts again.
|
||||||
func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error) {
|
func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
|
||||||
dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
|
dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
@@ -56,6 +56,16 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error)
|
|||||||
// exactly the failure the marker exists to prevent.
|
// exactly the failure the marker exists to prevent.
|
||||||
done := filepath.Join(dir, ".complete")
|
done := filepath.Join(dir, ".complete")
|
||||||
if _, err := os.Stat(done); err == nil {
|
if _, err := os.Stat(done); err == nil {
|
||||||
|
// Already here, maybe fetched for a model that allowed pickles, or
|
||||||
|
// before they were refused: looked at again, not trusted.
|
||||||
|
if !allowPickle {
|
||||||
|
entries, _ := os.ReadDir(dir)
|
||||||
|
for _, e := range entries {
|
||||||
|
if Pickled(e.Name()) {
|
||||||
|
return "", refusePickle(source + e.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
return dir, nil
|
return dir, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -66,6 +76,15 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error)
|
|||||||
if len(m.Files) == 0 {
|
if len(m.Files) == 0 {
|
||||||
return "", fmt.Errorf("%s holds no files", source)
|
return "", fmt.Errorf("%s holds no files", source)
|
||||||
}
|
}
|
||||||
|
// Refused whole rather than skipped: an engine given the directory
|
||||||
|
// loads the pickle when there is no safetensors beside it.
|
||||||
|
if !allowPickle {
|
||||||
|
for _, file := range m.Files {
|
||||||
|
if Pickled(file.Name) {
|
||||||
|
return "", refusePickle(source + file.Name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
|
|||||||
"config.json": "{}",
|
"config.json": "{}",
|
||||||
})
|
})
|
||||||
|
|
||||||
got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
got, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -74,11 +74,11 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
|
|||||||
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
|
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
|
||||||
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
|
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
|
||||||
|
|
||||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
after := *hits
|
after := *hits
|
||||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if *hits != after {
|
if *hits != after {
|
||||||
@@ -100,7 +100,7 @@ func TestAHalfFetchedSetIsNotTreatedAsFinished(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
got, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
got, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -117,7 +117,7 @@ func TestAFileAlreadyThereAtTheRightSizeIsNotRefetched(t *testing.T) {
|
|||||||
_ = os.MkdirAll(dir, 0o755)
|
_ = os.MkdirAll(dir, 0o755)
|
||||||
_ = os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644)
|
_ = os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644)
|
||||||
|
|
||||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// Resuming a set means not paying for the shards that already landed.
|
// Resuming a set means not paying for the shards that already landed.
|
||||||
@@ -133,7 +133,7 @@ func TestAFilenameThatEscapesTheDirectoryIsRefused(t *testing.T) {
|
|||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
|
|
||||||
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||||
// The mirror is a place an operator copies files into by hand, so what it
|
// The mirror is a place an operator copies files into by hand, so what it
|
||||||
// reports is not automatically safe to join onto a path.
|
// reports is not automatically safe to join onto a path.
|
||||||
if err == nil || !strings.Contains(err.Error(), "offered a file named") {
|
if err == nil || !strings.Contains(err.Error(), "offered a file named") {
|
||||||
@@ -148,7 +148,7 @@ func TestADirectoryListingIsNotAManifest(t *testing.T) {
|
|||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
|
|
||||||
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||||
// "invalid character '<'" sends somebody looking at the wrong thing.
|
// "invalid character '<'" sends somebody looking at the wrong thing.
|
||||||
if err == nil || !strings.Contains(err.Error(), "has to be a mirror version") {
|
if err == nil || !strings.Contains(err.Error(), "has to be a mirror version") {
|
||||||
t.Errorf("the reason should name the cause, got %v", err)
|
t.Errorf("the reason should name the cause, got %v", err)
|
||||||
@@ -182,12 +182,12 @@ func TestASetFileThatDoesNotMatchTheMirrorsChecksumIsRefused(t *testing.T) {
|
|||||||
}))
|
}))
|
||||||
t.Cleanup(srv.Close)
|
t.Cleanup(srv.Close)
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err == nil || !strings.Contains(err.Error(), "sha256") {
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err == nil || !strings.Contains(err.Error(), "sha256") {
|
||||||
t.Fatalf("a damaged shard was accepted: %v", err)
|
t.Fatalf("a damaged shard was accepted: %v", err)
|
||||||
}
|
}
|
||||||
served = good
|
served = good
|
||||||
// The size differs, so it is fetched again; this time it matches.
|
// The size differs, so it is fetched again; this time it matches.
|
||||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
|
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil {
|
||||||
t.Fatalf("the shard as mirrored: %v", err)
|
t.Fatalf("the shard as mirrored: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -198,7 +198,7 @@ func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) {
|
|||||||
}))
|
}))
|
||||||
t.Cleanup(srv.Close)
|
t.Cleanup(srv.Close)
|
||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||||
if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") ||
|
if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") ||
|
||||||
!strings.Contains(err.Error(), "model-00002-of-00002") {
|
!strings.Contains(err.Error(), "model-00002-of-00002") {
|
||||||
t.Fatalf("got %v", err)
|
t.Fatalf("got %v", err)
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ func TestAFetchSaysInItsTraceWhetherItDownloadedAndHowMuch(t *testing.T) {
|
|||||||
f := &Fetcher{CacheDir: t.TempDir()}
|
f := &Fetcher{CacheDir: t.TempDir()}
|
||||||
src := strings.Replace(srv.URL, "http://", "http://user:secret@", 1) + "/m/model.gguf?token=hunter2"
|
src := strings.Replace(srv.URL, "http://", "http://user:secret@", 1) + "/m/model.gguf?token=hunter2"
|
||||||
|
|
||||||
if _, err := f.Ensure(t.Context(), src); err != nil {
|
if _, err := f.Ensure(t.Context(), src, false); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
fetch := rec.One(t, [16]byte{}, "fetch weights")
|
fetch := rec.One(t, [16]byte{}, "fetch weights")
|
||||||
@@ -30,7 +30,7 @@ func TestAFetchSaysInItsTraceWhetherItDownloadedAndHowMuch(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The second start on this node finds it here.
|
// The second start on this node finds it here.
|
||||||
if _, err := f.Ensure(t.Context(), src); err != nil {
|
if _, err := f.Ensure(t.Context(), src, false); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
again := rec.Named("fetch weights")
|
again := rec.Named("fetch weights")
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
func newAnswerCacheCmd(open opener) *cobra.Command {
|
func newAnswerCacheCmd(open opener) *cobra.Command {
|
||||||
@@ -12,6 +14,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
|||||||
project string
|
project string
|
||||||
ttl time.Duration
|
ttl time.Duration
|
||||||
anyAnswer bool
|
anyAnswer bool
|
||||||
|
similar float64
|
||||||
off bool
|
off bool
|
||||||
)
|
)
|
||||||
cmd := &cobra.Command{
|
cmd := &cobra.Command{
|
||||||
@@ -28,15 +31,25 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
|||||||
"a request continuing a stored conversation or under a guardrails no_store\n" +
|
"a request continuing a stored conversation or under a guardrails no_store\n" +
|
||||||
"policy. The guardrails judge a reused answer as they judge any other.\n\n" +
|
"policy. The guardrails judge a reused answer as they judge any other.\n\n" +
|
||||||
"Off for every project until this is run. Without --ttl or --off, says how\n" +
|
"Off for every project until this is run. Without --ttl or --off, says how\n" +
|
||||||
"it stands. The answers are held in memory, router.cache bounds how much.",
|
"it stands. The answers are held in memory, router.cache bounds how much.\n\n" +
|
||||||
|
"--similar also answers a chat question that is like one already answered:\n" +
|
||||||
|
"its last user message embedded by router.cache.embedding_model, and given\n" +
|
||||||
|
"the answer whose question reaches that cosine similarity (0.95 is strict,\n" +
|
||||||
|
"0.9 looser), as x-llmgw-cache: similar. The system prompt, the earlier\n" +
|
||||||
|
"turns, tools and the sampling settings must still be the same.",
|
||||||
Example: " llmgw keys cache --project Acme/ml/train --ttl 1h\n" +
|
Example: " llmgw keys cache --project Acme/ml/train --ttl 1h\n" +
|
||||||
" llmgw keys cache --project Acme/ml/eval --ttl 24h --any\n" +
|
" llmgw keys cache --project Acme/ml/eval --ttl 24h --any\n" +
|
||||||
|
" llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95\n" +
|
||||||
" llmgw keys cache --project Acme/ml/train --off",
|
" llmgw keys cache --project Acme/ml/train --off",
|
||||||
Args: cobra.NoArgs,
|
Args: cobra.NoArgs,
|
||||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
switch {
|
switch {
|
||||||
case off && (ttl != 0 || anyAnswer):
|
case off && (ttl != 0 || anyAnswer || similar != 0):
|
||||||
return fmt.Errorf("--off, or --ttl with --any: not both")
|
return fmt.Errorf("--off, or --ttl with --any or --similar: not both")
|
||||||
|
case similar < 0 || similar > 1:
|
||||||
|
return fmt.Errorf("--similar is a cosine similarity between 0 and 1, like 0.95")
|
||||||
|
case similar != 0 && ttl == 0:
|
||||||
|
return fmt.Errorf("--similar goes with --ttl: say how long answers are reused")
|
||||||
case ttl < 0:
|
case ttl < 0:
|
||||||
return fmt.Errorf("--ttl is how long an answer is reused, like 1h")
|
return fmt.Errorf("--ttl is how long an answer is reused, like 1h")
|
||||||
case ttl > 0 && ttl < time.Second:
|
case ttl > 0 && ttl < time.Second:
|
||||||
@@ -54,7 +67,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if off || ttl > 0 {
|
if off || ttl > 0 {
|
||||||
if err := s.SetAnswerCache(cmd.Context(), p.ID, ttl, anyAnswer); err != nil {
|
if err := s.SetAnswerCache(cmd.Context(), p.ID, store.AnswerCache{TTL: ttl, Any: anyAnswer, Similar: similar}); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -71,6 +84,9 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
|||||||
fmt.Fprintf(cmd.OutOrStdout(), "%s reuses answers for %s: those asked at temperature 0 or with a seed, "+
|
fmt.Fprintf(cmd.OutOrStdout(), "%s reuses answers for %s: those asked at temperature 0 or with a seed, "+
|
||||||
"and embeddings, rerank and score.\n", p.Path, c.TTL)
|
"and embeddings, rerank and score.\n", p.Path, c.TTL)
|
||||||
}
|
}
|
||||||
|
if c.TTL > 0 && c.Similar > 0 {
|
||||||
|
fmt.Fprintf(cmd.OutOrStdout(), "Chat questions reaching %.2f similarity to one answered get its answer too.\n", c.Similar)
|
||||||
|
}
|
||||||
if off || ttl > 0 {
|
if off || ttl > 0 {
|
||||||
fmt.Fprintln(cmd.OutOrStdout(), "Gateways see the change within 30 seconds.")
|
fmt.Fprintln(cmd.OutOrStdout(), "Gateways see the change within 30 seconds.")
|
||||||
}
|
}
|
||||||
@@ -80,6 +96,7 @@ func newAnswerCacheCmd(open opener) *cobra.Command {
|
|||||||
cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
|
cmd.Flags().StringVar(&project, "project", "", "project: id, path or unique name")
|
||||||
cmd.Flags().DurationVar(&ttl, "ttl", 0, "how long an answer is reused, like 1h")
|
cmd.Flags().DurationVar(&ttl, "ttl", 0, "how long an answer is reused, like 1h")
|
||||||
cmd.Flags().BoolVar(&anyAnswer, "any", false, "reuse sampled answers too, not only reproducible ones")
|
cmd.Flags().BoolVar(&anyAnswer, "any", false, "reuse sampled answers too, not only reproducible ones")
|
||||||
|
cmd.Flags().Float64Var(&similar, "similar", 0, "also answer chat questions this similar to one answered, like 0.95")
|
||||||
cmd.Flags().BoolVar(&off, "off", false, "stop reusing answers for this project")
|
cmd.Flags().BoolVar(&off, "off", false, "stop reusing answers for this project")
|
||||||
return cmd
|
return cmd
|
||||||
}
|
}
|
||||||
@@ -150,6 +150,7 @@ func newBenchCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
|||||||
f.DurationVar(&cfg.SLO.TPOTp99, "tpot-p99", cfg.SLO.TPOTp99, "per-output-token target")
|
f.DurationVar(&cfg.SLO.TPOTp99, "tpot-p99", cfg.SLO.TPOTp99, "per-output-token target")
|
||||||
f.BoolVar(&save, "save", false,
|
f.BoolVar(&save, "save", false,
|
||||||
"record the capacity against the model and profile, so the number outlives this terminal")
|
"record the capacity against the model and profile, so the number outlives this terminal")
|
||||||
|
cmd.AddCommand(newBenchOverheadCmd(asJSON))
|
||||||
return cmd
|
return cmd
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,421 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"slices"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"syscall"
|
||||||
|
"text/tabwriter"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||||
|
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// What llmgw adds to a request, measured: the same load straight at an
|
||||||
|
// engine that answers at once, and through the gateway in front of it —
|
||||||
|
// bare, with authentication and usage accounting, and with the guardrails'
|
||||||
|
// pattern checks besides. The engine's own time is the same on both paths,
|
||||||
|
// so what differs is the gateway's.
|
||||||
|
//
|
||||||
|
// Everything runs in this process, on loopback: no cluster, no GPU, and the
|
||||||
|
// numbers are this machine's. On a machine shared with other work (a VM on
|
||||||
|
// an oversubscribed host) they move with it; compare modes run together,
|
||||||
|
// not figures from different runs.
|
||||||
|
|
||||||
|
// overheadMode is one way a request reaches the engine.
|
||||||
|
type overheadMode struct {
|
||||||
|
name string
|
||||||
|
gateway bool
|
||||||
|
auth, guard bool
|
||||||
|
}
|
||||||
|
|
||||||
|
var overheadModes = []overheadMode{
|
||||||
|
{name: "direct"},
|
||||||
|
{name: "gateway", gateway: true},
|
||||||
|
{name: "gateway+auth", gateway: true, auth: true},
|
||||||
|
{name: "gateway+auth+guard", gateway: true, auth: true, guard: true},
|
||||||
|
}
|
||||||
|
|
||||||
|
// overheadStep is one mode at one request rate.
|
||||||
|
type overheadStep struct {
|
||||||
|
Mode string `json:"mode"`
|
||||||
|
Rate int `json:"rate"`
|
||||||
|
Sent int `json:"sent"`
|
||||||
|
Failed int `json:"failed"`
|
||||||
|
P50 time.Duration `json:"p50"`
|
||||||
|
P99 time.Duration `json:"p99"`
|
||||||
|
AddedP50 time.Duration `json:"added_p50"`
|
||||||
|
AddedP99 time.Duration `json:"added_p99"`
|
||||||
|
// CPUPer1k is the process's CPU time per thousand requests beyond what
|
||||||
|
// the direct mode spent at the same rate: the gateway's.
|
||||||
|
CPUPer1k time.Duration `json:"cpu_per_1k"`
|
||||||
|
cpu time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
func newBenchOverheadCmd(asJSON *bool) *cobra.Command {
|
||||||
|
var (
|
||||||
|
rates []int
|
||||||
|
each time.Duration
|
||||||
|
)
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "overhead",
|
||||||
|
Short: "Measure what the gateway adds to a request, against an engine that answers at once",
|
||||||
|
Long: "Runs an engine that answers at once and a gateway in front of it, in this\n" +
|
||||||
|
"process on loopback, and sends the same load at each rate straight to the\n" +
|
||||||
|
"engine and through the gateway: bare, with authentication and usage\n" +
|
||||||
|
"accounting, and with the guardrails' pattern checks too. Reports each\n" +
|
||||||
|
"request's latency at p50 and p99, what the gateway added to it, and the\n" +
|
||||||
|
"gateway's CPU time per thousand requests.\n\n" +
|
||||||
|
"Needs no cluster. The figures are this machine's: on one shared with other\n" +
|
||||||
|
"work they move with it, so compare the modes of one run, and note the\n" +
|
||||||
|
"machine's load beside any figure you keep.",
|
||||||
|
Args: cobra.NoArgs,
|
||||||
|
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
|
steps, err := runOverhead(cmd.Context(), rates, each)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if *asJSON {
|
||||||
|
return json.NewEncoder(cmd.OutOrStdout()).Encode(steps)
|
||||||
|
}
|
||||||
|
printOverhead(cmd.OutOrStdout(), steps)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
f := cmd.Flags()
|
||||||
|
f.IntSliceVar(&rates, "rates", []int{100, 500, 1000, 2000}, "requests a second to send, one step each")
|
||||||
|
f.DurationVar(&each, "each", 3*time.Second, "how long each step sends for")
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func printOverhead(w io.Writer, steps []overheadStep) {
|
||||||
|
tw := tabwriter.NewWriter(w, 0, 0, 2, ' ', 0)
|
||||||
|
fmt.Fprintln(tw, "MODE\tRATE/S\tSENT\tFAILED\tP50\tP99\t+P50\t+P99\tCPU/1K REQ")
|
||||||
|
for _, s := range steps {
|
||||||
|
added := func(d time.Duration) string {
|
||||||
|
if s.Mode == "direct" {
|
||||||
|
return "-"
|
||||||
|
}
|
||||||
|
return msOf(d)
|
||||||
|
}
|
||||||
|
cpu := "-"
|
||||||
|
if s.Mode != "direct" {
|
||||||
|
cpu = msOf(s.CPUPer1k)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(tw, "%s\t%d\t%d\t%d\t%s\t%s\t%s\t%s\t%s\n", s.Mode, s.Rate, s.Sent, s.Failed,
|
||||||
|
msOf(s.P50), msOf(s.P99), added(s.AddedP50), added(s.AddedP99), cpu)
|
||||||
|
}
|
||||||
|
_ = tw.Flush()
|
||||||
|
fmt.Fprintf(w, "\n%d CPUs, %s. Loopback, in one process: the figures are this machine's.\n",
|
||||||
|
runtime.NumCPU(), loadAverage())
|
||||||
|
}
|
||||||
|
|
||||||
|
func msOf(d time.Duration) string { return fmt.Sprintf("%.2fms", float64(d)/float64(time.Millisecond)) }
|
||||||
|
|
||||||
|
// loadAverage is the machine's, where it can be read, to keep beside the
|
||||||
|
// figures.
|
||||||
|
func loadAverage() string {
|
||||||
|
if b, err := os.ReadFile("/proc/loadavg"); err == nil {
|
||||||
|
if f := strings.Fields(string(b)); len(f) >= 3 {
|
||||||
|
return "load " + strings.Join(f[:3], " ")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// macOS: "{ 1.23 1.10 0.98 }".
|
||||||
|
if b, err := exec.Command("sysctl", "-n", "vm.loadavg").Output(); err == nil {
|
||||||
|
if f := strings.Fields(strings.Trim(strings.TrimSpace(string(b)), "{}")); len(f) >= 3 {
|
||||||
|
return "load " + strings.Join(f[:3], " ")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "load not read"
|
||||||
|
}
|
||||||
|
|
||||||
|
// runOverhead measures every mode at every rate.
|
||||||
|
func runOverhead(ctx context.Context, rates []int, each time.Duration) ([]overheadStep, error) {
|
||||||
|
engine, err := instantEngine()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer engine.Close()
|
||||||
|
dir, err := os.MkdirTemp("", "llmgw-overhead-")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer os.RemoveAll(dir)
|
||||||
|
|
||||||
|
type target struct {
|
||||||
|
url, key string
|
||||||
|
stop func()
|
||||||
|
}
|
||||||
|
targets := map[string]target{}
|
||||||
|
defer func() {
|
||||||
|
for _, t := range targets {
|
||||||
|
if t.stop != nil {
|
||||||
|
t.stop()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
for _, m := range overheadModes {
|
||||||
|
if !m.gateway {
|
||||||
|
targets[m.name] = target{url: engine.URL}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
url, key, stop, err := overheadGateway(ctx, engine.URL, filepath.Join(dir, m.name+".db"), m)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("%s: %w", m.name, err)
|
||||||
|
}
|
||||||
|
targets[m.name] = target{url: url, key: key, stop: stop}
|
||||||
|
}
|
||||||
|
|
||||||
|
client := &http.Client{Transport: &http.Transport{MaxIdleConns: 1024, MaxIdleConnsPerHost: 1024}, Timeout: 30 * time.Second}
|
||||||
|
var out []overheadStep
|
||||||
|
for _, rate := range rates {
|
||||||
|
var direct overheadStep
|
||||||
|
for _, m := range overheadModes {
|
||||||
|
t := targets[m.name]
|
||||||
|
s := loadStep(ctx, client, t.url, t.key, rate, each)
|
||||||
|
s.Mode = m.name
|
||||||
|
if m.name == "direct" {
|
||||||
|
direct = s
|
||||||
|
} else {
|
||||||
|
s.AddedP50, s.AddedP99 = max(0, s.P50-direct.P50), max(0, s.P99-direct.P99)
|
||||||
|
if s.Sent > 0 {
|
||||||
|
s.CPUPer1k = max(0, s.cpu-direct.cpu) * 1000 / time.Duration(s.Sent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out = append(out, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadStep sends rate requests a second for d, each in its own goroutine,
|
||||||
|
// and measures each one's latency and the process's CPU meanwhile.
|
||||||
|
func loadStep(ctx context.Context, client *http.Client, url, key string, rate int, d time.Duration) overheadStep {
|
||||||
|
s := overheadStep{Rate: rate}
|
||||||
|
body := []byte(`{"model":"bench","max_tokens":8,"messages":[{"role":"user","content":"What is the capital of France?"}]}`)
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
lat []time.Duration
|
||||||
|
wg sync.WaitGroup
|
||||||
|
total = int(float64(rate) * d.Seconds())
|
||||||
|
)
|
||||||
|
interval := time.Second / time.Duration(max(rate, 1))
|
||||||
|
cpu0 := processCPU()
|
||||||
|
start := time.Now()
|
||||||
|
for i := range total {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if wait := time.Until(start.Add(time.Duration(i) * interval)); wait > 0 {
|
||||||
|
time.Sleep(wait)
|
||||||
|
}
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url+"/v1/chat/completions", bytes.NewReader(body))
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
if key != "" {
|
||||||
|
req.Header.Set("Authorization", "Bearer "+key)
|
||||||
|
}
|
||||||
|
t := time.Now()
|
||||||
|
resp, err := client.Do(req)
|
||||||
|
ok := err == nil && resp.StatusCode == http.StatusOK
|
||||||
|
if resp != nil {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
}
|
||||||
|
el := time.Since(t)
|
||||||
|
mu.Lock()
|
||||||
|
s.Sent++
|
||||||
|
if ok {
|
||||||
|
lat = append(lat, el)
|
||||||
|
} else {
|
||||||
|
s.Failed++
|
||||||
|
}
|
||||||
|
mu.Unlock()
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
s.cpu = processCPU() - cpu0
|
||||||
|
s.P50, s.P99 = percentile(lat, 0.50), percentile(lat, 0.99)
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func percentile(ds []time.Duration, p float64) time.Duration {
|
||||||
|
if len(ds) == 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
ds = slices.Clone(ds)
|
||||||
|
sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
|
||||||
|
i := int(p * float64(len(ds)-1))
|
||||||
|
return ds[i]
|
||||||
|
}
|
||||||
|
|
||||||
|
// processCPU is the CPU time this process has used, user and system.
|
||||||
|
func processCPU() time.Duration {
|
||||||
|
var ru syscall.Rusage
|
||||||
|
if err := syscall.Getrusage(syscall.RUSAGE_SELF, &ru); err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return time.Duration(ru.Utime.Nano() + ru.Stime.Nano())
|
||||||
|
}
|
||||||
|
|
||||||
|
// instantEngine is an OpenAI-compatible engine that answers at once, with
|
||||||
|
// the metrics the router scrapes.
|
||||||
|
func instantEngine() (*httpServer, error) {
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
fmt.Fprint(w, "# TYPE vllm:num_requests_running gauge\nvllm:num_requests_running 0\n"+
|
||||||
|
"# TYPE vllm:num_requests_waiting gauge\nvllm:num_requests_waiting 0\n"+
|
||||||
|
"# TYPE vllm:kv_cache_usage_perc gauge\nvllm:kv_cache_usage_perc 0.1\n")
|
||||||
|
})
|
||||||
|
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
|
||||||
|
mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_, _ = io.Copy(io.Discard, r.Body)
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
fmt.Fprint(w, `{"id":"b","object":"chat.completion","model":"bench","choices":[{"index":0,"message":{"role":"assistant","content":"Paris."},"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":2,"total_tokens":16}}`)
|
||||||
|
})
|
||||||
|
return serveLoopback(mux)
|
||||||
|
}
|
||||||
|
|
||||||
|
// httpServer is a handler served on a loopback port.
|
||||||
|
type httpServer struct {
|
||||||
|
URL string
|
||||||
|
srv *http.Server
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *httpServer) Close() { _ = s.srv.Close() }
|
||||||
|
|
||||||
|
func serveLoopback(h http.Handler) (*httpServer, error) {
|
||||||
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
srv := &http.Server{Handler: h, ReadHeaderTimeout: 10 * time.Second}
|
||||||
|
go func() { _ = srv.Serve(ln) }()
|
||||||
|
return &httpServer{URL: "http://" + ln.Addr().String(), srv: srv}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// overheadGateway starts a gateway in front of the engine as the mode asks,
|
||||||
|
// and returns its address and, with auth, a key.
|
||||||
|
func overheadGateway(ctx context.Context, engineURL, storePath string, m overheadMode) (string, string, func(), error) {
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.Listen = "127.0.0.1:0"
|
||||||
|
cfg.Router.ScrapeInterval = 200 * time.Millisecond
|
||||||
|
cfg.Endpoints = []config.Endpoint{{ID: "bench", URL: engineURL, Models: []string{"bench"}}}
|
||||||
|
key := ""
|
||||||
|
if m.auth {
|
||||||
|
var err error
|
||||||
|
if key, err = overheadKey(ctx, storePath); err != nil {
|
||||||
|
return "", "", nil, err
|
||||||
|
}
|
||||||
|
cfg.Auth.Enabled, cfg.Auth.Store = true, storePath
|
||||||
|
// Limits would measure themselves refusing: off.
|
||||||
|
cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM = 0, 0
|
||||||
|
}
|
||||||
|
rt, err := router.New(cfg, slog.New(slog.DiscardHandler))
|
||||||
|
if err != nil {
|
||||||
|
return "", "", nil, err
|
||||||
|
}
|
||||||
|
if m.guard {
|
||||||
|
g, err := baselineGuard()
|
||||||
|
if err != nil {
|
||||||
|
return "", "", nil, err
|
||||||
|
}
|
||||||
|
rt.SetGuard(g)
|
||||||
|
}
|
||||||
|
rctx, cancel := context.WithCancel(ctx)
|
||||||
|
rt.Start(rctx)
|
||||||
|
srv, err := serveLoopback(rt.Handler())
|
||||||
|
if err != nil {
|
||||||
|
cancel()
|
||||||
|
return "", "", nil, err
|
||||||
|
}
|
||||||
|
stop := func() { srv.Close(); cancel(); rt.Stop() }
|
||||||
|
// Ready once a request goes through: the endpoint is scraped first.
|
||||||
|
deadline := time.Now().Add(10 * time.Second)
|
||||||
|
for {
|
||||||
|
body := `{"model":"bench","messages":[{"role":"user","content":"hi"}]}`
|
||||||
|
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, srv.URL+"/v1/chat/completions", strings.NewReader(body))
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
if key != "" {
|
||||||
|
req.Header.Set("Authorization", "Bearer "+key)
|
||||||
|
}
|
||||||
|
resp, err := http.DefaultClient.Do(req)
|
||||||
|
if err == nil {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
if resp.StatusCode == http.StatusOK {
|
||||||
|
return srv.URL, key, stop, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
stop()
|
||||||
|
status := "no answer"
|
||||||
|
if err == nil {
|
||||||
|
status = resp.Status
|
||||||
|
}
|
||||||
|
return "", "", nil, fmt.Errorf("the gateway did not serve within 10s (%s)", status)
|
||||||
|
}
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// overheadKey makes a store with one project and a key in it.
|
||||||
|
func overheadKey(ctx context.Context, path string) (string, error) {
|
||||||
|
st, err := storepkg.Open(ctx, path)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
org, err := st.CreateOrg(ctx, "bench")
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
team, err := st.CreateTeam(ctx, org, "bench")
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
project, err := st.CreateProject(ctx, team, "bench")
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
secret, k, err := storepkg.NewKey("key-bench", "bench", project)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
k.CreatedAt = time.Now()
|
||||||
|
return secret, st.CreateKey(ctx, k)
|
||||||
|
}
|
||||||
|
|
||||||
|
// baselineGuard is the policy init installs, without its judge: the pattern
|
||||||
|
// checks every request passes.
|
||||||
|
func baselineGuard() (guard.Guard, error) {
|
||||||
|
m, err := config.Parse([]byte(guardSection(false)))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
cfg, err := config.FromMap(m)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return guard.NewGate(cfg.Guard, nil, slog.New(slog.DiscardHandler))
|
||||||
|
}
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The short version, so CI sees the gateway's overhead on every change: every
|
||||||
|
// mode answers every request, and what the gateway adds is measured, not
|
||||||
|
// guessed. No bound is asserted on it, as CI's machines vary; the table is
|
||||||
|
// in the test's log.
|
||||||
|
func TestTheGatewaysOverheadIsMeasured(t *testing.T) {
|
||||||
|
steps, err := runOverhead(t.Context(), []int{50}, time.Second)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(steps) != len(overheadModes) {
|
||||||
|
t.Fatalf("%d steps for %d modes", len(steps), len(overheadModes))
|
||||||
|
}
|
||||||
|
for _, s := range steps {
|
||||||
|
if s.Sent == 0 || s.Failed > 0 || s.P50 <= 0 || s.P99 < s.P50 {
|
||||||
|
t.Errorf("%s: sent %d failed %d p50 %s p99 %s", s.Mode, s.Sent, s.Failed, s.P50, s.P99)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var b bytes.Buffer
|
||||||
|
printOverhead(&b, steps)
|
||||||
|
if !strings.Contains(b.String(), "gateway+auth+guard") {
|
||||||
|
t.Errorf("table:\n%s", b.String())
|
||||||
|
}
|
||||||
|
t.Logf("\n%s", b.String())
|
||||||
|
}
|
||||||
@@ -186,6 +186,7 @@ func newCtlCmd() *cobra.Command {
|
|||||||
newCapacityCmd(client, &asJSON),
|
newCapacityCmd(client, &asJSON),
|
||||||
newQuotaCmd(client, &asJSON),
|
newQuotaCmd(client, &asJSON),
|
||||||
newGuardCmd(client, &asJSON),
|
newGuardCmd(client, &asJSON),
|
||||||
|
newMCPCmd(client, &asJSON),
|
||||||
newKBCmd(client, &asJSON),
|
newKBCmd(client, &asJSON),
|
||||||
newTrainCmd(client, &asJSON),
|
newTrainCmd(client, &asJSON),
|
||||||
newConfigCmd(client, &asJSON),
|
newConfigCmd(client, &asJSON),
|
||||||
@@ -320,6 +321,8 @@ func locationColumn(n adminapi.NodeView) string {
|
|||||||
// actually served, never what is merely catalogued.
|
// actually served, never what is merely catalogued.
|
||||||
func modelStatus(m adminapi.ModelView) string {
|
func modelStatus(m adminapi.ModelView) string {
|
||||||
switch {
|
switch {
|
||||||
|
case m.Stopped:
|
||||||
|
return "stopped (llmgw ctl model start)"
|
||||||
case m.Parked:
|
case m.Parked:
|
||||||
return "parked"
|
return "parked"
|
||||||
case m.External != "":
|
case m.External != "":
|
||||||
|
|||||||
@@ -0,0 +1,201 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newMCPCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||||
|
c := &cobra.Command{
|
||||||
|
Use: "mcp",
|
||||||
|
Short: "The company's own MCP servers, served through llmgw's /mcp",
|
||||||
|
Long: "An MCP server registered here — Jira, a file server, a database — is served\n" +
|
||||||
|
"through llmgw's /mcp beside llmgw's own tools, its tools named SERVER.tool.\n" +
|
||||||
|
"Its credentials stay on the control planes: a client gives only its llmgw\n" +
|
||||||
|
"key. A key sees and calls only the tools granted it, as patterns, with\n" +
|
||||||
|
"`llmgw keys limits --tools 'jira.*'` at any level of the tenancy tree;\n" +
|
||||||
|
"none granted anywhere is none. Every call is in `llmgw ctl audit`.",
|
||||||
|
}
|
||||||
|
c.AddCommand(newMCPAddCmd(client, asJSON), newMCPListCmd(client, asJSON), newMCPRemoveCmd(client),
|
||||||
|
newMCPTestCmd(client, asJSON))
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMCPAddCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||||
|
var req adminapi.MCPServerRequest
|
||||||
|
var headers, refs, secretFiles []string
|
||||||
|
var bearerFile string
|
||||||
|
var disabled bool
|
||||||
|
c := &cobra.Command{
|
||||||
|
Use: "add NAME --url URL",
|
||||||
|
Short: "Register an MCP server, or change one",
|
||||||
|
Example: " llmgw ctl mcp add jira --url https://jira.internal/mcp --bearer-file /root/jira-token\n" +
|
||||||
|
" llmgw ctl mcp add llmgw --url llmgw:admin --description \"this cluster, read only\"\n" +
|
||||||
|
" llmgw ctl mcp add wiki --url https://wiki.internal/mcp --header-ref Authorization=vault:secret/mcp/wiki#token\n" +
|
||||||
|
" llmgw ctl mcp add jira --disabled # off, kept",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
var err error
|
||||||
|
if req.Headers, err = pairs(headers, "--header"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if req.HeaderRefs, err = pairs(refs, "--header-ref"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
files, err := pairs(secretFiles, "--secret-file")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// A secret read here and kept in the store, sealed: the file
|
||||||
|
// need not be on every control plane, and is not read again.
|
||||||
|
for header, file := range files {
|
||||||
|
b, err := os.ReadFile(file) //nolint:gosec // a file the operator names
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if req.Secrets == nil {
|
||||||
|
req.Secrets = map[string]string{}
|
||||||
|
}
|
||||||
|
req.Secrets[header] = strings.TrimSpace(string(b))
|
||||||
|
}
|
||||||
|
if bearerFile != "" {
|
||||||
|
b, err := os.ReadFile(bearerFile) //nolint:gosec // a file the operator names
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if req.Secrets == nil {
|
||||||
|
req.Secrets = map[string]string{}
|
||||||
|
}
|
||||||
|
req.Secrets["Authorization"] = "Bearer " + strings.TrimSpace(string(b))
|
||||||
|
}
|
||||||
|
if cmd.Flags().Changed("disabled") {
|
||||||
|
on := !disabled
|
||||||
|
req.Enabled = &on
|
||||||
|
}
|
||||||
|
body, _ := json.Marshal(req)
|
||||||
|
var v adminapi.MCPServerView
|
||||||
|
if err := client().put(cmd.Context(), "/api/v1/mcp/servers/"+args[0], body, &v); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if *asJSON {
|
||||||
|
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||||
|
}
|
||||||
|
state := "on"
|
||||||
|
if !v.Enabled {
|
||||||
|
state = "off"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s, %s. Its tools are %s.TOOL in /mcp, to keys granted them "+
|
||||||
|
"(llmgw keys limits --tools '%s.*'). Check it: llmgw ctl mcp test %s\n", v.Name, v.URL, state, v.Name, v.Name, v.Name)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
f := c.Flags()
|
||||||
|
f.StringVar(&req.URL, "url", "", "the server's streamable HTTP address, https://…/mcp; llmgw:admin for this cluster, read only")
|
||||||
|
f.StringVar(&req.Description, "description", "", "what it is, shown beside its tools")
|
||||||
|
f.StringArrayVar(&headers, "header", nil, "NAME=VALUE sent as written, for what is not secret; repeatable")
|
||||||
|
f.StringArrayVar(&refs, "header-ref", nil, "NAME=REF read at each call: a file on every control plane, env:, vault: or store:; repeatable")
|
||||||
|
f.StringArrayVar(&secretFiles, "secret-file", nil, "NAME=FILE read now and kept in the store, sealed: a token; repeatable")
|
||||||
|
f.StringVar(&bearerFile, "bearer-file", "", "a token FILE, read now and kept sealed, sent as Authorization: Bearer")
|
||||||
|
f.BoolVar(&disabled, "disabled", false, "keep it but serve none of its tools")
|
||||||
|
f.IntVar(&req.TimeoutSeconds, "timeout", 0, "seconds one call may take (default 60)")
|
||||||
|
f.IntVar(&req.RatePerMinute, "rate", 0, "calls a minute each caller may make to its tools (default: no bound; -1 takes one away)")
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMCPListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "list",
|
||||||
|
Short: "The MCP servers /mcp serves",
|
||||||
|
Args: cobra.NoArgs,
|
||||||
|
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
|
var list []adminapi.MCPServerView
|
||||||
|
if err := client().get(cmd.Context(), "/api/v1/mcp/servers", &list); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if *asJSON {
|
||||||
|
return json.NewEncoder(cmd.OutOrStdout()).Encode(list)
|
||||||
|
}
|
||||||
|
if len(list) == 0 {
|
||||||
|
fmt.Fprintln(cmd.OutOrStdout(), "no MCP servers; add one: llmgw ctl mcp add NAME --url URL")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
w := tabTo(cmd.OutOrStdout())
|
||||||
|
fmt.Fprintln(w, "NAME\tSTATE\tURL\tCREDENTIALS\tDESCRIPTION")
|
||||||
|
for _, s := range list {
|
||||||
|
state := "on"
|
||||||
|
if !s.Enabled {
|
||||||
|
state = "off"
|
||||||
|
}
|
||||||
|
var creds []string
|
||||||
|
for h := range s.HeaderRefs {
|
||||||
|
creds = append(creds, h)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", s.Name, state, s.URL, dash(strings.Join(creds, ",")), s.Description)
|
||||||
|
}
|
||||||
|
return w.Flush()
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMCPRemoveCmd(client func() *ctlClient) *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "remove NAME",
|
||||||
|
Short: "Remove an MCP server: its tools leave /mcp",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
if err := client().delete(cmd.Context(), "/api/v1/mcp/servers/"+args[0], nil); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(cmd.OutOrStdout(), "%s removed; its tools leave /mcp within half a minute\n", args[0])
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMCPTestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "test NAME",
|
||||||
|
Short: "Ask an MCP server for its tools, with its credentials, as /mcp would",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
var v adminapi.MCPTestView
|
||||||
|
if err := client().post(cmd.Context(), "/api/v1/mcp/servers/"+args[0]+"/test", nil, &v); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if *asJSON {
|
||||||
|
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||||
|
}
|
||||||
|
if !v.OK {
|
||||||
|
return errors.New(args[0] + " did not answer: " + v.Error)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(cmd.OutOrStdout(), "%s answers with %d tool(s):\n", args[0], len(v.Tools))
|
||||||
|
for _, t := range v.Tools {
|
||||||
|
fmt.Fprintln(cmd.OutOrStdout(), " "+t)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// pairs reads NAME=VALUE flags.
|
||||||
|
func pairs(list []string, flag string) (map[string]string, error) {
|
||||||
|
if len(list) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
out := map[string]string{}
|
||||||
|
for _, p := range list {
|
||||||
|
k, v, ok := strings.Cut(p, "=")
|
||||||
|
if !ok || strings.TrimSpace(k) == "" {
|
||||||
|
return nil, fmt.Errorf("%s %q is not NAME=VALUE", flag, p)
|
||||||
|
}
|
||||||
|
out[strings.TrimSpace(k)] = v
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
@@ -33,7 +34,7 @@ func newGuardJudgeCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
|||||||
"then fetches and serves by itself: " + knownJudgeNames() + ".",
|
"then fetches and serves by itself: " + knownJudgeNames() + ".",
|
||||||
}
|
}
|
||||||
c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client),
|
c.AddCommand(newGuardJudgeStatusCmd(client, asJSON), newGuardJudgeEnableCmd(client),
|
||||||
newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON))
|
newGuardJudgeUseCmd(client), newGuardJudgeAskCmd(client, asJSON), newGuardJudgeCheckCmd(client, asJSON))
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -383,3 +384,70 @@ func newGuardJudgeAskCmd(client func() *ctlClient, asJSON *bool) *cobra.Command
|
|||||||
c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
|
c.Flags().StringVar(&answer, "answer", "", "an assistant's reply to TEXT, judged as output is")
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func newGuardJudgeCheckCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||||
|
var model string
|
||||||
|
var burst int
|
||||||
|
c := &cobra.Command{
|
||||||
|
Use: "check",
|
||||||
|
Short: "How a judge keeps up: a judgment alone and among others, against the time a request waits",
|
||||||
|
Long: "Times three judgments of a prompt of about 280 tokens, one after another, then\n" +
|
||||||
|
"a burst of them at once (two for each ready replica unless --burst says),\n" +
|
||||||
|
"and sets both against the policies' latency budget. A judgment past the\n" +
|
||||||
|
"budget is refused under fail: closed; past half of it alone, or past it\n" +
|
||||||
|
"among others, a busy moment refuses requests. Each request is judged once\n" +
|
||||||
|
"for its prompt and again every 600 characters of its answer.",
|
||||||
|
Args: cobra.NoArgs,
|
||||||
|
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
|
v, err := checkJudge(cmd.Context(), client(), model, burst)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if *asJSON {
|
||||||
|
return emitJSON(v)
|
||||||
|
}
|
||||||
|
fmt.Fprint(cmd.OutOrStdout(), judgeCheckText(v))
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
c.Flags().StringVar(&model, "model", "", "the judge (default: the one the policies ask)")
|
||||||
|
c.Flags().IntVar(&burst, "burst", 0, "judgments at once (default: two for each ready replica)")
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func checkJudge(ctx context.Context, c *ctlClient, model string, burst int) (adminapi.JudgeCheckView, error) {
|
||||||
|
body, _ := json.Marshal(adminapi.JudgeCheckRequest{Model: model, Burst: burst})
|
||||||
|
var v adminapi.JudgeCheckView
|
||||||
|
err := c.patient(3*time.Minute).post(ctx, "/api/v1/guard/judges/check", body, &v)
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// judgeCheckText says a check in a few lines, and what to do when it is not
|
||||||
|
// fine.
|
||||||
|
func judgeCheckText(v adminapi.JudgeCheckView) string {
|
||||||
|
var b strings.Builder
|
||||||
|
took := make([]string, len(v.TookMs))
|
||||||
|
for i, t := range v.TookMs {
|
||||||
|
took[i] = fmt.Sprintf("%.1f", float64(t)/1000)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&b, "%s, %d replica(s) ready, a request waits %.1f s for it\n", v.Model, v.Ready, float64(v.BudgetMs)/1000)
|
||||||
|
fmt.Fprintf(&b, " alone: %s s\n", strings.Join(took, ", "))
|
||||||
|
fmt.Fprintf(&b, " %d at once: slowest %.1f s", v.Burst, float64(v.BurstSlowestMs)/1000)
|
||||||
|
if v.BurstFailed > 0 {
|
||||||
|
fmt.Fprintf(&b, ", %d gave no verdict", v.BurstFailed)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&b, "; about %.1f judgments a second, so about %.1f requests a second with short answers\n",
|
||||||
|
v.PerSecond, v.PerSecond/2)
|
||||||
|
switch v.Verdict {
|
||||||
|
case "ok":
|
||||||
|
b.WriteString("ok: well inside the budget\n")
|
||||||
|
case "tight":
|
||||||
|
b.WriteString("tight: a busy moment passes the budget\n")
|
||||||
|
default:
|
||||||
|
b.WriteString("slow: a judgment alone passes the budget\n")
|
||||||
|
}
|
||||||
|
for _, a := range v.Advice {
|
||||||
|
b.WriteString(" - " + a + "\n")
|
||||||
|
}
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
@@ -32,12 +32,34 @@ type judgeKeeper struct {
|
|||||||
// tried is when each judge was last asked for, so one that cannot be
|
// tried is when each judge was last asked for, so one that cannot be
|
||||||
// fetched is not asked for again on every look.
|
// fetched is not asked for again on every look.
|
||||||
tried map[string]time.Time
|
tried map[string]time.Time
|
||||||
|
// grace is how long after its first look the keeper serves no judge
|
||||||
|
// itself: init adds its own judge just after the control plane starts,
|
||||||
|
// sized and placed as the install says, and one the leader added first
|
||||||
|
// was what init then kept. Zero waits for nothing.
|
||||||
|
grace time.Duration
|
||||||
|
first time.Time
|
||||||
|
|
||||||
|
// What refitting an earlier install's judge needs: the catalog to change
|
||||||
|
// its profile in, the replicas to restart, and the nodes' CPUs. Any nil
|
||||||
|
// leaves judges as they are.
|
||||||
|
profiles interface {
|
||||||
|
SetProfiles(ctx context.Context, name string, profiles []catalog.Profile, by string) (catalog.Model, error)
|
||||||
|
}
|
||||||
|
restart func(ids ...string) error
|
||||||
|
replicas func(model string) []string
|
||||||
|
nodeCPUs func() []int
|
||||||
|
// looked is each judge already refitted or found changed by hand, so
|
||||||
|
// each is said once.
|
||||||
|
looked map[string]bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// judgeRetry is how long a judge whose serving failed waits before it is
|
// judgeRetry is how long a judge whose serving failed waits before it is
|
||||||
// asked for again.
|
// asked for again.
|
||||||
const judgeRetry = 10 * time.Minute
|
const judgeRetry = 10 * time.Minute
|
||||||
|
|
||||||
|
// judgeGrace is the keeper's grace on a control plane just started.
|
||||||
|
const judgeGrace = 10 * time.Minute
|
||||||
|
|
||||||
func (k *judgeKeeper) run(ctx context.Context, every time.Duration) {
|
func (k *judgeKeeper) run(ctx context.Context, every time.Duration) {
|
||||||
t := time.NewTicker(every)
|
t := time.NewTicker(every)
|
||||||
defer t.Stop()
|
defer t.Stop()
|
||||||
@@ -69,6 +91,13 @@ func (k *judgeKeeper) check() {
|
|||||||
have := map[string]bool{}
|
have := map[string]bool{}
|
||||||
for _, m := range k.models() {
|
for _, m := range k.models() {
|
||||||
have[m.Name] = true
|
have[m.Name] = true
|
||||||
|
k.refit(m)
|
||||||
|
}
|
||||||
|
if k.first.IsZero() {
|
||||||
|
k.first = now()
|
||||||
|
}
|
||||||
|
if now().Sub(k.first) < k.grace {
|
||||||
|
return
|
||||||
}
|
}
|
||||||
busy := map[string]bool{}
|
busy := map[string]bool{}
|
||||||
for _, j := range k.serve.Jobs() {
|
for _, j := range k.serve.Jobs() {
|
||||||
@@ -96,6 +125,15 @@ func (k *judgeKeeper) check() {
|
|||||||
}
|
}
|
||||||
k.tried[name] = now()
|
k.tried[name] = now()
|
||||||
var opts judgeServeOptions
|
var opts judgeServeOptions
|
||||||
|
if !kj.GPU {
|
||||||
|
// The cores init would give the same number of judges.
|
||||||
|
var cpus []int
|
||||||
|
if k.nodeCPUs != nil {
|
||||||
|
cpus = k.nodeCPUs()
|
||||||
|
}
|
||||||
|
opts.Replicas = 2
|
||||||
|
opts.Cores = refitCores(opts.Replicas, cpus)
|
||||||
|
}
|
||||||
if kj.GPU {
|
if kj.GPU {
|
||||||
if opts.GPUClass = roomiestGPUClass(k.gpus()); opts.GPUClass == "" {
|
if opts.GPUClass = roomiestGPUClass(k.gpus()); opts.GPUClass == "" {
|
||||||
k.log.Warn("a policy names a judge that runs on a GPU, and this cluster has none",
|
k.log.Warn("a policy names a judge that runs on a GPU, and this cluster has none",
|
||||||
@@ -105,6 +143,8 @@ func (k *judgeKeeper) check() {
|
|||||||
}
|
}
|
||||||
req := kj.serveRequest(opts)
|
req := kj.serveRequest(opts)
|
||||||
req.By = "llmgw (a guard policy names it)"
|
req.By = "llmgw (a guard policy names it)"
|
||||||
|
// init, or an operator, may add it while this fetches: theirs stays.
|
||||||
|
req.IfAbsent = true
|
||||||
job, err := k.serve.Start(req)
|
job, err := k.serve.Start(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
k.log.Error("could not start serving a judge a policy names", "judge", name, "err", err)
|
k.log.Error("could not start serving a judge a policy names", "judge", name, "err", err)
|
||||||
@@ -114,3 +154,88 @@ func (k *judgeKeeper) check() {
|
|||||||
"judge", name, "repo", kj.Repo, "file", kj.File, "job", job.ID)
|
"judge", name, "repo", kj.Repo, "file", kj.File, "job", job.ID)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refit moves a judge an earlier `server init` installed to what init
|
||||||
|
// installs now, and restarts its replicas onto it. Those judges reserved no
|
||||||
|
// cores or one, 2 or 4 GB, and ran llama.cpp's prompt cache: on fewer than
|
||||||
|
// eight cores every judgment missed its 5 s budget and fail: closed refused
|
||||||
|
// every request, until fixed by hand. A judge whose profile differs in any
|
||||||
|
// other way was set by somebody, and is left as it is, said once.
|
||||||
|
func (k *judgeKeeper) refit(m catalog.Model) {
|
||||||
|
if k.profiles == nil || k.restart == nil || k.replicas == nil || m.Name != defaultJudge || !m.IsGuard() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if k.looked == nil {
|
||||||
|
k.looked = map[string]bool{}
|
||||||
|
}
|
||||||
|
if k.looked[m.Name] {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
k.looked[m.Name] = true
|
||||||
|
if !earlierInitJudge(m) {
|
||||||
|
if len(m.Profiles) != 1 || !slices.Contains(m.Profiles[0].Args, "--cache-ram") {
|
||||||
|
k.log.Info("the judge's profile is not one init wrote; left as it is",
|
||||||
|
"judge", m.Name, "see", "llmgw ctl model resources "+m.Name)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var cpus []int
|
||||||
|
if k.nodeCPUs != nil {
|
||||||
|
cpus = k.nodeCPUs()
|
||||||
|
}
|
||||||
|
p := m.Profiles[0]
|
||||||
|
p.MinMemoryGB = judgeMemoryGB
|
||||||
|
p.CPUCores = refitCores(m.Scaling.Min, cpus)
|
||||||
|
p.Args = []string{"--ctx-size", "8192", "--cache-ram", "0"}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if _, err := k.profiles.SetProfiles(ctx, m.Name, []catalog.Profile{p}, "llmgw (the judge an earlier init installed)"); err != nil {
|
||||||
|
k.log.Warn("could not refit the judge an earlier init installed", "judge", m.Name, "err", err)
|
||||||
|
k.looked[m.Name] = false
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ids := k.replicas(m.Name)
|
||||||
|
if len(ids) > 0 {
|
||||||
|
if err := k.restart(ids...); err != nil {
|
||||||
|
k.log.Warn("refitted the judge, but could not restart it", "judge", m.Name, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
k.log.Info("refitted the judge an earlier init installed", "judge", m.Name,
|
||||||
|
"memory_gb", p.MinMemoryGB, "cpu_cores", p.CPUCores, "replicas_restarted", len(ids))
|
||||||
|
}
|
||||||
|
|
||||||
|
// earlierInitJudge is whether a judge's entry is exactly what `server init`
|
||||||
|
// before v0.14.7 wrote: one llama.cpp profile on CPUs, an 8192 context and
|
||||||
|
// no other args, 2 or 4 GB, none to two cores.
|
||||||
|
func earlierInitJudge(m catalog.Model) bool {
|
||||||
|
if len(m.Profiles) != 1 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
p := m.Profiles[0]
|
||||||
|
return p.Name == "cpu" && p.Engine == "llamacpp" && p.GPUClass == "" && p.GPU == nil &&
|
||||||
|
len(p.NodeSelector) == 0 && slices.Equal(p.Args, []string{"--ctx-size", "8192"}) &&
|
||||||
|
(p.MinMemoryGB == 2 || p.MinMemoryGB == 4) && p.CPUCores >= 0 && p.CPUCores <= 2
|
||||||
|
}
|
||||||
|
|
||||||
|
// refitCores is a judge replica's cores by init's rule: up to four for the
|
||||||
|
// one judge of a cluster, two each where there are more, fewer where the
|
||||||
|
// smallest node cannot hold them all.
|
||||||
|
func refitCores(replicas int, cpus []int) int {
|
||||||
|
smallest := 0
|
||||||
|
for _, c := range cpus {
|
||||||
|
if c > 0 && (smallest == 0 || c < smallest) {
|
||||||
|
smallest = c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if replicas <= 1 {
|
||||||
|
if smallest == 0 {
|
||||||
|
return judgeCores
|
||||||
|
}
|
||||||
|
return min(smallest, judgeCoresAlone)
|
||||||
|
}
|
||||||
|
cores := judgeCores
|
||||||
|
if len(cpus) == 1 && smallest > 0 && replicas*cores > smallest {
|
||||||
|
cores = max(1, smallest/replicas)
|
||||||
|
}
|
||||||
|
return cores
|
||||||
|
}
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -51,7 +52,8 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
|
|||||||
// The GPU judge alone: the CPU one is there, and one llmgw does not know
|
// The GPU judge alone: the CPU one is there, and one llmgw does not know
|
||||||
// is not guessed at.
|
// is not guessed at.
|
||||||
if len(fs.started) != 1 || fs.started[0].Name != "qwen3guard-4b" || fs.started[0].GPUClass != "6000-24g" ||
|
if len(fs.started) != 1 || fs.started[0].Name != "qwen3guard-4b" || fs.started[0].GPUClass != "6000-24g" ||
|
||||||
fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU {
|
fs.started[0].Class != catalog.ClassStandard || fs.started[0].GuardFormat != "qwen3guard" || fs.started[0].CPU ||
|
||||||
|
!fs.started[0].IfAbsent {
|
||||||
t.Fatalf("started %+v", fs.started)
|
t.Fatalf("started %+v", fs.started)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -90,8 +92,12 @@ func TestAJudgeAPolicyNamesIsFetchedWhenTheCatalogLacksIt(t *testing.T) {
|
|||||||
lib = guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
|
lib = guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
|
||||||
inCatalog = nil
|
inCatalog = nil
|
||||||
k.gpus = func() []adminapi.ClassUse { return nil }
|
k.gpus = func() []adminapi.ClassUse { return nil }
|
||||||
|
k.nodeCPUs = func() []int { return []int{8, 4} }
|
||||||
k.check()
|
k.check()
|
||||||
if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 {
|
// Sized as init sizes its two judges: 2 GB, and two cores each on a
|
||||||
|
// cluster whose smallest node has four.
|
||||||
|
if last := fs.started[len(fs.started)-1]; last.Name != "qwen3guard-0.6b" || !last.CPU || last.Replicas != 2 ||
|
||||||
|
last.MemoryGB != judgeMemoryGB || last.CPUCores != refitCores(2, []int{8, 4}) || last.CPUCores == 0 {
|
||||||
t.Errorf("CPU judge: %+v", last)
|
t.Errorf("CPU judge: %+v", last)
|
||||||
}
|
}
|
||||||
fs.err = errors.New("the mirror is down")
|
fs.err = errors.New("the mirror is down")
|
||||||
@@ -124,3 +130,108 @@ func TestCategoriesFollowTheJudges(t *testing.T) {
|
|||||||
t.Errorf("no categories (every one counts) became %v", got)
|
t.Errorf("no categories (every one counts) became %v", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type fakeProfiles struct {
|
||||||
|
set map[string][]catalog.Profile
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeProfiles) SetProfiles(_ context.Context, name string, ps []catalog.Profile, _ string) (catalog.Model, error) {
|
||||||
|
if f.set == nil {
|
||||||
|
f.set = map[string][]catalog.Profile{}
|
||||||
|
}
|
||||||
|
f.set[name] = ps
|
||||||
|
return catalog.Model{Name: name, Profiles: ps}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// A judge that v0.14.6 or earlier's init installed is moved to today's
|
||||||
|
// figures, once, and its replicas restarted onto them; one somebody changed
|
||||||
|
// is left alone.
|
||||||
|
func TestAnEarlierInstallsJudgeIsRefittedOnce(t *testing.T) {
|
||||||
|
lib := guard.Config{Policies: map[string]guard.Policy{
|
||||||
|
"baseline": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}},
|
||||||
|
}}
|
||||||
|
earlier := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 1, nil)
|
||||||
|
earlier.Profiles[0].MinMemoryGB, earlier.Profiles[0].Args = 4, []string{"--ctx-size", "8192"}
|
||||||
|
inCatalog := []catalog.Model{earlier}
|
||||||
|
fp := &fakeProfiles{}
|
||||||
|
var restarted []string
|
||||||
|
k := &judgeKeeper{
|
||||||
|
library: func() guard.Config { return lib },
|
||||||
|
models: func() []catalog.Model { return inCatalog },
|
||||||
|
serve: &fakeServe{},
|
||||||
|
log: slog.New(slog.DiscardHandler),
|
||||||
|
profiles: fp,
|
||||||
|
restart: func(ids ...string) error { restarted = append(restarted, ids...); return nil },
|
||||||
|
replicas: func(string) []string { return []string{"j-1", "j-2"} },
|
||||||
|
nodeCPUs: func() []int { return []int{4} },
|
||||||
|
}
|
||||||
|
k.check()
|
||||||
|
got := fp.set["qwen3guard-0.6b"]
|
||||||
|
if len(got) != 1 || got[0].MinMemoryGB != 2 || got[0].CPUCores != 2 || !slices.Contains(got[0].Args, "--cache-ram") {
|
||||||
|
t.Fatalf("refitted to %+v", got)
|
||||||
|
}
|
||||||
|
if !slices.Equal(restarted, []string{"j-1", "j-2"}) {
|
||||||
|
t.Errorf("restarted %v", restarted)
|
||||||
|
}
|
||||||
|
k.check()
|
||||||
|
if len(restarted) != 2 {
|
||||||
|
t.Error("refitted twice")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Changed by somebody: an extra arg. Left as it is.
|
||||||
|
mine := guardModel("http://x/g.gguf", 2, 3, nil)
|
||||||
|
mine.Profiles[0].Args = []string{"--ctx-size", "16384"}
|
||||||
|
inCatalog = []catalog.Model{mine}
|
||||||
|
fp.set, k.looked = nil, nil
|
||||||
|
k.check()
|
||||||
|
if fp.set != nil {
|
||||||
|
t.Errorf("a judge set by hand was changed: %+v", fp.set)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAJudgeIsRefittedToTheCoresInitWouldGive(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
replicas int
|
||||||
|
cpus []int
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{1, []int{4}, 4},
|
||||||
|
{1, []int{2}, 2},
|
||||||
|
{1, []int{16}, 4},
|
||||||
|
{2, []int{4}, 2},
|
||||||
|
{2, []int{2}, 1},
|
||||||
|
{2, []int{4, 32}, 2},
|
||||||
|
{2, nil, 2},
|
||||||
|
} {
|
||||||
|
if got := refitCores(c.replicas, c.cpus); got != c.want {
|
||||||
|
t.Errorf("%d judges on %v: %d cores, want %d", c.replicas, c.cpus, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Just after the control plane starts, the judge init is adding is left to
|
||||||
|
// init: the keeper serves one itself only after its grace.
|
||||||
|
func TestTheKeeperLeavesAJustStartedInstallsJudgeToInit(t *testing.T) {
|
||||||
|
lib := guard.Config{Policies: map[string]guard.Policy{"b": {Judge: &guard.JudgePolicy{Model: "qwen3guard-0.6b", Input: "block"}}}}
|
||||||
|
now := time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC)
|
||||||
|
fs := &fakeServe{}
|
||||||
|
k := &judgeKeeper{
|
||||||
|
library: func() guard.Config { return lib },
|
||||||
|
models: func() []catalog.Model { return nil },
|
||||||
|
serve: fs, grace: judgeGrace,
|
||||||
|
gpus: func() []adminapi.ClassUse { return nil },
|
||||||
|
log: slog.New(slog.DiscardHandler),
|
||||||
|
now: func() time.Time { return now },
|
||||||
|
}
|
||||||
|
k.check()
|
||||||
|
now = now.Add(judgeGrace / 2)
|
||||||
|
k.check()
|
||||||
|
if len(fs.started) != 0 {
|
||||||
|
t.Fatalf("served within its grace: %+v", fs.started)
|
||||||
|
}
|
||||||
|
now = now.Add(judgeGrace)
|
||||||
|
k.check()
|
||||||
|
if len(fs.started) != 1 {
|
||||||
|
t.Errorf("never served after its grace: %d", len(fs.started))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -142,7 +142,9 @@ func (k knownJudge) serveRequest(o judgeServeOptions) modelserve.Request {
|
|||||||
}
|
}
|
||||||
return req
|
return req
|
||||||
}
|
}
|
||||||
req.CPU, req.CPUCores = true, o.Cores
|
// Sized as init sizes its judge: the memory it measured to need with
|
||||||
|
// no prompt cache, and the cores its rule gives.
|
||||||
|
req.CPU, req.CPUCores, req.MemoryGB = true, o.Cores, judgeMemoryGB
|
||||||
if req.Replicas == 0 {
|
if req.Replicas == 0 {
|
||||||
req.Replicas = 2
|
req.Replicas = 2
|
||||||
}
|
}
|
||||||
|
|||||||
+27
-1
@@ -7,6 +7,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"math"
|
"math"
|
||||||
|
"path"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"text/tabwriter"
|
"text/tabwriter"
|
||||||
@@ -301,7 +302,8 @@ func newLimitsCmd(open opener) *cobra.Command {
|
|||||||
var perMember bool
|
var perMember bool
|
||||||
var rpm, tpm int
|
var rpm, tpm int
|
||||||
var budget float64
|
var budget float64
|
||||||
var models []string
|
var models, tools []string
|
||||||
|
var replace bool
|
||||||
|
|
||||||
cmd := &cobra.Command{
|
cmd := &cobra.Command{
|
||||||
Use: "limits",
|
Use: "limits",
|
||||||
@@ -357,7 +359,19 @@ func newLimitsCmd(open opener) *cobra.Command {
|
|||||||
return errors.New("--per-member is for --scope org, team or project")
|
return errors.New("--per-member is for --scope org, team or project")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Only what is given changes: setting a tool grant must not
|
||||||
|
// clear the scope's rate or budget. --replace writes the row
|
||||||
|
// whole, a flag not given then no limit, as the console's
|
||||||
|
// form does with every field it shows.
|
||||||
l := store.Limits{Scope: level, ScopeID: scopeID}
|
l := store.Limits{Scope: level, ScopeID: scopeID}
|
||||||
|
if !replace {
|
||||||
|
switch cur, err := s.LimitsFor(cmd.Context(), level, scopeID); {
|
||||||
|
case err == nil:
|
||||||
|
l = cur
|
||||||
|
case !errors.Is(err, store.ErrNotFound):
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
if cmd.Flags().Changed("rpm") {
|
if cmd.Flags().Changed("rpm") {
|
||||||
l.RPM = &rpm
|
l.RPM = &rpm
|
||||||
}
|
}
|
||||||
@@ -379,6 +393,14 @@ func newLimitsCmd(open opener) *cobra.Command {
|
|||||||
}
|
}
|
||||||
l.AllowedModels = models
|
l.AllowedModels = models
|
||||||
}
|
}
|
||||||
|
if cmd.Flags().Changed("tools") {
|
||||||
|
for _, p := range tools {
|
||||||
|
if _, err := path.Match(p, "x.y"); err != nil {
|
||||||
|
return fmt.Errorf("--tools %q is not a pattern: %w", p, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
l.AllowedTools = tools
|
||||||
|
}
|
||||||
if err := s.PutLimits(cmd.Context(), l); err != nil {
|
if err := s.PutLimits(cmd.Context(), l); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -395,6 +417,10 @@ func newLimitsCmd(open opener) *cobra.Command {
|
|||||||
cmd.Flags().Float64Var(&budget, "budget", 0, "budget per period, in whole units as the quotas page shows them")
|
cmd.Flags().Float64Var(&budget, "budget", 0, "budget per period, in whole units as the quotas page shows them")
|
||||||
cmd.Flags().StringVar(&period, "period", "month", "day | month")
|
cmd.Flags().StringVar(&period, "period", "month", "day | month")
|
||||||
cmd.Flags().StringSliceVar(&models, "models", nil, "allowed models; empty allows none")
|
cmd.Flags().StringSliceVar(&models, "models", nil, "allowed models; empty allows none")
|
||||||
|
cmd.Flags().BoolVar(&replace, "replace", false,
|
||||||
|
"write every limit at the scope: one not given is then no limit there (default: only what is given changes)")
|
||||||
|
cmd.Flags().StringSliceVar(&tools, "tools", nil,
|
||||||
|
"the company MCP servers' tools granted here, as patterns: jira.*,db.query; none granted anywhere is none")
|
||||||
return cmd
|
return cmd
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -51,6 +51,7 @@ func newMirrorCmd() *cobra.Command {
|
|||||||
newMirrorRemoveCmd(&dir),
|
newMirrorRemoveCmd(&dir),
|
||||||
newMirrorServeCmd(&dir),
|
newMirrorServeCmd(&dir),
|
||||||
newMirrorInitCmd(&dir),
|
newMirrorInitCmd(&dir),
|
||||||
|
newMirrorAddPeerCmd(&dir),
|
||||||
)
|
)
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
@@ -237,6 +238,10 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
|
|||||||
if err := mirror.Remove(*dir, e); err != nil {
|
if err := mirror.Remove(*dir, e); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
// Kept for a mirror kept in step, which otherwise copies it back.
|
||||||
|
if err := mirror.RecordRemoval(*dir, model, version); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
fmt.Fprintf(cmd.OutOrStdout(), "removed %s/%s/%s\n", model, version, file)
|
fmt.Fprintf(cmd.OutOrStdout(), "removed %s/%s/%s\n", model, version, file)
|
||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
@@ -255,6 +260,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
|||||||
var tlsc tlsconf.Config
|
var tlsc tlsconf.Config
|
||||||
var allow, clients []string
|
var allow, clients []string
|
||||||
var allowPrivate bool
|
var allowPrivate bool
|
||||||
|
var peers []string
|
||||||
|
|
||||||
c := &cobra.Command{
|
c := &cobra.Command{
|
||||||
Use: "serve",
|
Use: "serve",
|
||||||
@@ -317,6 +323,21 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
|||||||
ReadTimeout: time.Minute,
|
ReadTimeout: time.Minute,
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// Another mirror kept in step with this one, each copying what
|
||||||
|
// the other has: the cluster's models while one machine is down.
|
||||||
|
if len(peers) > 0 {
|
||||||
|
p := &mirror.Peers{M: mgr, Log: log}
|
||||||
|
for _, addr := range peers {
|
||||||
|
c, err := mirror.NewClient(addr, tlsc, 30*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("--peer %s: %w", addr, err)
|
||||||
|
}
|
||||||
|
p.Clients = append(p.Clients, c)
|
||||||
|
}
|
||||||
|
go p.Run(cmd.Context())
|
||||||
|
fmt.Fprintf(out, "kept in step with %s, every %s\n", strings.Join(peers, ", "), mirror.PeerInterval)
|
||||||
|
}
|
||||||
|
|
||||||
errs := make(chan error, 2)
|
errs := make(chan error, 2)
|
||||||
go func() { errs <- fmt.Errorf("serving weights: %w", files.ListenAndServe()) }()
|
go func() { errs <- fmt.Errorf("serving weights: %w", files.ListenAndServe()) }()
|
||||||
fmt.Fprintf(out, "serving %s on %s\n", *dir, listen)
|
fmt.Fprintf(out, "serving %s on %s\n", *dir, listen)
|
||||||
@@ -343,6 +364,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
c.Flags().StringVar(&listen, "listen", ":8081", "address to serve the weights on")
|
c.Flags().StringVar(&listen, "listen", ":8081", "address to serve the weights on")
|
||||||
|
c.Flags().StringArrayVar(&peers, "peer", nil,
|
||||||
|
"another mirror's management API to keep in step with, each copying what the other has; repeatable. "+
|
||||||
|
"Give each the other's address in --api-allow (or its certificate in --api-client)")
|
||||||
c.Flags().StringVar(&apiListen, "api-listen", ":8082", "address to serve the management API on")
|
c.Flags().StringVar(&apiListen, "api-listen", ":8082", "address to serve the management API on")
|
||||||
c.Flags().StringVar(&base, "base", "http://localhost:8081",
|
c.Flags().StringVar(&base, "base", "http://localhost:8081",
|
||||||
"the address nodes use to reach this mirror, as it should appear in catalog sources")
|
"the address nodes use to reach this mirror, as it should appear in catalog sources")
|
||||||
|
|||||||
@@ -0,0 +1,224 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"regexp"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A second mirror, so models can still be added and fetched while the first
|
||||||
|
// machine is down. Set up from the machine of the first, which it is kept in
|
||||||
|
// step with: the first is told of it, it of the first, and the control
|
||||||
|
// planes of both.
|
||||||
|
|
||||||
|
func newMirrorAddPeerCmd(dir *string) *cobra.Command {
|
||||||
|
var controlPlanes []string
|
||||||
|
var base string
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "add-peer [user@]host",
|
||||||
|
Short: "Set up a second mirror on another machine, kept in step with this one",
|
||||||
|
Long: `Run on the machine of the cluster's mirror, as root. It logs in to HOST with
|
||||||
|
ssh, installs llmgw there if it is not, and sets up a mirror that keeps in
|
||||||
|
step with this one: each copies what the other holds, checksum checked, and
|
||||||
|
a model removed from one is removed from the other. This mirror is told of
|
||||||
|
it, and the cluster's settings name it (mirror.peers), so while either
|
||||||
|
machine is down the control planes use the other and nodes fetch from it.
|
||||||
|
On a mirror machine without the control plane's settings file, it prints
|
||||||
|
the llmgw ctl config set to run on a control plane instead.
|
||||||
|
|
||||||
|
HOST needs room for every model this mirror holds: it copies them all.`,
|
||||||
|
Example: " sudo llmgw mirror add-peer ubuntu@10.0.0.12 --control-plane 10.0.0.10 --control-plane 10.0.0.12",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
if os.Geteuid() != 0 {
|
||||||
|
return errors.New("add-peer changes this mirror's service and the cluster's settings: run it as root")
|
||||||
|
}
|
||||||
|
user, host, ok := strings.Cut(args[0], "@")
|
||||||
|
if !ok {
|
||||||
|
user, host = os.Getenv("SUDO_USER"), args[0]
|
||||||
|
}
|
||||||
|
return addMirrorPeer(cmd.Context(), cmd.OutOrStdout(), &onboard.SSH{Addr: host, User: user}, host,
|
||||||
|
*dir, base, controlPlanes, "/etc/llmgw/server.yaml")
|
||||||
|
},
|
||||||
|
}
|
||||||
|
cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
|
||||||
|
"address of a control plane that manages the mirrors; repeatable (default: this machine)")
|
||||||
|
cmd.Flags().StringVar(&base, "base", "", "the address nodes fetch from on HOST (default: http://HOST:8081)")
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
// addMirrorPeer sets up a mirror on a machine reached by remote, kept in
|
||||||
|
// step with the one on this machine.
|
||||||
|
func addMirrorPeer(ctx context.Context, out io.Writer, remote onboard.Remote, host, dir, base string,
|
||||||
|
controlPlanes []string, configPath string,
|
||||||
|
) error {
|
||||||
|
ip, err := ipv4Of(host)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
here := primaryAddr()
|
||||||
|
if len(controlPlanes) == 0 {
|
||||||
|
controlPlanes = []string{here}
|
||||||
|
}
|
||||||
|
if base == "" {
|
||||||
|
base = "http://" + net.JoinHostPort(ip, "8081")
|
||||||
|
}
|
||||||
|
thisAPI := "http://" + net.JoinHostPort(here, "8082")
|
||||||
|
peerAPI := "http://" + net.JoinHostPort(ip, "8082")
|
||||||
|
|
||||||
|
// llmgw there, the same as here, when it has none.
|
||||||
|
if _, err := remote.Run(ctx, "command -v llmgw >/dev/null", nil); err != nil {
|
||||||
|
exe, err := os.Executable()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
bin, err := os.ReadFile(exe) //nolint:gosec // this program
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, " .. llmgw to %s\n", host)
|
||||||
|
if _, err := remote.Run(ctx, "sudo install -m 755 /dev/stdin /usr/local/bin/llmgw", bin); err != nil {
|
||||||
|
return fmt.Errorf("installing llmgw on %s: %w", host, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The mirror there: its management API for the control planes and this
|
||||||
|
// mirror, which reads it; kept in step with this one.
|
||||||
|
allow := append([]string{}, controlPlanes...)
|
||||||
|
if !slices.Contains(allow, here) {
|
||||||
|
allow = append(allow, here)
|
||||||
|
}
|
||||||
|
script := "sudo llmgw mirror init --dir " + shellQuote(dir) + " --base " + shellQuote(base) + " --peer " + shellQuote(thisAPI)
|
||||||
|
for _, a := range allow {
|
||||||
|
script += " --control-plane " + shellQuote(a)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, " .. the mirror on %s\n", host)
|
||||||
|
if b, err := remote.Run(ctx, script, nil); err != nil {
|
||||||
|
return fmt.Errorf("the mirror on %s: %w\n%s", host, err, b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// This mirror told of it, and to let it read this one's API.
|
||||||
|
changed, err := addToMirrorUnit(mirrorUnitPath, peerAPI, ip)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if changed {
|
||||||
|
in := &serverInit{out: out}
|
||||||
|
if err := in.systemctl(ctx, "daemon-reload"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, " ok this mirror keeps in step with %s\n", peerAPI)
|
||||||
|
|
||||||
|
// The control planes told of both.
|
||||||
|
return nameMirrorPeer(out, configPath, peerAPI)
|
||||||
|
}
|
||||||
|
|
||||||
|
// nameMirrorPeer names a peer in the cluster's settings. A mirror on a
|
||||||
|
// machine of its own has neither the settings file nor the store: it says
|
||||||
|
// what to run on a control plane.
|
||||||
|
func nameMirrorPeer(out io.Writer, configPath, peerAPI string) error {
|
||||||
|
if _, err := os.Stat(configPath); errors.Is(err, os.ErrNotExist) {
|
||||||
|
fmt.Fprintf(out, " !! this machine has no %s, so the cluster's settings are not changed here. On a control plane:\n"+
|
||||||
|
" sudo llmgw ctl config set mirror.peers '[%s]'\n"+
|
||||||
|
" (with any peers already named, which llmgw ctl config get mirror.peers shows)\n", configPath, peerAPI)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err := addMirrorPeerSetting(configPath, peerAPI); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, " ok mirror.peers names %s; the control planes use it while this one is down\n", peerAPI)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// addToMirrorUnit adds a peer and the address allowed to read this mirror's
|
||||||
|
// API to its systemd unit, and says whether it changed.
|
||||||
|
func addToMirrorUnit(path, peer, allow string) (bool, error) {
|
||||||
|
b, err := os.ReadFile(path) //nolint:gosec // the mirror's unit
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("this machine has no mirror to keep in step (%s): run add-peer on the mirror's machine", path)
|
||||||
|
}
|
||||||
|
unit := string(b)
|
||||||
|
re := regexp.MustCompile(`(?m)^ExecStart=.*$`)
|
||||||
|
line := re.FindString(unit)
|
||||||
|
if line == "" {
|
||||||
|
return false, fmt.Errorf("%s has no ExecStart", path)
|
||||||
|
}
|
||||||
|
next := line
|
||||||
|
if !strings.Contains(next, "--peer "+peer) {
|
||||||
|
next += " --peer " + peer
|
||||||
|
}
|
||||||
|
if !strings.Contains(next, "--api-allow "+allow+" ") && !strings.HasSuffix(next, "--api-allow "+allow) &&
|
||||||
|
strings.Contains(next, "--api-listen :") {
|
||||||
|
next += " --api-allow " + allow
|
||||||
|
}
|
||||||
|
if next == line {
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
return true, os.WriteFile(path, []byte(strings.Replace(unit, line, next, 1)), 0o644)
|
||||||
|
}
|
||||||
|
|
||||||
|
// addMirrorPeerSetting names a peer in mirror.peers: in the cluster's
|
||||||
|
// settings when the store holds them, else in server.yaml.
|
||||||
|
func addMirrorPeerSetting(path, peer string) error {
|
||||||
|
add := func(doc *settings.Doc) error {
|
||||||
|
var peers []string
|
||||||
|
if n := doc.Node([]string{"mirror", "peers"}); n != nil {
|
||||||
|
_ = n.Decode(&peers)
|
||||||
|
}
|
||||||
|
if slices.Contains(peers, peer) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
list := "[" + strings.Join(append(peers, peer), ", ") + "]"
|
||||||
|
return doc.SetYAML([]string{"mirror", "peers"}, list)
|
||||||
|
}
|
||||||
|
changed, err := changeStoredSettings(path, "a second mirror (llmgw mirror add-peer)", add)
|
||||||
|
if err != nil || changed {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
doc, err := settings.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := add(doc); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_, err = doc.Save(nil)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// ensureSecondMirror sets up the mirror kept in step with this machine's on
|
||||||
|
// the store machine chosen, reached as the store's setup reached it.
|
||||||
|
func (in *serverInit) ensureSecondMirror(ctx context.Context) error {
|
||||||
|
if !in.standbys {
|
||||||
|
return errors.New("it needs --standbys: without them this mirror's API answers this machine alone, which the second cannot read")
|
||||||
|
}
|
||||||
|
i := in.secondMirror - 1
|
||||||
|
if i < 0 || i >= len(in.patroniNodes) {
|
||||||
|
return fmt.Errorf("there is no database machine %d to put it on", in.secondMirror)
|
||||||
|
}
|
||||||
|
if in.noMirror || in.mirrorURL != "" {
|
||||||
|
return errors.New("it keeps in step with a mirror on this machine, and there is none")
|
||||||
|
}
|
||||||
|
n := in.patroniNodes[i]
|
||||||
|
fmt.Fprintf(in.out, " .. a second mirror on %s\n", n.Name)
|
||||||
|
if err := addMirrorPeer(ctx, in.out, n.Remote, n.IP, in.mirrorRoot(), "", in.mirrorAllow(), in.configPath()); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
in.done("a second mirror on %s (%s), kept in step with this one", n.Name, n.IP)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAPeerIsAddedToTheMirrorsUnitOnce(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "llmgw-mirror.service")
|
||||||
|
unit := mirrorUnitFor("/srv/models", ":8082", "http://10.0.0.10:8081", []string{"10.0.0.10"}, nil, "test")
|
||||||
|
if err := os.WriteFile(path, []byte(unit), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
changed, err := addToMirrorUnit(path, "http://10.0.0.12:8082", "10.0.0.12")
|
||||||
|
if err != nil || !changed {
|
||||||
|
t.Fatalf("changed %v %v", changed, err)
|
||||||
|
}
|
||||||
|
b, _ := os.ReadFile(path)
|
||||||
|
if !strings.Contains(string(b), "--peer http://10.0.0.12:8082") || !strings.Contains(string(b), "--api-allow 10.0.0.12") {
|
||||||
|
t.Errorf("unit:\n%s", b)
|
||||||
|
}
|
||||||
|
if changed, _ := addToMirrorUnit(path, "http://10.0.0.12:8082", "10.0.0.12"); changed {
|
||||||
|
t.Error("added twice")
|
||||||
|
}
|
||||||
|
// A mirror whose API is on loopback alone is not opened by the back door.
|
||||||
|
lo := mirrorUnitFor("/srv/models", "127.0.0.1:8082", "http://10.0.0.10:8081", nil, nil, "test")
|
||||||
|
_ = os.WriteFile(path, []byte(lo), 0o644)
|
||||||
|
_, _ = addToMirrorUnit(path, "http://10.0.0.12:8082", "10.0.0.12")
|
||||||
|
if b, _ := os.ReadFile(path); strings.Contains(string(b), "--api-allow") {
|
||||||
|
t.Errorf("a loopback mirror was opened:\n%s", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestASecondMirrorIsNamedInTheSettings(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "server.yaml")
|
||||||
|
if err := os.WriteFile(path, []byte("listen: \":8080\"\nmirror:\n api: http://10.0.0.10:8082\n"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for range 2 {
|
||||||
|
if err := addMirrorPeerSetting(path, "http://10.0.0.12:8082"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
b, _ := os.ReadFile(path)
|
||||||
|
if strings.Count(string(b), "10.0.0.12:8082") != 1 || !strings.Contains(string(b), "peers") {
|
||||||
|
t.Errorf("server.yaml:\n%s", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheSetupOffersASecondMirrorOnAStoreMachine(t *testing.T) {
|
||||||
|
in := newDefaultServerInit()
|
||||||
|
f := newInstallForm(&in, "ctl", []string{"10.0.0.5"}, true)
|
||||||
|
f.secondMirror.Choice = 1
|
||||||
|
if !f.secondMirror.Show() {
|
||||||
|
t.Fatal("not offered with the mirror on this machine")
|
||||||
|
}
|
||||||
|
f.mirrorWhere.Choice = 1
|
||||||
|
if f.secondMirror.Show() {
|
||||||
|
t.Error("offered with the mirror elsewhere")
|
||||||
|
}
|
||||||
|
f.mirrorWhere.Choice = 0
|
||||||
|
in.secondMirror = 1
|
||||||
|
in.patroni = []string{"ubuntu@10.0.0.11", "ubuntu@10.0.0.12"}
|
||||||
|
if got := strings.Join(in.initFlags(), " "); !strings.Contains(got, "--second-mirror 1") {
|
||||||
|
t.Errorf("flags %s", got)
|
||||||
|
}
|
||||||
|
// Without standbys this mirror answers itself alone: the second could not read it.
|
||||||
|
in.standbys = false
|
||||||
|
if err := in.ensureSecondMirror(t.Context()); err == nil || !strings.Contains(err.Error(), "standbys") {
|
||||||
|
t.Errorf("without standbys: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAMirrorOnAMachineOfItsOwnSaysWhatToRunOnAControlPlane(t *testing.T) {
|
||||||
|
var out bytes.Buffer
|
||||||
|
if err := nameMirrorPeer(&out, filepath.Join(t.TempDir(), "server.yaml"), "http://10.0.0.12:8082"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out.String(), "llmgw ctl config set mirror.peers '[http://10.0.0.12:8082]'") {
|
||||||
|
t.Errorf("%s", out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -29,6 +29,7 @@ import (
|
|||||||
func newMirrorInitCmd(dir *string) *cobra.Command {
|
func newMirrorInitCmd(dir *string) *cobra.Command {
|
||||||
var (
|
var (
|
||||||
controlPlanes []string
|
controlPlanes []string
|
||||||
|
peers []string
|
||||||
base string
|
base string
|
||||||
noFirewall bool
|
noFirewall bool
|
||||||
)
|
)
|
||||||
@@ -61,7 +62,7 @@ Safe to run again: it changes only what differs.`,
|
|||||||
if _, err := parseAllow(controlPlanes); err != nil {
|
if _, err := parseAllow(controlPlanes); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes); err != nil {
|
if err := mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes, peers); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if !noFirewall {
|
if !noFirewall {
|
||||||
@@ -80,6 +81,9 @@ Safe to run again: it changes only what differs.`,
|
|||||||
}
|
}
|
||||||
cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
|
cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
|
||||||
"address of the control plane that manages this mirror; repeatable")
|
"address of the control plane that manages this mirror; repeatable")
|
||||||
|
cmd.Flags().StringArrayVar(&peers, "peer", nil,
|
||||||
|
"another mirror's management API to keep in step with, http://host:8082; repeatable. "+
|
||||||
|
"Its machine must be among --control-plane, to read this one")
|
||||||
cmd.Flags().BoolVar(&noFirewall, "no-firewall", false,
|
cmd.Flags().BoolVar(&noFirewall, "no-firewall", false,
|
||||||
"leave the firewall alone, instead of keeping the mirror's ports to the cluster's networks")
|
"leave the firewall alone, instead of keeping the mirror's ports to the cluster's networks")
|
||||||
cmd.Flags().StringVar(&base, "base", "",
|
cmd.Flags().StringVar(&base, "base", "",
|
||||||
@@ -87,7 +91,7 @@ Safe to run again: it changes only what differs.`,
|
|||||||
return cmd
|
return cmd
|
||||||
}
|
}
|
||||||
|
|
||||||
func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPlanes []string) error {
|
func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPlanes, peers []string) error {
|
||||||
in := &serverInit{user: "llmgw", stateDir: "/var/lib/llmgw", out: out}
|
in := &serverInit{user: "llmgw", stateDir: "/var/lib/llmgw", out: out}
|
||||||
if err := in.ensureUser(ctx); err != nil {
|
if err := in.ensureUser(ctx); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -105,7 +109,7 @@ func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPla
|
|||||||
}
|
}
|
||||||
base = "http://" + net.JoinHostPort(addr, "8081")
|
base = "http://" + net.JoinHostPort(addr, "8081")
|
||||||
}
|
}
|
||||||
unit := mirrorUnitFor(dir, ":8082", base, controlPlanes,
|
unit := mirrorUnitFor(dir, ":8082", base, controlPlanes, peers,
|
||||||
"'llmgw mirror init'. Bytes on :8081 for every node; the\n# management API on :8082, for the control plane only.")
|
"'llmgw mirror init'. Bytes on :8081 for every node; the\n# management API on :8082, for the control plane only.")
|
||||||
changed, err := in.writeUnit(mirrorUnitPath, unit)
|
changed, err := in.writeUnit(mirrorUnitPath, unit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
+200
-7
@@ -18,6 +18,7 @@ import (
|
|||||||
|
|
||||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -33,7 +34,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
|||||||
Use: "model",
|
Use: "model",
|
||||||
Short: "Add or remove models while the cluster is running",
|
Short: "Add or remove models while the cluster is running",
|
||||||
}
|
}
|
||||||
c.AddCommand(newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
|
c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||||
newTargetCmd(client, asJSON), newAdaptersCmd(client),
|
newTargetCmd(client, asJSON), newAdaptersCmd(client),
|
||||||
newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
|
newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
|
||||||
@@ -162,21 +163,104 @@ func readModelFile(path string) (catalog.Model, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
|
func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
|
||||||
return &cobra.Command{
|
var purge bool
|
||||||
|
c := &cobra.Command{
|
||||||
Use: "remove <name>",
|
Use: "remove <name>",
|
||||||
Short: "Remove a model that was added at runtime",
|
Short: "Remove a model that was added at runtime, and with --purge its files",
|
||||||
Long: "Only models added at runtime. One declared in server.yaml would come\n" +
|
Long: "Only models added at runtime. One declared in server.yaml would come\n" +
|
||||||
"back on the next restart, so removing it here would leave the cluster\n" +
|
"back on the next restart, so removing it here would leave the cluster\n" +
|
||||||
"disagreeing with its own configuration until then.\n\n" +
|
"disagreeing with its own configuration until then.\n\n" +
|
||||||
"Its replicas are stopped on the next pass, after the endpoint has been\n" +
|
"Its replicas are stopped on the next pass, after the endpoint has been\n" +
|
||||||
"withdrawn, so traffic stops before the engine does.",
|
"withdrawn, so traffic stops before the engine does.\n\n" +
|
||||||
|
"Its files stay on the mirror and the nodes, for adding it again. With\n" +
|
||||||
|
"--purge they go too: every version of it on the mirror (a mirror kept in\n" +
|
||||||
|
"step follows), and the nodes' copies once its replicas have stopped.\n" +
|
||||||
|
"What another model uses stays. To take a model out of service and keep\n" +
|
||||||
|
"it, use llmgw ctl model stop.",
|
||||||
|
Example: " llmgw ctl model remove qwen-7b\n" +
|
||||||
|
" llmgw ctl model remove qwen-7b --purge # its files on the mirror and the nodes too",
|
||||||
Args: cobra.ExactArgs(1),
|
Args: cobra.ExactArgs(1),
|
||||||
RunE: func(cmd *cobra.Command, args []string) error {
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
var res map[string]string
|
out := cmd.OutOrStdout()
|
||||||
if err := client().delete(cmd.Context(), "/api/v1/models/"+args[0], &res); err != nil {
|
if !purge {
|
||||||
|
var res map[string]string
|
||||||
|
if err := client().delete(cmd.Context(), "/api/v1/models/"+args[0], &res); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "%s removed. Its files stay on the mirror and the nodes; --purge deletes them.\n", args[0])
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var v adminapi.PurgeView
|
||||||
|
if err := client().patient(3*time.Minute).delete(cmd.Context(), "/api/v1/models/"+args[0]+"?purge=true", &v); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
fmt.Fprintf(cmd.OutOrStdout(), "%s removed.\n", args[0])
|
fmt.Fprintf(out, "%s removed.\n", v.Removed)
|
||||||
|
for _, m := range v.Mirror {
|
||||||
|
fmt.Fprintf(out, " mirror %s deleted\n", m)
|
||||||
|
}
|
||||||
|
for _, m := range v.Kept {
|
||||||
|
fmt.Fprintf(out, " mirror %s kept: another model uses it\n", m)
|
||||||
|
}
|
||||||
|
var freed int64
|
||||||
|
for _, n := range v.Nodes {
|
||||||
|
if n.Error != "" {
|
||||||
|
fmt.Fprintf(out, " node %s: %s\n", n.Node, n.Error)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
freed += n.Bytes
|
||||||
|
fmt.Fprintf(out, " node %s: %d file(s), %s\n", n.Node, n.Files, humanBytes(n.Bytes))
|
||||||
|
}
|
||||||
|
if len(v.Mirror) == 0 && len(v.Nodes) == 0 && v.Note == "" {
|
||||||
|
fmt.Fprintln(out, " it had no files of its own on the mirror or the nodes")
|
||||||
|
}
|
||||||
|
if freed > 0 {
|
||||||
|
fmt.Fprintf(out, " %s freed on the nodes\n", humanBytes(freed))
|
||||||
|
}
|
||||||
|
if v.Note != "" {
|
||||||
|
fmt.Fprintln(out, " "+v.Note)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
c.Flags().BoolVar(&purge, "purge", false, "delete its files from the mirror and the nodes as well")
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func newModelStopCmd(client func() *ctlClient) *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "stop <model>",
|
||||||
|
Short: "Take a model out of service and keep it: none of its replicas run until it is started",
|
||||||
|
Long: "Its replicas stop; it stays in the catalog with its files, and a request\n" +
|
||||||
|
"for it is told an administrator stopped it. What it was scaled to is kept:\n" +
|
||||||
|
"llmgw ctl model start runs it again as it was.",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
var v adminapi.ScaleResponse
|
||||||
|
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/stop", nil, &v); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(cmd.OutOrStdout(), "%s stopped: its replicas stop on the next pass. It is kept; "+
|
||||||
|
"llmgw ctl model start %s runs it again.\n", v.Model, v.Model)
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newModelStartCmd(client func() *ctlClient) *cobra.Command {
|
||||||
|
return &cobra.Command{
|
||||||
|
Use: "start <model>",
|
||||||
|
Short: "Run a stopped model again, scaled as it was when it was stopped",
|
||||||
|
Args: cobra.ExactArgs(1),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
var v adminapi.ScaleResponse
|
||||||
|
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/start", nil, &v); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
how := fmt.Sprintf("%d replica(s)", v.Min)
|
||||||
|
if v.OnDemand {
|
||||||
|
how = "on demand"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(cmd.OutOrStdout(), "%s started: %s, max %d. Watch it with llmgw ctl models.\n", v.Model, how, v.Max)
|
||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -386,3 +470,112 @@ func newAdaptersCmd(client func() *ctlClient) *cobra.Command {
|
|||||||
c.Flags().StringArrayVar(&remove, "remove", nil, "an adapter's name; repeatable")
|
c.Flags().StringArrayVar(&remove, "remove", nil, "an adapter's name; repeatable")
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func newModelContextCmd(client func() *ctlClient) *cobra.Command {
|
||||||
|
var slots int
|
||||||
|
c := &cobra.Command{
|
||||||
|
Use: "context <model> <tokens>",
|
||||||
|
Short: "Change how much one request to a served model may use",
|
||||||
|
Long: "The context lives in three places: the catalog (what a request may use, and\n" +
|
||||||
|
"what it may answer), the engine's arguments (llama.cpp's --ctx-size and\n" +
|
||||||
|
"--parallel, vLLM's --max-model-len), and the memory the KV cache takes. This\n" +
|
||||||
|
"changes all three together. It refuses more than the model was trained for,\n" +
|
||||||
|
"and a context its cards would not hold, saying what each would need. Its\n" +
|
||||||
|
"replicas are then started again on it, each new one before the old stops.\n\n" +
|
||||||
|
"--slots is how many requests a llama.cpp replica answers at once; its pool is\n" +
|
||||||
|
"slots × tokens. Left out, it keeps what it has.",
|
||||||
|
Example: " llmgw ctl model context qwen3-coder-30b 65536\n" +
|
||||||
|
" llmgw ctl model context qwen2.5-7b 16384 --slots 4",
|
||||||
|
Args: cobra.ExactArgs(2),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
tokens, err := strconv.Atoi(args[1])
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("the context is a whole number of tokens, not %q", args[1])
|
||||||
|
}
|
||||||
|
body, _ := json.Marshal(modelserve.ContextRequest{Tokens: tokens, Slots: slots})
|
||||||
|
var v modelserve.ContextChange
|
||||||
|
if err := client().patient(time.Minute).post(cmd.Context(), "/api/v1/models/"+args[0]+"/context", body, &v); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
out := cmd.OutOrStdout()
|
||||||
|
fmt.Fprintf(out, "%s: %d → %d tokens a request, answers up to %d", v.Model, v.Before, v.After, v.MaxOutput)
|
||||||
|
if v.Trained > 0 {
|
||||||
|
fmt.Fprintf(out, " (trained for %d)", v.Trained)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out)
|
||||||
|
for _, p := range v.Profiles {
|
||||||
|
line := " " + p.Profile + " (" + p.Engine + "): " + strings.Join(p.Args, " ")
|
||||||
|
switch {
|
||||||
|
case p.Before != "" && p.After != "":
|
||||||
|
line += "; " + p.Before + " → " + p.After
|
||||||
|
case p.After != "":
|
||||||
|
line += "; " + p.After
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, line)
|
||||||
|
}
|
||||||
|
if v.Note != "" {
|
||||||
|
fmt.Fprintln(out, " note: "+v.Note)
|
||||||
|
}
|
||||||
|
if v.Restarted > 0 {
|
||||||
|
fmt.Fprintf(out, "%d replica(s) started again on it, each new one before the old stops: llmgw ctl replicas --model %s\n", v.Restarted, v.Model)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
c.Flags().IntVar(&slots, "slots", 0, "llama.cpp: requests a replica answers at once (default: as it is)")
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func newModelGPUsCmd(client func() *ctlClient) *cobra.Command {
|
||||||
|
var profile string
|
||||||
|
var inPlace bool
|
||||||
|
c := &cobra.Command{
|
||||||
|
Use: "gpus <model> <cards>",
|
||||||
|
Short: "Spread a served model over more cards, or fewer",
|
||||||
|
Long: "How many cards of one node each replica uses. vLLM, SGLang and TensorRT-LLM\n" +
|
||||||
|
"spread the model by tensor parallel, which needs the count to divide the\n" +
|
||||||
|
"model's KV heads or be divided by them; llama.cpp splits its layers over the\n" +
|
||||||
|
"cards. It refuses a count a node does not have, and one whose cards would not\n" +
|
||||||
|
"hold the model and its context, saying what each would need. Its replicas are\n" +
|
||||||
|
"then started again on it, each new one beside the old before it stops.\n\n" +
|
||||||
|
"With no node free enough beside the running replicas — a node of two cards,\n" +
|
||||||
|
"the model on one — --in-place stops each replica first: the model does not\n" +
|
||||||
|
"answer while the new one loads.\n\n" +
|
||||||
|
"--profile names the profile for a model with more than one on cards.",
|
||||||
|
Example: " llmgw ctl model gpus qwen3-coder-30b 2\n" +
|
||||||
|
" llmgw ctl model gpus llama-70b 4 --profile h100 --in-place",
|
||||||
|
Args: cobra.ExactArgs(2),
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
n, err := strconv.Atoi(args[1])
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("the cards are a whole number, not %q", args[1])
|
||||||
|
}
|
||||||
|
body, _ := json.Marshal(modelserve.GPUsRequest{GPUs: n, Profile: profile, InPlace: inPlace})
|
||||||
|
var v modelserve.GPUsChange
|
||||||
|
if err := client().patient(time.Minute).post(cmd.Context(), "/api/v1/models/"+args[0]+"/gpus", body, &v); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
out := cmd.OutOrStdout()
|
||||||
|
fmt.Fprintf(out, "%s: %d → %d %s card(s) a replica (%s, %s)\n", v.Model, v.Before, v.After, v.Class, v.Profile, v.Engine)
|
||||||
|
if v.PerCard != "" {
|
||||||
|
fmt.Fprintln(out, " "+v.PerCard)
|
||||||
|
}
|
||||||
|
if len(v.Args) > 0 {
|
||||||
|
fmt.Fprintln(out, " args: "+strings.Join(v.Args, " "))
|
||||||
|
}
|
||||||
|
if v.Note != "" {
|
||||||
|
fmt.Fprintln(out, " note: "+v.Note)
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case v.Restarted > 0 && v.InPlace:
|
||||||
|
fmt.Fprintf(out, "%d replica(s) stopped and started again on it, one by one: llmgw ctl replicas --model %s\n", v.Restarted, v.Model)
|
||||||
|
case v.Restarted > 0:
|
||||||
|
fmt.Fprintf(out, "%d replica(s) started again on it, each new one before the old stops: llmgw ctl replicas --model %s\n", v.Restarted, v.Model)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
c.Flags().StringVar(&profile, "profile", "", "the profile to change, for a model with several on cards")
|
||||||
|
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before its replacement starts, when there is no room beside it")
|
||||||
|
return c
|
||||||
|
}
|
||||||
@@ -69,6 +69,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
|||||||
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
|
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
|
||||||
f.IntVar(&req.TP, "tp", 0, "the older name of --gpus")
|
f.IntVar(&req.TP, "tp", 0, "the older name of --gpus")
|
||||||
_ = f.MarkHidden("tp")
|
_ = f.MarkHidden("tp")
|
||||||
|
f.StringToStringVar(&req.Env, "env", nil,
|
||||||
|
"KEY=VALUE in the engine's environment on every node, e.g. NCCL_DEBUG=INFO; repeatable")
|
||||||
f.BoolVar(&req.Fallbacks, "fallbacks", false, "also add the next best ways to run it, used when the first is busy")
|
f.BoolVar(&req.Fallbacks, "fallbacks", false, "also add the next best ways to run it, used when the first is busy")
|
||||||
f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
|
f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
|
||||||
f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
|
f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
|
||||||
|
|||||||
@@ -291,14 +291,14 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
|||||||
}
|
}
|
||||||
fmt.Fprintln(out)
|
fmt.Fprintln(out)
|
||||||
w := tabTo(out)
|
w := tabTo(out)
|
||||||
fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tHOLDS\tOF\tNOW")
|
fmt.Fprintln(w, "FILE\tSIZE\tBEST WAY\tA CARD\tOF\tALL CARDS\tNOW")
|
||||||
for _, a := range p.Artifacts {
|
for _, a := range p.Artifacts {
|
||||||
name := a.Name
|
name := a.Name
|
||||||
if a.Shards > 1 && a.Format == "gguf" {
|
if a.Shards > 1 && a.Format == "gguf" {
|
||||||
name += fmt.Sprintf(" (%d parts)", a.Shards)
|
name += fmt.Sprintf(" (%d parts)", a.Shards)
|
||||||
}
|
}
|
||||||
if a.Best == nil {
|
if a.Best == nil {
|
||||||
fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\n", name, bytesShort(a.Bytes))
|
fmt.Fprintf(w, "%s\t%s\tfits nowhere\t\t\t\t\n", name, bytesShort(a.Bytes))
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
b := a.Best
|
b := a.Best
|
||||||
@@ -306,11 +306,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
|||||||
if b.FreeNow {
|
if b.FreeNow {
|
||||||
now = "free"
|
now = "free"
|
||||||
}
|
}
|
||||||
holds := fmt.Sprintf("%.1f GB a card", b.NeedGB)
|
holds, all := fmt.Sprintf("%.1f GB", b.NeedGB), fmt.Sprintf("%.0f GB", b.TotalGB())
|
||||||
if b.CPU {
|
if b.CPU {
|
||||||
holds = fmt.Sprintf("%.1f GB of RAM", b.NeedGB)
|
holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, now)
|
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
|
||||||
}
|
}
|
||||||
_ = w.Flush()
|
_ = w.Flush()
|
||||||
rec, ok := p.Recommended()
|
rec, ok := p.Recommended()
|
||||||
@@ -318,7 +318,11 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
|||||||
fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
|
fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
|
||||||
return errDoesNotFit
|
return errDoesNotFit
|
||||||
}
|
}
|
||||||
fmt.Fprintf(out, "\nBest: %s on %s.\n", rec.Artifact, rec.Label())
|
fmt.Fprintf(out, "\nBest: %s on %s: %s.\n", rec.Artifact, rec.Label(), rec.MemoryText())
|
||||||
|
if p.KVBytesPerToken > 0 {
|
||||||
|
fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n",
|
||||||
|
modelplan.GB(p.KVBytesPerToken*1000))
|
||||||
|
}
|
||||||
if len(p.Options) > 1 {
|
if len(p.Options) > 1 {
|
||||||
fmt.Fprintln(out, "\nEvery way, best first:")
|
fmt.Fprintln(out, "\nEvery way, best first:")
|
||||||
w = tabTo(out)
|
w = tabTo(out)
|
||||||
@@ -327,7 +331,7 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
|||||||
if o.FreeNow {
|
if o.FreeNow {
|
||||||
now = "free"
|
now = "free"
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, " %s\t%s\t%.1f of %.0f GB\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.NeedGB, o.HaveGB, o.Hosts, now)
|
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now)
|
||||||
}
|
}
|
||||||
_ = w.Flush()
|
_ = w.Flush()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -186,7 +186,7 @@ func (na *nodeAdd) prepare() error {
|
|||||||
}
|
}
|
||||||
// Where the mirror serves model files, for the node to be asked to reach.
|
// Where the mirror serves model files, for the node to be asked to reach.
|
||||||
if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && cfg.Mirror.API != "" {
|
if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && cfg.Mirror.API != "" {
|
||||||
if mc, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 5*time.Second); err == nil {
|
if mc, err := mirror.NewFailoverClient(append([]string{cfg.Mirror.API}, cfg.Mirror.Peers...), cfg.Mirror.TLS, 5*time.Second); err == nil {
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
if list, err := mc.List(ctx); err == nil {
|
if list, err := mc.List(ctx); err == nil {
|
||||||
o.MirrorBase = list.Base
|
o.MirrorBase = list.Base
|
||||||
|
|||||||
@@ -131,3 +131,33 @@ func TestKeyBudgetsAreWholeUnits(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A limit given alone changes that limit alone: granting a tool keeps the
|
||||||
|
// scope's rate. --replace writes the row whole, as the console's form does.
|
||||||
|
func TestALimitGivenAloneLeavesTheOthersAtTheScope(t *testing.T) {
|
||||||
|
db := filepath.Join(t.TempDir(), "s.db")
|
||||||
|
mustRun(t, db, "org", "add", "org", "Acme")
|
||||||
|
mustRun(t, db, "org", "add", "team", "platform")
|
||||||
|
mustRun(t, db, "org", "add", "project", "gateway")
|
||||||
|
mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--rpm", "30")
|
||||||
|
mustRun(t, db, "keys", "limits", "--scope", "project", "--id", "gateway", "--tools", "jira.*")
|
||||||
|
read := func() store.Limits {
|
||||||
|
s, err := store.Open(context.Background(), db)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer s.Close()
|
||||||
|
all, _ := s.AllLimits(context.Background())
|
||||||
|
for _, l := range all {
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
return store.Limits{}
|
||||||
|
}
|
||||||
|
if l := read(); l.RPM == nil || *l.RPM != 30 || len(l.AllowedTools) != 1 {
|
||||||
|
t.Errorf("after a tool grant: rpm %v, tools %v", l.RPM, l.AllowedTools)
|
||||||
|
}
|
||||||
|
mustRun(t, db, "keys", "limits", "--replace", "--scope", "project", "--id", "gateway", "--tpm", "1000")
|
||||||
|
if l := read(); l.RPM != nil || l.AllowedTools != nil || l.TPM == nil {
|
||||||
|
t.Errorf("after --replace: %+v", l)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -55,6 +55,7 @@ func Root() *cobra.Command {
|
|||||||
newBundleCmd(),
|
newBundleCmd(),
|
||||||
newSimEngineCmd(),
|
newSimEngineCmd(),
|
||||||
newComfyUIGateCmd(),
|
newComfyUIGateCmd(),
|
||||||
|
newStrataServeCmd(),
|
||||||
newDoctorCmd(),
|
newDoctorCmd(),
|
||||||
newNodeCmd(),
|
newNodeCmd(),
|
||||||
newSelfUpgradeCmd(),
|
newSelfUpgradeCmd(),
|
||||||
|
|||||||
+62
-18
@@ -76,6 +76,7 @@ func newServerCmd() *cobra.Command {
|
|||||||
|
|
||||||
cmd := &cobra.Command{
|
cmd := &cobra.Command{
|
||||||
Use: "server",
|
Use: "server",
|
||||||
|
Args: cobra.NoArgs,
|
||||||
Short: "Run the router (phase 1; the control plane arrives in phase 3)",
|
Short: "Run the router (phase 1; the control plane arrives in phase 3)",
|
||||||
Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
|
Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
|
||||||
"In this phase the replica list is static: give it in the config file, or\n" +
|
"In this phase the replica list is static: give it in the config file, or\n" +
|
||||||
@@ -138,6 +139,26 @@ func newServerCmd() *cobra.Command {
|
|||||||
_ = shutdownTracing(sctx)
|
_ = shutdownTracing(sctx)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
|
// The ports before anything else: a second server on this machine,
|
||||||
|
// whose ports are taken, must stop here, not go on to compete for
|
||||||
|
// the lease and lead a cluster no request can reach.
|
||||||
|
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||||
|
}
|
||||||
|
if cfg.MaxConnections > 0 {
|
||||||
|
// Past this, new connections wait in the kernel's backlog
|
||||||
|
// rather than each taking a goroutine and its buffers.
|
||||||
|
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||||
|
}
|
||||||
|
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||||
|
if err != nil {
|
||||||
|
_ = apiLn.Close()
|
||||||
|
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||||
|
}
|
||||||
|
defer apiLn.Close()
|
||||||
|
defer adminLn.Close()
|
||||||
|
|
||||||
rt, err := router.New(cfg, log)
|
rt, err := router.New(cfg, log)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -418,6 +439,9 @@ func newServerCmd() *cobra.Command {
|
|||||||
}
|
}
|
||||||
// The cluster's settings, each change checked against this
|
// The cluster's settings, each change checked against this
|
||||||
// machine's own keys before it is kept.
|
// machine's own keys before it is kept.
|
||||||
|
api.SetMCP(st)
|
||||||
|
rt.SetAdminMCP(http.HandlerFunc(api.MCP))
|
||||||
|
api.SetAuth(&adminapi.AuthAdmin{Status: rt.AuthStatus, TestLDAP: rt.TestDirectory, Secrets: st, Log: log})
|
||||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||||
Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
|
Check: func(body []byte) error { return checkClusterSettings(configPath, body) }})
|
||||||
// And the same document on the portal, for administrators.
|
// And the same document on the portal, for administrators.
|
||||||
@@ -543,18 +567,33 @@ func newServerCmd() *cobra.Command {
|
|||||||
// rather than turning into a broken admin page hours later when
|
// rather than turning into a broken admin page hours later when
|
||||||
// somebody finally opens it.
|
// somebody finally opens it.
|
||||||
if cfg.Mirror.API != "" {
|
if cfg.Mirror.API != "" {
|
||||||
mc, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, cfg.Mirror.Timeout)
|
mirrors := append([]string{cfg.Mirror.API}, cfg.Mirror.Peers...)
|
||||||
|
mc, err := mirror.NewFailoverClient(mirrors, cfg.Mirror.TLS, cfg.Mirror.Timeout)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("the model mirror: %w", err)
|
return fmt.Errorf("the model mirror: %w", err)
|
||||||
}
|
}
|
||||||
|
// Mirrors kept in step: a node is given a model's file on one
|
||||||
|
// that answers while the one the catalog names is down.
|
||||||
|
if len(cfg.Mirror.Peers) > 0 {
|
||||||
|
src, err := mirror.NewSources(mirrors, cfg.Mirror.TLS, log)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("the model mirrors: %w", err)
|
||||||
|
}
|
||||||
|
go src.Run(ctx)
|
||||||
|
agentsvc.SetSourceRewrite(src.Rewrite)
|
||||||
|
}
|
||||||
rt.SetMirror(mc)
|
rt.SetMirror(mc)
|
||||||
|
if adminAPI != nil {
|
||||||
|
// A model removed with its files is removed from it too.
|
||||||
|
adminAPI.SetPurgeMirror(mc)
|
||||||
|
}
|
||||||
if fwPub != nil {
|
if fwPub != nil {
|
||||||
fwPub.setMirror(mc)
|
fwPub.setMirror(mc)
|
||||||
}
|
}
|
||||||
// Adapters trained on the nodes are kept there too; sending one
|
// Adapters trained on the nodes are kept there too; sending one
|
||||||
// takes longer than the mirror's usual answers.
|
// takes longer than the mirror's usual answers.
|
||||||
if adminAPI != nil {
|
if adminAPI != nil {
|
||||||
if up, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 10*time.Minute); err == nil {
|
if up, err := mirror.NewFailoverClient(mirrors, cfg.Mirror.TLS, 10*time.Minute); err == nil {
|
||||||
adminAPI.SetTrainMirror(up)
|
adminAPI.SetTrainMirror(up)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -572,7 +611,7 @@ func newServerCmd() *cobra.Command {
|
|||||||
// Updates: the catalog in force, the service that
|
// Updates: the catalog in force, the service that
|
||||||
// changes it, and what replaces replicas.
|
// changes it, and what replaces replicas.
|
||||||
Current: func() []catalog.Model { return admin.Catalog().Models() },
|
Current: func() []catalog.Model { return admin.Catalog().Models() },
|
||||||
Models: catalogSvc, Restart: admin.Reconciler().Restart,
|
Models: catalogSvc, Restart: admin.Reconciler().Restart, RestartInPlace: admin.Reconciler().RestartInPlace,
|
||||||
Pool: adminAPI.PoolNodes, Notify: notifier,
|
Pool: adminAPI.PoolNodes, Notify: notifier,
|
||||||
// Canaries: how the router saw each side serve.
|
// Canaries: how the router saw each side serve.
|
||||||
CanaryStats: rt.CanaryStats, ResetCanary: rt.ResetCanary,
|
CanaryStats: rt.CanaryStats, ResetCanary: rt.ResetCanary,
|
||||||
@@ -585,6 +624,7 @@ func newServerCmd() *cobra.Command {
|
|||||||
// A judge a policy names and the catalog lacks is
|
// A judge a policy names and the catalog lacks is
|
||||||
// fetched and served, on the leader.
|
// fetched and served, on the leader.
|
||||||
keeper := &judgeKeeper{
|
keeper := &judgeKeeper{
|
||||||
|
grace: judgeGrace,
|
||||||
library: gate.Library,
|
library: gate.Library,
|
||||||
models: func() []catalog.Model { return admin.Catalog().Models() },
|
models: func() []catalog.Model { return admin.Catalog().Models() },
|
||||||
serve: serve,
|
serve: serve,
|
||||||
@@ -593,6 +633,25 @@ func newServerCmd() *cobra.Command {
|
|||||||
},
|
},
|
||||||
gpus: func() []adminapi.ClassUse { return adminAPI.Summary().GPUs },
|
gpus: func() []adminapi.ClassUse { return adminAPI.Summary().GPUs },
|
||||||
log: log,
|
log: log,
|
||||||
|
// An earlier install's judge, moved to today's figures.
|
||||||
|
profiles: catalogSvc,
|
||||||
|
restart: admin.Reconciler().Restart,
|
||||||
|
replicas: func(model string) []string {
|
||||||
|
var ids []string
|
||||||
|
for _, r := range admin.Reconciler().ReadyReplicas() {
|
||||||
|
if r.Model == model {
|
||||||
|
ids = append(ids, r.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ids
|
||||||
|
},
|
||||||
|
nodeCPUs: func() []int {
|
||||||
|
var out []int
|
||||||
|
for _, n := range admin.Nodes() {
|
||||||
|
out = append(out, n.CPUCount)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
},
|
||||||
}
|
}
|
||||||
go keeper.run(ctx, 30*time.Second)
|
go keeper.run(ctx, 30*time.Second)
|
||||||
}
|
}
|
||||||
@@ -642,21 +701,6 @@ func newServerCmd() *cobra.Command {
|
|||||||
ReadTimeout: time.Minute,
|
ReadTimeout: time.Minute,
|
||||||
})
|
})
|
||||||
|
|
||||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
|
||||||
}
|
|
||||||
if cfg.MaxConnections > 0 {
|
|
||||||
// Past this, new connections wait in the kernel's backlog
|
|
||||||
// rather than each taking a goroutine and its buffers.
|
|
||||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
|
||||||
}
|
|
||||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
|
||||||
if err != nil {
|
|
||||||
_ = apiLn.Close()
|
|
||||||
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A metrics-only listener, when one is configured. Started after
|
// A metrics-only listener, when one is configured. Started after
|
||||||
// AdminHandler, which is where the registry it serves is built.
|
// AdminHandler, which is where the registry it serves is built.
|
||||||
var metricsSrv *http.Server
|
var metricsSrv *http.Server
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// `llmgw server check` once ran a whole second server: words after server
|
||||||
|
// that name no subcommand are refused.
|
||||||
|
func TestServerTakesNoWordsItDoesNotKnow(t *testing.T) {
|
||||||
|
cmd := newServerCmd()
|
||||||
|
cmd.SetArgs([]string{"check", "--config", "/nonexistent"})
|
||||||
|
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||||
|
if err := cmd.Execute(); err == nil || !strings.Contains(err.Error(), "unknown command") {
|
||||||
|
t.Errorf("err %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A second server whose port is taken stops at once, before the router or
|
||||||
|
// any control plane is built.
|
||||||
|
func TestAServerWhosePortIsTakenStopsFirst(t *testing.T) {
|
||||||
|
taken, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer taken.Close()
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
cmd := newServerCmd()
|
||||||
|
cmd.SetArgs([]string{"--listen", taken.Addr().String(), "--admin-listen", "127.0.0.1:0",
|
||||||
|
"--endpoint", "a=http://127.0.0.1:1=m"})
|
||||||
|
cmd.SilenceUsage, cmd.SilenceErrors = true, true
|
||||||
|
if err := cmd.ExecuteContext(ctx); err == nil || !strings.Contains(err.Error(), "listening on") {
|
||||||
|
t.Errorf("err %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"encoding/pem"
|
"encoding/pem"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"hash/fnv"
|
||||||
"io"
|
"io"
|
||||||
"net"
|
"net"
|
||||||
"os"
|
"os"
|
||||||
@@ -44,6 +45,11 @@ type serverAdd struct {
|
|||||||
// addr is the machine's address as given; ip what the others reach it at.
|
// addr is the machine's address as given; ip what the others reach it at.
|
||||||
addr, ip, name string
|
addr, ip, name string
|
||||||
out io.Writer
|
out io.Writer
|
||||||
|
// priority is its keepalived priority, under this machine's 110; 0 is
|
||||||
|
// 100. Two standbys get two, so which takes the address is not a tie.
|
||||||
|
priority int
|
||||||
|
// quiet leaves out the closing sentence, for init, which says its own.
|
||||||
|
quiet bool
|
||||||
}
|
}
|
||||||
|
|
||||||
func newServerAddCmd() *cobra.Command {
|
func newServerAddCmd() *cobra.Command {
|
||||||
@@ -148,7 +154,9 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
|||||||
return fmt.Errorf("%s: %w", s.name, err)
|
return fmt.Errorf("%s: %w", s.name, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
fmt.Fprintf(sa.out, "\n%s is a control plane of this cluster, standing by while this one leads.\n", sa.name)
|
if !sa.quiet {
|
||||||
|
fmt.Fprintf(sa.out, "\n%s is a control plane of this cluster, standing by while this one leads.\n", sa.name)
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -511,18 +519,31 @@ func (sa *serverAdd) floating(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
here := pgha.Local{}
|
here := pgha.Local{}
|
||||||
|
priority := sa.priority
|
||||||
|
if priority <= 0 {
|
||||||
|
priority = 100
|
||||||
|
}
|
||||||
for _, m := range []struct {
|
for _, m := range []struct {
|
||||||
r onboard.Remote
|
r onboard.Remote
|
||||||
name string
|
name string
|
||||||
priority int
|
priority int
|
||||||
}{{here, "this machine", 110}, {sa.remote, sa.name, 100}} {
|
}{{here, "this machine", 110}, {sa.remote, sa.name, priority}} {
|
||||||
|
// The interface on the address's network, by its route: not the
|
||||||
|
// route to the address, which is lo on the machine holding it
|
||||||
|
// already, as a run after the first finds it.
|
||||||
out, err := m.r.Run(ctx, fmt.Sprintf(`set -e
|
out, err := m.r.Run(ctx, fmt.Sprintf(`set -e
|
||||||
command -v keepalived >/dev/null 2>&1 || { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y keepalived >/dev/null; }
|
command -v keepalived >/dev/null 2>&1 || { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y keepalived >/dev/null; }
|
||||||
ip -o route get %s | awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}'`, vip), nil)
|
dev() { awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}'; }
|
||||||
|
d=$(ip -o -4 route show %s | dev)
|
||||||
|
[ -n "$d" ] && [ "$d" != lo ] || d=$(ip -o route get %s | dev)
|
||||||
|
echo "$d"`, nw.String(), vip), nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
|
return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
|
||||||
}
|
}
|
||||||
iface := strings.TrimSpace(string(out))
|
iface := strings.TrimSpace(string(out))
|
||||||
|
if iface == "" || iface == "lo" {
|
||||||
|
return fmt.Errorf("on %s: no interface on %s's network", m.name, sa.vip)
|
||||||
|
}
|
||||||
conf := keepalivedConf(iface, vip.String(), prefix, m.priority, pass)
|
conf := keepalivedConf(iface, vip.String(), prefix, m.priority, pass)
|
||||||
if out, err := m.r.Run(ctx, `set -e
|
if out, err := m.r.Run(ctx, `set -e
|
||||||
sudo install -d -m 0755 /etc/keepalived
|
sudo install -d -m 0755 /etc/keepalived
|
||||||
@@ -560,6 +581,17 @@ func vrrpPassword(etcDir string) (string, error) {
|
|||||||
return pass, nil
|
return pass, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// vrrpRouterID is the VRRP router id for a floating address: from the
|
||||||
|
// address, so two clusters on one network, each with its own address, do
|
||||||
|
// not answer each other's adverts — with one id for all, each logged the
|
||||||
|
// other's as an invalid password, and dropped its own address's
|
||||||
|
// announcements with them.
|
||||||
|
func vrrpRouterID(vip string) int {
|
||||||
|
h := fnv.New32a()
|
||||||
|
_, _ = h.Write([]byte(vip))
|
||||||
|
return int(h.Sum32()%255) + 1
|
||||||
|
}
|
||||||
|
|
||||||
func keepalivedConf(iface, vip string, prefix, priority int, pass string) string {
|
func keepalivedConf(iface, vip string, prefix, priority int, pass string) string {
|
||||||
return fmt.Sprintf(`# llmgw: the gateway's floating address, on whichever control plane leads.
|
return fmt.Sprintf(`# llmgw: the gateway's floating address, on whichever control plane leads.
|
||||||
vrrp_script llmgw_leader {
|
vrrp_script llmgw_leader {
|
||||||
@@ -572,7 +604,7 @@ vrrp_script llmgw_leader {
|
|||||||
vrrp_instance llmgw {
|
vrrp_instance llmgw {
|
||||||
state BACKUP
|
state BACKUP
|
||||||
interface %s
|
interface %s
|
||||||
virtual_router_id 51
|
virtual_router_id %d
|
||||||
priority %d
|
priority %d
|
||||||
advert_int 1
|
advert_int 1
|
||||||
authentication {
|
authentication {
|
||||||
@@ -586,7 +618,7 @@ vrrp_instance llmgw {
|
|||||||
llmgw_leader
|
llmgw_leader
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
`, iface, priority, pass, vip, prefix)
|
`, iface, vrrpRouterID(vip), priority, pass, vip, prefix)
|
||||||
}
|
}
|
||||||
|
|
||||||
// clientCapable says a control plane's certificate can also present itself
|
// clientCapable says a control plane's certificate can also present itself
|
||||||
|
|||||||
+158
-28
@@ -57,10 +57,12 @@ WantedBy=multi-user.target
|
|||||||
// it again finishes an install that stopped halfway and changes nothing on
|
// it again finishes an install that stopped halfway and changes nothing on
|
||||||
// one that did not.
|
// one that did not.
|
||||||
type serverInit struct {
|
type serverInit struct {
|
||||||
etcDir string // /etc/llmgw
|
// serviceActive says whether llmgw-server runs; nil asks systemctl.
|
||||||
stateDir string // /var/lib/llmgw
|
serviceActive func() bool
|
||||||
pkiDir string // where the authority lives
|
etcDir string // /etc/llmgw
|
||||||
unitPath string // "" leaves systemd alone
|
stateDir string // /var/lib/llmgw
|
||||||
|
pkiDir string // where the authority lives
|
||||||
|
unitPath string // "" leaves systemd alone
|
||||||
|
|
||||||
// user owns the state and reads the key; "" means no user is created and
|
// user owns the state and reads the key; "" means no user is created and
|
||||||
// nothing is chowned, which is what a test wants.
|
// nothing is chowned, which is what a test wants.
|
||||||
@@ -80,6 +82,16 @@ type serverInit struct {
|
|||||||
// setup screen does with the password it was given; when set, patroni
|
// setup screen does with the password it was given; when set, patroni
|
||||||
// only names them.
|
// only names them.
|
||||||
patroniNodes []pgha.Node
|
patroniNodes []pgha.Node
|
||||||
|
// judgeMachines are machines for the guardrails' judges alone, added as
|
||||||
|
// nodes labelled for them; judgeCPUs their cores, once added.
|
||||||
|
judgeMachines []judgeMachine
|
||||||
|
judgeCPUs []int
|
||||||
|
// secondMirror is which machine of the store, 1 or 2, keeps a second
|
||||||
|
// mirror in step with this machine's; 0 is none.
|
||||||
|
secondMirror int
|
||||||
|
// judgeNote is what timing the judge found, said before the next
|
||||||
|
// steps when it is not fine.
|
||||||
|
judgeNote string
|
||||||
// noMirror, noNode and noModel turn off the parts of a first install that
|
// noMirror, noNode and noModel turn off the parts of a first install that
|
||||||
// make it serve something: a mirror, this machine as a node, and a small
|
// make it serve something: a mirror, this machine as a node, and a small
|
||||||
// model on it.
|
// model on it.
|
||||||
@@ -95,9 +107,34 @@ type serverInit struct {
|
|||||||
// bundle is an offline bundle to install from, and bundleSHA its
|
// bundle is an offline bundle to install from, and bundleSHA its
|
||||||
// manifest's sha256: the cluster is then set up offline.
|
// manifest's sha256: the cluster is then set up offline.
|
||||||
bundle, bundleSHA string
|
bundle, bundleSHA string
|
||||||
// mirrorPath is the mirror's directory here; "" is /srv/models. A seam
|
// mirrorPath is the mirror's directory here; "" is /srv/models.
|
||||||
// for tests.
|
|
||||||
mirrorPath string
|
mirrorPath string
|
||||||
|
|
||||||
|
// What setup lets a person choose besides the parts: zero is the
|
||||||
|
// default (serverinit_choices.go).
|
||||||
|
starter starterSpec
|
||||||
|
starterCtx int
|
||||||
|
judge judgeSpec
|
||||||
|
// language is the portal's for a browser that asks for neither of its
|
||||||
|
// own; "" leaves it English.
|
||||||
|
language string
|
||||||
|
signupOpen bool
|
||||||
|
defaultRPM, defaultTPM int
|
||||||
|
maxKeyTTL time.Duration
|
||||||
|
// media has the nodes install ComfyUI and the leader add a picture model
|
||||||
|
// and a 3D model once a card fits one.
|
||||||
|
media bool
|
||||||
|
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
|
||||||
|
evictAfterIdle, minLifetime time.Duration
|
||||||
|
// standbys makes each machine of the replicated store a control plane
|
||||||
|
// too, standing by while this one leads: three machines, each able to
|
||||||
|
// take over. vip is a floating address, as 10.0.0.160/24, kept on
|
||||||
|
// whichever leads.
|
||||||
|
standbys bool
|
||||||
|
vip string
|
||||||
|
// cpus is this machine's CPU count for dividing it between init's
|
||||||
|
// models; zero asks the machine. A seam for tests.
|
||||||
|
cpus int
|
||||||
// pgSetup prepares PostgreSQL and returns the store address; a seam for
|
// pgSetup prepares PostgreSQL and returns the store address; a seam for
|
||||||
// tests. Nil does it for real.
|
// tests. Nil does it for real.
|
||||||
pgSetup func(context.Context) (string, error)
|
pgSetup func(context.Context) (string, error)
|
||||||
@@ -122,7 +159,10 @@ func newDefaultServerInit() serverInit {
|
|||||||
|
|
||||||
func newServerInitCmd() *cobra.Command {
|
func newServerInitCmd() *cobra.Command {
|
||||||
in := newDefaultServerInit()
|
in := newDefaultServerInit()
|
||||||
var plain bool
|
var (
|
||||||
|
plain bool
|
||||||
|
starter, judgeCats string
|
||||||
|
)
|
||||||
cmd := &cobra.Command{
|
cmd := &cobra.Command{
|
||||||
Use: "init",
|
Use: "init",
|
||||||
Short: "Set this machine up as a control plane and start it",
|
Short: "Set this machine up as a control plane and start it",
|
||||||
@@ -158,6 +198,9 @@ func newServerInitCmd() *cobra.Command {
|
|||||||
if os.Geteuid() != 0 {
|
if os.Geteuid() != 0 {
|
||||||
return errors.New("init creates a user, writes /etc and starts a service: run it as root")
|
return errors.New("init creates a user, writes /etc and starts a service: run it as root")
|
||||||
}
|
}
|
||||||
|
if err := in.readChoices(starter, judgeCats); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
in.out = cmd.OutOrStdout()
|
in.out = cmd.OutOrStdout()
|
||||||
if !plain && isTerminal(in.out) {
|
if !plain && isTerminal(in.out) {
|
||||||
return in.runInitTUI(cmd.Context(), in.out)
|
return in.runInitTUI(cmd.Context(), in.out)
|
||||||
@@ -191,6 +234,38 @@ func newServerInitCmd() *cobra.Command {
|
|||||||
f.StringArrayVar(&in.patroni, "patroni", nil,
|
f.StringArrayVar(&in.patroni, "patroni", nil,
|
||||||
"user@host of a machine to keep the store on, replicated under Patroni; give two or more. "+
|
"user@host of a machine to keep the store on, replicated under Patroni; give two or more. "+
|
||||||
"Installs PostgreSQL, Patroni and etcd there over ssh, and moves the existing store in")
|
"Installs PostgreSQL, Patroni and etcd there over ssh, and moves the existing store in")
|
||||||
|
|
||||||
|
// What server.yaml is written with; each is in it only when init writes
|
||||||
|
// it, never into one already there.
|
||||||
|
f.StringVar(&in.mirrorPath, "mirror-dir", mirrorDir, "the directory the mirror on this machine keeps model files in")
|
||||||
|
f.StringVar(&starter, "starter", starterName,
|
||||||
|
"the starter model: "+strings.Join(starterNames(), ", ")+", or hf:<owner>/<repo>/<file>.gguf")
|
||||||
|
f.IntVar(&in.starterCtx, "starter-context", defaultContext, "the context, in tokens, a client may use of the starter model")
|
||||||
|
f.StringVar(&in.judge.Fail, "judge-fail", "closed",
|
||||||
|
"a request no judge read in time: closed refuses it, open serves it flagged")
|
||||||
|
f.StringVar(&judgeCats, "judge-categories", strings.Join(defaultJudgeCategories, ","),
|
||||||
|
"what the judge refuses, comma-separated, of: "+strings.Join(judgeCategories, ", "))
|
||||||
|
f.IntVar(&in.judge.Replicas, "judge-replicas", 2, "how many copies of the judge run, each on CPUs")
|
||||||
|
f.IntVar(&in.secondMirror, "second-mirror", 0,
|
||||||
|
"with --patroni, a second mirror on the store's machine 1 or 2, kept in step with this one's (0: none)")
|
||||||
|
f.Var(judgeMachineFlag{&in.judgeMachines}, "judge-node",
|
||||||
|
"user@host of a machine for the guardrails' judges alone: added as a node, and the judges run only there; repeatable")
|
||||||
|
f.DurationVar(&in.judge.Budget, "judge-budget", 5*time.Second, "how long a request waits for the judge")
|
||||||
|
f.StringVar(&in.language, "language", "", "the portal's language for a browser that asks for neither: en or tr")
|
||||||
|
f.BoolVar(&in.signupOpen, "signup-open", false, "let anyone ask for an account on the portal, to be approved by an administrator")
|
||||||
|
f.IntVar(&in.defaultRPM, "default-rpm", defaultRPM, "requests a minute a key may make when nothing gives it a limit")
|
||||||
|
f.IntVar(&in.defaultTPM, "default-tpm", defaultTPM, "tokens a minute a key may use when nothing gives it a limit")
|
||||||
|
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
|
||||||
|
f.BoolVar(&in.media, "media", false,
|
||||||
|
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
|
||||||
|
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
|
||||||
|
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
|
||||||
|
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
|
||||||
|
"how long a model just loaded on demand is safe from that (0: 5m)")
|
||||||
|
f.BoolVar(&in.standbys, "standbys", false,
|
||||||
|
"with --patroni, make each of those machines a control plane too, standing by while this one leads")
|
||||||
|
f.StringVar(&in.vip, "vip", "",
|
||||||
|
"with --standbys, a floating address for whichever control plane leads, as 10.0.0.160/24 (keepalived on each)")
|
||||||
return cmd
|
return cmd
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -198,6 +273,9 @@ func (in *serverInit) run(ctx context.Context) error {
|
|||||||
if err := in.checkBundle(); err != nil {
|
if err := in.checkBundle(); err != nil {
|
||||||
return fmt.Errorf("offline bundle: %w", err)
|
return fmt.Errorf("offline bundle: %w", err)
|
||||||
}
|
}
|
||||||
|
if err := in.checkMemory(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
steps := []struct {
|
steps := []struct {
|
||||||
name string
|
name string
|
||||||
fn func(context.Context) error
|
fn func(context.Context) error
|
||||||
@@ -258,6 +336,11 @@ func (in *serverInit) run(ctx context.Context) error {
|
|||||||
if err := in.ensureWebhooks(ctx); err != nil {
|
if err := in.ensureWebhooks(ctx); err != nil {
|
||||||
fmt.Fprintf(in.out, " !! webhooks: %v\n", err)
|
fmt.Fprintf(in.out, " !! webhooks: %v\n", err)
|
||||||
}
|
}
|
||||||
|
if hasSystemd && len(in.judgeMachines) > 0 {
|
||||||
|
if err := in.ensureJudgeNodes(ctx); err != nil {
|
||||||
|
return fmt.Errorf("judge nodes: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
if in.judgeWanted() && in.configJudgesWith(guardModelName) {
|
if in.judgeWanted() && in.configJudgesWith(guardModelName) {
|
||||||
// Not fatal: without its judge the baseline policy still checks for
|
// Not fatal: without its judge the baseline policy still checks for
|
||||||
// secrets and injection, and a judged request is served flagged.
|
// secrets and injection, and a judged request is served flagged.
|
||||||
@@ -274,6 +357,20 @@ func (in *serverInit) run(ctx context.Context) error {
|
|||||||
return fmt.Errorf("starter model: %w", err)
|
return fmt.Errorf("starter model: %w", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if in.standbys {
|
||||||
|
if err := in.ensureStandbys(ctx); err != nil {
|
||||||
|
return fmt.Errorf("standby control planes: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if hasSystemd && in.secondMirror > 0 {
|
||||||
|
if err := in.ensureSecondMirror(ctx); err != nil {
|
||||||
|
// Not fatal: the first mirror serves; said, and how to try again.
|
||||||
|
fmt.Fprintf(in.out, " !! second mirror: %v; later: sudo llmgw mirror add-peer <user>@<machine>\n", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if hasSystemd && in.judgeWanted() && in.configJudgesWith(guardModelName) {
|
||||||
|
in.checkJudge(ctx)
|
||||||
|
}
|
||||||
in.nextSteps()
|
in.nextSteps()
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -281,12 +378,7 @@ func (in *serverInit) run(ctx context.Context) error {
|
|||||||
// judgeWanted is whether this install fetches the judge model, and so
|
// judgeWanted is whether this install fetches the judge model, and so
|
||||||
// whether the baseline policy asks it.
|
// whether the baseline policy asks it.
|
||||||
func (in *serverInit) judgeWanted() bool {
|
func (in *serverInit) judgeWanted() bool {
|
||||||
if in.noGuardModel || (in.noMirror && in.mirrorURL == "") || in.unitPath == "" {
|
if !in.judgeComes() || in.unitPath == "" {
|
||||||
return false
|
|
||||||
}
|
|
||||||
// Offline, only a judge the bundle carries: a policy asking a judge
|
|
||||||
// that never comes refuses every request.
|
|
||||||
if in.bundle != "" && !in.bundleCarries(guardModelName) {
|
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
_, err := exec.LookPath("systemctl")
|
_, err := exec.LookPath("systemctl")
|
||||||
@@ -310,29 +402,45 @@ func (in *serverInit) configJudgesWith(model string) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// guardSection is server.yaml's guard: the baseline every request passes,
|
// guardSection is server.yaml's guard: the baseline every request passes,
|
||||||
// with the judge model when there is one.
|
// with the judge model, as init sets it up unless told otherwise, when there
|
||||||
|
// is one.
|
||||||
func guardSection(judge bool) string {
|
func guardSection(judge bool) string {
|
||||||
j := ` # A judge model reads what patterns cannot: violence, weapons, abuse,
|
if !judge {
|
||||||
|
return guardSectionFor(nil)
|
||||||
|
}
|
||||||
|
j := (&serverInit{}).judgeFor()
|
||||||
|
return guardSectionFor(&j)
|
||||||
|
}
|
||||||
|
|
||||||
|
// guardSectionFor is server.yaml's guard with this judge, or none.
|
||||||
|
func guardSectionFor(j *judgeSpec) string {
|
||||||
|
fail := "closed"
|
||||||
|
jb := ` # A judge model reads what patterns cannot: violence, weapons, abuse,
|
||||||
# self-harm. Installed by init; --no-guard-model leaves it out.
|
# self-harm. Installed by init; --no-guard-model leaves it out.
|
||||||
# judge: {model: ` + guardModelName + `, input: block, output: block, ...}
|
# judge: {model: ` + guardModelName + `, input: block, output: block, ...}
|
||||||
`
|
`
|
||||||
if judge {
|
if j != nil {
|
||||||
j = ` # A judge model reads what patterns cannot: violence, crime, sexual
|
fail = j.Fail
|
||||||
# content, self-harm, jailbreaks. Two replicas run on CPUs, about two
|
copies := map[int]string{1: "One replica runs", 2: "Two replicas run", 3: "Three replicas run"}[j.Replicas]
|
||||||
|
if copies == "" {
|
||||||
|
copies = fmt.Sprintf("%d replicas run", j.Replicas)
|
||||||
|
}
|
||||||
|
jb = ` # A judge model reads what patterns cannot: violence, crime, sexual
|
||||||
|
# content, self-harm, jailbreaks. ` + copies + ` on CPUs, about two
|
||||||
# seconds for a short text; an answer is read every 600 characters as
|
# seconds for a short text; an answer is read every 600 characters as
|
||||||
# it streams. With fail: closed, a request no judge can read within
|
# it streams. With fail: closed, a request no judge can read within
|
||||||
# latency_budget is refused (503); open serves it, flagged, unjudged.
|
# latency_budget is refused (503); open serves it, flagged, unjudged.
|
||||||
# PII is the patterns' (above), and copyright, unethical acts and
|
# PII is the patterns' (above), and copyright, unethical acts and
|
||||||
# politically sensitive topics are left out: they refuse too much
|
# politically sensitive topics are left out unless chosen: they refuse
|
||||||
# ordinary work. Controversial counts as safe; controversial: unsafe
|
# too much ordinary work. Controversial counts as safe; controversial:
|
||||||
# makes it count. With a GPU: llmgw ctl guard judge enable --gpu, then
|
# unsafe makes it count. With a GPU: llmgw ctl guard judge enable --gpu,
|
||||||
# llmgw ctl guard judge use ` + defaultGPUJudge + ` ` + guardModelName + `.
|
# then llmgw ctl guard judge use ` + defaultGPUJudge + ` ` + guardModelName + `.
|
||||||
judge:
|
judge:
|
||||||
models: [` + guardModelName + `]
|
models: [` + guardModelName + `]
|
||||||
input: block
|
input: block
|
||||||
output: block
|
output: block
|
||||||
categories: [Violent, Non-violent Illegal Acts, Sexual Content or Sexual Acts, Suicide & Self-Harm, Jailbreak]
|
categories: [` + strings.Join(j.Categories, ", ") + `]
|
||||||
latency_budget: 5s
|
latency_budget: ` + durationYAML(j.Budget) + `
|
||||||
`
|
`
|
||||||
}
|
}
|
||||||
return `# Guardrails (docs/guardrails.md). The baseline applies to every request;
|
return `# Guardrails (docs/guardrails.md). The baseline applies to every request;
|
||||||
@@ -342,13 +450,13 @@ guard:
|
|||||||
default: [baseline]
|
default: [baseline]
|
||||||
policies:
|
policies:
|
||||||
baseline:
|
baseline:
|
||||||
fail: closed
|
fail: ` + fail + `
|
||||||
input:
|
input:
|
||||||
pii: {secret: block, card: redact}
|
pii: {secret: block, card: redact}
|
||||||
injection: flag
|
injection: flag
|
||||||
output:
|
output:
|
||||||
pii: {secret: block, card: redact}
|
pii: {secret: block, card: redact}
|
||||||
` + j
|
` + jb
|
||||||
}
|
}
|
||||||
|
|
||||||
func (in *serverInit) done(format string, a ...any) {
|
func (in *serverInit) done(format string, a ...any) {
|
||||||
@@ -527,6 +635,10 @@ func (in *serverInit) certHosts() []string {
|
|||||||
for _, h := range in.hosts {
|
for _, h := range in.hosts {
|
||||||
add(h)
|
add(h)
|
||||||
}
|
}
|
||||||
|
// The floating address is this machine's whenever it leads.
|
||||||
|
if ip, _, err := net.ParseCIDR(in.vip); err == nil {
|
||||||
|
add(ip.String())
|
||||||
|
}
|
||||||
if name, err := os.Hostname(); err == nil {
|
if name, err := os.Hostname(); err == nil {
|
||||||
add(name)
|
add(name)
|
||||||
}
|
}
|
||||||
@@ -565,8 +677,13 @@ func (in *serverInit) ensureConfig(context.Context) error {
|
|||||||
if contact == "" {
|
if contact == "" {
|
||||||
contact = "platform@example.com"
|
contact = "platform@example.com"
|
||||||
}
|
}
|
||||||
body := fmt.Sprintf(serverConfigTemplate, in.etcDir, in.etcDir, in.etcDir,
|
var judge *judgeSpec
|
||||||
in.storeAddress(), contact, guardSection(in.judgeWanted()))
|
if in.judgeWanted() {
|
||||||
|
j := in.judgeFor()
|
||||||
|
judge = &j
|
||||||
|
}
|
||||||
|
body := in.tuneConfig(fmt.Sprintf(serverConfigTemplate, in.etcDir, in.etcDir, in.etcDir,
|
||||||
|
in.storeAddress(), contact, guardSectionFor(judge)))
|
||||||
if err := os.WriteFile(path, []byte(body), 0o640); err != nil {
|
if err := os.WriteFile(path, []byte(body), 0o640); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -813,6 +930,9 @@ func (in *serverInit) waitHealthy(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (in *serverInit) nextSteps() {
|
func (in *serverInit) nextSteps() {
|
||||||
|
if in.judgeNote != "" {
|
||||||
|
fmt.Fprint(in.out, "\n"+in.judgeNote)
|
||||||
|
}
|
||||||
fmt.Fprint(in.out, `
|
fmt.Fprint(in.out, `
|
||||||
The control plane is up. Next:
|
The control plane is up. Next:
|
||||||
|
|
||||||
@@ -888,3 +1008,13 @@ func virtualInterface(name string) bool {
|
|||||||
}
|
}
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// judgeComes is whether this setup has a judge to install: one not left
|
||||||
|
// out, a mirror to fetch it from, and, offline, a bundle that carries it —
|
||||||
|
// a policy asking a judge that never comes refuses every request.
|
||||||
|
func (in *serverInit) judgeComes() bool {
|
||||||
|
if in.noGuardModel || (in.noMirror && in.mirrorURL == "") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return in.bundle == "" || in.bundleCarries(guardModelName)
|
||||||
|
}
|
||||||
@@ -0,0 +1,497 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"math"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path"
|
||||||
|
"slices"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||||
|
)
|
||||||
|
|
||||||
|
// What a first install lets a person choose beyond which parts it has: which
|
||||||
|
// starter model, how the judge decides, what the portal speaks, the limits a
|
||||||
|
// key starts with, and whether pictures and 3D come by themselves. Each zero
|
||||||
|
// value is what init did before it could be chosen, so `llmgw server init`
|
||||||
|
// with no flags installs what it always did.
|
||||||
|
//
|
||||||
|
// All of it is written into server.yaml when init writes it, and the cluster's
|
||||||
|
// keys among it become the cluster's first settings when the control plane
|
||||||
|
// first starts. A server.yaml already there is never overwritten, so a second
|
||||||
|
// run changes none of it: the settings page and `llmgw ctl config` do.
|
||||||
|
|
||||||
|
// starterSpec is a model init can fetch and serve so there is something to ask.
|
||||||
|
type starterSpec struct {
|
||||||
|
Name, Alias, Source string
|
||||||
|
// SizeGB is the weights' size, for the memory a node needs; 0 is not known.
|
||||||
|
SizeGB float64
|
||||||
|
About string
|
||||||
|
}
|
||||||
|
|
||||||
|
// starterChoices are the starters offered, smallest first: the first is the
|
||||||
|
// default. Each is a single GGUF file, Q4_K_M.
|
||||||
|
var starterChoices = []starterSpec{
|
||||||
|
{starterName, starterAlias, starterSource, 0.5, "answers at once on any CPU; fast, not clever"},
|
||||||
|
{"qwen2.5-1.5b-instruct", "qwen-1.5b", "hf:Qwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf", 1.1,
|
||||||
|
"still quick on a CPU, and noticeably better"},
|
||||||
|
{"qwen2.5-3b-instruct", "qwen-3b", "hf:Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf", 2.1,
|
||||||
|
"slow on a CPU, quick on any GPU"},
|
||||||
|
{"qwen3-4b", "", "hf:Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf", 2.5,
|
||||||
|
"thinks before it answers; a GPU is worth having"},
|
||||||
|
{"qwen3-8b", "", "hf:Qwen/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf", 5.0,
|
||||||
|
"a real assistant; a GPU with 8 GB or more"},
|
||||||
|
{"qwen3-14b", "", "hf:Qwen/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf", 9.0,
|
||||||
|
"better still; a GPU with 16 GB or more"},
|
||||||
|
}
|
||||||
|
|
||||||
|
// starterFrom reads --starter: a name from starterChoices, or the hf: source
|
||||||
|
// of any GGUF file, named after the file.
|
||||||
|
func starterFrom(s string) (starterSpec, error) {
|
||||||
|
s = strings.TrimSpace(s)
|
||||||
|
for _, c := range starterChoices {
|
||||||
|
if s == c.Name || (c.Alias != "" && s == c.Alias) {
|
||||||
|
return c, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if rest, ok := strings.CutPrefix(s, "hf:"); ok && strings.Count(rest, "/") >= 2 && strings.HasSuffix(strings.ToLower(rest), ".gguf") {
|
||||||
|
name := strings.ToLower(strings.TrimSuffix(path.Base(rest), path.Ext(rest)))
|
||||||
|
return starterSpec{Name: name, Source: s, About: "chosen at install"}, nil
|
||||||
|
}
|
||||||
|
var names []string
|
||||||
|
for _, c := range starterChoices {
|
||||||
|
names = append(names, c.Name)
|
||||||
|
}
|
||||||
|
return starterSpec{}, fmt.Errorf("starter %q: one of %s, or hf:<owner>/<repo>/<file>.gguf", s, strings.Join(names, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func starterNames() []string {
|
||||||
|
var out []string
|
||||||
|
for _, c := range starterChoices {
|
||||||
|
out = append(out, c.Name)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// readChoices takes the flags that are not read straight into init: the
|
||||||
|
// starter by its name, the judge's categories by commas. Then it checks them
|
||||||
|
// all.
|
||||||
|
func (in *serverInit) readChoices(starter, categories string) error {
|
||||||
|
if strings.TrimSpace(starter) != "" {
|
||||||
|
s, err := starterFrom(starter)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
in.starter = s
|
||||||
|
}
|
||||||
|
in.judge.Categories = nil
|
||||||
|
for _, c := range strings.Split(categories, ",") {
|
||||||
|
if c = strings.TrimSpace(c); c != "" {
|
||||||
|
in.judge.Categories = append(in.judge.Categories, c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return in.checkChoices()
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkChoices says what among the choices cannot be installed.
|
||||||
|
func (in *serverInit) checkChoices() error {
|
||||||
|
switch in.language {
|
||||||
|
case "", "en", "tr":
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("language %q: en or tr", in.language)
|
||||||
|
}
|
||||||
|
switch in.judge.Fail {
|
||||||
|
case "", "closed", "open":
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("judge fail %q: closed or open", in.judge.Fail)
|
||||||
|
}
|
||||||
|
for _, c := range in.judge.Categories {
|
||||||
|
if !slices.Contains(judgeCategories, c) {
|
||||||
|
return fmt.Errorf("judge category %q: one of %s", c, strings.Join(judgeCategories, ", "))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case in.judge.Replicas < 0 || in.judge.Replicas > 8:
|
||||||
|
return fmt.Errorf("judge replicas %d: from 1 to 8", in.judge.Replicas)
|
||||||
|
case in.judge.Budget < 0 || in.judge.Budget > time.Minute:
|
||||||
|
return fmt.Errorf("judge budget %s: at most a minute", in.judge.Budget)
|
||||||
|
case in.starterCtx < 0 || in.starterCtx > 262144:
|
||||||
|
return fmt.Errorf("starter context %d: at most 262144 tokens", in.starterCtx)
|
||||||
|
case in.starterCtx > 0 && in.starterCtx < 512:
|
||||||
|
return fmt.Errorf("starter context %d: at least 512 tokens", in.starterCtx)
|
||||||
|
case in.defaultRPM < 0 || in.defaultTPM < 0:
|
||||||
|
return fmt.Errorf("default limits cannot be negative")
|
||||||
|
case in.maxKeyTTL < 0 || (in.maxKeyTTL > 0 && in.maxKeyTTL < time.Hour):
|
||||||
|
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
|
||||||
|
case in.evictAfterIdle < 0 || in.minLifetime < 0:
|
||||||
|
return fmt.Errorf("on-demand times cannot be negative")
|
||||||
|
case in.vip != "" && !in.standbys:
|
||||||
|
return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
|
||||||
|
}
|
||||||
|
if in.vip != "" {
|
||||||
|
ip, _, err := net.ParseCIDR(in.vip)
|
||||||
|
if err != nil || ip.To4() == nil {
|
||||||
|
return fmt.Errorf("floating address %q: an IPv4 address with its prefix, as 10.0.0.160/24", in.vip)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// starterModelSpec is the starter this install serves.
|
||||||
|
func (in *serverInit) starterModelSpec() starterSpec {
|
||||||
|
if in.starter.Source == "" {
|
||||||
|
return starterChoices[0]
|
||||||
|
}
|
||||||
|
return in.starter
|
||||||
|
}
|
||||||
|
|
||||||
|
// minMemoryGB is what a node needs free for the starter: its weights and
|
||||||
|
// room for its context, never less than the 2 GB it always asked.
|
||||||
|
func (s starterSpec) minMemoryGB() int {
|
||||||
|
return max(2, int(math.Ceil(s.SizeGB+1.5)))
|
||||||
|
}
|
||||||
|
|
||||||
|
// servingMemoryMB is what this machine needs, beyond what the control plane
|
||||||
|
// does, for the models it will serve itself: the starter's and the judges'
|
||||||
|
// reservations, and a gigabyte more because the scheduler counts a machine's
|
||||||
|
// memory in whole gigabytes, rounded down — 7.75 GB is 7. Zero when it serves
|
||||||
|
// none of them; what names those it does.
|
||||||
|
//
|
||||||
|
// Asked before anything is installed: the 8 GB machine the guide was first
|
||||||
|
// recorded on finished every step and then had no room for the starter, the
|
||||||
|
// one thing a person asks first.
|
||||||
|
func (in *serverInit) servingMemoryMB(judgesElsewhere bool) (mb int, what string) {
|
||||||
|
if in.noNode {
|
||||||
|
return 0, ""
|
||||||
|
}
|
||||||
|
gb := 0
|
||||||
|
var parts []string
|
||||||
|
if !in.noModel && (!in.noMirror || in.mirrorURL != "") {
|
||||||
|
gb += in.starterModelSpec().minMemoryGB()
|
||||||
|
parts = append(parts, "the starter model")
|
||||||
|
}
|
||||||
|
// Only judges that will be installed: none without a mirror to fetch
|
||||||
|
// them from, or from a bundle that does not carry one.
|
||||||
|
if !judgesElsewhere && in.judgeComes() {
|
||||||
|
n := in.judgeFor().Replicas
|
||||||
|
gb += n * judgeMemoryGB
|
||||||
|
parts = append(parts, fmt.Sprintf("%d judges", n))
|
||||||
|
}
|
||||||
|
if gb == 0 {
|
||||||
|
return 0, ""
|
||||||
|
}
|
||||||
|
return (gb + 1) * 1024, strings.Join(parts, " and ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkMemory refuses, before anything is installed, a machine too small for
|
||||||
|
// the models it would serve as well as the control plane.
|
||||||
|
func (in *serverInit) checkMemory() error {
|
||||||
|
if in.unitPath == "" {
|
||||||
|
return nil // a development run, served by nothing
|
||||||
|
}
|
||||||
|
// Run again on a machine already serving — to change --host, say, as
|
||||||
|
// the docs advise — it installs no models it does not already serve. A
|
||||||
|
// first run that stopped half way is checked again: it still would.
|
||||||
|
if in.alreadyServing() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
meminfo, err := os.ReadFile("/proc/meminfo")
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
total, ok := pgha.MemTotal(string(meminfo))
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
extra, what := in.servingMemoryMB(len(in.judgeMachines) > 0)
|
||||||
|
if extra == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
have, need := int(total>>20), preflight.Needs[preflight.RoleControl].MemoryMB+extra
|
||||||
|
if have >= need {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return fmt.Errorf("memory: this machine has %d MB; the control plane and %s it would serve need %d. "+
|
||||||
|
"Give it more, or leave them off it: --no-model, --no-guard-model, or --no-node to serve models only on nodes added later", have, what, need)
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultContext is the context a client may use of the starter.
|
||||||
|
const defaultContext = 4096
|
||||||
|
|
||||||
|
// starterContext is the context a client may use of the starter.
|
||||||
|
func (in *serverInit) starterContext() int {
|
||||||
|
if in.starterCtx > 0 {
|
||||||
|
return in.starterCtx
|
||||||
|
}
|
||||||
|
return defaultContext
|
||||||
|
}
|
||||||
|
|
||||||
|
// judgeSpec is how the guardrails' judge decides.
|
||||||
|
type judgeSpec struct {
|
||||||
|
// Fail is what is done with a request no judge read in time: closed
|
||||||
|
// refuses it, open serves it flagged.
|
||||||
|
Fail string
|
||||||
|
Categories []string
|
||||||
|
Replicas int
|
||||||
|
Budget time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// judgeCategories are the ones a judge can be asked about, as Qwen3Guard
|
||||||
|
// names them; defaultJudgeCategories those it is asked about unless told.
|
||||||
|
var (
|
||||||
|
judgeCategories = []string{"Violent", "Non-violent Illegal Acts", "Sexual Content or Sexual Acts",
|
||||||
|
"Suicide & Self-Harm", "Jailbreak", "PII", "Unethical Acts", "Politically Sensitive Topics", "Copyright Violation"}
|
||||||
|
defaultJudgeCategories = judgeCategories[:5]
|
||||||
|
)
|
||||||
|
|
||||||
|
// judgeFor is the judge as this install sets it up, the defaults filled in.
|
||||||
|
func (in *serverInit) judgeFor() judgeSpec {
|
||||||
|
j := in.judge
|
||||||
|
if j.Fail == "" {
|
||||||
|
j.Fail = "closed"
|
||||||
|
}
|
||||||
|
if len(j.Categories) == 0 {
|
||||||
|
j.Categories = defaultJudgeCategories
|
||||||
|
}
|
||||||
|
if j.Replicas <= 0 {
|
||||||
|
j.Replicas = 2
|
||||||
|
if in.singleMachine() {
|
||||||
|
j.Replicas = 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if j.Budget <= 0 {
|
||||||
|
j.Budget = 5 * time.Second
|
||||||
|
}
|
||||||
|
return j
|
||||||
|
}
|
||||||
|
|
||||||
|
// singleMachine is an install where this machine is the only node there
|
||||||
|
// will be, as far as init knows: a second judge replica there shares the
|
||||||
|
// same cores and is lost with the same machine, so it buys nothing but
|
||||||
|
// half the speed of each.
|
||||||
|
func (in *serverInit) singleMachine() bool {
|
||||||
|
return !in.noNode && !in.standbys && len(in.patroni) == 0 && len(in.patroniNodes) == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// The configuration's limits, as init writes them unless told otherwise.
|
||||||
|
const (
|
||||||
|
defaultRPM = 60
|
||||||
|
defaultTPM = 200000
|
||||||
|
defaultMaxKeyTTL = 2160 * time.Hour
|
||||||
|
)
|
||||||
|
|
||||||
|
// tuneConfig writes this install's choices into the server.yaml template:
|
||||||
|
// the lines the template has, changed, and the blocks it has not, added.
|
||||||
|
// Unchosen, the template comes back as it was.
|
||||||
|
func (in *serverInit) tuneConfig(body string) string {
|
||||||
|
set := func(old, now string) {
|
||||||
|
if old != now {
|
||||||
|
body = strings.Replace(body, old, now, 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if in.signupOpen {
|
||||||
|
set(" signup_open: false", " signup_open: true")
|
||||||
|
}
|
||||||
|
if in.defaultRPM > 0 {
|
||||||
|
set(" default_rpm: 60", " default_rpm: "+strconv.Itoa(in.defaultRPM))
|
||||||
|
}
|
||||||
|
if in.defaultTPM > 0 {
|
||||||
|
set(" default_tpm: 200000", " default_tpm: "+strconv.Itoa(in.defaultTPM))
|
||||||
|
}
|
||||||
|
if in.maxKeyTTL > 0 && in.maxKeyTTL != defaultMaxKeyTTL {
|
||||||
|
set(" max_key_ttl: 2160h # 90 days", fmt.Sprintf(" max_key_ttl: %s # %s", durationYAML(in.maxKeyTTL), days(in.maxKeyTTL)))
|
||||||
|
}
|
||||||
|
if in.language != "" {
|
||||||
|
set(" # language: tr", " language: "+in.language)
|
||||||
|
}
|
||||||
|
var extra []string
|
||||||
|
if in.media {
|
||||||
|
extra = append(extra, "# Pictures and 3D, chosen at install: the nodes install ComfyUI, and the\n"+
|
||||||
|
"# leader adds a picture model and a 3D model by itself once a card fits\n"+
|
||||||
|
"# one ('llmgw ctl model presets auto off' stops it).\n"+
|
||||||
|
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
|
||||||
|
}
|
||||||
|
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
|
||||||
|
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
|
||||||
|
if in.evictAfterIdle > 0 {
|
||||||
|
b += " evict_after_idle: " + durationYAML(in.evictAfterIdle) + "\n"
|
||||||
|
}
|
||||||
|
if in.minLifetime > 0 {
|
||||||
|
b += " min_lifetime: " + durationYAML(in.minLifetime) + "\n"
|
||||||
|
}
|
||||||
|
extra = append(extra, b)
|
||||||
|
}
|
||||||
|
if len(extra) > 0 {
|
||||||
|
body += "\n" + strings.Join(extra, "\n")
|
||||||
|
}
|
||||||
|
return body
|
||||||
|
}
|
||||||
|
|
||||||
|
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
|
||||||
|
func durationYAML(d time.Duration) string {
|
||||||
|
switch {
|
||||||
|
case d == 0:
|
||||||
|
return "0s"
|
||||||
|
case d%time.Hour == 0:
|
||||||
|
return fmt.Sprintf("%dh", d/time.Hour)
|
||||||
|
case d%time.Minute == 0:
|
||||||
|
return fmt.Sprintf("%dm", d/time.Minute)
|
||||||
|
}
|
||||||
|
return d.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func days(d time.Duration) string {
|
||||||
|
n := int(d.Hours() / 24)
|
||||||
|
if n == 1 {
|
||||||
|
return "1 day"
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%d days", n)
|
||||||
|
}
|
||||||
|
|
||||||
|
// initFlags is this install as `llmgw server init`'s flags, for doing it
|
||||||
|
// again from a script; what is left at its default is left out.
|
||||||
|
func (in *serverInit) initFlags() []string {
|
||||||
|
var out []string
|
||||||
|
add := func(flag string, v ...string) {
|
||||||
|
if len(v) == 0 {
|
||||||
|
out = append(out, flag)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
out = append(out, flag+" "+shellWord(v[0]))
|
||||||
|
}
|
||||||
|
if in.admin != "" {
|
||||||
|
add("--admin", in.admin)
|
||||||
|
}
|
||||||
|
for _, h := range in.hosts {
|
||||||
|
add("--host", h)
|
||||||
|
}
|
||||||
|
if in.language != "" {
|
||||||
|
add("--language", in.language)
|
||||||
|
}
|
||||||
|
if in.signupOpen {
|
||||||
|
add("--signup-open")
|
||||||
|
}
|
||||||
|
if in.defaultRPM > 0 && in.defaultRPM != defaultRPM {
|
||||||
|
add("--default-rpm", strconv.Itoa(in.defaultRPM))
|
||||||
|
}
|
||||||
|
if in.defaultTPM > 0 && in.defaultTPM != defaultTPM {
|
||||||
|
add("--default-tpm", strconv.Itoa(in.defaultTPM))
|
||||||
|
}
|
||||||
|
if in.maxKeyTTL > 0 && in.maxKeyTTL != defaultMaxKeyTTL {
|
||||||
|
add("--max-key-ttl", durationYAML(in.maxKeyTTL))
|
||||||
|
}
|
||||||
|
if in.postgres {
|
||||||
|
add("--postgres")
|
||||||
|
}
|
||||||
|
for _, t := range in.patroni {
|
||||||
|
add("--patroni", t)
|
||||||
|
}
|
||||||
|
for _, m := range in.judgeMachines {
|
||||||
|
add("--judge-node", m.target)
|
||||||
|
}
|
||||||
|
if in.secondMirror > 0 {
|
||||||
|
add("--second-mirror", fmt.Sprint(in.secondMirror))
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case in.mirrorURL != "":
|
||||||
|
add("--mirror", in.mirrorURL)
|
||||||
|
case in.noMirror:
|
||||||
|
add("--no-mirror")
|
||||||
|
case in.mirrorPath != "" && in.mirrorPath != mirrorDir:
|
||||||
|
add("--mirror-dir", in.mirrorPath)
|
||||||
|
}
|
||||||
|
if in.bundle != "" {
|
||||||
|
add("--bundle", in.bundle)
|
||||||
|
add("--bundle-sha256", in.bundleSHA)
|
||||||
|
}
|
||||||
|
if in.noNode {
|
||||||
|
add("--no-node")
|
||||||
|
}
|
||||||
|
if in.noModel {
|
||||||
|
add("--no-model")
|
||||||
|
} else {
|
||||||
|
if s := in.starterModelSpec(); s.Name != starterName {
|
||||||
|
v := s.Name
|
||||||
|
if !slices.ContainsFunc(starterChoices, func(c starterSpec) bool { return c.Name == s.Name }) {
|
||||||
|
v = s.Source
|
||||||
|
}
|
||||||
|
add("--starter", v)
|
||||||
|
}
|
||||||
|
if in.starterCtx > 0 && in.starterCtx != defaultContext {
|
||||||
|
add("--starter-context", strconv.Itoa(in.starterCtx))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if in.noGuardModel {
|
||||||
|
add("--no-guard-model")
|
||||||
|
} else {
|
||||||
|
// Against the defaults for an install of the same shape: one judge
|
||||||
|
// on one machine, two otherwise.
|
||||||
|
shape := serverInit{noNode: in.noNode, standbys: in.standbys, patroni: in.patroni, patroniNodes: in.patroniNodes}
|
||||||
|
j, d := in.judgeFor(), shape.judgeFor()
|
||||||
|
if j.Fail != d.Fail {
|
||||||
|
add("--judge-fail", j.Fail)
|
||||||
|
}
|
||||||
|
if !slices.Equal(j.Categories, d.Categories) {
|
||||||
|
add("--judge-categories", strings.Join(j.Categories, ","))
|
||||||
|
}
|
||||||
|
if j.Replicas != d.Replicas {
|
||||||
|
add("--judge-replicas", strconv.Itoa(j.Replicas))
|
||||||
|
}
|
||||||
|
if j.Budget != d.Budget {
|
||||||
|
add("--judge-budget", durationYAML(j.Budget))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if in.noFirewall {
|
||||||
|
add("--no-firewall")
|
||||||
|
}
|
||||||
|
if in.standbys {
|
||||||
|
add("--standbys")
|
||||||
|
}
|
||||||
|
if in.vip != "" {
|
||||||
|
add("--vip", in.vip)
|
||||||
|
}
|
||||||
|
for _, w := range in.webhooks {
|
||||||
|
for _, s := range strings.Split(w, ",") {
|
||||||
|
if s = strings.TrimSpace(s); s != "" {
|
||||||
|
add("--webhook", s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if in.media {
|
||||||
|
add("--media")
|
||||||
|
}
|
||||||
|
if in.evictAfterIdle > 0 {
|
||||||
|
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
|
||||||
|
}
|
||||||
|
if in.minLifetime > 0 {
|
||||||
|
add("--min-lifetime", durationYAML(in.minLifetime))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// shellWord quotes a word for a shell when it needs it.
|
||||||
|
func shellWord(s string) string {
|
||||||
|
if s != "" && !strings.ContainsAny(s, " \t\n'\"\\$`&;|<>(){}*?[]#~!") {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
||||||
|
}
|
||||||
|
|
||||||
|
// alreadyServing reports whether this machine's gateway is set up and
|
||||||
|
// running: a server.yaml, and its unit active.
|
||||||
|
func (in *serverInit) alreadyServing() bool {
|
||||||
|
if _, err := os.Stat(in.configPath()); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if in.serviceActive != nil {
|
||||||
|
return in.serviceActive()
|
||||||
|
}
|
||||||
|
return exec.Command("systemctl", "is-active", "--quiet", "llmgw-server").Run() == nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,241 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||||
|
in := &serverInit{
|
||||||
|
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
|
||||||
|
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||||
|
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
|
||||||
|
}
|
||||||
|
j := in.judgeFor()
|
||||||
|
body := in.tuneConfig(fmt.Sprintf(serverConfigTemplate, "/etc/llmgw", "/etc/llmgw", "/etc/llmgw",
|
||||||
|
"/var/lib/llmgw/llmgw.db", "ops@example.com", guardSectionFor(&j)))
|
||||||
|
path := filepath.Join(t.TempDir(), "server.yaml")
|
||||||
|
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cfg, err := config.Load(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%v\n%s", err, body)
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case cfg.Landing.Language != "tr" || !cfg.Auth.SignupOpen:
|
||||||
|
t.Errorf("language %q signup %v", cfg.Landing.Language, cfg.Auth.SignupOpen)
|
||||||
|
case cfg.Auth.DefaultRPM != 120 || cfg.Auth.DefaultTPM != 50000 || cfg.Auth.MaxKeyTTL != 30*24*time.Hour:
|
||||||
|
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
|
||||||
|
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
|
||||||
|
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
|
||||||
|
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
|
||||||
|
t.Errorf("on demand %+v", cfg.OnDemand)
|
||||||
|
}
|
||||||
|
base := cfg.Guard.Policies["baseline"]
|
||||||
|
if base.Fail != "open" || base.Judge == nil || !slices.Equal(base.Judge.Categories, []string{"Violent", "PII"}) ||
|
||||||
|
base.Judge.LatencyBudget != 3*time.Second {
|
||||||
|
t.Errorf("the judge as written: fail %q %+v", base.Fail, base.Judge)
|
||||||
|
}
|
||||||
|
if !strings.Contains(body, "Three replicas run on CPUs") {
|
||||||
|
t.Error("the comment says another number of judges than run")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNothingChosenIsTheTemplateAsItWas(t *testing.T) {
|
||||||
|
body := fmt.Sprintf(serverConfigTemplate, "/e", "/e", "/e", "/s.db", "a@b", guardSection(true))
|
||||||
|
if got := (&serverInit{}).tuneConfig(body); got != body {
|
||||||
|
t.Error("an install with nothing chosen writes another configuration than before")
|
||||||
|
}
|
||||||
|
// Each line tuneConfig changes is in the template, or a choice would be
|
||||||
|
// lost without a word.
|
||||||
|
for _, line := range []string{" signup_open: false", " default_rpm: 60", " default_tpm: 200000",
|
||||||
|
" max_key_ttl: 2160h # 90 days", " # language: tr"} {
|
||||||
|
if !strings.Contains(serverConfigTemplate, line) {
|
||||||
|
t.Errorf("the template has no %q", line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
|
||||||
|
s, err := starterFrom("qwen3-8b")
|
||||||
|
if err != nil || s.SizeGB != 5.0 || !strings.HasPrefix(s.Source, "hf:Qwen/Qwen3-8B-GGUF/") {
|
||||||
|
t.Errorf("by name: %+v %v", s, err)
|
||||||
|
}
|
||||||
|
s, err = starterFrom("hf:bartowski/Phi-4-GGUF/Phi-4-Q4_K_M.gguf")
|
||||||
|
if err != nil || s.Name != "phi-4-q4_k_m" {
|
||||||
|
t.Errorf("by file: %+v %v", s, err)
|
||||||
|
}
|
||||||
|
if _, err := starterFrom("gpt-5"); err == nil {
|
||||||
|
t.Error("a model nobody offers was taken")
|
||||||
|
}
|
||||||
|
m := starterModel(s, 8192, 0, "http://x/p.gguf", []string{"6000-24g"}, true)
|
||||||
|
if m.Name != "phi-4-q4_k_m" || m.ContextLength != 8192 || len(m.Aliases) != 0 ||
|
||||||
|
!slices.Equal(m.Profiles[0].Args, []string{"--ctx-size", "32768"}) {
|
||||||
|
t.Errorf("entry %+v", m)
|
||||||
|
}
|
||||||
|
big, _ := starterFrom("qwen3-14b")
|
||||||
|
if m := starterModel(big, 16384, 0, "http://x/q.gguf", nil, true); m.Profiles[0].MinMemoryGB != 11 ||
|
||||||
|
!slices.Equal(m.Profiles[0].Args, []string{"--ctx-size", "65536"}) {
|
||||||
|
t.Errorf("a 9 GB starter asks for %d GB, %v", m.Profiles[0].MinMemoryGB, m.Profiles[0].Args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
|
||||||
|
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
|
||||||
|
mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
|
||||||
|
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
|
||||||
|
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
|
||||||
|
in.starter, _ = starterFrom("qwen3-4b")
|
||||||
|
got := strings.Join(in.initFlags(), " ")
|
||||||
|
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
|
||||||
|
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||||
|
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("flags\n got %s\nwant %s", got, want)
|
||||||
|
}
|
||||||
|
if got := strings.Join((&serverInit{}).initFlags(), " "); got != "" {
|
||||||
|
t.Errorf("nothing chosen is %q", got)
|
||||||
|
}
|
||||||
|
if w := shellWord("Non-violent Illegal Acts,PII"); w != "'Non-violent Illegal Acts,PII'" {
|
||||||
|
t.Errorf("quoted %s", w)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
|
||||||
|
for _, in := range []serverInit{
|
||||||
|
{language: "de"},
|
||||||
|
{judge: judgeSpec{Fail: "maybe"}},
|
||||||
|
{judge: judgeSpec{Categories: []string{"Rudeness"}}},
|
||||||
|
{judge: judgeSpec{Replicas: 9}},
|
||||||
|
{starterCtx: 100},
|
||||||
|
{maxKeyTTL: time.Minute},
|
||||||
|
{vip: "10.0.0.160/24"},
|
||||||
|
{standbys: true, vip: "10.0.0.160"},
|
||||||
|
} {
|
||||||
|
if err := in.checkChoices(); err == nil {
|
||||||
|
t.Errorf("taken: %+v", in)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := (&serverInit{standbys: true, vip: "10.0.0.160/24"}).checkChoices(); err != nil {
|
||||||
|
t.Error(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWithStandbysTheMirrorAnswersEveryControlPlane(t *testing.T) {
|
||||||
|
in := &serverInit{hosts: []string{"10.0.0.5"}, patroniNodes: []pgha.Node{
|
||||||
|
{Member: pgha.Member{Name: "db1", IP: "10.0.0.6"}}, {Member: pgha.Member{Name: "db2", IP: "10.0.0.7"}},
|
||||||
|
}}
|
||||||
|
if u := in.mirrorUnit(); !strings.Contains(u, "--api-listen 127.0.0.1:8082") || strings.Contains(u, "--api-allow") {
|
||||||
|
t.Errorf("one control plane: the API is for this machine only:\n%s", u)
|
||||||
|
}
|
||||||
|
in.standbys = true
|
||||||
|
u := in.mirrorUnit()
|
||||||
|
for _, want := range []string{"--api-listen :8082", "--api-allow 10.0.0.5", "--api-allow 10.0.0.6", "--api-allow 10.0.0.7"} {
|
||||||
|
if !strings.Contains(u, want) {
|
||||||
|
t.Errorf("the unit lacks %q:\n%s", want, u)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := in.mirrorAPIURL(); got != "http://10.0.0.5:8082" {
|
||||||
|
t.Errorf("the control planes reach the mirror at %s", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheFloatingAddressIsInTheCertificate(t *testing.T) {
|
||||||
|
in := &serverInit{vip: "10.0.0.160/24"}
|
||||||
|
if !slices.Contains(in.certHosts(), "10.0.0.160") {
|
||||||
|
t.Errorf("hosts %v", in.certHosts())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStandbysWantTheStoresMachines(t *testing.T) {
|
||||||
|
in := &serverInit{standbys: true, out: &strings.Builder{}}
|
||||||
|
if err := in.ensureStandbys(context.Background()); err == nil || !strings.Contains(err.Error(), "--patroni") {
|
||||||
|
t.Errorf("standbys with nowhere to go: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTwoClustersOnOneNetworkKeepApartFloatingAddresses(t *testing.T) {
|
||||||
|
a, b := vrrpRouterID("10.0.0.161"), vrrpRouterID("10.0.0.164")
|
||||||
|
if a == b || a < 1 || a > 255 || b < 1 || b > 255 {
|
||||||
|
t.Errorf("router ids %d and %d", a, b)
|
||||||
|
}
|
||||||
|
conf := keepalivedConf("ens18", "10.0.0.164", 24, 100, "secret12")
|
||||||
|
if !strings.Contains(conf, fmt.Sprintf("virtual_router_id %d\n", b)) || !strings.Contains(conf, "interface ens18") {
|
||||||
|
t.Errorf("conf:\n%s", conf)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheJudgesAndTheStarterShareTheCPUsRatherThanEachTakingAll(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
in serverInit
|
||||||
|
judge, starter int
|
||||||
|
}{
|
||||||
|
// One machine: one judge, on up to four cores; the starter has the rest.
|
||||||
|
{serverInit{cpus: 4}, 4, 0},
|
||||||
|
{serverInit{cpus: 8}, 4, 4},
|
||||||
|
{serverInit{cpus: 32}, 4, 8},
|
||||||
|
{serverInit{cpus: 2}, 2, 0},
|
||||||
|
{serverInit{cpus: 1}, 0, 0},
|
||||||
|
{serverInit{cpus: 4, noGuardModel: true}, judgeCores, 4}, // no judge: all for the starter
|
||||||
|
// Two asked for on one machine: two cores each, one each where two
|
||||||
|
// would not fit both.
|
||||||
|
{serverInit{cpus: 8, judge: judgeSpec{Replicas: 2}}, 2, 4},
|
||||||
|
{serverInit{cpus: 2, judge: judgeSpec{Replicas: 2}}, 1, 0},
|
||||||
|
{serverInit{cpus: 8, judge: judgeSpec{Replicas: 3}}, 2, 2},
|
||||||
|
{serverInit{cpus: 4, noNode: true}, 2, 0}, // on nodes not known yet
|
||||||
|
} {
|
||||||
|
got := c.in.cpuPlan()
|
||||||
|
if got.judge != c.judge || got.starter != c.starter {
|
||||||
|
t.Errorf("%+v: judge %d starter %d, want %d %d", c.in, got.judge, got.starter, c.judge, c.starter)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
m := guardModel("http://x/g.gguf", 2, 1, nil)
|
||||||
|
if p := m.Profiles[0]; p.CPUCores != 1 || p.MinMemoryGB != 2 || !slices.Contains(p.Args, "--cache-ram") {
|
||||||
|
t.Errorf("judge profile: %d cores, %d GB, args %v", p.CPUCores, p.MinMemoryGB, p.Args)
|
||||||
|
}
|
||||||
|
s := starterModel(starterChoices[0], defaultContext, 2, "http://x/q.gguf", []string{"6000-24g"}, true)
|
||||||
|
if cpu := s.Profiles[len(s.Profiles)-1]; cpu.Name != "cpu" || cpu.CPUCores != 2 || s.Profiles[0].CPUCores != 0 {
|
||||||
|
t.Errorf("starter: GPU profile %d cores, CPU %+v", s.Profiles[0].CPUCores, cpu)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A second judge on the same machine shares its cores and is lost with it:
|
||||||
|
// one machine has one, more machines two.
|
||||||
|
func TestOneMachineHasOneJudgeAndMoreHaveTwo(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
in serverInit
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{serverInit{}, 1},
|
||||||
|
{serverInit{noNode: true}, 2},
|
||||||
|
{serverInit{patroni: []string{"a@db1", "b@db2"}}, 2},
|
||||||
|
{serverInit{standbys: true}, 2},
|
||||||
|
{serverInit{judge: judgeSpec{Replicas: 3}}, 3},
|
||||||
|
} {
|
||||||
|
if got := c.in.judgeFor().Replicas; got != c.want {
|
||||||
|
t.Errorf("%+v: %d judges, want %d", c.in, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The defaults are left out of the command; a choice against them is in.
|
||||||
|
ha := serverInit{noNode: true, patroni: []string{"a@db1", "b@db2"}, judge: judgeSpec{Replicas: 2}}
|
||||||
|
if f := strings.Join(ha.initFlags(), " "); strings.Contains(f, "--judge-replicas") {
|
||||||
|
t.Errorf("two judges on three machines named: %s", f)
|
||||||
|
}
|
||||||
|
one := serverInit{judge: judgeSpec{Replicas: 2}}
|
||||||
|
if f := strings.Join(one.initFlags(), " "); !strings.Contains(f, "--judge-replicas 2") {
|
||||||
|
t.Errorf("two judges on one machine not named: %s", f)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -9,7 +9,9 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -120,17 +122,46 @@ func (in *serverInit) writeUnit(path, body string) (bool, error) {
|
|||||||
|
|
||||||
func (in *serverInit) mirrorUnit() string {
|
func (in *serverInit) mirrorUnit() string {
|
||||||
base := "http://" + net.JoinHostPort(in.reachableAddr(), "8081")
|
base := "http://" + net.JoinHostPort(in.reachableAddr(), "8081")
|
||||||
return mirrorUnitFor(mirrorDir, mirrorAPI, base, nil,
|
if in.standbys {
|
||||||
|
return mirrorUnitFor(in.mirrorRoot(), ":8082", base, in.mirrorAllow(), nil,
|
||||||
|
"'llmgw server init'. Bytes on :8081 for every node; the\n# management API for the control planes only, whichever leads.")
|
||||||
|
}
|
||||||
|
return mirrorUnitFor(in.mirrorRoot(), mirrorAPI, base, nil, nil,
|
||||||
"'llmgw server init'. Bytes on :8081 for every node; the\n# management API on loopback only, for the control plane beside it.")
|
"'llmgw server init'. Bytes on :8081 for every node; the\n# management API on loopback only, for the control plane beside it.")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// localMirrorAPI is the management API of the mirror beside this control
|
||||||
|
// plane: on loopback, or, with standbys that may lead instead, on an address
|
||||||
|
// they reach.
|
||||||
|
func (in *serverInit) localMirrorAPI() string {
|
||||||
|
if in.standbys {
|
||||||
|
return "http://" + net.JoinHostPort(in.reachableAddr(), "8082")
|
||||||
|
}
|
||||||
|
return "http://" + mirrorAPI
|
||||||
|
}
|
||||||
|
|
||||||
|
// mirrorAllow are the control planes the mirror's management API answers
|
||||||
|
// besides loopback: this one by its own address, and each standby.
|
||||||
|
func (in *serverInit) mirrorAllow() []string {
|
||||||
|
out := []string{in.reachableAddr()}
|
||||||
|
for _, n := range in.patroniNodes {
|
||||||
|
if !slices.Contains(out, n.IP) {
|
||||||
|
out = append(out, n.IP)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
// mirrorUnitFor is a mirror's systemd unit. allow is who, besides the
|
// mirrorUnitFor is a mirror's systemd unit. allow is who, besides the
|
||||||
// machine itself, may use the management API.
|
// machine itself, may use the management API.
|
||||||
func mirrorUnitFor(dir, apiListen, base string, allow []string, about string) string {
|
func mirrorUnitFor(dir, apiListen, base string, allow, peers []string, about string) string {
|
||||||
exec := "/usr/local/bin/llmgw mirror serve --dir " + dir + " --listen :8081 --api-listen " + apiListen + " --base " + base
|
exec := "/usr/local/bin/llmgw mirror serve --dir " + dir + " --listen :8081 --api-listen " + apiListen + " --base " + base
|
||||||
for _, a := range allow {
|
for _, a := range allow {
|
||||||
exec += " --api-allow " + a
|
exec += " --api-allow " + a
|
||||||
}
|
}
|
||||||
|
for _, p := range peers {
|
||||||
|
exec += " --peer " + p
|
||||||
|
}
|
||||||
return `[Unit]
|
return `[Unit]
|
||||||
Description=llmgw model mirror
|
Description=llmgw model mirror
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
@@ -160,7 +191,7 @@ func (in *serverInit) mirrorAPIURL() string {
|
|||||||
if in.mirrorURL != "" {
|
if in.mirrorURL != "" {
|
||||||
return in.mirrorURL
|
return in.mirrorURL
|
||||||
}
|
}
|
||||||
return "http://" + mirrorAPI
|
return in.localMirrorAPI()
|
||||||
}
|
}
|
||||||
|
|
||||||
// useMirror points the control plane at a mirror on another machine, set up
|
// useMirror points the control plane at a mirror on another machine, set up
|
||||||
@@ -202,15 +233,16 @@ func (in *serverInit) ensureMirror(ctx context.Context) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if cfg.Mirror.API != "" && cfg.Mirror.API != "http://"+mirrorAPI {
|
if cfg.Mirror.API != "" && cfg.Mirror.API != "http://"+mirrorAPI && cfg.Mirror.API != in.localMirrorAPI() {
|
||||||
in.kept("mirror at %s, as server.yaml says", cfg.Mirror.API)
|
in.kept("mirror at %s, as server.yaml says", cfg.Mirror.API)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := os.MkdirAll(mirrorDir, 0o755); err != nil {
|
dir := in.mirrorRoot()
|
||||||
|
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := in.chown(mirrorDir, true); err != nil {
|
if err := in.chown(dir, true); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
changed, err := in.writeUnit(mirrorUnitPath, in.mirrorUnit())
|
changed, err := in.writeUnit(mirrorUnitPath, in.mirrorUnit())
|
||||||
@@ -227,21 +259,25 @@ func (in *serverInit) ensureMirror(ctx context.Context) error {
|
|||||||
if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
|
if err := in.systemctl(ctx, "restart", "llmgw-mirror"); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
in.done("model mirror in %s, serving on :8081", mirrorDir)
|
in.done("model mirror in %s, serving on :8081", dir)
|
||||||
} else {
|
} else {
|
||||||
in.kept("model mirror in %s", mirrorDir)
|
in.kept("model mirror in %s", dir)
|
||||||
}
|
}
|
||||||
|
|
||||||
if cfg.Mirror.API == "" {
|
if cfg.Mirror.API == "" {
|
||||||
if err := appendMirrorBlockFor(in.configPath(), "http://"+mirrorAPI,
|
comment := "# The model mirror on this machine, added by 'llmgw server init'. Its\n" +
|
||||||
"# The model mirror on this machine, added by 'llmgw server init'. Its\n"+
|
"# management API is on loopback, so it needs no certificate."
|
||||||
"# management API is on loopback, so it needs no certificate."); err != nil {
|
if in.standbys {
|
||||||
|
comment = "# The model mirror on this machine, added by 'llmgw server init'. Its\n" +
|
||||||
|
"# management API answers the control planes' addresses only."
|
||||||
|
}
|
||||||
|
if err := appendMirrorBlockFor(in.configPath(), in.localMirrorAPI(), comment); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
in.done("server.yaml points at the mirror")
|
in.done("server.yaml points at the mirror")
|
||||||
}
|
}
|
||||||
|
|
||||||
m, err := mirror.NewClient("http://"+mirrorAPI, tlsconf.Config{}, 5*time.Second)
|
m, err := mirror.NewClient(in.localMirrorAPI(), tlsconf.Config{}, 5*time.Second)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -414,8 +450,9 @@ func hasNVIDIADevice() bool {
|
|||||||
// --- the starter model ---
|
// --- the starter model ---
|
||||||
|
|
||||||
// starterModel is the catalog entry: a GPU profile first when there is a card,
|
// starterModel is the catalog entry: a GPU profile first when there is a card,
|
||||||
// and a CPU profile always, both pinned to this machine.
|
// and a CPU profile always, both pinned to this machine. ctx is the context a
|
||||||
func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model {
|
// client may use; each engine holds four times that, for four at once.
|
||||||
|
func starterModel(spec starterSpec, ctx, cores int, source string, gpuClasses []string, pinned bool) catalog.Model {
|
||||||
var pin map[string]string
|
var pin map[string]string
|
||||||
if pinned {
|
if pinned {
|
||||||
pin = map[string]string{localLabelKey: localLabelValue}
|
pin = map[string]string{localLabelKey: localLabelValue}
|
||||||
@@ -424,21 +461,25 @@ func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model
|
|||||||
for _, class := range gpuClasses {
|
for _, class := range gpuClasses {
|
||||||
profiles = append(profiles, catalog.Profile{
|
profiles = append(profiles, catalog.Profile{
|
||||||
Name: "gpu-" + class, Engine: "llamacpp", GPUClass: class,
|
Name: "gpu-" + class, Engine: "llamacpp", GPUClass: class,
|
||||||
Priority: 200, MinMemoryGB: 2, NodeSelector: pin,
|
Priority: 200, MinMemoryGB: spec.minMemoryGB(), NodeSelector: pin,
|
||||||
Args: []string{"--ctx-size", "32768"},
|
Args: []string{"--ctx-size", strconv.Itoa(max(32768, 4*ctx))},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
profiles = append(profiles, catalog.Profile{
|
profiles = append(profiles, catalog.Profile{
|
||||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2, NodeSelector: pin,
|
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: spec.minMemoryGB(), CPUCores: cores, NodeSelector: pin,
|
||||||
Args: []string{"--ctx-size", "16384"},
|
Args: []string{"--ctx-size", strconv.Itoa(max(16384, 4*ctx))},
|
||||||
})
|
})
|
||||||
|
var aliases []string
|
||||||
|
if spec.Alias != "" {
|
||||||
|
aliases = []string{spec.Alias}
|
||||||
|
}
|
||||||
return catalog.Model{
|
return catalog.Model{
|
||||||
Name: starterName,
|
Name: spec.Name,
|
||||||
Aliases: []string{starterAlias},
|
Aliases: aliases,
|
||||||
Description: "A small model init put here so there is something to try; fast, not clever.",
|
Description: "The model init put here so there is something to try: " + spec.About + ".",
|
||||||
Source: source,
|
Source: source,
|
||||||
RoutingProfile: "chat",
|
RoutingProfile: "chat",
|
||||||
ContextLength: 4096,
|
ContextLength: ctx,
|
||||||
MaxOutput: 1024,
|
MaxOutput: 1024,
|
||||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||||
Profiles: profiles,
|
Profiles: profiles,
|
||||||
@@ -446,8 +487,8 @@ func starterModel(source string, gpuClasses []string, pinned bool) catalog.Model
|
|||||||
}
|
}
|
||||||
|
|
||||||
// guardModel is the judge's catalog entry: CPU only, so it takes no card from
|
// guardModel is the judge's catalog entry: CPU only, so it takes no card from
|
||||||
// the models people use, on whichever node has room.
|
// the models people use, on whichever node has room; replicas copies of it.
|
||||||
func guardModel(source string) catalog.Model {
|
func guardModel(source string, replicas, cores int, selector map[string]string) catalog.Model {
|
||||||
return catalog.Model{
|
return catalog.Model{
|
||||||
Name: guardModelName,
|
Name: guardModelName,
|
||||||
Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
|
Description: "The guardrails' judge: reads prompts and answers for unsafe content. Not offered to clients.",
|
||||||
@@ -456,10 +497,17 @@ func guardModel(source string) catalog.Model {
|
|||||||
Guard: &catalog.GuardModel{Format: "qwen3guard"},
|
Guard: &catalog.GuardModel{Format: "qwen3guard"},
|
||||||
// Two: every judged request waits on it, and with fail: closed a
|
// Two: every judged request waits on it, and with fail: closed a
|
||||||
// judge that is not there refuses everything.
|
// judge that is not there refuses everything.
|
||||||
Scaling: catalog.Scaling{Min: 2, Max: 2},
|
Scaling: catalog.Scaling{Min: replicas, Max: replicas},
|
||||||
Profiles: []catalog.Profile{{
|
Profiles: []catalog.Profile{{
|
||||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 2,
|
// About 1.6 GB with its context, which stays so with no prompt
|
||||||
Args: []string{"--ctx-size", "8192"},
|
// cache: llama.cpp keeps up to 8 GB of prompts by default, and a
|
||||||
|
// judge's, each a different text, are not asked again. With the
|
||||||
|
// cache one grew to 3.2 GB over a day of traffic; reserving 4 for
|
||||||
|
// that left a 7 GB machine room for only one of the two.
|
||||||
|
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: judgeMemoryGB, CPUCores: cores,
|
||||||
|
Args: []string{"--ctx-size", "8192", "--cache-ram", "0"},
|
||||||
|
// The judge machines, when the install was given some.
|
||||||
|
NodeSelector: selector,
|
||||||
}},
|
}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -469,15 +517,21 @@ func (in *serverInit) ensureStarter(ctx context.Context) error {
|
|||||||
// Served here when this machine is a node, pinned so it does not wander
|
// Served here when this machine is a node, pinned so it does not wander
|
||||||
// to the first GPU node that joins; otherwise on whichever node joins.
|
// to the first GPU node that joins; otherwise on whichever node joins.
|
||||||
pinned := !in.noNode
|
pinned := !in.noNode
|
||||||
return in.ensureServed(ctx, starterName, starterSource, starterVersion, pinned,
|
spec, n := in.starterModelSpec(), in.starterContext()
|
||||||
func(url string, classes []string) catalog.Model { return starterModel(url, classes, pinned) })
|
return in.ensureServed(ctx, spec.Name, spec.Source, starterVersion, pinned,
|
||||||
|
func(url string, classes []string) catalog.Model {
|
||||||
|
return starterModel(spec, n, in.cpuPlan().starter, url, classes, pinned)
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// ensureGuardModel fetches the guardrails' judge model into the mirror and
|
// ensureGuardModel fetches the guardrails' judge model into the mirror and
|
||||||
// adds it to the catalog, to run on any node with room.
|
// adds it to the catalog, to run on any node with room.
|
||||||
func (in *serverInit) ensureGuardModel(ctx context.Context) error {
|
func (in *serverInit) ensureGuardModel(ctx context.Context) error {
|
||||||
return in.ensureServed(ctx, guardModelName, guardModelSource, guardModelVersion, false,
|
return in.ensureServed(ctx, guardModelName, guardModelSource, guardModelVersion, false,
|
||||||
func(url string, _ []string) catalog.Model { return guardModel(url) })
|
func(url string, _ []string) catalog.Model {
|
||||||
|
cores, selector := in.judgePlacement()
|
||||||
|
return guardModel(url, in.judgeFor().Replicas, cores, selector)
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// ensureServed fetches a model into the mirror, adds it to the catalog, and
|
// ensureServed fetches a model into the mirror, adds it to the catalog, and
|
||||||
@@ -661,3 +715,55 @@ func waitFor(ctx context.Context, limit time.Duration, what string, done func()
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// cpuSplit is the cores init's models reserve: each judge replica's, and
|
||||||
|
// the starter's on CPU.
|
||||||
|
type cpuSplit struct{ judge, starter int }
|
||||||
|
|
||||||
|
// cpuPlan divides the CPUs between the judges and the starter, so that the
|
||||||
|
// engines do not each take every core. Without it, two judge replicas and
|
||||||
|
// the starter on one 4-core control plane all ran on all four: under load
|
||||||
|
// most judgments missed their budget and the requests were refused.
|
||||||
|
//
|
||||||
|
// A judge needs cores: a judgment of about 280 tokens took 8.0 s on one,
|
||||||
|
// 4.2 s on two and 2.3 s on four, against a 5 s budget. On one machine its
|
||||||
|
// one replica gets up to four; where there are two, two each (one each only
|
||||||
|
// where two would not fit both). The starter has what is left, up to 8, or
|
||||||
|
// no reservation at all when nothing is, rather than a starter that fits
|
||||||
|
// nowhere. A machine that is not the node reserves two a judge on whichever
|
||||||
|
// node takes it, and the starter, on a node not known yet, none.
|
||||||
|
func (in *serverInit) cpuPlan() cpuSplit {
|
||||||
|
if in.noNode {
|
||||||
|
return cpuSplit{judge: judgeCores}
|
||||||
|
}
|
||||||
|
n := in.cpus
|
||||||
|
if n <= 0 {
|
||||||
|
n = runtime.NumCPU()
|
||||||
|
}
|
||||||
|
if n < 2 {
|
||||||
|
return cpuSplit{}
|
||||||
|
}
|
||||||
|
judges := 0
|
||||||
|
if !in.noGuardModel {
|
||||||
|
judges = in.judgeFor().Replicas
|
||||||
|
}
|
||||||
|
judge := judgeCores
|
||||||
|
switch {
|
||||||
|
case judges == 1:
|
||||||
|
judge = min(n, judgeCoresAlone)
|
||||||
|
case judges > 0 && judges*judge > n:
|
||||||
|
judge = max(1, n/judges)
|
||||||
|
}
|
||||||
|
left := max(n-judges*judge, 0)
|
||||||
|
return cpuSplit{judge: judge, starter: min(left, 8)}
|
||||||
|
}
|
||||||
|
|
||||||
|
// judgeCores and judgeMemoryGB are what a judge replica needs: two cores
|
||||||
|
// for a judgment within its budget, and 2 GB for its 1.6 with no prompt
|
||||||
|
// cache. judgeCoresAlone is what the one judge of a single machine takes,
|
||||||
|
// past which a model this small hardly gets faster.
|
||||||
|
const (
|
||||||
|
judgeCores = 2
|
||||||
|
judgeCoresAlone = 4
|
||||||
|
judgeMemoryGB = 2
|
||||||
|
)
|
||||||
@@ -28,7 +28,7 @@ func TestTheInitAgentUnitIsTheShippedUnit(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
|
func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
|
||||||
m := starterModel("http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"}, true)
|
m := starterModel(starterChoices[0], defaultContext, 0, "http://10.0.0.5:8081/qwen2.5-0.5b-instruct/starter/q.gguf", []string{"4090-24g"}, true)
|
||||||
c, err := catalog.New([]catalog.Model{m})
|
c, err := catalog.New([]catalog.Model{m})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("the starter is not a valid catalog entry: %v", err)
|
t.Fatalf("the starter is not a valid catalog entry: %v", err)
|
||||||
@@ -53,7 +53,7 @@ func TestTheStarterUsesTheGPUAndFallsBackToCPUOnThisMachine(t *testing.T) {
|
|||||||
t.Error("the starter should be running, not waiting to be asked for")
|
t.Error("the starter should be running, not waiting to be asked for")
|
||||||
}
|
}
|
||||||
|
|
||||||
if m := starterModel("http://x/q.gguf", nil, true); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
|
if m := starterModel(starterChoices[0], defaultContext, 0, "http://x/q.gguf", nil, true); len(m.Profiles) != 1 || m.Profiles[0].Name != "cpu" {
|
||||||
t.Errorf("without a GPU only the CPU profile: %+v", m.Profiles)
|
t.Errorf("without a GPU only the CPU profile: %+v", m.Profiles)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -96,7 +96,7 @@ func TestTheMirrorUnitAdvertisesAnAddressNodesCanReach(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestWithNoNodeHereTheStarterGoesToTheFirstNodeThatJoins(t *testing.T) {
|
func TestWithNoNodeHereTheStarterGoesToTheFirstNodeThatJoins(t *testing.T) {
|
||||||
m := starterModel("http://x/q.gguf", nil, false)
|
m := starterModel(starterChoices[0], defaultContext, 0, "http://x/q.gguf", nil, false)
|
||||||
for _, p := range m.Profiles {
|
for _, p := range m.Profiles {
|
||||||
if len(p.NodeSelector) > 0 {
|
if len(p.NodeSelector) > 0 {
|
||||||
t.Errorf("profile %s is pinned to a machine that is not a node: %v", p.Name, p.NodeSelector)
|
t.Errorf("profile %s is pinned to a machine that is not a node: %v", p.Name, p.NodeSelector)
|
||||||
@@ -171,7 +171,7 @@ func TestTheInstalledConfigurationIsHardenedAndLoads(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestTheGuardModelIsAValidJudge(t *testing.T) {
|
func TestTheGuardModelIsAValidJudge(t *testing.T) {
|
||||||
m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf")
|
m := guardModel("http://10.0.0.5:8081/qwen3guard-0.6b/guard/g.gguf", 2, 2, nil)
|
||||||
c, err := catalog.New([]catalog.Model{m})
|
c, err := catalog.New([]catalog.Model{m})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("not a valid catalog entry: %v", err)
|
t.Fatalf("not a valid catalog entry: %v", err)
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Machines of the judges' own. Under load on a 4-core control plane that was
|
||||||
|
// also a node, 38 of 60 requests were refused for want of a judgment; with the
|
||||||
|
// judges on a node of their own the same test refused 2. A judge machine is
|
||||||
|
// added as a node carrying judgeRoleLabel, and the judge's profile selects
|
||||||
|
// that label, so nothing else is placed beside it and it on nothing else.
|
||||||
|
|
||||||
|
// judgeRoleLabel marks a node as the judges'.
|
||||||
|
const (
|
||||||
|
judgeRoleLabel = placement.RoleLabel
|
||||||
|
judgeRoleValue = "judge"
|
||||||
|
)
|
||||||
|
|
||||||
|
// judgeMachine is a machine for the judges: user@host, and the connection the
|
||||||
|
// setup screen already made to it, when it made one.
|
||||||
|
type judgeMachine struct {
|
||||||
|
target string
|
||||||
|
remote onboard.Remote
|
||||||
|
}
|
||||||
|
|
||||||
|
// ensureJudgeNodes adds each judge machine as a node labelled for the
|
||||||
|
// judges, and says how many cores the smallest of them has.
|
||||||
|
func (in *serverInit) ensureJudgeNodes(ctx context.Context) error {
|
||||||
|
if len(in.judgeMachines) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
na := nodeAdd{serverCfg: in.configPath(), plain: true}
|
||||||
|
na.opts.PKI = in.pkiDir
|
||||||
|
na.opts.NoFirewall = in.noFirewall
|
||||||
|
if err := na.prepare(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
in.judgeCPUs = nil
|
||||||
|
for _, jm := range in.judgeMachines {
|
||||||
|
o := na.target(jm.target)
|
||||||
|
fmt.Fprintf(in.out, " .. adding %s as a node for the judges\n", o.Addr)
|
||||||
|
r := onboard.New(o, nil)
|
||||||
|
if jm.remote != nil {
|
||||||
|
r.Remote = jm.remote
|
||||||
|
}
|
||||||
|
res, err := r.Run(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("%s: %w", o.Addr, err)
|
||||||
|
}
|
||||||
|
if err := na.watch(ctx, res.Node, func(onboard.Event) {}); err != nil && err != errNoAdmin {
|
||||||
|
return fmt.Errorf("%s: %w", res.Node, err)
|
||||||
|
}
|
||||||
|
if out, err := r.Remote.Run(ctx, "nproc", nil); err == nil {
|
||||||
|
if n, err := strconv.Atoi(strings.TrimSpace(string(out))); err == nil {
|
||||||
|
in.judgeCPUs = append(in.judgeCPUs, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, err := changeStoredSettings(in.configPath(), "a node for the judges, from init", func(doc *settings.Doc) error {
|
||||||
|
return doc.SetYAML([]string{"agents", "nodes", res.Node, "labels", judgeRoleLabel}, judgeRoleValue)
|
||||||
|
}); err != nil {
|
||||||
|
return fmt.Errorf("labelling %s for the judges: %w", res.Node, err)
|
||||||
|
}
|
||||||
|
in.done("%s is a node for the judges alone (%s: %s)", res.Node, judgeRoleLabel, judgeRoleValue)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// judgePlacement is the judge's cores and where it may run: on the judge
|
||||||
|
// machines when there are some, by init's rule over their cores; otherwise
|
||||||
|
// on any node, with this machine's plan.
|
||||||
|
func (in *serverInit) judgePlacement() (cores int, selector map[string]string) {
|
||||||
|
if len(in.judgeMachines) == 0 {
|
||||||
|
return in.cpuPlan().judge, nil
|
||||||
|
}
|
||||||
|
return refitCores(in.judgeFor().Replicas, in.judgeCPUs), map[string]string{judgeRoleLabel: judgeRoleValue}
|
||||||
|
}
|
||||||
|
|
||||||
|
// judgeMachineFlag is --judge-node, given again for each machine.
|
||||||
|
type judgeMachineFlag struct{ to *[]judgeMachine }
|
||||||
|
|
||||||
|
func (f judgeMachineFlag) String() string {
|
||||||
|
if f.to == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var out []string
|
||||||
|
for _, m := range *f.to {
|
||||||
|
out = append(out, m.target)
|
||||||
|
}
|
||||||
|
return strings.Join(out, ",")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f judgeMachineFlag) Set(v string) error {
|
||||||
|
if strings.TrimSpace(v) == "" {
|
||||||
|
return fmt.Errorf("--judge-node wants user@host")
|
||||||
|
}
|
||||||
|
*f.to = append(*f.to, judgeMachine{target: strings.TrimSpace(v)})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (judgeMachineFlag) Type() string { return "user@host" }
|
||||||
|
|
||||||
|
// judgeReadyWait is how long init waits for its judge to load before timing
|
||||||
|
// it: the weights come from the mirror, and llama.cpp loads in seconds.
|
||||||
|
const judgeReadyWait = 15 * time.Minute
|
||||||
|
|
||||||
|
// checkJudge times the judge once it is up, and keeps what it found for the
|
||||||
|
// closing text: a machine too slow for its judge is said before its first
|
||||||
|
// refused request, not after. The budget is not changed without asking.
|
||||||
|
func (in *serverInit) checkJudge(ctx context.Context) {
|
||||||
|
c := in.adminAPI()
|
||||||
|
deadline := time.Now().Add(judgeReadyWait)
|
||||||
|
for {
|
||||||
|
var judges []adminapi.JudgeView
|
||||||
|
if err := c.get(ctx, "/api/v1/guard/judges", &judges); err == nil {
|
||||||
|
ready := false
|
||||||
|
for _, j := range judges {
|
||||||
|
ready = ready || (j.Model == guardModelName && j.Ready > 0)
|
||||||
|
}
|
||||||
|
if ready {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) || ctx.Err() != nil {
|
||||||
|
in.judgeNote = " !! the judge was not up to be timed; once it is: llmgw ctl guard judge check\n"
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Second)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(in.out, " .. timing the judge\n")
|
||||||
|
v, err := checkJudge(ctx, c, guardModelName, 0)
|
||||||
|
if err != nil {
|
||||||
|
in.judgeNote = " !! the judge could not be timed (" + err.Error() + "); later: llmgw ctl guard judge check\n"
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if v.Verdict == "ok" {
|
||||||
|
in.done("the judge reads a prompt in %.1f s, of the %.1f s a request waits; about %.1f requests a second",
|
||||||
|
float64(v.SlowestMs)/1000, float64(v.BudgetMs)/1000, v.PerSecond/2)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
in.judgeNote = "The judge is " + v.Verdict + " on this cluster:\n" + indent(judgeCheckText(v), " ") + "\n"
|
||||||
|
}
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestJudgesGoOnTheirOwnMachinesWhenTheInstallHasSome(t *testing.T) {
|
||||||
|
in := newDefaultServerInit()
|
||||||
|
if _, sel := in.judgePlacement(); sel != nil {
|
||||||
|
t.Errorf("no judge machines, yet a selector: %v", sel)
|
||||||
|
}
|
||||||
|
if err := (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.21"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = (judgeMachineFlag{&in.judgeMachines}).Set("ubuntu@10.0.0.22")
|
||||||
|
in.judge.Replicas, in.judgeCPUs = 2, []int{4, 4}
|
||||||
|
cores, sel := in.judgePlacement()
|
||||||
|
if sel[judgeRoleLabel] != judgeRoleValue || cores != refitCores(2, []int{4, 4}) {
|
||||||
|
t.Errorf("cores %d, selector %v", cores, sel)
|
||||||
|
}
|
||||||
|
m := guardModel("http://x/g.gguf", 2, cores, sel)
|
||||||
|
if m.Profiles[0].NodeSelector[judgeRoleLabel] != judgeRoleValue {
|
||||||
|
t.Errorf("the judge's profile does not select its machines: %+v", m.Profiles[0])
|
||||||
|
}
|
||||||
|
// The same from a script.
|
||||||
|
flags := strings.Join(in.initFlags(), " ")
|
||||||
|
if !strings.Contains(flags, "--judge-node ubuntu@10.0.0.21") || !strings.Contains(flags, "--judge-node ubuntu@10.0.0.22") {
|
||||||
|
t.Errorf("flags %s", flags)
|
||||||
|
}
|
||||||
|
if err := (judgeMachineFlag{&in.judgeMachines}).Set(" "); err == nil {
|
||||||
|
t.Error("an empty --judge-node was taken")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheSetupAsksForJudgeMachinesWhenChosen(t *testing.T) {
|
||||||
|
m := newSetupModel("ctl", []string{"10.0.0.5"})
|
||||||
|
m.choice, m.page = 1, pMachines
|
||||||
|
m.mfocus = m.shown() * mfCount // the judges' choice
|
||||||
|
m = press(m, "right").(setupModel)
|
||||||
|
if m.judges != 1 || m.shown() != 3 || m.mfocus != 3*mfCount {
|
||||||
|
t.Fatalf("judges %d, shown %d, focus %d", m.judges, m.shown(), m.mfocus)
|
||||||
|
}
|
||||||
|
if v := m.machinesView(); !strings.Contains(v, "Judge machine 1") || !strings.Contains(v, "Check the 4 machines") {
|
||||||
|
t.Errorf("the page lacks the judge machine:\n%s", v)
|
||||||
|
}
|
||||||
|
for i, addr := range []string{"10.0.0.11", "10.0.0.12", ""} {
|
||||||
|
m.db[i].addr.SetValue(addr)
|
||||||
|
m.db[i].user.SetValue("ubuntu")
|
||||||
|
m.db[i].auth = authAgent
|
||||||
|
}
|
||||||
|
if err := m.validateMachines(); !strings.Contains(err, "judge machine 1 has no address") {
|
||||||
|
t.Errorf("err %q", err)
|
||||||
|
}
|
||||||
|
m.db[2].addr.SetValue("10.0.0.21")
|
||||||
|
if err := m.validateMachines(); err != "" {
|
||||||
|
t.Errorf("err %q", err)
|
||||||
|
}
|
||||||
|
if cmd := m.command(); !strings.Contains(cmd, "--judge-node ubuntu@10.0.0.21") || strings.Contains(cmd, "--patroni ubuntu@10.0.0.21") {
|
||||||
|
t.Errorf("command %s", cmd)
|
||||||
|
}
|
||||||
|
// None again: the judge machine is no longer asked for.
|
||||||
|
m = press(m, "left").(setupModel)
|
||||||
|
if m.judges != 0 || strings.Contains(m.machinesView(), "Judge machine 1") {
|
||||||
|
t.Errorf("judges %d after choosing none", m.judges)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestTheMemoryAMachineNeedsCountsWhatItServes(t *testing.T) {
|
||||||
|
in := &serverInit{}
|
||||||
|
mb, what := in.servingMemoryMB(false)
|
||||||
|
need := preflight.Needs[preflight.RoleControl].MemoryMB + mb
|
||||||
|
if what == "" || need <= 7941 || need > 11961 {
|
||||||
|
t.Errorf("node, starter and judges here: need %d MB (%q); an 8 GB machine must fail and a 12 GB one pass", need, what)
|
||||||
|
}
|
||||||
|
// Judges on machines of their own take their share off this one.
|
||||||
|
if mb2, _ := in.servingMemoryMB(true); mb2 >= mb {
|
||||||
|
t.Errorf("with judge machines %d MB, without %d", mb2, mb)
|
||||||
|
}
|
||||||
|
// A control plane that serves nothing needs only its own.
|
||||||
|
if mb, what := (&serverInit{noNode: true}).servingMemoryMB(false); mb != 0 || what != "" {
|
||||||
|
t.Errorf("no node: %d %q", mb, what)
|
||||||
|
}
|
||||||
|
if mb, _ := (&serverInit{noModel: true, noGuardModel: true}).servingMemoryMB(false); mb != 0 {
|
||||||
|
t.Errorf("nothing served: %d", mb)
|
||||||
|
}
|
||||||
|
// Without a mirror no judge is fetched, so none is counted.
|
||||||
|
if mb, what := (&serverInit{noMirror: true}).servingMemoryMB(false); mb != 0 {
|
||||||
|
t.Errorf("no mirror, nothing to fetch: %d %q", mb, what)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSetupRunAgainOnAMachineAlreadySetUpIsNotRefusedItsMemory(t *testing.T) {
|
||||||
|
// The docs fix a certificate with server init --host on a machine set up
|
||||||
|
// before the check: it installs nothing it does not already serve.
|
||||||
|
dir := t.TempDir()
|
||||||
|
cfg := dir + "/server.yaml"
|
||||||
|
if err := os.WriteFile(cfg, []byte("listen: :8080\n"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
in := &serverInit{unitPath: dir + "/llmgw-server.service", etcDir: dir, serviceActive: func() bool { return true }}
|
||||||
|
if err := in.checkMemory(); err != nil {
|
||||||
|
t.Errorf("refused: %v", err)
|
||||||
|
}
|
||||||
|
// A first run that stopped half way left server.yaml and serves nothing:
|
||||||
|
// it still would install the models, and is checked.
|
||||||
|
in.serviceActive = func() bool { return false }
|
||||||
|
if in.alreadyServing() {
|
||||||
|
t.Error("a machine whose gateway is not running was taken for one serving")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -51,6 +51,8 @@ func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
|
|||||||
Remote: remote,
|
Remote: remote,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
// Kept: the standbys are made control planes on the same machines.
|
||||||
|
in.patroniNodes = setup.DB
|
||||||
self, err := sourceIPToward(setup.DB[0].IP)
|
self, err := sourceIPToward(setup.DB[0].IP)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Three machines, each able to take over: the store's two machines are made
|
||||||
|
// control planes too, standing by while this one leads, as 'llmgw server add'
|
||||||
|
// makes any machine one. With the store replicated across them and etcd's
|
||||||
|
// three votes on the three, losing any one machine loses neither the store
|
||||||
|
// nor the gateway. With a floating address, clients keep the one address.
|
||||||
|
|
||||||
|
// ensureStandbys makes each machine of the replicated store a control plane,
|
||||||
|
// one after the other.
|
||||||
|
func (in *serverInit) ensureStandbys(ctx context.Context) error {
|
||||||
|
if len(in.patroniNodes) == 0 {
|
||||||
|
return errors.New("the standbys go on the store's machines: give them with --patroni")
|
||||||
|
}
|
||||||
|
var hosts []string
|
||||||
|
if ip, _, err := net.ParseCIDR(in.vip); err == nil {
|
||||||
|
hosts = []string{ip.String()}
|
||||||
|
}
|
||||||
|
for i, n := range in.patroniNodes {
|
||||||
|
fmt.Fprintf(in.out, " .. making %s a control plane too\n", n.Name)
|
||||||
|
sa := serverAdd{
|
||||||
|
etcDir: in.etcDir, pkiDir: in.pkiDir, hosts: hosts, vip: in.vip,
|
||||||
|
remote: n.Remote, noFirewall: in.noFirewall, addr: n.IP, ip: n.IP,
|
||||||
|
out: in.out, priority: 100 - 10*i, quiet: true,
|
||||||
|
}
|
||||||
|
if err := sa.run(ctx); err != nil {
|
||||||
|
return fmt.Errorf("%s: %w", n.Name, err)
|
||||||
|
}
|
||||||
|
in.done("%s (%s) is a control plane too, standing by while this one leads", n.Name, n.IP)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os/signal"
|
||||||
|
"syscall"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newStrataServeCmd() *cobra.Command {
|
||||||
|
var cfg strata.Config
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "strata-serve [flags] [-- setup.py flags]",
|
||||||
|
Short: "Run Strata as an engine: prepare the model, then serve it (started by the agent)",
|
||||||
|
Hidden: true,
|
||||||
|
Long: "Started by the node agent for a replica whose engine is strata. Prepares the\n" +
|
||||||
|
"model from the fetched GGUF files with Strata's setup, which skips what it has\n" +
|
||||||
|
"done before, then becomes Strata's server on the replica's address. The key is\n" +
|
||||||
|
"read from " + strata.KeyEnv + ".",
|
||||||
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
|
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||||
|
defer stop()
|
||||||
|
cfg.Args = args
|
||||||
|
cfg.Log = logger()
|
||||||
|
return strata.Run(ctx, cfg)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
f := cmd.Flags()
|
||||||
|
f.StringVar(&cfg.Listen, "listen", "", "the replica's address")
|
||||||
|
f.StringVar(&cfg.Model, "model", "", "the model's name, for /v1/models")
|
||||||
|
f.StringVar(&cfg.Weights, "weights", "", "the fetched GGUF files' directory")
|
||||||
|
f.StringVar(&cfg.State, "state", "", "the replica's own directory")
|
||||||
|
f.StringVar(&cfg.Data, "data", "", "the node's directory for prepared models, shared by its replicas")
|
||||||
|
f.StringVar(&cfg.Python, "python", "python3", "the interpreter of Strata's environment")
|
||||||
|
f.StringVar(&cfg.Dir, "strata", "", "Strata's directory, where setup.py is")
|
||||||
|
f.StringVar(&cfg.Engine, "engine", "", "Strata's engine directory, where the strata binary is")
|
||||||
|
for _, n := range []string{"listen", "weights", "state", "data", "strata", "engine"} {
|
||||||
|
_ = cmd.MarkFlagRequired(n)
|
||||||
|
}
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
+160
-2
@@ -97,10 +97,21 @@ func (m *consoleModel) openForm(f *formDialog) {
|
|||||||
|
|
||||||
// call runs one admin API request in the background and reports it.
|
// call runs one admin API request in the background and reports it.
|
||||||
func (m *consoleModel) call(title, method, path string, body any, say func(raw json.RawMessage) string) tea.Cmd {
|
func (m *consoleModel) call(title, method, path string, body any, say func(raw json.RawMessage) string) tea.Cmd {
|
||||||
|
return m.callWaiting(title, method, path, body, say, 60*time.Second)
|
||||||
|
}
|
||||||
|
|
||||||
|
// callWaiting is call for a change the server may spend longer on than the
|
||||||
|
// console's usual wait.
|
||||||
|
func (m *consoleModel) callWaiting(title, method, path string, body any, say func(raw json.RawMessage) string,
|
||||||
|
wait time.Duration,
|
||||||
|
) tea.Cmd {
|
||||||
m.busy = title
|
m.busy = title
|
||||||
c := m.client
|
c := m.client
|
||||||
|
if wait > 60*time.Second {
|
||||||
|
c = c.patient(wait)
|
||||||
|
}
|
||||||
return func() tea.Msg {
|
return func() tea.Msg {
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
ctx, cancel := context.WithTimeout(context.Background(), wait)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
var raw json.RawMessage
|
var raw json.RawMessage
|
||||||
// A change may answer with no body at all.
|
// A change may answer with no body at all.
|
||||||
@@ -332,6 +343,55 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
|
|||||||
}})
|
}})
|
||||||
return nil
|
return nil
|
||||||
}},
|
}},
|
||||||
|
{"Context length…", func(m *consoleModel) tea.Cmd {
|
||||||
|
m.openForm(&formDialog{title: "Context of " + md.Name,
|
||||||
|
note: "What one request may use. The engine's arguments and the memory change with it, checked against " +
|
||||||
|
"what the cards hold; its replicas are then started again on it, one by one.",
|
||||||
|
fields: []formField{
|
||||||
|
textF("tokens", strconv.Itoa(md.ContextLength), "a request's prompt and answer together"),
|
||||||
|
textF("slots", "", "llama.cpp: requests at once; empty keeps it"),
|
||||||
|
},
|
||||||
|
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||||
|
tokens, err := strconv.Atoi(v[0])
|
||||||
|
if err != nil || tokens < 256 {
|
||||||
|
return nil, fmt.Errorf("the context is a whole number of tokens, at least 256")
|
||||||
|
}
|
||||||
|
slots := 0
|
||||||
|
if v[1] != "" {
|
||||||
|
if slots, err = strconv.Atoi(v[1]); err != nil || slots < 1 {
|
||||||
|
return nil, fmt.Errorf("slots is a whole number from 1")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m.call("Context of "+md.Name, http.MethodPost, path+"/context",
|
||||||
|
modelserve.ContextRequest{Tokens: tokens, Slots: slots}, contextSays), nil
|
||||||
|
}})
|
||||||
|
return nil
|
||||||
|
}},
|
||||||
|
{"GPUs…", func(m *consoleModel) tea.Cmd {
|
||||||
|
profiles := append([]string{"(its only one on cards)"}, md.Profiles...)
|
||||||
|
m.openForm(&formDialog{title: "Cards of " + md.Name,
|
||||||
|
note: "How many cards of one node each replica uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, " +
|
||||||
|
"the layers split for llama.cpp. Checked against the model's heads, its cards' memory and the nodes; " +
|
||||||
|
"its replicas are then started again on it, one by one.",
|
||||||
|
fields: []formField{
|
||||||
|
choiceF("profile", profiles, profiles[0]),
|
||||||
|
textF("cards", "", "a replica's, on one node"),
|
||||||
|
choiceF("in place", []string{"no", "yes"}, "no"),
|
||||||
|
},
|
||||||
|
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||||
|
n, err := strconv.Atoi(v[1])
|
||||||
|
if err != nil || n < 1 {
|
||||||
|
return nil, fmt.Errorf("the cards are a whole number from 1")
|
||||||
|
}
|
||||||
|
profile := v[0]
|
||||||
|
if profile == profiles[0] {
|
||||||
|
profile = ""
|
||||||
|
}
|
||||||
|
return m.call("Cards of "+md.Name, http.MethodPost, path+"/gpus",
|
||||||
|
modelserve.GPUsRequest{GPUs: n, Profile: profile, InPlace: v[2] == "yes"}, gpusSays), nil
|
||||||
|
}})
|
||||||
|
return nil
|
||||||
|
}},
|
||||||
{"Resources…", func(m *consoleModel) tea.Cmd {
|
{"Resources…", func(m *consoleModel) tea.Cmd {
|
||||||
profiles := md.Profiles
|
profiles := md.Profiles
|
||||||
if len(profiles) == 0 {
|
if len(profiles) == 0 {
|
||||||
@@ -430,13 +490,40 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
|
|||||||
return m.call("Retry "+md.Name, http.MethodPost, path+"/retry", map[string]string{}, nil)
|
return m.call("Retry "+md.Name, http.MethodPost, path+"/retry", map[string]string{}, nil)
|
||||||
}})
|
}})
|
||||||
}
|
}
|
||||||
|
if md.Stopped {
|
||||||
|
items = append(items, menuItem{"Start", func(m *consoleModel) tea.Cmd {
|
||||||
|
return m.call("Start "+md.Name, http.MethodPost, path+"/start", nil, func(json.RawMessage) string {
|
||||||
|
return md.Name + " runs again, scaled as it was when it was stopped."
|
||||||
|
})
|
||||||
|
}})
|
||||||
|
} else {
|
||||||
|
items = append(items, menuItem{"Stop…", func(m *consoleModel) tea.Cmd {
|
||||||
|
m.confirmD = &confirmDialog{title: "Stop " + md.Name,
|
||||||
|
text: "Its replicas stop; it stays in the catalog with its files, and Start runs it again as it was.",
|
||||||
|
yes: func(m *consoleModel) tea.Cmd {
|
||||||
|
return m.call("Stop "+md.Name, http.MethodPost, path+"/stop", nil, func(json.RawMessage) string {
|
||||||
|
return md.Name + " is stopped and kept."
|
||||||
|
})
|
||||||
|
}}
|
||||||
|
return nil
|
||||||
|
}})
|
||||||
|
}
|
||||||
items = append(items, menuItem{"Remove…", func(m *consoleModel) tea.Cmd {
|
items = append(items, menuItem{"Remove…", func(m *consoleModel) tea.Cmd {
|
||||||
m.confirmD = &confirmDialog{title: "Remove " + md.Name,
|
m.confirmD = &confirmDialog{title: "Remove " + md.Name,
|
||||||
text: "Its replicas stop and it leaves the catalog. A model from server.yaml cannot be removed here.",
|
text: "Its replicas stop and it leaves the catalog; its files stay on the mirror and the nodes. " +
|
||||||
|
"A model from server.yaml cannot be removed here.",
|
||||||
yes: func(m *consoleModel) tea.Cmd {
|
yes: func(m *consoleModel) tea.Cmd {
|
||||||
return m.call("Remove "+md.Name, http.MethodDelete, path, nil, nil)
|
return m.call("Remove "+md.Name, http.MethodDelete, path, nil, nil)
|
||||||
}}
|
}}
|
||||||
return nil
|
return nil
|
||||||
|
}}, menuItem{"Remove with its files…", func(m *consoleModel) tea.Cmd {
|
||||||
|
m.confirmD = &confirmDialog{title: "Remove " + md.Name + " and its files",
|
||||||
|
text: "It leaves the catalog, every version of it is deleted from the mirror, and the nodes' copies once " +
|
||||||
|
"its replicas have stopped. What another model uses stays. This cannot be undone: adding it again fetches it again.",
|
||||||
|
yes: func(m *consoleModel) tea.Cmd {
|
||||||
|
return m.callWaiting("Remove "+md.Name, http.MethodDelete, path+"?purge=true", nil, purgeSays, 3*time.Minute)
|
||||||
|
}}
|
||||||
|
return nil
|
||||||
}})
|
}})
|
||||||
return &menuDialog{title: md.Name, items: items}
|
return &menuDialog{title: md.Name, items: items}
|
||||||
}
|
}
|
||||||
@@ -898,3 +985,74 @@ func locationRequest(n adminapi.NodeView, vals []string) adminapi.NodeLabelsRequ
|
|||||||
}
|
}
|
||||||
return req
|
return req
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// contextSays is a context change in a sentence or two.
|
||||||
|
func contextSays(raw json.RawMessage) string {
|
||||||
|
var v modelserve.ContextChange
|
||||||
|
if json.Unmarshal(raw, &v) != nil {
|
||||||
|
return "Done."
|
||||||
|
}
|
||||||
|
s := fmt.Sprintf("%s: %d → %d tokens a request, answers up to %d.", v.Model, v.Before, v.After, v.MaxOutput)
|
||||||
|
for _, p := range v.Profiles {
|
||||||
|
if p.After != "" {
|
||||||
|
s += "\n" + p.Profile + ": " + p.After
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if v.Note != "" {
|
||||||
|
s += "\n" + v.Note
|
||||||
|
}
|
||||||
|
if v.Restarted > 0 {
|
||||||
|
s += fmt.Sprintf("\n%d replica(s) started again on it, one by one.", v.Restarted)
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// gpusSays is a change of cards in a sentence or two.
|
||||||
|
func gpusSays(raw json.RawMessage) string {
|
||||||
|
var v modelserve.GPUsChange
|
||||||
|
if json.Unmarshal(raw, &v) != nil {
|
||||||
|
return "Done."
|
||||||
|
}
|
||||||
|
s := fmt.Sprintf("%s: %d → %d %s card(s) a replica.", v.Model, v.Before, v.After, v.Class)
|
||||||
|
if v.PerCard != "" {
|
||||||
|
s += "\n" + v.PerCard
|
||||||
|
}
|
||||||
|
if v.Note != "" {
|
||||||
|
s += "\n" + v.Note
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case v.Restarted > 0 && v.InPlace:
|
||||||
|
s += fmt.Sprintf("\n%d replica(s) stopped and started again on it, one by one.", v.Restarted)
|
||||||
|
case v.Restarted > 0:
|
||||||
|
s += fmt.Sprintf("\n%d replica(s) started again on it, one by one.", v.Restarted)
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// purgeSays is a model removed with its files, in a few lines.
|
||||||
|
func purgeSays(raw json.RawMessage) string {
|
||||||
|
var v adminapi.PurgeView
|
||||||
|
if json.Unmarshal(raw, &v) != nil {
|
||||||
|
return "Done."
|
||||||
|
}
|
||||||
|
s := v.Removed + " removed."
|
||||||
|
if len(v.Mirror) > 0 {
|
||||||
|
s += fmt.Sprintf("\nMirror: %d version(s) deleted.", len(v.Mirror))
|
||||||
|
}
|
||||||
|
if len(v.Kept) > 0 {
|
||||||
|
s += "\nKept on the mirror, another model uses them: " + strings.Join(v.Kept, ", ")
|
||||||
|
}
|
||||||
|
var files int
|
||||||
|
var freed int64
|
||||||
|
for _, n := range v.Nodes {
|
||||||
|
files += n.Files
|
||||||
|
freed += n.Bytes
|
||||||
|
}
|
||||||
|
if files > 0 {
|
||||||
|
s += fmt.Sprintf("\nNodes: %d file(s), %s freed.", files, humanBytes(freed))
|
||||||
|
}
|
||||||
|
if v.Note != "" {
|
||||||
|
s += "\n" + v.Note
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
Loaded 100 of 380 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user