Compare commits
4
Commits
main
..
presets-ui
| Author | SHA1 | Date | |
|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
85a2bdd0e3 | ||
|
Ilker ManapandClaude Opus 5.5
|
e0adcc5c4e | ||
|
Ilker ManapandClaude Opus 5.5
|
a1974c2b45 | ||
|
Ilker ManapandClaude Opus 5.5
|
92b588452e |
No files matched your search
@@ -34,11 +34,7 @@ jobs:
|
||||
run: go vet ./...
|
||||
|
||||
- name: Test with the race detector
|
||||
# Under the job's limit, so a package that hangs panics with the
|
||||
# test it was in and where every goroutine waited, rather than the
|
||||
# job ending at its deadline with nothing said: from 2026-10-15 every
|
||||
# run did, printing nothing after internal/release.
|
||||
run: go test -race -timeout 12m ./...
|
||||
run: go test -race ./...
|
||||
|
||||
- name: Build
|
||||
run: go build ./...
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
# Fails when an engine llmgw follows release by release (FreeToken) has
|
||||
# published a version newer than the one DefaultEngines installs: the failed
|
||||
# run is the notice to check the new version and move to it. See
|
||||
# scripts/engine_releases.py.
|
||||
name: Engine releases
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Mondays 06:00 UTC.
|
||||
- cron: "0 6 * * 1"
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
check:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Compare with PyPI
|
||||
run: python3 scripts/engine_releases.py
|
||||
@@ -9,5 +9,3 @@ coverage.out
|
||||
/terraform-provider-llmgw/terraform-provider-llmgw
|
||||
/terraform-provider-llmgw/examples/.terraform*
|
||||
/terraform-provider-llmgw/examples/terraform.tfstate*
|
||||
# What a development server run from the checkout writes beside it.
|
||||
/secrets/
|
||||
+13
-753
@@ -15,66 +15,6 @@ Conventions this codebase keeps, which every item here must keep too:
|
||||
- `go test ./...`, `go vet ./...` and `govulncheck ./...` pass. No release is
|
||||
cut and nothing is deployed to the PoC cluster without being told to.
|
||||
|
||||
## Open, in order (2026-10-20)
|
||||
|
||||
What is left, most needed first. The detail is in the item each names.
|
||||
|
||||
**Now: what goes wrong where it is installed.**
|
||||
1. ~~**J8**~~: done 2026-10-20. `ctl guard judge check`, and init times the judge. Left of **J3**: its load test
|
||||
against judges on the control plane, which was stopped half way. Judges on a 4-core node of their own refused
|
||||
0/60 at 2 at once, 13/60 at 4 and 42/60 at 10.
|
||||
2. ~~**I2 P2**~~: done 2026-10-20: the guardrails both ways, a rate per caller and server, a breaker. A portal page
|
||||
for the servers is left, with P3 (stdio, per-person OAuth).
|
||||
3. **K**: fixes from the fabric test the operator runs with docs/large-models.md.
|
||||
|
||||
**Soon: the customer's experience, and HA.**
|
||||
4. ~~**C24**~~: done 2026-10-21. The store's, catalog's and mirror's errors, percent signs, footer times and stale
|
||||
forms are in the page's language. Left: the /docs documents, and a language kept per person (cookie only today).
|
||||
5. ~~**J4**~~: done 2026-10-29. The done-criterion held on the PoC: with mirror 146 stopped, `ctl model serve` of a
|
||||
new model was fetched by the second mirror (VM 167, 10.0.0.152), placed and answered; a node without a file
|
||||
146's address names was given 152's; what was added meanwhile reached 146 when it came back; a removal was
|
||||
carried both ways. Found and fixed there: add-peer on a mirror machine of its own, files without checksums never
|
||||
copied, and `mirror remove` neither recorded nor seen by the running mirror.
|
||||
6. **F5**: large imports. Done 2026-10-29: the gateway embeds what `llmgw-kb` sends as text; the slots and
|
||||
halfvec were done before. Left: a larger store machine before a million chunks (the operator's), and the
|
||||
embedding model on a GPU for a real import.
|
||||
|
||||
**From Kong** (section L): L1–L5 and L9 done (v0.15.0); left of L4 is routing by cost between outside
|
||||
providers, of L5 cache-write and audio prices. L6 and L7 are llmgw-80's; L8 (A2A) when asked.
|
||||
|
||||
**Models we can trust** (section M, 2026-10-25): M1 done (pickles refused; the explorer lists only what this
|
||||
cluster's hardware holds, with CPU and GPU tok/s). Next M2 (every download pinned and checked against Hugging
|
||||
Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
|
||||
|
||||
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
|
||||
|
||||
**Governance.**
|
||||
7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
|
||||
projects, people and keys from /org; viewers are drawn no forms; forms made compact.
|
||||
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||
|
||||
**When there is room.**
|
||||
9. ~~**I1**~~: done 2026-10-25 (in memory, not pgvector); left: the 40-rewordings run on the PoC.
|
||||
10. **H1**: tried on the PoC.
|
||||
11. **C4**: Parquet, and a scheduled push for usage export.
|
||||
|
||||
**When needed.**
|
||||
12. **H5**: licences in video and audio. 13. **H3**: replicas shared, the catalog side. 14. **J7**: nginx in front
|
||||
of several control planes. 15. **J6**: rate limits per process (a note).
|
||||
|
||||
**Done since 2026-10-16:**
|
||||
- J1, J5, I3, I4, J2.
|
||||
- Active Directory and Entra ID, tried against a Samba DC, with the console's Sign-in screen.
|
||||
- J3's code.
|
||||
- K's code and docs/large-models.md.
|
||||
- I2 P1 and the cluster's own read-only MCP server (llmgw:admin).
|
||||
- The node console's Overview; a node's page showing its version and the models on each GPU.
|
||||
|
||||
**Not code, the operator's to decide or do:**
|
||||
- Where Alertmanager sends (its receiver is "none"): with a judge down, nobody hears.
|
||||
- The fabric test for K.
|
||||
- The PoC runs a dev build of main (v0.15.0-7-ga583359) on its control planes and nodes, the mirror v0.15.0.
|
||||
|
||||
---
|
||||
|
||||
## A. Security — do first
|
||||
@@ -436,9 +376,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
API middleware and the /admin pages (with each change's result), `ctl
|
||||
audit`, the console's Audit view; staff roles viewer / operator /
|
||||
billing / admin. Left: before/after values (the request's fields are
|
||||
recorded, not the state they replaced). Done 2026-11-08: an organization's
|
||||
admin who changes its own organization from /org (managers still only view
|
||||
it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
recorded, not the state they replaced); a tenant-admin role that changes
|
||||
its own organization (managers still only view it); hiding the forms a
|
||||
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
|
||||
plus an org "manager" who can view. Add `audit_events` (who, what,
|
||||
before/after, source) written by the `adminAction`/`tenancyAction`
|
||||
@@ -463,9 +403,8 @@ portal is not done — the page shows; the console and ctl change.
|
||||
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. `server init` and both setup screens ask for webhooks and test
|
||||
each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
|
||||
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
@@ -1134,19 +1073,13 @@ four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
|
||||
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
|
||||
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
|
||||
at a third of its CPUs.
|
||||
- Done for `ctl model serve` (32 slots of 1024 tokens; `ctl model context
|
||||
MODEL 1024 --slots 32` resizes one served before): an embedding model's profile with many slots by default (`--parallel`
|
||||
- An embedding model's profile with many slots by default (`--parallel`
|
||||
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
|
||||
longest chunk), or `ctl model serve` choosing that for a model with
|
||||
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
|
||||
- Done (2026-10-29): embedding on the gateway's side of a document PUT.
|
||||
`llmgw-kb` sends text only to a gateway that says `server_embeds`; the
|
||||
gateway embeds through its own `/v1/embeddings` as the caller, 32 a batch,
|
||||
four batches at once, waiting out a busy engine. On 40 DTIC files the
|
||||
upload went from 25.7 to 3.5 MiB, and the ~21 KB a chunk that came down
|
||||
is gone; on 6 files both ways gave the same 32 chunks and the same search
|
||||
answer. Speed is the engine's either way: the PoC's bge-m3 on node02's two
|
||||
CPU cores embeds ~0.2 chunks/s, so a real import still wants it on a GPU.
|
||||
- Embedding on the gateway's side of a document PUT, so the client sends
|
||||
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
|
||||
the text and vector go up again (~11 KB).
|
||||
- The store next: HNSW inserts took half of one of pg01's two cores; at a
|
||||
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
|
||||
0.7) and a larger store machine.
|
||||
@@ -1452,7 +1385,7 @@ gate, so no token is on the node.
|
||||
- Done when: SF3D runs on cuda01 with nothing installed into the engine's
|
||||
venv by hand (the hand install of 2026-10-10 removed first).
|
||||
|
||||
### H5. Licence on every output (S) — done 2026-10-14: the whole licence in GLB asset.extras and PNG iTXt; video and audio not yet
|
||||
### H5. Licence on every output (S)
|
||||
- `License` gains `id` (SPDX where there is one, else a lowercase name:
|
||||
stability-community, tencent-hunyuan-community), `commercial` (unset is
|
||||
not said), `excluded_regions` (ISO 3166 codes, `EU` for the union),
|
||||
@@ -1465,7 +1398,7 @@ gate, so no token is on the node.
|
||||
audio formats in their metadata where ffmpeg is at hand on the router,
|
||||
otherwise beside it in the job's answer only — said in the docs.
|
||||
|
||||
### H6. Licence choices (S) — done 2026-10-11 (4bbac08), live on the PoC
|
||||
### H6. Licence choices (S) — the other session's (llmgw-b7), at the user's request
|
||||
The user asked (2026-10-11) for the choice to be a person's: on their
|
||||
profile they pick the licences they will not use, and their keys and
|
||||
portal session neither see those models in `/v1/models` nor may call
|
||||
@@ -1474,689 +1407,16 @@ them (403, saying which licence). Matched on `license.id`, and on
|
||||
store table is migration 0046. A site-wide or team-wide default can sit
|
||||
on top of it later.
|
||||
|
||||
### H7. A job worker per model, not one for all (S) — done 2026-10-14: one job per model at a time, eight in all
|
||||
### H7. A job worker per model, not one for all (S)
|
||||
The worker runs one job at a time: a job waiting for the card holds up a
|
||||
job for another model that could run now (seen on the PoC, 2026-10-10).
|
||||
Jobs are claimed and run concurrently up to a limit per model (or per
|
||||
environment, after H3) and a cluster-wide cap, with the lease and
|
||||
leader-only rules of G1 kept.
|
||||
|
||||
### H8. Presets that add themselves (S) — done 2026-10-13 (v0.14.5)
|
||||
flux-schnell and triposr come with llmgw (`ctl model add <preset>`), and
|
||||
the leader adds them on demand where a node fits one, the cluster installs
|
||||
ComfyUI and the catalog makes nothing of that kind (`presets.auto`); a
|
||||
preset removed is declined and not added back. The cluster page and the
|
||||
console show and add them.
|
||||
|
||||
### H9. Small things learned on the PoC (2026-10-14)
|
||||
- Done: `albedo_match` on sf3d (its delighting made dark objects light);
|
||||
`on_demand.evict_after_idle` / `min_lifetime` as live settings; an
|
||||
upgrade that leaves nodes behind says which; `ctl model serve` serves an
|
||||
embedding model for embedding; refreshes wait while a form is filled in.
|
||||
- Done 2026-10-15: llmgw's `/mcp` with picture, 3D, VLM and workflow tools,
|
||||
so a client needs no server of its own (asked by the game team); the
|
||||
shared ComfyUI replica (H3's placement side).
|
||||
- Done 2026-10-16 (v0.14.6, v0.14.7): setup lists every part with what it
|
||||
is and its settings; three machines each run the control plane, with a
|
||||
floating address (VRRP id from the address); a lost store primary costs no
|
||||
request (3 s connect per member, keepalive and TCP_USER_TIMEOUT, Patroni
|
||||
ttl 20, a standby leaves the lease alone a round after the store returns);
|
||||
a new leader waits for its judge within the budget; a judge is asked again
|
||||
on another machine at half its budget; init sizes the judge (2 GB, no
|
||||
prompt cache, two cores, one judge on up to four cores on one machine).
|
||||
|
||||
## I. What other gateways do that llmgw does not (written 2026-10-16)
|
||||
|
||||
From a comparison with Bifrost (maximhq/bifrost, Apache 2.0, Go), with
|
||||
LiteLLM and Portkey in mind. Bifrost routes to 23+ cloud providers and runs
|
||||
no model; llmgw runs the models, so most of Bifrost is either in llmgw
|
||||
already (keys, budgets, fallbacks, guardrails, tracing, SSO, HA — the last
|
||||
three paid in Bifrost) or does not apply. These four are worth taking for
|
||||
a site that runs its own models. Left until a customer asks: native
|
||||
Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
|
||||
routing by what a request holds. Not taken: a realtime (WebSocket audio)
|
||||
API, prompt management, an `npx` install.
|
||||
|
||||
### I1. Answers to similar questions, not only the same one (M) — done 2026-10-25
|
||||
`keys cache --similar`, `router.cache.embedding_model`, `internal/router/similar.go`. Changed
|
||||
from the plan: the vectors are in the gateway's memory beside the exact answers, not a pgvector
|
||||
table. The exact cache keeps answer text out of the store on purpose (data.md), and a table of
|
||||
answers would undo that; it also works on SQLite. The usage row marks a similar hit as a hit
|
||||
(no column of its own). Open: the 40-rewordings and eval-suite runs on the PoC.
|
||||
The original item follows.
|
||||
The answer cache (`internal/router/cache.go`, C17) answers a request whose
|
||||
body is the same byte for byte. A help desk asked "how do I reset my
|
||||
password" in forty wordings asks the model forty times.
|
||||
- Per project, off by default, beside the exact cache: `llmgw keys cache
|
||||
--project P --similar 0.95` (the cosine similarity a question must reach;
|
||||
higher is stricter). The exact cache is asked first.
|
||||
- The question embedded by a model the operator names
|
||||
(`router.cache.embedding_model`, an embedding model in the catalog), into a
|
||||
pgvector table per project with the answer, the model and the TTL; the
|
||||
store's halfvec and text index work from F applies. SQLite stores keep the
|
||||
exact cache only.
|
||||
- What counts as the question: the last user turn, with the system prompt
|
||||
and the earlier turns hashed and required to match exactly — a similar
|
||||
question in another conversation is not the same question. Tools,
|
||||
`response_format` and the sampling settings also exact.
|
||||
- Said: `x-llmgw-cache: similar` with the similarity, and the usage row
|
||||
says so; guardrails judge the request before a hit, as for the exact
|
||||
cache. A project with `no_store` gets neither.
|
||||
- Done when: forty rewordings of one question on the PoC cost one model call
|
||||
and 39 lookups, and an eval suite run with the cache on scores within a
|
||||
point of the run with it off.
|
||||
|
||||
### I2. MCP gateway: the customer's tool servers behind llmgw (L)
|
||||
P1 done 2026-10-20: HTTP servers registered (`ctl mcp`, `/api/v1/mcp/servers`,
|
||||
store `mcp_servers`), tools as `server.tool` in `/mcp`, `allowed_tools`
|
||||
grants in the tree (deny by default), credentials in the store or by
|
||||
reference, the caller named in `X-Llmgw-*` headers, time and size bounds,
|
||||
progress passed on, audit (Source `mcp`) and metrics. Left: P2, the
|
||||
guardrails both ways, a per-tool rate limit, a breaker, and a portal page;
|
||||
P3, stdio servers and per-person OAuth. The "done when" with Claude Code
|
||||
against a real server is still to try.
|
||||
`/mcp` serves llmgw's own tools. An agent inside a company also wants Jira,
|
||||
the file server, a database — each an MCP server, each today configured
|
||||
in every client with its own credentials and nobody seeing what was called.
|
||||
- Servers registered once, by an administrator: `llmgw ctl mcp add jira
|
||||
--url https://… --header-file /etc/llmgw/secrets/jira` (HTTP transport;
|
||||
stdio servers run on the control plane under their own user, like an
|
||||
engine). Their credentials stay on the control plane, never in a client.
|
||||
- `/mcp` lists every server's tools, prefixed by its name (`jira.search`),
|
||||
filtered by what the key may use: an allowlist per org, team, project or
|
||||
key, by server or by tool, in the tenancy tree as model allowlists are.
|
||||
- Each call goes to the audit log — who, which tool, the arguments' size,
|
||||
the outcome and how long — and through the guardrails both ways (a tool's
|
||||
answer is prompt input). A per-tool rate limit.
|
||||
- Later, if it earns it: Bifrost's "code mode" (the tools as one typed API
|
||||
the model writes code against), which cuts the tokens of a long tool list.
|
||||
- Done when: Claude Code given only llmgw's `/mcp` reaches a test MCP server
|
||||
through it, a key without the grant does not see the server's tools, and
|
||||
every call is in `ctl audit`.
|
||||
|
||||
### I3. What llmgw adds to a request, measured (S) — done 2026-10-16
|
||||
`llmgw ctl bench overhead` (`internal/cli/bench_overhead.go`): an instant
|
||||
engine and the gateway in one process, four modes (direct, bare, auth, auth
|
||||
and guard patterns) at rising rates. On an Apple M5 the gateway added
|
||||
0.05–0.42 ms at p50 and at most 0.49 ms at p99, in the README.
|
||||
`TestTheGatewaysOverheadIsMeasured` runs a second of it in CI with no bound
|
||||
asserted. Left: the CPU figure is rough with everything in one process; a
|
||||
gateway in a process of its own would measure it properly. The original
|
||||
item follows.
|
||||
|
||||
Bifrost's headline is "11 µs at 5k RPS" (its own figure, and its own pages
|
||||
disagree with it). llmgw has none, so any question about its overhead is
|
||||
answered with a guess.
|
||||
- `llmgw ctl bench --overhead`: the same load through the gateway and straight
|
||||
at a `sim-engine` (`internal/cli/simengine.go`) that answers at once, at
|
||||
rising request rates, with auth, accounting and the guardrails' pattern
|
||||
checks on and off. Reports the added latency at p50/p99 and the gateway's
|
||||
CPU per thousand requests.
|
||||
- On the PoC the CPU is oversubscribed (pve1, 28 threads for ~125 vCPUs):
|
||||
report it as a ratio and record the host's load beside it, and repeat on
|
||||
an idle machine before a figure goes into the README.
|
||||
- Done when: the README states the overhead, with how it was measured, and
|
||||
CI runs the short version so a change that doubles it is seen.
|
||||
|
||||
### I4. Several keys for one outside provider (S) — done 2026-10-17
|
||||
`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
|
||||
each in `control.externalEndpoints`, 401 retried on another key and
|
||||
`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
|
||||
still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
|
||||
original item follows.
|
||||
|
||||
An external model has one `api_key_file` (`internal/control/control.go`,
|
||||
where external endpoints are made). A provider's limit is per key; one key
|
||||
answering 429 makes the whole fallback answer 429.
|
||||
- `api_key_files: [a, b, c]`, each with an optional `weight`: one endpoint
|
||||
per key (`external:<model>#<n>`), so the router's scoring, retries and
|
||||
breakers work per key as they already do per replica. A key answering 429
|
||||
or 401 is passed over while its breaker is open; a 401 also notifies
|
||||
(`external_key_refused`).
|
||||
- The usage row says which key answered (by its index, never the key).
|
||||
- Done when: with two keys and one returning 429, every request is answered
|
||||
by the other, and the console shows the one breaker open.
|
||||
|
||||
## J. Found testing HA and single-machine installs (written 2026-10-16)
|
||||
|
||||
Three VMs installed as HA from the setup screens, a fourth as a judge node,
|
||||
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
|
||||
|
||||
### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
|
||||
Not a wait for ever, as first written: on the 4-core VM the judge's restart
|
||||
was *refused* ("fits on no node even with this replica stopped") because the
|
||||
starter, restarted at the same time to reserve fewer cores, still held its
|
||||
old ones. Once refused, nothing tried again, so the judges kept their old
|
||||
figures. Two fixes in `internal/control/reconciler`:
|
||||
- a restart whose replacement fits once other restarts under way have gone
|
||||
(queued, retiring or draining) waits for them, up to DrainGrace, instead
|
||||
of being refused (`fitsAfterRestarts`, restart.go);
|
||||
- a drained replica counts a retiring one that still answers as serving, so
|
||||
one restart after another no longer leaves the first's drained replica,
|
||||
and the room it holds, waiting out its grace (reconciler.go).
|
||||
`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
|
||||
|
||||
### J2. A judge the leader adds is sized as init's is (S)
|
||||
When a policy names a judge the catalog lacks, the leader serves it itself
|
||||
(`internal/cli/judgekeeper.go` via `knownJudge.serveRequest` in
|
||||
`judges_known.go`, through modelserve). That judge reserves 3 GB (the
|
||||
weights, an 8192-token KV cache and a fixed gigabyte), no cores, and runs
|
||||
with llama.cpp's prompt cache, which grew one to 3.2 GB. init's judge
|
||||
(`guardModel` in `internal/cli/serverinit_cluster.go`) reserves 2 GB and
|
||||
its cores and runs with `--cache-ram 0`.
|
||||
- `modelserve` gives a guard model `--cache-ram 0`, and the keeper's
|
||||
request the same memory and cores rule as `cpuPlan`.
|
||||
- Done when: a judge removed from the catalog comes back with the same
|
||||
figures init would have given it.
|
||||
|
||||
### J3. HA setup puts the judges on nodes of their own (S)
|
||||
Under load on a 4-core control plane that was also a node, 38 of 60
|
||||
requests were refused for want of a judgment. With the judges on a node of
|
||||
their own (4 cores, 32 GB, two cores each) the same test refused 2.
|
||||
- The HA setup offers a judge node or two: machines given like the store's,
|
||||
added as nodes with a label (`llmgw.role: judge`), and the judge's
|
||||
profile selects that label.
|
||||
- Done when: the setup installs the judges on the machines given for them,
|
||||
and the load test above refuses no more than it did by hand.
|
||||
|
||||
### J4. The model mirror on more than one machine (M) — done 2026-10-29, tried on the PoC
|
||||
In an HA install the mirror stays on the first machine. While it is down,
|
||||
models the nodes already have keep serving and new ones wait for it.
|
||||
- A second mirror kept in step (each machine of the store, or one more), the
|
||||
control planes and nodes given both, the first that answers used.
|
||||
- Done when: with the first machine powered off, a model added then is
|
||||
fetched and served.
|
||||
|
||||
### J5. An upgrade corrects an earlier install's judge (S) — done 2026-10-16
|
||||
Built in the judge keeper (`internal/cli/judgekeeper.go`, `refit`), on the
|
||||
leader: a judge whose entry is exactly what init before v0.14.7 wrote is
|
||||
given 2 GB, `--cache-ram 0` and cores by init's rule (`refitCores`: up to
|
||||
four for one judge, two each for more), through `modelsvc.SetProfiles`,
|
||||
and its ready replicas are restarted (after J1). Its replica count is left
|
||||
as it is. One that differs is left alone and said once in the log. The
|
||||
original item follows.
|
||||
|
||||
v0.14.6 and earlier installed two judges reserving no cores or one, with
|
||||
2 or 4 GB and the prompt cache on. On fewer than eight cores every judgment
|
||||
missed its 5 s budget and every request was refused. v0.14.7's notes say
|
||||
how to fix it by hand (`ctl model resources`, then `ctl restart`).
|
||||
- On upgrade, or on `server init` run again, a judge whose profile is still
|
||||
exactly what an earlier init wrote is moved to today's figures and
|
||||
restarted (after J1). One changed by hand is left alone and named.
|
||||
- Done when: a v0.14.6 single-machine install upgraded to it judges within
|
||||
budget without anything typed.
|
||||
|
||||
### J6. Rate limits per process (note)
|
||||
Rate limits are counted in each control plane's memory. A failover starts
|
||||
every key's minute afresh. Documented in install.md; worth a shared count in
|
||||
the store only if a customer's limits must hold across a failover.
|
||||
With `router.active_active` (O2, 2026-11-01) the counts are shared every 2 s,
|
||||
so a standby taking over keeps the minute; without it, as above.
|
||||
|
||||
### J7. nginx in front of several control planes (S)
|
||||
`deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
|
||||
nginx's open-source build has no active health checks, so its `upstream`
|
||||
block would send to standbys and to a lost leader alike.
|
||||
- nginx.conf: the floating address as its one upstream, and why; or the
|
||||
`max_fails` passive approach with what it costs.
|
||||
|
||||
### J8. A judge budget that fits the machine (S)
|
||||
A judgment takes 2.3 s on four cores and 4.2 on two, against the 5 s
|
||||
default: two cores are close to it, and a busy machine passes it.
|
||||
- At install, after the judge is up, time three judgments. If the slowest is
|
||||
more than half the budget, say so in the plan's closing text with the
|
||||
choices (more cores, a GPU judge, a longer `latency_budget`, `fail:
|
||||
open`). The budget itself is not changed without being asked.
|
||||
- Done when: an install on a machine too slow for its judge says so before
|
||||
its first refused request.
|
||||
|
||||
## L. Taken from Kong's AI Gateway (written 2026-10-25)
|
||||
|
||||
From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
|
||||
3.9, and only six of its AI plugins are open (proxy, prompt template,
|
||||
decorator, regex prompt guard, request and response transformer); the rest
|
||||
is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
|
||||
plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
|
||||
prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
|
||||
unshown), prompt templates and decorators (no one has asked), Konnect's
|
||||
entity model. MCP items are llmgw-80's (I2); they are listed for it.
|
||||
|
||||
### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
|
||||
`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
|
||||
operation, request and response model, max_tokens, temperature, top_p, the
|
||||
session as conversation id, input/output/cached tokens, the engine as
|
||||
provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
|
||||
Prometheus already counted tokens by direction, cached apart. Left: the MCP
|
||||
calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
|
||||
|
||||
### L2. Redacted personal data put back in the answer (S) — done 2026-10-25
|
||||
`restore: true` (`guard.Rules.Restore`), `Verdict.Restore`, `internal/router/restore.go`
|
||||
ahead of the output filter. The original item follows.
|
||||
The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
|
||||
`placeholders`), the same value always the same placeholder. A model that
|
||||
answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
|
||||
- Per policy, `restore: true`: the placeholders of one request kept in memory
|
||||
for its life, and swapped back in the answer, streamed too (a placeholder
|
||||
split across two chunks held until it is whole).
|
||||
- Only into the answer to the request that held them; never stored, never in
|
||||
usage rows or traces.
|
||||
- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
|
||||
policy reaches the model as placeholders and the client reads the real name
|
||||
and address back.
|
||||
|
||||
### L3. A guardrail that is someone else's service (S) — done 2026-10-25
|
||||
`guard.Service` (`internal/guard/service.go`), policy `services`, asked
|
||||
before the judge on input and once on the whole answer. The original item follows.
|
||||
Customers may be required to use one: Azure Content Safety, Lakera, NeMo
|
||||
Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
|
||||
templated HTTP call with an expression for "block".
|
||||
- A detector kind `http`: URL, headers from secret files, a body template
|
||||
with the text, a JSON path that says blocked and one that says why;
|
||||
timeout and `fail` as the judge's. Checked on input, output or both.
|
||||
- Done when: a policy with an `http` detector against a test service blocks
|
||||
what it says to, and a service that does not answer is treated as `fail`
|
||||
says.
|
||||
|
||||
### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
|
||||
`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
|
||||
what `fallbacks` already does (local, then an outside model when no room). Left:
|
||||
routing by cost between several outside providers. The original item follows.
|
||||
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
|
||||
and KV cache. Kong also balances by measured latency, by tokens or cost
|
||||
used, and in priority tiers.
|
||||
- A `latency` scorer: a moving average of time to first token per endpoint.
|
||||
- Fallbacks in tiers: local replicas first, then an outside provider, with
|
||||
the tier said in the usage row.
|
||||
- Done when: with two replicas of one model, one on a slower card, `latency`
|
||||
sends most of the traffic to the faster and the slower still gets some.
|
||||
|
||||
### L5. Prices by kind and by cache (S–M) — done 2026-10-25
|
||||
`cached_per_1k`, charged on the answer's `cached_tokens`. A budget in money was already
|
||||
there (`keys limits --budget`). Not done: cache-write and audio/video token prices —
|
||||
no engine llmgw runs reports them, and no provider it reaches is billed by them yet.
|
||||
The original item follows.
|
||||
`catalog` prices are per million input/output tokens and per picture/mesh.
|
||||
Kong prices cached reads and cache writes apart, and audio, image and video
|
||||
tokens apart, and limits by cost.
|
||||
- Price fields for cached input and by modality; the usage row's cost from
|
||||
them; a budget limit in money as well as tokens (`keys limits --cost`).
|
||||
- Done when: a request with 90% cached prompt costs what the cached rate says.
|
||||
|
||||
### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
|
||||
Kong turns OpenAPI operations into MCP tools. A company with an internal
|
||||
REST API and no MCP server could offer it to agents through `/mcp`:
|
||||
`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
|
||||
operation a tool, under the same grants, audit, guardrails and rate as I2.
|
||||
|
||||
### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
|
||||
Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
|
||||
resource metadata, tokens checked by JWKS against the company's identity
|
||||
provider), and a token exchanged for a short-lived one per upstream server
|
||||
rather than one long-lived secret.
|
||||
|
||||
### L8. A2A (S, when asked)
|
||||
Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
|
||||
llmgw's, calls audited and traced. The standard is young; wait for a
|
||||
customer.
|
||||
|
||||
### L9. llmgw behind Kong, written down (S) — done 2026-10-25
|
||||
docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on;
|
||||
Kong 3.9 counts no completion tokens of a stream. The original item follows.
|
||||
A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
|
||||
`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
|
||||
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
|
||||
container, and what Kong's token accounting sees.
|
||||
|
||||
## M. Models from Hugging Face we can trust (written 2026-10-25)
|
||||
|
||||
The user asked how we know a model fetched from Hugging Face is safe. Two
|
||||
questions: whether its files can run code on our machines, which we can
|
||||
close, and whether the model itself was trained to misbehave, which no scan
|
||||
proves — only who published it, an eval before it serves, and the
|
||||
guardrails narrow that. What existed: `--trust-remote-code` refused, no
|
||||
repository Python fetched, ComfyUI nodes pinned to a commit, engines
|
||||
hash-pinned and run as `llmgw-engine`, mirror versions pinned to a commit
|
||||
and never overwritten. What did not: everything below.
|
||||
|
||||
### M1. Weights in a pickle format refused (S) — done 2026-10-25
|
||||
`.bin`, `.pt`, `.pth`, `.ckpt`, `.pkl` run code as they are loaded, which is
|
||||
how the malicious models found on Hugging Face work. The node refuses such a
|
||||
source, or a mirror set holding one, before fetching anything; a model being
|
||||
added or given such a source is refused; vLLM and SGLang downloading a model
|
||||
by its id are given `--load-format safetensors`. The explorer hides a repository with neither GGUF nor safetensors
|
||||
(under "runs here", ticked by default) and does not offer a repository's
|
||||
pickle files. `allow_pickle: true` on a model lets it through, on purpose. On the
|
||||
PoC, `triposr` (`hf:stabilityai/TripoSR/model.ckpt`) is the one model that
|
||||
needs it.
|
||||
|
||||
### M2. Every download pinned and checked against Hugging Face (M) — done 2026-10-30
|
||||
Done: what is added from Hugging Face is pinned (the file's commit URL, or
|
||||
`--revision` for vLLM/SGLang) with each file's sha256 in the origin; the node
|
||||
refuses and deletes a copy that is not it, naming both, and keeps the
|
||||
checksum beside a cached file; the mirror checks what `ctl model serve`
|
||||
fetches, `mirror add` a Hugging Face URL. Left: models declared in the
|
||||
settings are not pinned, the admin page's and the console's mirror fetches
|
||||
send no sha256, and adapters by hub id are not pinned.
|
||||
- `hf:org/repo/file` follows `main` and is checked by nothing; a bare
|
||||
`org/repo` given to vLLM is downloaded by vLLM, of whatever `main` is.
|
||||
Both: resolve to the commit when the model is added, keep it in the
|
||||
catalog's origin, and fetch that commit only (`--revision` for vLLM).
|
||||
- The sha256 Hugging Face publishes for each LFS file compared on every
|
||||
path, and a file that differs removed: `mirror add`, the manager, the
|
||||
node's `hf:` and http sources, mirror set files with no sha recorded.
|
||||
- Done when: a file changed on Hugging Face after a model was added is
|
||||
refused by the node with both checksums named.
|
||||
|
||||
### M3. Hugging Face's own scan (S)
|
||||
Hugging Face scans every file (pickle imports, ClamAV, partners' scanners)
|
||||
and marks a file unsafe. Ask for it with the repository listing; refuse an
|
||||
unsafe file, show the result on the mirror page and in the catalog's origin.
|
||||
The API field has to be confirmed first.
|
||||
|
||||
### M4. Publishers we trust, and the rest asked about (S)
|
||||
An allow-list of organisations in the cluster's settings (Qwen, meta-llama,
|
||||
mistralai, google, microsoft, …). A model from anyone else is added only by
|
||||
an admin who says so, and who said it is kept in the origin. The PoC already
|
||||
serves four community models whose training nobody here checked.
|
||||
|
||||
### M5. Engines without the internet (M)
|
||||
The firewall is inbound only, so an engine that was made to run code can
|
||||
send what it reads anywhere. With weights from the mirror the engines need
|
||||
no egress: an nft output rule for the engine user (allow the mirror, the
|
||||
cluster, loopback), systemd hardening of the agent's engines
|
||||
(`NoNewPrivileges`, `ProtectSystem`, private /tmp). Agent itself still root.
|
||||
|
||||
### M6. An eval before a new model serves (S)
|
||||
`eval` and `canary` exist; make a new model from an untrusted publisher
|
||||
(M4) serve only after its eval suite passes, so a model trained to answer
|
||||
wrongly on a trigger has at least been asked the suite.
|
||||
|
||||
## N. Strata, all the way up (written 2026-10-30)
|
||||
|
||||
Strata (opt-in engine since 2026-10-29) is wired at the agent: adapter, `llmgw strata-serve`, install pinned by
|
||||
digest, metrics in vLLM's names, GPU memory of child processes. Above the agent it is invisible: a Strata model is
|
||||
served only from a definition written by hand. In order:
|
||||
|
||||
**A. What is broken.**
|
||||
- N1. A Strata named by hand (`engines.strata.dir`) was never reported: no Probe. Done 2026-10-30.
|
||||
- N2. Its engine runs in a session of its own and survived a stop, holding the card and the RAM: the supervisor
|
||||
now finds what left the group before signalling it. Done 2026-10-30.
|
||||
- N3. Host memory: the experts live in RAM (40 GB for Qwen3.8-Flash-Next IQ2_XS, 72–136 GB for Unsloth's) and
|
||||
nothing reserved it. A Strata profile added now reserves what its size takes, from the table (N5), unless it says
|
||||
`min_memory_gb`. Done 2026-10-30.
|
||||
- N4. Pascal and Volta: installed (CUDA 12.9 build) and then refused (7.0 floor). Floor now 6.0. Done 2026-10-30.
|
||||
|
||||
**B. Offered where models are chosen.**
|
||||
- N5. A table of Strata's families and sizes (qwen, coder, swift, unsloth, gyro, nvfp4, qwen36, deepseek41) with
|
||||
download and RAM, from Strata 0.2.7's setup (internal/strata/models.go); the catalog refuses an unknown family or
|
||||
size, and requires `--size`. Done 2026-10-30. Speeds measured: Qwen3.8-Flash-Next IQ2_XS 70–87 tok/s on an RTX 6000.
|
||||
- N6. `modelplan`, `ctl model plan` and `ctl model serve` offer "Strata on one card + N GB RAM" for a model it runs;
|
||||
a large MoE that fits no card is not "fits nowhere". Done 2026-10-30, without a speed (one measurement so far).
|
||||
- N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
|
||||
- N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
|
||||
`ctl model context` takes Strata (setup's `--context`), and `--max-output` changes the longest answer alone,
|
||||
in place, with no restart: done 2026-11-01 (31f440a). Open: a Strata model served without `--context` still runs
|
||||
at what Strata's setup picks (4096 on cuda01) while the catalog says the model's trained context (32768), so
|
||||
serve should write `--context` from `context_length`; and `--parallel`.
|
||||
|
||||
- N7b. The mirror page's Strata tab: every family and size, fetch (GGUF) or serve (packs Strata fetches itself:
|
||||
NVFP4, DeepSeek V4.1). Done 2026-10-30. Community `strata-ple` packs Strata 0.2.7 does not list stay unservable;
|
||||
running them would mean starting the engine past Strata's setup (`--pack`, `--native`, `--ple-gguf`).
|
||||
|
||||
**C. Operating it.**
|
||||
- N9. `ctl engine-check strata`: the flags of setup.py and server.py, which drift between versions.
|
||||
- N10. `ctl engine-build NODE strata cuda12|cuda13`, as vLLM has.
|
||||
- N11a. Strata's per-replica state directories (engine-home/strata/<replica>) are left behind when a replica stops;
|
||||
remove them with it.
|
||||
- N11. Disk for the prepared pack and draft layer (and DeepSeek V4.1's 240–280 GB), cleaned with the weights; a
|
||||
first start's preparation within the startup timeout.
|
||||
- N12. docs/install.md: Strata in the engines table, a section with a profile, its memory and first-run time;
|
||||
`doctor` and `server init` checking or offering it.
|
||||
|
||||
**D. What Strata can do that llmgw does not ask of it yet.**
|
||||
- N13. Several cards of one node (`--gpus`, `--gpu-roles`): larger packs on a customer's multi-GPU machines.
|
||||
- N14. `--calibrate` once per node and model, kept.
|
||||
- N15. Vision, KV types, experts from SSD when RAM is short (`--low-ram`) in the plan and forms.
|
||||
- N16. Versions apart from llmgw's releases (Strata's signed update feed, `--rollback-engine`); offline bundles.
|
||||
- N17. `cache_salt` for KB isolation if Strata caches prefixes; `ctl bench` with its metrics path.
|
||||
|
||||
## O. More than one place answering clients (written 2026-10-30)
|
||||
|
||||
Every client request goes through the leader's router today: the floating address is on the leader, and a standby
|
||||
passes `/v1` on to it (a standby's router sees an empty cluster). That is failover, not scale: the cluster serves what
|
||||
one control plane's router can. In order:
|
||||
|
||||
- O1. Measured 2026-10-30/31 on the PoC: the leader (2 vCPU, 2 GB, pve1 at load 6–10 of 28 threads) in front of
|
||||
a fake engine on node06 (`sim-engine`, 500 tok/s a stream, no batch slowdown, as an external model), 64-token
|
||||
streams from mirror 146, the default `observe` guardrails and usage written to PostgreSQL:
|
||||
|
||||
| at once | req/s | tok/s | first token p50/p95 | leader llmgw CPU |
|
||||
|---|---|---|---|---|
|
||||
| 64 | 465 | 29,752 | 9 / 34 ms | 159% |
|
||||
| 128 | 572 | 36,607 | 91 / 167 ms | 181% |
|
||||
| 256 | 577 | 36,900 | 310 / 392 ms | 182% |
|
||||
| 512 | 551 | 35,240 | 793 / 883 ms | 181% |
|
||||
|
||||
The ceiling is the router's CPU: ~575 requests/s, ~37k streamed tokens/s on two cores, no errors; past it the
|
||||
first token waits. Found on the way and fixed: an external model held to 3–4 at once by admission; a valid key
|
||||
throttled as "unrecognised" when its cache entry expired under load.
|
||||
|
||||
Profiled under 128 streams (2026-10-31): a third of the CPU was one write a token to the client; joining the
|
||||
events already in hand into one write took the ceiling to ~700 req/s and ~45k tok/s, first token p50 45 ms at 128.
|
||||
Now the largest part is the output guardrail of the default `observe` policy, ~30%: decoding each event (~20%;
|
||||
a token-level reader with encoding/json/jsontext was measured slower than json.Unmarshal, 1.66 vs 1.27 µs) and
|
||||
the PII patterns (~12%). Ways on, a guardrail decision: judge shadow output in larger windows, or a sample of it.
|
||||
Next: the same run on a machine the size of a customer's control plane, then O2.
|
||||
- O2. Every control plane answers `/v1` itself (active-active):
|
||||
- [x] the leader's replica table followed live by the standbys, so their routers route (2026-11-01, behind
|
||||
`router.active_active`, off by default). Every 2 s a standby posts what its router used to the leader's
|
||||
`POST /api/v1/routing/activity` over the agent port with its control-plane certificate, and is answered with
|
||||
the Ready replicas: no keys travel, each control plane derives a replica's key from the cluster secret. The
|
||||
leader takes the last uses in (else a replica busy behind a standby is evicted as idle) and the on-demand models
|
||||
asked for; a catalog changed on the leader is read again from the store. Unheard from the leader for a minute,
|
||||
a standby routes nothing and passes `/v1` on as before. The portal, `/admin` and `/mcp` still go to the leader.
|
||||
- [x] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
|
||||
- [x] rate limits cluster-wide (2026-11-01): each router's per-counter request and token counts, this minute's
|
||||
and the last as absolute figures, ride the same 2 s exchange; the leader answers each standby with everyone
|
||||
else's sum, its own included, and puts the standbys' sum in force on its own router. Each limiter adds the
|
||||
others' counts to its sliding window. Overshoot is bounded by what N routers allow in 2 s; a standby quiet for
|
||||
10 s stops counting. (J6's single-process case was never the problem.)
|
||||
- [ ] a session's requests to one router and one replica: the session ring already picks the same replica on
|
||||
every router for the same replica set; what is left is the same router, which is the balancer's (source hash).
|
||||
- [ ] in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer
|
||||
(J7, L9). Until then the standbys' own addresses are what spread the load.
|
||||
- [x] tried on the PoC (2026-11-01, v0.16.1-14-g8b7eefe, setting on since): 140 leads, 150 follows. A request
|
||||
through 150 is answered and logged by 150; a model added on the leader is routable on 150 within a tick; a
|
||||
CPU replica used only through 150 shows idle 6 s on the leader. Load from 146, 64-token streams, a sim engine on
|
||||
node06 at 5,000 tok/s a stream (2,300 req/s on its own; at 500 tok/s it was the ceiling, ~490 req/s, routers or
|
||||
not), pve1 at load 14 of 28 threads:
|
||||
|
||||
| streams | leader alone | split over 140 + 150 | first token p50, alone / split |
|
||||
|---|---|---|---|
|
||||
| 128 | 971 req/s (140 at 176%) | 832 + 692 = 1,524 req/s (178% / 166%) | 115 / 59–75 ms |
|
||||
| 256 | 990 req/s (177%) | 816 + 677 = 1,493 req/s (177% / 164%) | 243 / 135–166 ms |
|
||||
|
||||
1.5× with two routers on an oversubscribed host, no errors. Not 2×: both routers' VMs and the load generator
|
||||
share pve1's cores.
|
||||
- Known and accepted for now, per router: the precise KV index is empty on a standby (agents stream to the leader
|
||||
only), a revoked key works on a standby until `auth.cache_ttl`, canary promotion sees the leader's share only,
|
||||
the answer cache and prefix tree are split, an MCP caller's rate is per router.
|
||||
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
|
||||
control plane, as many as the traffic needs.
|
||||
|
||||
## P. Training beyond one card (written 2026-11-11) — decision open
|
||||
|
||||
F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
|
||||
Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
|
||||
|
||||
**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
|
||||
venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
|
||||
draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
|
||||
storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
|
||||
the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
|
||||
several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
|
||||
inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
|
||||
multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
|
||||
our job format.
|
||||
|
||||
**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
|
||||
planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
|
||||
llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
|
||||
or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
|
||||
M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
|
||||
multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
|
||||
upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
|
||||
data are Slurm's and the file system's, not the KB's.
|
||||
|
||||
Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
|
||||
|
||||
Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
|
||||
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
|
||||
mid-size jobs without wanting Slurm.
|
||||
|
||||
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
|
||||
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
|
||||
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
|
||||
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
|
||||
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
|
||||
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
|
||||
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
|
||||
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
|
||||
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
|
||||
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
|
||||
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
|
||||
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
|
||||
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
|
||||
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
|
||||
Later: a backup slurmctld on a second node, accounting (slurmdbd; it takes MySQL/MariaDB only, not the Patroni PG), users from LDAP/AD.
|
||||
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
|
||||
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
|
||||
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
|
||||
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
|
||||
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
|
||||
ending. Found and fixed on the way: a client command needs its files even for --version (version now
|
||||
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
|
||||
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
|
||||
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
|
||||
want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES
|
||||
mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the
|
||||
partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
|
||||
23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
|
||||
backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
|
||||
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
|
||||
- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
|
||||
MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
|
||||
key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
|
||||
the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
|
||||
AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
|
||||
submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
|
||||
a Red Hat controller.
|
||||
- Fresh install tried 2026-11-12 on VMs (3 control planes with Patroni, 2 CPU nodes, `server init
|
||||
--slurm`): Slurm, slurmdbd and MariaDB came up with the install; two-node srun/sbatch on the lent nodes,
|
||||
`sacct` has them; leader failover kept serving. Open from it: `--slurm` needs the first machine to be a
|
||||
node (the controller is its node), so a site wanting control planes without nodes chooses the
|
||||
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
|
||||
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
|
||||
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
|
||||
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
|
||||
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
|
||||
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
|
||||
before it takes requests.
|
||||
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
|
||||
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
|
||||
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
|
||||
own mount only looked for; `slurm.users: llmgw` — every active account a Linux user everywhere
|
||||
(`slurm_users`: login from the e-mail, UID from 200000, never reused; homes made on the exporting
|
||||
machine, else the controller; keys in /etc/llmgw/ssh-keys/%u through an sshd drop-in), SSH keys
|
||||
from the portal's Profile or `ctl slurm users keys`. A user already on a machine with another UID
|
||||
is left alone and said. The site has AD (not yet live): users from it, through sssd, are the next
|
||||
step (`slurm.users` empty today leaves them to the site).
|
||||
- Logging in to a node, done 2026-11-14: llmgw's users reach the controller freely, another Slurm
|
||||
machine only while a job of theirs runs there (pam_slurm_adopt, installed by the agent; PrologFlags=
|
||||
Contain). Only the users listed in the agent's users file are held: root and the machine's own users
|
||||
log in as before, so a wrong line never locks out who would fix it. pam_systemd would move the session
|
||||
out of the job, so sshd's stack takes the session file in place of its include, with pam_systemd
|
||||
skipped for the listed users alone; where that cannot be done safely (nested includes, jumps across
|
||||
it) they are held to having a job, not to its cores and cards, and the agent says so. Tried on the
|
||||
VMs: ayse without a job refused ("no active jobs on this node"), with one in
|
||||
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
|
||||
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
|
||||
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
|
||||
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
|
||||
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
|
||||
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
|
||||
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
|
||||
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
|
||||
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
|
||||
Done (c3a75dc): it says the time.
|
||||
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
|
||||
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
|
||||
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
|
||||
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
|
||||
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
|
||||
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
|
||||
once every machine reports both, since a machine without the plugin would fail every job. Tried on
|
||||
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
|
||||
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
|
||||
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
|
||||
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
|
||||
- AD end to end, tried 2026-11-14 (decided: one AD per install, the existing auth.ldap group mapping, no
|
||||
tree from AD, no sssd unless a site asks): the VMs' auth.ldap against Samba DC VM 165 (ldaps by IP,
|
||||
CA in the settings, ml-team -> team:Test/arge). ada signed in on the portal with her AD password: an
|
||||
account, in Test/arge, Slurm user ada UID 200003; her key added in the portal's Profile, ssh to the
|
||||
controller ~20 s later, sbatch ran on n1 as ada. Disabled in AD: suspended in 56 s, the Linux user
|
||||
gone in 89 s, ssh refused, home kept. Enabled: back in 32 s, same UID, key kept. Taken out of
|
||||
ml-team: out of the team in 11 s; put back: in again in 56 s.
|
||||
- P6, done 2026-11-15 but for budgets: each project a Slurm account (org and team accounts above it,
|
||||
ids keyed by kind: the first install's org, team and project share the id "default"), its members and its
|
||||
team's and org's on it; AccountingStorageEnforce=associations; the controller's agent keeps them with
|
||||
sacctmgr (only what it made; a user's default account moved before an association goes, the user taken out
|
||||
when none is left) and reports the jobs that ended in the last day; the leader records each once
|
||||
(slurm_jobs, keyed cluster/id) with a usage row, model slurm/PARTITION, priced at gpu_hour_cost or
|
||||
core_hour_cost. Tried on the VMs: accounts made, ayse's job on her project's account ran and another
|
||||
project's was refused, both jobs in deneme's usage at 345 and 0 micro-units. Left: budgets (a job's cost
|
||||
is not counted against a budget, which binds keys; GrpTRESMins from a project's budget would hold its jobs).
|
||||
- P6 as written 2026-11-12: The tenancy tree to Slurm:
|
||||
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
|
||||
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
|
||||
`sacct --json` for jobs ended since the last read (through the controller's agent) into a
|
||||
`batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
|
||||
per GPU-hour and core-hour (auth.gpu_hour_cost, core_hour_cost: the same basis `ctl prices suggest`
|
||||
prices tokens from, 2026-11-15) into cost_micros, summed with the requests' usage on the usage pages and in
|
||||
budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
|
||||
Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
|
||||
llmgw-made). Also found: slurm.conf
|
||||
wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
|
||||
until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
|
||||
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
|
||||
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
|
||||
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
|
||||
controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
|
||||
nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
|
||||
nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
|
||||
again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
|
||||
ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
|
||||
SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
|
||||
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
|
||||
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
|
||||
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
|
||||
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
|
||||
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
|
||||
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
|
||||
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: ~~the TUI screen~~
|
||||
(done 2026-11-11: console **Slurm**, lend/take back on a row), P5,
|
||||
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
|
||||
taken to be nvidia-smi's index.
|
||||
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
|
||||
screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
|
||||
`slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
|
||||
inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
|
||||
not take.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
`/var/lib/llmgw/admin-token` (A5), `ctlClient` in `internal/cli/ctl.go`.
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
Metrics on `:9091` with `/prometheus/targets` beside them.
|
||||
- Store: `internal/store`, migrations under `internal/store/migrations/{sqlite,postgres}`;
|
||||
the PoC runs PostgreSQL under Patroni.
|
||||
|
||||
+1
-1090
File diff suppressed because it is too large.
Load diff
@@ -1,105 +0,0 @@
|
||||
# Functional Source License, Version 1.1, ALv2 Future License
|
||||
|
||||
## Abbreviation
|
||||
|
||||
FSL-1.1-ALv2
|
||||
|
||||
## Notice
|
||||
|
||||
Copyright 2026 Ilker Manap
|
||||
|
||||
## Terms and Conditions
|
||||
|
||||
### Licensor ("We")
|
||||
|
||||
The party offering the Software under these Terms and Conditions.
|
||||
|
||||
### The Software
|
||||
|
||||
The "Software" is each version of the software that we make available under
|
||||
these Terms and Conditions, as indicated by our inclusion of these Terms and
|
||||
Conditions with the Software.
|
||||
|
||||
### License Grant
|
||||
|
||||
Subject to your compliance with this License Grant and the Patents,
|
||||
Redistribution and Trademark clauses below, we hereby grant you the right to
|
||||
use, copy, modify, create derivative works, publicly perform, publicly display
|
||||
and redistribute the Software for any Permitted Purpose identified below.
|
||||
|
||||
### Permitted Purpose
|
||||
|
||||
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
|
||||
means making the Software available to others in a commercial product or
|
||||
service that:
|
||||
|
||||
1. substitutes for the Software;
|
||||
|
||||
2. substitutes for any other product or service we offer using the Software
|
||||
that exists as of the date we make the Software available; or
|
||||
|
||||
3. offers the same or substantially similar functionality as the Software.
|
||||
|
||||
Permitted Purposes specifically include using the Software:
|
||||
|
||||
1. for your internal use and access;
|
||||
|
||||
2. for non-commercial education;
|
||||
|
||||
3. for non-commercial research; and
|
||||
|
||||
4. in connection with professional services that you provide to a licensee
|
||||
using the Software in accordance with these Terms and Conditions.
|
||||
|
||||
### Patents
|
||||
|
||||
To the extent your use for a Permitted Purpose would necessarily infringe our
|
||||
patents, the license grant above includes a license under our patents. If you
|
||||
make a claim against any party that the Software infringes or contributes to
|
||||
the infringement of any patent, then your patent license to the Software ends
|
||||
immediately.
|
||||
|
||||
### Redistribution
|
||||
|
||||
The Terms and Conditions apply to all copies, modifications and derivatives of
|
||||
the Software.
|
||||
|
||||
If you redistribute any copies, modifications or derivatives of the Software,
|
||||
you must include a copy of or a link to these Terms and Conditions and not
|
||||
remove any copyright notices provided in or with the Software.
|
||||
|
||||
### Disclaimer
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
|
||||
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
|
||||
|
||||
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
|
||||
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
|
||||
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
|
||||
|
||||
### Trademarks
|
||||
|
||||
Except for displaying the License Details and identifying us as the origin of
|
||||
the Software, you have no right under these Terms and Conditions to use our
|
||||
trademarks, trade names, service marks or product names.
|
||||
|
||||
## Grant of Future License
|
||||
|
||||
We hereby irrevocably grant you an additional license to use the Software under
|
||||
the Apache License, Version 2.0 that is effective on the second anniversary of
|
||||
the date we make the Software available. On or after that date, you may use the
|
||||
Software under the Apache License, Version 2.0, in which case the following
|
||||
will apply:
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
||||
this file except in compliance with the License.
|
||||
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed
|
||||
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
||||
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
@@ -40,9 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences) |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -52,24 +50,6 @@ budgets and rate limits decide who may use any of it.
|
||||
|
||||
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
|
||||
|
||||
- **Setup that explains, and high availability that is whole** (v0.14.6).
|
||||
`sudo llmgw` lists the parts of an install, each with what it is for and
|
||||
what is missing without it, and every one can be turned off or set: the
|
||||
starter model and its context, the mirror's directory, the judge's
|
||||
categories and what it does when it does not answer, limits, the
|
||||
portal's language, pictures and 3D. With three machines each runs the
|
||||
control plane, with a floating address on the leader, and any one can
|
||||
be lost: on three VMs, cutting the store primary's power lost no
|
||||
request. One ComfyUI serves an environment's models, pictures, 3D and
|
||||
workflows are MCP tools, and `/v1/messages` gives a model's reasoning as
|
||||
thinking blocks.
|
||||
- **Picture and 3D models that add themselves** (v0.14.5). Where a node's
|
||||
card fits flux-schnell or triposr and the catalog makes nothing of that
|
||||
kind, the leader adds it on demand, so a new cluster's portal has its
|
||||
Images and 3D pages; the cluster page and the console show each preset
|
||||
and add it by hand, and `presets.auto` turns it off. Stable Fast 3D's
|
||||
colours can be brought to the picture's (`albedo_match`), and every mesh
|
||||
carries its whole licence.
|
||||
- **Any ComfyUI workflow, and models by name** (v0.14.4). `POST
|
||||
/v1/workflows/runs` runs a ComfyUI model's workflow as a job, its inputs
|
||||
(text, numbers, choices, images, masks, audio, video, files) declared in
|
||||
@@ -185,8 +165,6 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
cluster that exists
|
||||
- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
|
||||
Kubernetes with the Helm chart, and why GPU nodes stay outside it
|
||||
- [docs/kong.md](docs/kong.md) — llmgw behind Kong Gateway, or Kong behind
|
||||
llmgw: the configuration, what Kong counts, and which to choose
|
||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||
when a replica spanning nodes will not run
|
||||
@@ -261,33 +239,6 @@ Add `--json` to any command for machine-readable output.
|
||||
says nothing about which of six constraints bit, so it prints every candidate
|
||||
considered and the reason it was rejected.
|
||||
|
||||
## What llmgw adds to a request
|
||||
|
||||
Measured with `llmgw ctl bench overhead`. It starts an engine that answers
|
||||
at once and a gateway in front of it, in one process on loopback. It sends
|
||||
the same load straight to the engine and through the gateway, and reports
|
||||
what the gateway added. Run on an Apple M5 (10 CPUs, load about 2) on
|
||||
2026-10-16, 10 s a step, 72,000 requests with none failed:
|
||||
|
||||
| Through the gateway | added at p50 | added at p99 |
|
||||
|---|---|---|
|
||||
| bare | 0.06–0.24 ms | 0.10–0.33 ms |
|
||||
| with authentication and usage accounting | 0.05–0.32 ms | 0.09–0.37 ms |
|
||||
| with the guardrails' pattern checks as well | 0.10–0.42 ms | 0.15–0.49 ms |
|
||||
|
||||
The range is across 100 to 2,000 requests a second. The overhead was
|
||||
*lowest* at 2,000, most likely because a busier processor keeps its cores
|
||||
awake. The guardrails' judge model is not in these figures: a judgment
|
||||
takes seconds on CPUs (see [install.md](docs/install.md)) and is a model's
|
||||
time, not the gateway's.
|
||||
|
||||
The command also prints the gateway's CPU per thousand requests. Gateway,
|
||||
engine and load share one process there, so that figure is rough (0.04 to
|
||||
0.65 s per thousand in the run above); the latency is the one to quote.
|
||||
Figures from a VM on a busy host move with the host: run it where you will
|
||||
deploy, and keep the load it prints beside the result. CI runs a one-second
|
||||
version on every change (`TestTheGatewaysOverheadIsMeasured`).
|
||||
|
||||
## Running the router
|
||||
|
||||
```sh
|
||||
@@ -315,9 +266,8 @@ within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
||||
admission off.
|
||||
|
||||
Routing profiles select how endpoints are scored: `chat` favours session
|
||||
affinity and the replica that has been answering soonest (its time to first
|
||||
token, smoothed), `batch` favours the shortest queue, `long-context` favours
|
||||
free KV cache. Pick one per model in the config, or per request with the
|
||||
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
||||
cache. Pick one per model in the config, or per request with the
|
||||
`x-llmgw-profile` header.
|
||||
|
||||
## sim-engine
|
||||
@@ -372,10 +322,3 @@ make rename NEW=git.example.com/team/llmgw
|
||||
## Language
|
||||
|
||||
Code, comments, tests, commit messages and documentation are in English.
|
||||
|
||||
## Licence
|
||||
|
||||
Functional Source License 1.1 with an Apache 2.0 future licence
|
||||
(FSL-1.1-ALv2); see [LICENSE](LICENSE). Each release may be used, read and
|
||||
changed for anything except offering it as a competing product or service,
|
||||
and becomes Apache 2.0 two years after it is published.
|
||||
+99
-1080
File diff suppressed because it is too large.
Load diff
@@ -30,7 +30,6 @@ message AgentMessage {
|
||||
KVEvents kv_events = 8;
|
||||
TrainStatus train_status = 9;
|
||||
TrainArtifact train_artifact = 10;
|
||||
SlurmStatus slurm_status = 11;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,7 +78,6 @@ message ControlMessage {
|
||||
AgentSettings agent_settings = 15;
|
||||
TrainStart train_start = 16;
|
||||
TrainCancel train_cancel = 17;
|
||||
Slurm slurm = 18;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -240,13 +238,6 @@ message Host {
|
||||
// nvidia_driver is the driver version, empty when there is none. A card
|
||||
// without a driver is not a GPU anything can use.
|
||||
string nvidia_driver = 6;
|
||||
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
|
||||
// the agent started: what decides a model's speed on the CPU. 0 unknown.
|
||||
double mem_bandwidth_gbs = 7;
|
||||
// mem_bandwidth_core_gbs is what one core reads alone, in GB/s: a replica
|
||||
// given a few threads of a large machine reads at a few times this, not at
|
||||
// the whole machine's rate. 0 unknown.
|
||||
double mem_bandwidth_core_gbs = 8;
|
||||
}
|
||||
|
||||
// Storage is one filesystem a node keeps weights on.
|
||||
@@ -343,13 +334,6 @@ message GPU {
|
||||
// memory the instance's share; utilization_percent stays zero, because no
|
||||
// driver measures it per instance.
|
||||
MIGDevice mig = 22;
|
||||
// How the card is linked to its host: the fastest generation card and
|
||||
// slot allow together, the card's own fastest, and the lanes in use and
|
||||
// the card's most. Zero when the driver did not say.
|
||||
int32 pcie_gen = 23;
|
||||
int32 pcie_gpu_gen = 24;
|
||||
int32 pcie_width = 25;
|
||||
int32 pcie_width_max = 26;
|
||||
}
|
||||
|
||||
// MIGDevice is which card a MIG instance is carved from, and how.
|
||||
@@ -454,21 +438,6 @@ message GPUSensor {
|
||||
int32 last_xid = 26;
|
||||
string last_xid_text = 27;
|
||||
int64 last_xid_unix = 28;
|
||||
// How much of the chip the work on it uses, which utilization_percent
|
||||
// does not say: that is the share of time any kernel ran, and a kernel on
|
||||
// one SM of a hundred counts in full. has_profile says these are present;
|
||||
// -1 is not reported. Read from NVML's GPM on Hopper and later, or from a
|
||||
// dcgm-exporter on the node; profile_source says which ("gpm", "dcgm").
|
||||
bool has_profile = 29;
|
||||
// sm_active_percent is the share of time the SMs had work, averaged over
|
||||
// all of them; sm_occupancy_percent how full those SMs were with warps.
|
||||
double sm_active_percent = 30;
|
||||
double sm_occupancy_percent = 31;
|
||||
// tensor_active_percent is the tensor cores; dram_active_percent the
|
||||
// memory interface, which decode is bound by.
|
||||
double tensor_active_percent = 32;
|
||||
double dram_active_percent = 33;
|
||||
string profile_source = 34;
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
@@ -619,13 +588,6 @@ message Heartbeat {
|
||||
// is not handed out. Empty from an agent that does not measure it, or from
|
||||
// one that could not this time.
|
||||
repeated int32 gpu_foreign_memory_mb = 19;
|
||||
|
||||
// mem_bandwidth_gbs is how fast the node's memory reads, in GB/s, once
|
||||
// the agent has measured it — in the background after it starts, so a
|
||||
// registration usually has none yet. 0 means not measured.
|
||||
double mem_bandwidth_gbs = 20;
|
||||
// mem_bandwidth_core_gbs is one core's read rate, as in Host.
|
||||
double mem_bandwidth_core_gbs = 21;
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
@@ -728,9 +690,6 @@ message FetchProgress {
|
||||
// is a real answer: a percentage would then be invented, and an invented
|
||||
// percentage that stops moving is worse than a byte count that keeps rising.
|
||||
int64 bytes_total = 3;
|
||||
// phase is what the bytes are: empty while they arrive, "checking" while the
|
||||
// node checks a set's files against the mirror's checksums.
|
||||
string phase = 4;
|
||||
}
|
||||
|
||||
// StartShard launches one shard. It is idempotent by shard_id: the control
|
||||
@@ -790,13 +749,6 @@ message EngineProfile {
|
||||
// kv_events has a vLLM engine publish its KV cache events to the agent, for
|
||||
// the router's precise index.
|
||||
bool kv_events = 13;
|
||||
// allow_pickle lets the node fetch weights in a pickle format, which run
|
||||
// code when loaded: the catalog's allow_pickle, set on purpose.
|
||||
bool allow_pickle = 14;
|
||||
// model_sha256 is the sha256 the registry published for the file
|
||||
// model_path names, from the catalog's origin: the node refuses a copy
|
||||
// that is not it. Empty when nothing is known to check against.
|
||||
string model_sha256 = 15;
|
||||
}
|
||||
|
||||
// RemoveWeights deletes cached model files a node no longer needs.
|
||||
@@ -988,129 +940,3 @@ message TrainArtifact {
|
||||
bytes data = 4;
|
||||
bool last = 5;
|
||||
}
|
||||
|
||||
// Slurm is what this node is in the Slurm cluster run beside llmgw
|
||||
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
|
||||
// an agent registers and whenever it changes; the agent installs what is
|
||||
// missing and keeps the daemons running as it says.
|
||||
message Slurm {
|
||||
// controller runs slurmctld here, on files.
|
||||
bool controller = 1;
|
||||
// node runs slurmd here, configless, fetching its files from conf_server.
|
||||
bool node = 2;
|
||||
// name is this node's NodeName in slurm.conf.
|
||||
string name = 3;
|
||||
// conf_server is slurmctld's host:port.
|
||||
string conf_server = 4;
|
||||
// munge_key is the key every machine of the cluster shares.
|
||||
bytes munge_key = 5;
|
||||
// files are the controller's, by name in /etc/slurm; empty for a node
|
||||
// that is not the controller.
|
||||
map<string, string> files = 6;
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
bool lent = 7;
|
||||
// shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path; unset or with no path, none.
|
||||
SlurmShared shared = 8;
|
||||
// manage_users has llmgw keep the Linux users below on this machine,
|
||||
// and take away those it made that are no longer among them.
|
||||
bool manage_users = 9;
|
||||
repeated SlurmUser users = 10;
|
||||
// make_homes has this machine make the users' home directories: the one
|
||||
// that exports the shared path, else the controller.
|
||||
bool make_homes = 11;
|
||||
// containers has the machine install pyxis and enroot, so jobs may run
|
||||
// in containers (srun --container-image), and set them up.
|
||||
bool containers = 12;
|
||||
// manage_accounts has the controller keep Slurm's accounts as accounts
|
||||
// says: one for each organization, team and project, nested, the users
|
||||
// on each project's. Only those llmgw made are changed or taken away.
|
||||
bool manage_accounts = 13;
|
||||
repeated SlurmAccount accounts = 14;
|
||||
}
|
||||
|
||||
// SlurmAccount is an account in Slurm's accounting, parents before their
|
||||
// children.
|
||||
message SlurmAccount {
|
||||
string name = 1;
|
||||
string parent = 2;
|
||||
repeated string users = 3;
|
||||
}
|
||||
|
||||
// SlurmJob is a job that ended, as sacct has it, for llmgw's accounting.
|
||||
message SlurmJob {
|
||||
string id = 1;
|
||||
string user = 2;
|
||||
string account = 3;
|
||||
string partition = 4;
|
||||
string state = 5;
|
||||
int64 start_unix = 6;
|
||||
int64 end_unix = 7;
|
||||
int64 elapsed_seconds = 8;
|
||||
int32 cpus = 9;
|
||||
int32 gpus = 10;
|
||||
}
|
||||
|
||||
// SlurmShared is where the Slurm cluster's shared files come from.
|
||||
message SlurmShared {
|
||||
// path is where every machine sees them, as /shared.
|
||||
string path = 1;
|
||||
// nfs is an export, host:/path, to mount at path; empty when this
|
||||
// machine exports path itself (serve) or the site mounts it.
|
||||
string nfs = 2;
|
||||
// serve has this machine export path over NFS to clients.
|
||||
bool serve = 3;
|
||||
// clients are the addresses the export answers.
|
||||
repeated string clients = 4;
|
||||
}
|
||||
|
||||
// SlurmUser is a Linux user that runs jobs, the same on every machine.
|
||||
message SlurmUser {
|
||||
string login = 1;
|
||||
// uid is also the user's own group's gid.
|
||||
uint32 uid = 2;
|
||||
// name is the account's, for the GECOS field.
|
||||
string name = 3;
|
||||
string home = 4;
|
||||
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
|
||||
repeated string ssh_keys = 5;
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
message SlurmStatus {
|
||||
// version is Slurm's, as "23.11.4"; empty when it is not installed.
|
||||
string version = 1;
|
||||
// hardware is `slurmd -C` on a node, which slurm.conf must match.
|
||||
string hardware = 2;
|
||||
bool slurmd_running = 3;
|
||||
bool slurmctld_running = 4;
|
||||
// state and reason are the controller's for this node, from scontrol.
|
||||
string state = 5;
|
||||
string reason = 6;
|
||||
// jobs is how many job steps run on this node now, counted from their
|
||||
// slurmstepd so a controller that is down does not hide them; -1 when the
|
||||
// agent cannot tell.
|
||||
int32 jobs = 7;
|
||||
// error is the last thing that went wrong keeping Slurm, empty when
|
||||
// nothing did.
|
||||
string error = 8;
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
string files_sha256 = 9;
|
||||
// slurmdbd_running is the accounting daemon on the controller, which keeps
|
||||
// the jobs' records in a MariaDB beside it.
|
||||
bool slurmdbd_running = 10;
|
||||
// shared is the shared path as this machine has it: "mounted",
|
||||
// "exported", "missing", or empty when there is none.
|
||||
string shared = 11;
|
||||
// users is how many of llmgw's users this machine has.
|
||||
int32 users = 12;
|
||||
// containers is what this machine has for jobs in containers, as
|
||||
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
|
||||
string containers = 13;
|
||||
// ended are, on the controller, the jobs that ended in the last day; the
|
||||
// leader records each once.
|
||||
repeated SlurmJob ended = 14;
|
||||
}
|
||||
@@ -331,8 +331,6 @@ hidden files, version control folders and Office lock files are not.`,
|
||||
f.BoolVar(&o.Delete, "delete", false, "remove documents this machine sent whose files are gone")
|
||||
f.BoolVar(&o.Retry, "retry", false, "read again files skipped before")
|
||||
f.IntVar(&o.Workers, "workers", 4, "files read and sent at once")
|
||||
f.BoolVar(&o.EmbedHere, "embed-here", false,
|
||||
"embed on this machine and send the vectors, even to a gateway that embeds text it is sent")
|
||||
f.StringVar(&ocr, "ocr-languages", "tur+eng", "Tesseract's languages for images and scanned pages")
|
||||
f.Int64Var(&maxMiB, "max-size", 256, "skip files larger than this many MiB")
|
||||
f.BoolVarP(&verbose, "verbose", "v", false, "list unchanged files too")
|
||||
|
||||
+12
-37
@@ -1,4 +1,4 @@
|
||||
# More than one control plane
|
||||
# Two control planes
|
||||
|
||||
llmgw handles the hard half of this itself: only one control plane places work,
|
||||
and agents find whichever one is leading. What it does not do is move client
|
||||
@@ -37,7 +37,7 @@ nothing in front of the two had moved.
|
||||
So something has to. The options, in the order most people should consider
|
||||
them:
|
||||
|
||||
**A load balancer you already run.** Health-check `/leader` on each control
|
||||
**A load balancer you already run.** Health-check `/ready` on each control
|
||||
plane and send traffic to whichever answers. This is the right answer if you
|
||||
have one, and it needs nothing else from this directory.
|
||||
|
||||
@@ -50,50 +50,25 @@ in whatever your resolvers actually honour rather than what you set, which on
|
||||
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
||||
one an agent is driving.
|
||||
|
||||
## Check `/leader`, not `/health`
|
||||
## Check `/ready`, not `/health`
|
||||
|
||||
This is the one thing to get right, and the obvious choice is the wrong one.
|
||||
|
||||
`/health` says the process is up. It says nothing about whether it leads,
|
||||
`/health` says the process is up. It says nothing about whether it can serve,
|
||||
and a standby is exactly that case: alive, answering, and holding no endpoints
|
||||
at all — its reconciler placed nothing and it has no agent streams to adopt
|
||||
from. A standby passes client requests on to the leader, so traffic sent to
|
||||
one still works — until the leader is the machine that was lost, and there
|
||||
is nobody to pass them to.
|
||||
|
||||
`/leader` is 200 on the control plane that leads and 503 on a standby. It is
|
||||
what keepalived follows for the floating address, and what a load balancer
|
||||
should send traffic by: to the one machine that answers 200.
|
||||
|
||||
`/ready` is narrower: 200 only when this process has somewhere to send a
|
||||
request, so 503 on a standby and also on a leader with no model running yet.
|
||||
Checking it takes the portal and the admin pages away with the models — on a
|
||||
new cluster with no node yet, nobody can sign in to add one. Use it to stop
|
||||
sending inference only, if your load balancer can tell the paths apart.
|
||||
from. A load balancer checking `/health` would send it half the traffic, and
|
||||
half the traffic would be a 503.
|
||||
|
||||
`/ready` says this process has somewhere to send a request. It is 503 on a
|
||||
standby, 503 on a leader whose replicas have all failed, and 200 when there is
|
||||
something to serve — which is the question a load balancer is actually asking.
|
||||
Both are open, because a load balancer has no key.
|
||||
|
||||
During a failover nothing passes for about the length of the lease handover:
|
||||
During a failover neither passes for about the length of the lease handover:
|
||||
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
||||
is not. A proxy that waits that long for a healthy machine rather than
|
||||
failing at once (Caddy's `lb_try_duration`) hides it from clients.
|
||||
|
||||
**Caddy**, in front of three control planes — the end of
|
||||
[deploy/proxy/Caddyfile](../proxy/Caddyfile) has it with the rest of the
|
||||
settings:
|
||||
|
||||
```
|
||||
reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
|
||||
lb_policy first
|
||||
health_uri /leader
|
||||
health_interval 2s
|
||||
lb_try_duration 15s
|
||||
flush_interval -1
|
||||
}
|
||||
```
|
||||
|
||||
With a floating address the address already follows the leader: give Caddy
|
||||
that one address instead.
|
||||
is not. That gap is real and there is no way to have it both ways — traffic
|
||||
cannot go to a control plane that has nothing to route to.
|
||||
|
||||
Not the admin port, for a different reason: it serves pprof, and pprof can
|
||||
dump the process heap, where API keys sit in plaintext.
|
||||
|
||||
@@ -41,9 +41,7 @@ vrrp_instance llmgw {
|
||||
# MASTER on the machine you would rather serve; BACKUP on the other.
|
||||
state MASTER
|
||||
interface eth0
|
||||
# Any number both machines agree on and nothing else on the segment uses:
|
||||
# another cluster's keepalived with the same one reads as an invalid
|
||||
# password. `llmgw server add --vip` derives one from the address.
|
||||
# Any number both machines agree on and nothing else on the segment uses.
|
||||
virtual_router_id 51
|
||||
# The higher one wins while both are healthy. The weight above is enough
|
||||
# to invert this when the preferred machine's gateway is not answering.
|
||||
|
||||
@@ -504,7 +504,7 @@
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU busy, by card",
|
||||
"description": "Share of each sample period in which any kernel ran (nvidia-smi's utilization.gpu), sampled every heartbeat. Not how much of the chip: one SM busy reads 100%. The SMs panel and power against its limit say how much.",
|
||||
"description": "Share of each sample period a kernel was running, sampled every heartbeat.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
@@ -908,51 +908,6 @@
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "GPU SMs and tensor cores active, by card",
|
||||
"description": "From the cards' profiling counters: NVML's GPM on Hopper and later, or a dcgm-exporter on the node. SMs active is the share of time the SMs had work, averaged over all of them; the gap to busy is chip a running kernel leaves idle. Tensor cores stay low in decode at batch one; prefill and batches are where they rise. Empty for cards nothing can profile.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "llmgw-prom"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 73,
|
||||
"w": 24,
|
||||
"h": 7
|
||||
},
|
||||
"id": 25,
|
||||
"targets": [
|
||||
{
|
||||
"expr": "llmgw_gpu_sm_active_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}} SMs",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "llmgw_gpu_tensor_active_percent",
|
||||
"legendFormat": "{{node}} gpu {{gpu}} tensor",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -21,18 +21,11 @@ scrape_configs:
|
||||
# The control plane's dedicated metrics port. Not its admin listener: that one
|
||||
# is loopback because it also serves pprof, and pprof can dump the heap where
|
||||
# API keys sit in plaintext.
|
||||
#
|
||||
# Every control plane, one line each: the leader moves on restarts and the
|
||||
# router's metrics are the leading one's, so scraping one alone leaves gaps
|
||||
# — and alerts on them — whenever another leads.
|
||||
- job_name: llmgw
|
||||
static_configs:
|
||||
- targets: ['10.0.0.140:9091']
|
||||
labels:
|
||||
instance: control-plane-140
|
||||
# - targets: ['10.0.0.150:9091']
|
||||
# labels:
|
||||
# instance: control-plane-150
|
||||
instance: control-plane
|
||||
|
||||
# Every machine of the cluster, for its node exporter: network and disk
|
||||
# traffic, filesystems, CPU time. The control plane lists them — nodes,
|
||||
@@ -40,15 +33,10 @@ scrape_configs:
|
||||
# nothing edited here, and each carries its llmgw id as the node label.
|
||||
# `llmgw node add` installs the exporter; on a machine added before that,
|
||||
# apt-get install prometheus-node-exporter.
|
||||
# Asked of every control plane too: a standby lists only the control
|
||||
# planes and the store, and they label every machine alike, so the same
|
||||
# target from two of them is scraped once.
|
||||
- job_name: nodes
|
||||
http_sd_configs:
|
||||
- url: http://10.0.0.140:9091/prometheus/targets
|
||||
refresh_interval: 1m
|
||||
# - url: http://10.0.0.150:9091/prometheus/targets
|
||||
# refresh_interval: 1m
|
||||
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
|
||||
@@ -14,7 +14,6 @@ llm.example.com {
|
||||
max_size 32MB
|
||||
}
|
||||
|
||||
# One control plane. With more than one, see the end of this file.
|
||||
reverse_proxy 127.0.0.1:8080 {
|
||||
# Tokens are written the moment they are generated. Without this a
|
||||
# streamed reply can arrive in bursts, and a client measuring time to
|
||||
@@ -40,36 +39,3 @@ llm.example.com {
|
||||
# Access log to stderr, which under systemd is the journal.
|
||||
log
|
||||
}
|
||||
|
||||
# More than one control plane (server init --standbys, server add): one of
|
||||
# them leads, the others stand by. Name them all and send to the one that
|
||||
# leads — /leader is 200 there and 503 on a standby. Not /ready: that is 503
|
||||
# on a leader with no model running yet, and would take the portal away with
|
||||
# it. A standby would pass a request on to the leader anyway, but not while
|
||||
# there is none, and the hop is for nothing.
|
||||
#
|
||||
# With a floating address (--vip) the address already follows the leader:
|
||||
# give reverse_proxy that one address instead, and keep the rest as above.
|
||||
#
|
||||
# llm.example.com {
|
||||
# request_body {
|
||||
# max_size 32MB
|
||||
# }
|
||||
# reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
|
||||
# # The first that is healthy: the leader, whichever it is.
|
||||
# lb_policy first
|
||||
# health_uri /leader
|
||||
# health_interval 2s
|
||||
# health_timeout 2s
|
||||
# # While leadership moves — a few seconds when a leader is lost —
|
||||
# # none is healthy; a request waits that long for one rather than
|
||||
# # failing at once.
|
||||
# lb_try_duration 15s
|
||||
# lb_try_interval 250ms
|
||||
# flush_interval -1
|
||||
# transport http {
|
||||
# response_header_timeout 15m
|
||||
# }
|
||||
# }
|
||||
# log
|
||||
# }
|
||||
@@ -2,9 +2,6 @@
|
||||
Description=llmgw node agent
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never given up on: a new release that fails its first starts, or an upgrade
|
||||
# that restarts it twice in a minute, would otherwise leave the node off.
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
|
||||
|
||||
+2
-1949
File diff suppressed because it is too large.
Load diff
@@ -82,7 +82,7 @@
|
||||
"/mcp": {
|
||||
"post": {
|
||||
"operationId": "postMcp",
|
||||
"summary": "The gateway's MCP server: the memory notes, and pictures, 3D models, vision and workflows, as tools",
|
||||
"summary": "The gateway's MCP server: the memory notes as tools",
|
||||
"tags": [
|
||||
"mcp"
|
||||
],
|
||||
@@ -2890,10 +2890,6 @@
|
||||
"writable": {
|
||||
"type": "boolean",
|
||||
"description": "Writable is whether this caller may add and remove documents."
|
||||
},
|
||||
"server_embeds": {
|
||||
"type": "boolean",
|
||||
"description": "ServerEmbeds says a document may be sent as text alone: the gateway embeds its chunks, as the caller, with the collection's model."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -534,60 +534,6 @@ curl https://llm.example.com/mcp \
|
||||
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
|
||||
```
|
||||
|
||||
**Pictures, 3D models, vision and workflows.** Where the gateway has the
|
||||
models for them, the same server offers more tools — only those it has a
|
||||
model for:
|
||||
|
||||
| Tool | Does |
|
||||
|---|---|
|
||||
| `generate_image` | A picture from a prompt. `style` adds what a picture for a mesh wants: `concept` (one object, three-quarter view, grey background), `side` (an exact side view), `texture` (a seamless tile) or `none`. Answers the PNG's `file_id`, `url` and `seed`, with a 512 px preview to look at. |
|
||||
| `generate_mesh` | A GLB from a picture — `image` is a `file-…` id (a `generate_image` result, say), a data URL or base64 — or from a `prompt`, for which a concept picture is made first. `seed`, `resolution`, `texture_resolution`, and any other input the model's workflow declares in `inputs`. Answers the GLB's `file_id`, `url` and licence. |
|
||||
| `review_images` | Asks a vision model about one to four pictures (each shrunk to 1280 px on its long side); `json_schema` makes it answer that JSON. |
|
||||
| `run_workflow` | Runs a ComfyUI workflow from the inputs its catalog entry declares, and answers its result file. |
|
||||
| `job_status` | Where a mesh or a workflow run stands, and its file when done; `wait_seconds` waits for it. |
|
||||
| `gateway_status` | The picture, 3D, vision and workflow models, whether each is loaded on a GPU now, a workflow's inputs, and your last jobs. |
|
||||
|
||||
Each tool makes its request of the gateway under your key, so your
|
||||
allowlist, limits, left-out licences and budget apply, and it is charged
|
||||
and listed like any other request. What it makes is kept as your file for
|
||||
as long as job results are: fetch it from `url` with the same key (`curl -H
|
||||
"Authorization: Bearer $LLMGW_API_KEY" -o rock.glb <url>`). A tool server
|
||||
over HTTP cannot write to your disk, so an agent that needs the file
|
||||
locally — to import it in Blender — fetches it, or goes through a small
|
||||
local bridge that does.
|
||||
|
||||
A picture can be uploaded once and named by its id after: `POST /v1/files`
|
||||
with `purpose=vision` (or `user_data`) keeps it, for as long as job results
|
||||
are kept, as a file of yours that `generate_mesh`, `review_images` and a
|
||||
workflow's file inputs take as `file-…`. A call is at most 64 MB.
|
||||
|
||||
A model that is not loaded is waited for: a picture or a look at one whose
|
||||
model is starting, or waiting for another to give up the GPU, waits as a
|
||||
mesh does, saying so, rather than failing with "model loading".
|
||||
|
||||
A mesh takes from half a minute to a few minutes. A client that takes the
|
||||
answer as a stream (`Accept: text/event-stream`, as Claude Code does) is
|
||||
answered that way: `notifications/progress` for the call's
|
||||
`progressToken` while it runs — queued, waiting for the GPU, the node and
|
||||
its percent — then the result, and the call waits up to ten minutes. One
|
||||
answered in one piece waits 50 seconds; a job not done by then is answered
|
||||
with its `job_id`, to ask after with `job_status`. `wait_seconds` (up to
|
||||
900) sets either.
|
||||
|
||||
```
|
||||
claude mcp add --transport http llmgw https://llm.example.com/mcp \
|
||||
--header "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
**Your company's tools.** Where an administrator has registered the
|
||||
company's own MCP servers (Jira, a file server, a database) and your key is
|
||||
granted their tools, the same `/mcp` lists them too, each as `server.tool`
|
||||
(`jira.search`). Give your agent only llmgw's `/mcp` and your key: the
|
||||
servers' own credentials stay with llmgw. A tool your key is not granted is
|
||||
not listed, and calling it by name is refused. Each call is in the audit
|
||||
log under your key: which tool, the size of what was sent, how it ended and
|
||||
how long it took, never what it said.
|
||||
|
||||
### Licences you leave out
|
||||
|
||||
A model's licence is in `/v1/models` (`license`: its `id`, `name`, whether
|
||||
|
||||
+6
-73
@@ -2,8 +2,8 @@
|
||||
|
||||
`sudo llmgw`, with nothing after it, on a machine llmgw runs on. On a machine
|
||||
with nothing installed it opens the setup screens; on a control plane, the
|
||||
console described here; on a node, the node's own console, which shows
|
||||
what runs on that machine and edits `agent.yaml`. Everything on these screens can also be done by command — each
|
||||
console described here; on a node, the node's own console, which edits
|
||||
`agent.yaml`. Everything on these screens can also be done by command — each
|
||||
screen says which — so a script never needs them and a person never needs
|
||||
to remember the commands.
|
||||
|
||||
@@ -67,8 +67,7 @@ or with the mirror unticked, is said in the form and nothing starts.
|
||||
## Watch
|
||||
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
models, the store's members, and anything parked, unplaced or idle. With no
|
||||
webhook in the cluster's settings it says the alerts go nowhere.
|
||||
models, the store's members, and anything parked, unplaced or idle.
|
||||
|
||||
**Monitor** draws the cluster the way btop draws a machine: CPU, memory,
|
||||
GPU busy and GPU memory, requests and tokens a second, time to first token,
|
||||
@@ -174,23 +173,6 @@ search…** as an administrator, or **Delete…** it. People
|
||||
fill collections from their own machines with `llmgw-kb`
|
||||
([knowledge.md](knowledge.md)). Off on a SQLite store, and says why.
|
||||
|
||||
**MCP servers** lists the company's own MCP servers that agents reach through
|
||||
the gateway's `/mcp`, their tools named `server.tool`: whether each is on,
|
||||
who has been granted its tools, its rate per caller, the names of the
|
||||
credentials it is sent (never their values) and its address. **A** adds one:
|
||||
its name, which is its tools' prefix, its address, and a bearer token or
|
||||
another header, kept sealed in the store. On a row: **Test, and list its
|
||||
tools**, with its credential, as a call would; **Grant its tools to…** a
|
||||
team, project, organization, person or key — all of them, `jira.*`, or some
|
||||
by name — beside whatever else that level is granted, and saying when a
|
||||
level above keeps them out of reach; **Revoke** a grant;
|
||||
**Change…** its address, timeout or rate; **Replace its credential…**;
|
||||
**Turn off** (its tools are listed nowhere and every call refused) or on;
|
||||
**Remove…**. A server's tools are seen nowhere until granted. The screen
|
||||
also says how a person points an agent at the gateway, with a key of their
|
||||
own from `/profile/keys`. `llmgw ctl mcp` does the same from a shell, and
|
||||
the web page **MCP servers** (`/admin/mcp`, administrators only) in a browser.
|
||||
|
||||
**Mirror** is what the model mirror holds, one row for each version of each
|
||||
model — a file, or the set of files a model of several is: shards, a vision
|
||||
projector, a config and tokenizer — with how far its download has got and
|
||||
@@ -208,19 +190,6 @@ starting — and whether it ended done, failed, or waiting for room on busy
|
||||
cards. The entry shows how many are running. Enter on one shows its steps.
|
||||
`llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.
|
||||
|
||||
**Slurm** is the Slurm cluster run beside llmgw, when there is one: its
|
||||
controller and whether slurmctld runs, and each node that runs slurmd —
|
||||
whose it is now (llmgw's, Slurm's, or on its way between them), its state in
|
||||
Slurm, the jobs running there. On a row, **Lend to Slurm…** moves the node's
|
||||
replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
|
||||
starts no new job there and gives it back to llmgw when its jobs have ended.
|
||||
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
|
||||
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
|
||||
setup`, or the Slurm part of the first install. Above the nodes it says where
|
||||
the shared files come from and how many users llmgw made; the SHARED column
|
||||
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
|
||||
`llmgw ctl slurm users` change them.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
|
||||
@@ -234,8 +203,7 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
|
||||
dashboard, and what they belong to. **A** creates an account, already
|
||||
approved. On a row: approve one waiting, suspend or reinstate, give or take
|
||||
the cluster dashboard, add to or take out of an organization, team or
|
||||
project (on an organization as a member, its manager, who sees its page, or
|
||||
its admin, who also changes it), their limits, a new password (their sessions close), delete, and
|
||||
project, their limits, a new password (their sessions close), delete, and
|
||||
**Forget them…** for a request to be forgotten (`llmgw users forget`).
|
||||
|
||||
**API keys** — every key, its project, owner, state and last use. **A**
|
||||
@@ -278,28 +246,6 @@ long a key made there may live, and is the expiry of one made without any
|
||||
Limits are requests and tokens a minute, a budget per day or month, and the
|
||||
models allowed. Each level only narrows what is above it.
|
||||
|
||||
**Sign-in** — how people get in besides a password kept here: single
|
||||
sign-on, Active Directory and provisioning, each on or off, with the
|
||||
directory's last sync (how many people, accounts kept in step, suspended,
|
||||
reinstated).
|
||||
- **E** sets up Entra ID. Give the tenant ID, the client ID and secret,
|
||||
and the groups for administrators and operators. A provider other than
|
||||
Entra takes its issuer URL in place of the tenant. The form shows the
|
||||
redirect address to register.
|
||||
- **D** sets up Active Directory: the servers, where people are, the
|
||||
service account and its password, the domain's CA certificate as a PEM
|
||||
file on this machine, the group whose members may sign in, and groups
|
||||
for roles and teams.
|
||||
- Before it is kept, the directory is tried from the control plane: each
|
||||
server reached and bound, how many people may sign in, how many are in
|
||||
each group. If any step fails, nothing is kept and the steps are shown.
|
||||
**T** tries the settings in force.
|
||||
- On the provisioning row, Enter makes a SCIM token, shown once.
|
||||
- The password, the client secret and the token are kept in the store,
|
||||
sealed by the master key when one is set. The settings name them as
|
||||
`store:auth/…`, so each control plane has them with no file put on it.
|
||||
The CA is written into the settings themselves.
|
||||
|
||||
**Guardrails…** on a row lists the guardrail policies server.yaml defines.
|
||||
Each policy is marked in one of three ways:
|
||||
|
||||
@@ -389,18 +335,5 @@ to restart it.
|
||||
|
||||
## The node's console
|
||||
|
||||
On a node, `sudo llmgw` opens on **Overview**: what this machine is doing, read
|
||||
on the machine itself, so it works without an admin token and while the control
|
||||
plane is out of reach. Refreshed every three seconds, it shows
|
||||
|
||||
- llmgw-agent's state and since when, and the version of llmgw;
|
||||
- each control plane the agent dials, from `agent.yaml` and the list the
|
||||
leader last sent, and whether it can be reached;
|
||||
- each GPU: how busy, its memory, its heat, and the models on it;
|
||||
- the engines running here, from the agent's own records: replica, model,
|
||||
engine, GPUs, port, whether the process is still there, and for how long;
|
||||
- the free space where weights and engines are kept;
|
||||
- the agent's warnings and errors of the last hour.
|
||||
|
||||
Below it, `agent.yaml` in the same settings view, and the manual. The cluster
|
||||
is watched and changed from a control plane.
|
||||
On a node, `sudo llmgw` opens `agent.yaml` in the same settings view, and the
|
||||
manual. The cluster is watched and changed from a control plane.
|
||||
+1
-3
@@ -72,9 +72,7 @@ each answer that may be reused, as the client got it, and a hash of the
|
||||
request it answers, in the gateway process's memory — never in the store,
|
||||
never on disk — for the project's TTL or until pushed out by newer ones
|
||||
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
||||
policy. A project that reuses answers to similar questions (`--similar`)
|
||||
also has each question's last user message embedded, and the vector kept
|
||||
beside its answer, in the same memory and for as long. Forgetting a person does not reach into it; the answers age out by
|
||||
policy. Forgetting a person does not reach into it; the answers age out by
|
||||
the TTL. The store keeps only the setting, and a usage record's mark that it
|
||||
was answered from the cache.
|
||||
|
||||
|
||||
@@ -91,21 +91,6 @@ guard:
|
||||
value is the same placeholder throughout a request).
|
||||
- **`block`** stops it.
|
||||
|
||||
**Putting them back.** With `restore: true` in a policy's `input`, the
|
||||
values a request had redacted are put back in its answer: the model reads
|
||||
"write to [PERSON_1] at [EMAIL_1]", the client reads the name and address
|
||||
it sent. Streamed answers too — a placeholder split across two events is
|
||||
held until it is whole. Only that request's own values, only into its
|
||||
answer; the map is kept in memory for the request and nowhere else. The
|
||||
output rules then read the answer as the client will, so a policy that
|
||||
redacts personal data in answers still does.
|
||||
|
||||
```yaml
|
||||
input:
|
||||
pii: {email: redact, phone: redact}
|
||||
restore: true
|
||||
```
|
||||
|
||||
**Shapes.**
|
||||
|
||||
- `system_prompt` is put first.
|
||||
@@ -273,47 +258,6 @@ themselves.
|
||||
context rather than overflowing it and going unjudged.
|
||||
- **Cache:** the same text is not asked about twice within five minutes.
|
||||
|
||||
## An outside service
|
||||
|
||||
A site required to use a particular guardrail — Azure Content Safety,
|
||||
Lakera Guard, NeMo Guardrails, a classifier of its own — names it in a
|
||||
policy's `services`. It is asked over HTTP beside the patterns and before
|
||||
the judge; its "unsafe" blocks or flags like theirs, and a service that does
|
||||
not answer is what the policy's `fail` says.
|
||||
|
||||
```yaml
|
||||
policies:
|
||||
strict:
|
||||
fail: closed
|
||||
services:
|
||||
- name: lakera
|
||||
url: https://api.lakera.ai/v2/guard
|
||||
header_files: {Authorization: /etc/llmgw/secrets/lakera} # "Bearer …", read each call
|
||||
body: '{"messages": [{"role": "user", "content": {{json .Text}}}]}'
|
||||
blocked: flagged # where the answer says unsafe
|
||||
reason: breakdown.0.detector_type
|
||||
on: both # input, output or both
|
||||
action: block # or flag
|
||||
timeout: 2s
|
||||
```
|
||||
|
||||
`body` is a template given `.Text` (`{{json .Text}}` writes it as a JSON
|
||||
string; empty sends `{"input": …}`). `blocked` and `reason` are dotted paths
|
||||
into the JSON answer, numbers for list items; true, a number above zero, a
|
||||
non-empty list or a string other than "false" is unsafe. On input the
|
||||
service reads the latest user turn as the model will, with what `redact`
|
||||
took out still out; on output the whole answer, once, at its end, as the
|
||||
client reads it — with `restore`, the values put back. Being at the end, a
|
||||
service can stop a streamed answer only where it is held back
|
||||
(`stream: buffer_all`): released a sentence at a time, the client has read
|
||||
it by then, and what the service finds ends the stream with
|
||||
`content_filter`, flags it and is recorded. The rule recorded is
|
||||
`service:<name>:<reason>`. The service is told the text only, never who
|
||||
sent it. Under `fail: open`, the default, a service that cannot be asked
|
||||
lets the text through — and a caller can make it fail, sending what it
|
||||
refuses to read; give `fail: closed` where the service must decide. Of an
|
||||
answer asked for with `n` above 1, the service reads the first.
|
||||
|
||||
## Where a policy applies
|
||||
|
||||
`guard.default` applies everywhere. Others are attached to a part of the
|
||||
|
||||
+33
-781
@@ -89,36 +89,13 @@ CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
|
||||
|
||||
**The short way is the screens.** On the machine that will be the control
|
||||
plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
|
||||
whether this is one machine or three for high availability, then lists what
|
||||
an install is made of, each part with what it is for and what is missing
|
||||
without it:
|
||||
|
||||
| Part | | Its settings (Enter) |
|
||||
|---|---|---|
|
||||
| Control plane | always | the administrator, other names for the certificate, the portal's language, the sign-up form |
|
||||
| Store | always | SQLite or PostgreSQL on this machine; with three machines, PostgreSQL under Patroni on the other two |
|
||||
| Control plane on all three | three machines, on | a floating address for whichever leads |
|
||||
| Model mirror | on | here, and in which directory, or on another machine |
|
||||
| This machine as a node | on for one machine | |
|
||||
| Starter model | on for one machine | which: Qwen2.5 0.5B, 1.5B or 3B, Qwen3 4B, 8B or 14B, or any GGUF on Hugging Face; its context |
|
||||
| Pictures and 3D | off | |
|
||||
| Guardrails' judge | on | refuse or serve flagged when it does not answer; copies; how long it is waited for; what it refuses |
|
||||
| Firewall | on | |
|
||||
| Notifications | off | the webhooks |
|
||||
| Limits and keys | always | requests and tokens a minute; the longest a key lives |
|
||||
| Taking turns on a card | always | how long an on-demand model goes unused, and is safe after loading |
|
||||
| Offline bundle | off | the bundle and its sha256 |
|
||||
|
||||
Space turns a part on or off; Enter opens its settings, which say more. What
|
||||
cannot go together is said before anything is done — a starter model with no
|
||||
mirror to fetch it into — and closing the message opens the setting that is
|
||||
wrong. For three machines it then asks for the other two — address, user, and
|
||||
an ssh key, the ssh agent, or a password, and the sudo password if sudo asks
|
||||
for one; or, for a machine nothing may log into, a join code to run there
|
||||
with `sudo llmgw join <code>`, which dials out to this one on port 7444 —
|
||||
checks all three before installing anything (system, room, clock, ports,
|
||||
internet, whether they reach one another). Last it shows the plan, part by
|
||||
part, what is left out and what that costs, and installs. Passwords are held in memory for
|
||||
whether this is one machine or three for high availability; for three, it asks
|
||||
for the other two — address, user, and an ssh key, the ssh agent, or a
|
||||
password, and the sudo password if sudo asks for one; or, for a machine
|
||||
nothing may log into, a join code to run there with `sudo llmgw join <code>`,
|
||||
which dials out to this one on port 7444 — checks all three before
|
||||
installing anything (system, room, clock, ports, internet, whether they reach
|
||||
one another), shows the plan, and installs. Passwords are held in memory for
|
||||
the run only; a sudo password lets the user sudo without one until the install
|
||||
finishes, and is then taken back. At the end it shows the administrator's
|
||||
password and the command that does the same from a script.
|
||||
@@ -152,99 +129,18 @@ That is the whole step. In order, it:
|
||||
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
||||
engine
|
||||
9. closes llmgw's internal ports to everything but the cluster's networks
|
||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
|
||||
CPUs: one replica on up to four cores when this machine is the only
|
||||
node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
|
||||
replicas on the CPUs of nodes with room — this one, or the first that
|
||||
join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||
first thing anyone opens `/chat` to find is a model that answers. A
|
||||
machine without room for it is told so within two minutes, and why.
|
||||
|
||||
`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9,
|
||||
`--no-guard-model` 10 and `--no-model` 11. Every setting the screens ask for
|
||||
is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
|
||||
(or `hf:<owner>/<repo>/<file>.gguf`) and `--starter-context`, `--mirror-dir`,
|
||||
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
|
||||
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
|
||||
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
|
||||
nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
|
||||
go into `server.yaml` as init writes it, and become the cluster's first
|
||||
settings when it starts; a `server.yaml` already there is never overwritten,
|
||||
so run again they change nothing — the settings page does. An NVIDIA card with no working
|
||||
`--no-guard-model` 10 and `--no-model` 11. An NVIDIA card with no working
|
||||
driver is reported rather than silently ignored; install the driver and run
|
||||
init again to serve on it.
|
||||
|
||||
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
|
||||
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
|
||||
machine's node, slurmd on every node, those that join later too
|
||||
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
|
||||
from the machines' own package sources, and on this machine slurmdbd, which
|
||||
keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
|
||||
answers this machine alone. Slurm keeps its records in MySQL or MariaDB
|
||||
only, not in the cluster's PostgreSQL; the database's password is made from
|
||||
the cluster's munge key, so nothing new is stored. A MariaDB the machine
|
||||
already had keeps its settings and only gets the database `slurm_acct_db`
|
||||
and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
|
||||
records older than a year are purged. No node is Slurm's at first: each is
|
||||
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
|
||||
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
|
||||
takes it back once its jobs end. Jobs are submitted on this machine with
|
||||
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
|
||||
needs this machine to be a node, and its hostname to be a name Slurm takes.
|
||||
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
|
||||
networks only, as llmgw's own do; a firewall of the site's own between the
|
||||
nodes must let them through.
|
||||
|
||||
**Slurm's shared files and users.** Jobs need the same files on every
|
||||
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
|
||||
exports `/shared` over NFS to every Slurm machine and they mount it
|
||||
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
|
||||
export the Slurm machines alone. That is one machine, not highly available
|
||||
and no faster than its disk and network: enough for a few nodes. A site with
|
||||
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
|
||||
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
|
||||
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
|
||||
shared --path /lustre/hpc` only has llmgw look that it is there. Every
|
||||
active llmgw account is a Linux user on every Slurm machine
|
||||
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
|
||||
200000 up that is never given to anyone else, its home under
|
||||
`/shared/home`. People add their SSH public keys in the portal's
|
||||
**Profile** and log in to this machine to run `sbatch`; an account suspended
|
||||
or deleted loses its user, its files stay. On the other Slurm machines they
|
||||
are let in only while a job of theirs runs there, and their ssh session is
|
||||
put in that job, held to its cores and cards (pam_slurm_adopt, which the
|
||||
agents install); root and the machines' own users log in as before. A site whose users come from its
|
||||
directory (sssd against Active Directory or LDAP) passes `--slurm-users
|
||||
site`, or later `llmgw ctl slurm users off`.
|
||||
|
||||
With llmgw's users, Slurm's accounts are llmgw's projects: an account for
|
||||
each organization, team and project, nested (`acme`, `acme-ml`,
|
||||
`acme-ml-training`), and on each project's account its members and its
|
||||
team's and organization's. A job runs on one of its user's accounts and no
|
||||
other (`sbatch --account=acme-ml-training`; without it, the user's first),
|
||||
and when it ends its cards' hours (a job without cards, its cores') are
|
||||
priced at `auth.gpu_hour_cost` (`core_hour_cost`) into the project's usage,
|
||||
where it shows as model `slurm/PARTITION` beside the project's requests.
|
||||
Accounts a Slurm administrator made are left alone. A job's cost does not
|
||||
yet count against a budget, which binds API keys only.
|
||||
|
||||
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
|
||||
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
|
||||
versions llmgw pins and checks; pyxis is built against each machine's own
|
||||
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
|
||||
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
|
||||
`containers: on`:
|
||||
|
||||
```sh
|
||||
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
|
||||
```
|
||||
|
||||
The shared path is in every container where it is on the host, and the job
|
||||
runs as its user, held to its cores and cards. Each node pulls the image
|
||||
from its registry for each job; for a large image used often, import it
|
||||
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
|
||||
and give that file to `--container-image`.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
@@ -259,43 +155,15 @@ demonstration one:
|
||||
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
||||
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
||||
|
||||
The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
|
||||
cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
|
||||
streamed answer every 600 characters. The policy fails **closed**: a request
|
||||
the judge cannot read within 5 seconds is refused with 503 rather than
|
||||
served unjudged, and `judge_unavailable` goes to the notification webhooks.
|
||||
So a judge gets cores of its own: on one machine one replica on up to four,
|
||||
and on a larger cluster two replicas of two cores each, spread over its nodes
|
||||
so that losing one leaves the other. On a machine of fewer than four cores,
|
||||
a busy one, or a VM without AVX2 (see
|
||||
The judge runs as two replicas, each on a CPU — about half a second for a
|
||||
short text on any CPU with AVX2 — and reads a streamed answer every 600
|
||||
characters. The policy fails **closed**: a request the judge cannot read
|
||||
within 5 seconds is refused with 503 rather than served unjudged, and
|
||||
`judge_unavailable` goes to the notification webhooks. Two replicas are what
|
||||
make that bearable; on one small machine, or a VM without AVX2 (see
|
||||
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
||||
first thing to fail. The choices there are a longer `latency_budget`,
|
||||
`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
|
||||
or `--no-guard-model`.
|
||||
|
||||
init times the judge once it is up. If a judgment takes more than half the
|
||||
wait, or more than the wait when several come at once, it says so before the
|
||||
next steps, with what to do. The wait itself is not changed without being
|
||||
asked. `llmgw ctl guard judge check` does the same at any time:
|
||||
|
||||
```
|
||||
qwen3guard-0.6b, 2 replica(s) ready, a request waits 5.0 s for it
|
||||
alone: 2.4, 2.3, 2.3 s
|
||||
4 at once: slowest 6.1 s; about 0.7 judgments a second, so about 0.3 requests a second with short answers
|
||||
tight: a busy moment passes the budget
|
||||
```
|
||||
|
||||
In a three-machine install the judges are better on machines of their own.
|
||||
On a 4-core control plane that was also a node, 38 of 60 requests under load
|
||||
were refused for want of a judgment. With the judges on a node of their own
|
||||
(4 cores, two judges of two cores each), 2 were refused. The setup screen asks
|
||||
for up to two **judge machines** after the store's two, or give
|
||||
`--judge-node user@host` to `server init`, once for each. Each is added as a
|
||||
node labelled `llmgw.role: judge`, and the judge's profile selects that
|
||||
label, so the judges run only there and nothing else does. Four cores and
|
||||
8 GB are enough for two judges.
|
||||
|
||||
Defamation, specialized advice,
|
||||
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
|
||||
`--no-guard-model` is the choice there. Defamation, specialized advice,
|
||||
intellectual property and elections are left out of what it refuses: they
|
||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||
|
||||
@@ -463,24 +331,7 @@ by itself. Three things are worth knowing before you edit it:
|
||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||
budgets mean anything. Without a price every request costs zero and a budget
|
||||
never binds; without the defaults, a project created without limits is
|
||||
metered by nothing at all. A price may set `cached_per_1k` beside
|
||||
`prompt_per_1k`: the prompt tokens the engine or provider read from its
|
||||
cache (the answer's `cached_tokens`) are charged at it, the rest at the
|
||||
prompt rate. A provider that bills cached reads at a tenth is
|
||||
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
|
||||
- What to price a self-hosted model at follows from what the cluster costs.
|
||||
Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
|
||||
are in: the hardware written off over its years, its power and room, and the
|
||||
people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
|
||||
server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
|
||||
`auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
|
||||
hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
|
||||
prices suggest` spreads them over the tokens the model served: an output
|
||||
token counted as four input ones (making one is that much dearer), a cached
|
||||
prompt token as a tenth of one. Idle hours are in the cost, so a model held
|
||||
all month and seldom used comes out dear, which is the point; one held and
|
||||
billed nothing (a guard's judge) is shown with what it held. `--apply`
|
||||
writes the prices into `auth.pricing`.
|
||||
metered by nothing at all.
|
||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||
is empty. Without it, a file with no models is refused as having nothing to
|
||||
serve.
|
||||
@@ -788,8 +639,6 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
|
||||
| FreeToken | nodes with a GPU of compute capability 8.0 or newer (RTX 30, A100 and later), driver 580+, x86-64 and the CUDA 13 toolkit | a Python environment built by uv from PyPI (`freetoken[accel]`); see [FreeToken](#freetoken-models-larger-than-the-card) |
|
||||
| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
|
||||
|
||||
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
|
||||
node gets the same dependencies whenever it is built, and installed with
|
||||
@@ -832,100 +681,6 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
|
||||
`engines.trtllm.python`): the agent then installs none of its own beside it.
|
||||
|
||||
#### Strata: a large model of experts on one card
|
||||
|
||||
Strata runs a few mixture-of-experts models — Qwen3.8-Flash-Next (125B) and its
|
||||
Swift, Coder, Unsloth, Gyro and NVFP4 versions, Qwen3.6-35B-A3B, DeepSeek
|
||||
V4.1-Flash — with the experts in the node's RAM and the rest on one card of 12
|
||||
GB or more. Where it is installed, the plan offers it for a repository of one
|
||||
of those families:
|
||||
|
||||
```
|
||||
llmgw ctl model plan ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
|
||||
llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
|
||||
--file Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --engine strata
|
||||
```
|
||||
|
||||
The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
|
||||
and the mirror's serve form serves such a version on Strata.
|
||||
|
||||
The mirror page's **Strata** tab lists every family and size Strata runs, with
|
||||
what each downloads and keeps in RAM and where it would run here:
|
||||
|
||||
- **Fetch into the mirror**, for the GGUF families: the size's files, checked
|
||||
against Hugging Face's checksums; the version's serve form then serves it on
|
||||
Strata.
|
||||
- **Serve**, for the packs Strata fetches itself — the NVFP4 packs
|
||||
(`Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` and its NVIDIA
|
||||
sibling, NVIDIA's FP4-Local) and DeepSeek V4.1: the model is added with the
|
||||
repository as its source, and its first start fetches tens of gigabytes onto
|
||||
the node, each file pinned and checked by Strata, before preparing it.
|
||||
`ctl model serve` does the same for these repositories.
|
||||
|
||||
Repositories of architecture `strata-ple` that Strata 0.2.7 does not list —
|
||||
community packs such as `…-Q8_0-Strata` — are named as such in the explorer:
|
||||
Strata's setup refuses them, and llama.cpp cannot read them. A Strata profile
|
||||
says its family and size (`--family qwen --size IQ2_XS`), which the catalog
|
||||
checks against what Strata runs, and reserves the RAM the size keeps its experts
|
||||
in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
|
||||
first start prepares the model on the node, which takes minutes, and fetches a
|
||||
draft layer of about 5 GB from the internet.
|
||||
|
||||
#### FreeToken: models larger than the card
|
||||
|
||||
[FreeToken](https://github.com/FlashML-org/FreeToken) serves mixture-of-experts
|
||||
models several times a card's size from one card: the experts stay in the
|
||||
node's RAM, the ones used most are cached in VRAM, and the rest stream over
|
||||
PCIe. Unlike Strata it reads Hugging Face safetensors checkpoints as they are
|
||||
published — DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8 (Flash-Next among them),
|
||||
gpt-oss, Gemma 4, MiniMax M2.5/M3 — in FP8, NVFP4, MXFP4 or BF16, and batches
|
||||
several requests (`--max-running-requests`, 4 by default).
|
||||
|
||||
Every node that can run it installs it, as with vLLM: an NVIDIA GPU of compute
|
||||
capability 8.0 or newer, driver 580 or newer, x86-64. Older cards are passed
|
||||
over without a word — on a Quadro RTX 6000 (7.5) version 0.2.1 had no kernel
|
||||
image for the card's first prefill, and in fp16 its first request ended in an
|
||||
illegal memory access. Its FP8 checkpoints need 8.9 (Ada, Hopper) and its NVFP4
|
||||
ones 12.0 (RTX 50, RTX PRO 6000 Blackwell). It compiles some kernels the first
|
||||
time a model loads, so the node also needs the **CUDA 13 toolkit** (`nvcc` in
|
||||
`CUDA_HOME`, `/usr/local/cuda` or `/usr/local/cuda-13.*`); a node of new cards
|
||||
without one says so on the cluster page instead of installing an engine that
|
||||
fails on its first model.
|
||||
|
||||
A FreeToken profile runs on one card of one node (`llmgw ctl model add -f`):
|
||||
|
||||
```yaml
|
||||
name: glm-5.2
|
||||
source: /var/lib/llmgw-weights/GLM-5.2-NVFP4 # a safetensors checkpoint's directory
|
||||
profiles:
|
||||
- name: one-card
|
||||
engine: freetoken
|
||||
gpu_class: rtx-pro-6000-96g
|
||||
min_memory_gb: 200 # the experts live in RAM: say how much
|
||||
args: ["--max-running-requests", "8"]
|
||||
```
|
||||
|
||||
A Hugging Face id as the source works too — FreeToken downloads the
|
||||
repository itself — but it is not pinned to a commit or checked against
|
||||
Hugging Face's checksums as vLLM's are, FreeToken having no `--revision`; a
|
||||
directory the mirror fetched, or one put on the node, is.
|
||||
|
||||
The agent gives it the model, its address, its card (`--gpu`) and the key (in
|
||||
`FREETOKEN_API_KEY`, not on the command line); `--mmproj`, media paths and
|
||||
domains, `--hf-overrides` and `--moe-cache-split` are refused as they read
|
||||
files or fetch. `ctl model context` sets `--max-seq-len-override`. Readiness
|
||||
waits for its `/health` to stop saying `loading`, since it answers 200 from the
|
||||
moment its HTTP server is up. It serves no Prometheus metrics, so the router
|
||||
counts its requests itself.
|
||||
|
||||
llmgw follows FreeToken release by release: the version is
|
||||
`DefaultEngines["freetoken"]`, and a weekly workflow
|
||||
(`.gitea/workflows/engine-releases.yml`) fails when PyPI has a newer one, the
|
||||
notice to check it and move to it. A version of your own is named under
|
||||
`engines` like any other (`freetoken: "0.2.2"`), and `freetoken: ""` installs
|
||||
none; one node is turned off with its build set to *none* (console, the node's
|
||||
menu).
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
@@ -1057,45 +812,6 @@ Nothing to do here if you built the mirror in step 3 — the agent fetches what
|
||||
model's `source` names, and that source is your mirror. This section is for the
|
||||
arrangements where the files are already on the machine.
|
||||
|
||||
**Weights in a pickle format are refused.** `.bin`, `.pt`, `.pth`, `.ckpt` and
|
||||
`.pkl` are Python pickles, and loading one runs whatever it holds, as the
|
||||
engine's user, on the node — which is how the malicious models found on Hugging
|
||||
Face work. A model whose source is one, or a mirror version holding one, is
|
||||
refused when it is added and by the node before it fetches anything; vLLM and
|
||||
SGLang given a model by its Hugging Face id read only its safetensors. Take the
|
||||
repository's safetensors or GGUF. A model that has no other form, and whose
|
||||
files were checked, can say so:
|
||||
|
||||
```yaml
|
||||
- name: triposr
|
||||
source: http://mirror:8081/triposr/<rev>/model.ckpt
|
||||
allow_pickle: true
|
||||
```
|
||||
|
||||
A local path is judged by its name as any source is; a local directory is
|
||||
not looked into. What an engine downloads itself by a Hugging Face id, past
|
||||
the node — an adapter given by its id, a TensorRT-LLM model's — is refused
|
||||
when the model is added, unless it says `allow_pickle`: mirror its
|
||||
safetensors and give that instead.
|
||||
|
||||
**Weights are pinned to a commit and checked.** A model added with
|
||||
`ctl model add`, the API or the console, whose weights come from Hugging Face,
|
||||
is pinned to the commit the repository is at when it is added:
|
||||
|
||||
- `hf:org/repo/file` and a `…/resolve/main/…` URL become that commit's URL;
|
||||
- a repository vLLM or SGLang downloads itself is given `--revision`;
|
||||
- the sha256 Hugging Face published for each file goes in the model's origin
|
||||
(its definition, `GET /api/v1/models/NAME`, shows it).
|
||||
|
||||
A node then refuses a copy of the file that is not it, naming both checksums,
|
||||
and deletes it; a cached copy is checked once and its checksum kept beside it.
|
||||
The mirror does the same for a model `ctl model serve` fetches, and
|
||||
`llmgw mirror add` for a Hugging Face URL (`--sha256` for any other). A new
|
||||
upload to the repository reaches the cluster only through `ctl model update`,
|
||||
which says what changed. Not pinned: models declared in the cluster's settings,
|
||||
which are not added this way, and a vLLM model whose arguments already say
|
||||
`--revision` keeps that one.
|
||||
|
||||
Put them where the node can read them. `scp`, a shared filesystem, or the
|
||||
engine's own downloader are all valid, and which you want depends on how many
|
||||
nodes you have.
|
||||
@@ -1254,27 +970,6 @@ A node with no location is placed as before, as a rack of its own.
|
||||
See [facility-aware-placement.md](facility-aware-placement.md) for power caps
|
||||
and holds.
|
||||
|
||||
### A card in the wrong slot
|
||||
|
||||
An x16 card in a slot wired x8 or x4, behind a riser, or in a slot of an
|
||||
older PCIe generation than the card's, links slower than it was built to and
|
||||
works otherwise. A model on the card still decodes at the card's own memory
|
||||
speed, but it loads slower, a replica split over several cards by PCIe is
|
||||
slower, and so is anything that spills to the host. Each card's agent reads
|
||||
how its link stands — the fastest generation card and slot allow together,
|
||||
and the lanes in use — against what the card could do, and the node says so
|
||||
among its warnings (the dashboard, the console's Nodes, `llmgw doctor`):
|
||||
|
||||
```
|
||||
GPU 1 (NVIDIA H100 80GB HBM3) linked at x8 of its x16 lanes: 31.5 GB/s where it
|
||||
could have 63.0. Check the slot (wired fewer lanes, or an older generation), a
|
||||
riser, or the BIOS's PCIe setting
|
||||
```
|
||||
|
||||
The generation a card reports now drops while it idles, to save power; what is
|
||||
compared is the fastest it may reach, which does not. `lspci -vv -s BUS-ID`
|
||||
shows the same: `LnkCap` is the card's, `LnkSta` the link's.
|
||||
|
||||
## 6. Setting the node's CPU model, if it is a VM
|
||||
|
||||
Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
|
||||
@@ -1592,26 +1287,6 @@ catalog, and a key rotated in the file is used within seconds. A model whose key
|
||||
file cannot be read is not offered. Everything else applies as to any model:
|
||||
prices and budgets, rate limits, usage records, and key allowlists.
|
||||
|
||||
A provider's limits are per key. With several keys, one answering 429 need not
|
||||
hold up the model: give them all, each with an optional weight (its share
|
||||
against the others'), in place of `api_key_file`:
|
||||
|
||||
```yaml
|
||||
external:
|
||||
url: https://api.openai.com
|
||||
model: gpt-4o-mini
|
||||
api_key_files:
|
||||
- {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
|
||||
- {file: /etc/llmgw/secrets/openai-2.key}
|
||||
```
|
||||
|
||||
Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
|
||||
answers 429 is passed over while its breaker is open, and the request goes to
|
||||
another. One that answers 401 (revoked, expired, mistyped) is passed over too,
|
||||
and `external_key_refused` goes to the notification webhooks, naming the
|
||||
endpoint and never the key. The usage records say which key answered by the
|
||||
same name. A key whose file cannot be read is left out, and the others stay.
|
||||
|
||||
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
@@ -1828,93 +1503,6 @@ Not yet: models declared in server.yaml cannot name a peer's model as a
|
||||
fallback (the catalog in the file is checked before the peers are read), and
|
||||
a peer behind a private CA needs that CA in the system's trust store.
|
||||
|
||||
### Stopping a model, and removing one with its files
|
||||
|
||||
```
|
||||
llmgw ctl model stop qwen-7b # none of its replicas run; it is kept
|
||||
llmgw ctl model start qwen-7b # runs again, scaled as it was
|
||||
llmgw ctl model remove qwen-7b # leaves the catalog; its files stay
|
||||
llmgw ctl model remove qwen-7b --purge # its files on the mirror and the nodes too
|
||||
```
|
||||
|
||||
- **Stop** keeps the model in the catalog with its files and what it was
|
||||
scaled to; a request for it is told an administrator stopped it. `ctl
|
||||
models` shows it as stopped. A `ctl scale` while it is stopped starts it at
|
||||
that number.
|
||||
- **Remove** leaves its files on the mirror and the nodes, for adding it
|
||||
again without fetching it. `ctl weights` lists such files on the nodes.
|
||||
- **Remove --purge** deletes every version of it on the mirror (a second
|
||||
mirror follows), its adapters and the weights its last update replaced,
|
||||
and the nodes' copies once its replicas have stopped. What another model
|
||||
uses — a canary's weights, a shared adapter — stays. It cannot be undone:
|
||||
adding it again fetches it again.
|
||||
|
||||
The console has the same in a model's menu: Stop… or Start, Remove…, and
|
||||
Remove with its files….
|
||||
|
||||
### Changing a served model's context and longest answer
|
||||
|
||||
How much one request may use, and how long an answer may be, are set when a
|
||||
model is served. To change them afterwards, in place:
|
||||
|
||||
```
|
||||
llmgw ctl model context qwen-7b 32768 # tokens a request
|
||||
llmgw ctl model context qwen-7b 32768 --slots 4 # llama.cpp: four at once
|
||||
llmgw ctl model context qwen-7b 32768 --max-output 8192 # and the longest answer
|
||||
llmgw ctl model context qwen-7b --max-output 4096 # the longest answer alone
|
||||
```
|
||||
|
||||
The longest answer is the gateway's: a request asking for more `max_tokens`
|
||||
is refused, `/v1/models` lists it as `max_output_tokens` for clients to read,
|
||||
and the account page's chat asks for no more. Changed alone it is only
|
||||
the catalog's, in force at once, and no replica is started again. With a
|
||||
context it is set with it; left out, it grows and shrinks with the context
|
||||
as before, unless it was set to something else. It never exceeds the context.
|
||||
|
||||
For a context, the catalog, the engine's arguments (`--ctx-size` and
|
||||
`--parallel` for llama.cpp, `--max-model-len` for vLLM, `--context-length` for
|
||||
SGLang, `--context` for Strata's setup, `--max-seq-len-override` for
|
||||
FreeToken) and the memory its KV cache takes change together. Strata and
|
||||
FreeToken size their caches to the card themselves, so their cards are not
|
||||
checked here. A context longer than the
|
||||
model was trained for is refused, and so is one its cards cannot hold — with
|
||||
the figures, so you know how far to go. Its replicas are then started again on
|
||||
it, one by one, a new one ready before the old one stops. The console has the
|
||||
same under a model's menu, Context length…, with the longest answer beside
|
||||
it. The admin API is `POST /api/v1/models/{name}/context` with `tokens`,
|
||||
`slots` and `max_output`, any of them left out. A model declared in the cluster's
|
||||
settings is changed there (`llmgw ctl config edit`).
|
||||
|
||||
### Moving a served model to more cards
|
||||
|
||||
How many cards of one node each replica uses can be changed while the model
|
||||
is served:
|
||||
|
||||
```
|
||||
llmgw ctl model gpus qwen3-coder-30b 2 # two cards a replica
|
||||
llmgw ctl model gpus llama-70b 4 --profile h100 # the model has several profiles on cards
|
||||
llmgw ctl model gpus qwen3-coder-30b 2 --in-place # no room beside the running replica
|
||||
```
|
||||
|
||||
vLLM, SGLang and TensorRT-LLM spread the model by tensor parallel (`tp`),
|
||||
which needs the count to divide the model's KV heads or be divided by them:
|
||||
a model with 8 splits 2, 4, 8 or 16 ways, not 3 or 12. llama.cpp has no
|
||||
tensor parallel; it splits the layers over the cards (`gpus_per_node`), and
|
||||
a `--tensor-split` written for the old cards is dropped. ComfyUI and Strata
|
||||
run on one card. A count no node has is refused, and so is one whose cards
|
||||
would not hold the weights and the context — with the figures; fewer cards
|
||||
with a long context may need `llmgw ctl model context` first.
|
||||
|
||||
Its replicas are then started again on it, each new one ready beside the old
|
||||
before the old stops. That needs the cards free beside it: on a node of two
|
||||
cards with the model on one, two do not fit until it is gone, and the command
|
||||
says so. `--in-place` stops each replica first and starts its replacement on
|
||||
the cards it held and the free ones, and the model does not answer while the
|
||||
new one loads — a model with several replicas keeps answering on the others.
|
||||
A replica spread over several nodes is laid out by hand, with a fabric
|
||||
(`nodes`, `gpus_per_node` in the profile). The console has the same under a
|
||||
model's menu, GPUs….
|
||||
|
||||
### Trying new weights on some of the traffic first
|
||||
|
||||
When a model's repository publishes new weights, `llmgw ctl model update`
|
||||
@@ -2174,31 +1762,12 @@ dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
|
||||
to run it, which is not the control plane: monitoring that dies with the thing
|
||||
it monitors tells you nothing at the moment you need it.
|
||||
|
||||
Three figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
|
||||
Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
|
||||
rather than zero when the driver does not report it — a flat zero across a
|
||||
fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
The third is what "busy" means. `llmgw_gpu_utilization_percent` is
|
||||
nvidia-smi's `utilization.gpu`: the share of time *any* kernel ran, however
|
||||
little of the chip it ran on — a kernel on one SM of 142 reads 100%. A Quadro
|
||||
RTX 6000 decoding read 92% busy while its SMs had work 69% of the time and
|
||||
issued an instruction on 18% of their cycles. How much of the chip is at work
|
||||
is in `llmgw_gpu_sm_active_percent`, `llmgw_gpu_sm_occupancy_percent`,
|
||||
`llmgw_gpu_tensor_active_percent` and `llmgw_gpu_dram_active_percent`, from
|
||||
the card's profiling counters. The agent reads them from NVML's GPM on
|
||||
Hopper and later, with nothing to install; on older cards it reads them from
|
||||
a [dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter) on the node, at
|
||||
`http://127.0.0.1:9400/metrics` unless `discovery.dcgm_exporter` in
|
||||
agent.yaml says otherwise (`off` reads none). Where neither is there, the
|
||||
dashboard's GPU map and the console colour cards by their draw against the
|
||||
power limit (`llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts`), which
|
||||
rises with how much of the chip works rather than with whether anything
|
||||
does. Low tensor-core activity in decode is not waste: a quantised
|
||||
matrix-vector product at batch one runs on the integer units, and it is
|
||||
prefill and batches that use the tensor cores.
|
||||
|
||||
### Token by token, and SLOs
|
||||
|
||||
Beside time to the first token, every streamed answer is timed per token:
|
||||
@@ -2258,23 +1827,6 @@ same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
Setup asks for them: **Notification webhooks** on its first page, or
|
||||
`--webhook` to `llmgw server init`, as often as there are webhooks:
|
||||
|
||||
```
|
||||
sudo llmgw server init --admin ops@example.com \
|
||||
--webhook https://hooks.slack.com/services/… \
|
||||
--webhook json:https://ops.example.com/llmgw
|
||||
```
|
||||
|
||||
An address of Slack's, or one with `/hooks/` in it (Mattermost, Rocket.Chat)
|
||||
or ending in `/slack`, is sent Slack's `{"text": …}`; any other the event as
|
||||
JSON; `slack:` or `json:` in front says which. They are added to the
|
||||
cluster's settings beside any already there — run again, nothing is added
|
||||
twice — and once the control plane runs on them, each is sent a test, whose
|
||||
answer setup prints. Events to choose and a signing secret are set in the
|
||||
settings afterwards (`llmgw ctl config edit`).
|
||||
|
||||
### Tracing
|
||||
|
||||
Metrics say that a model took four minutes to start; a trace says where they
|
||||
@@ -2308,18 +1860,6 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
An inference request and each `upstream` attempt also carry the
|
||||
[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
|
||||
so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
|
||||
llmgw's calls as model calls without a dashboard made for llmgw:
|
||||
`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
|
||||
`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
|
||||
when the client set them, `gen_ai.conversation.id` (the session),
|
||||
`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
|
||||
`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
|
||||
engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
|
||||
what was answered are never in a span.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
@@ -2484,10 +2024,8 @@ llmgw keys cache --project Acme/ml/train --off
|
||||
- **Policies still apply:** the limits are checked and the guardrails judge
|
||||
the request before the cache is looked in, and the output policy judges a
|
||||
reused answer as it would a new one. A request continuing a stored
|
||||
conversation, under a `no_store` policy, or whose redacted values are put
|
||||
back in its answer (`restore: true`) is never answered from the cache nor
|
||||
kept in it: redacted, two callers' requests read the same, and the answer
|
||||
has the first caller's values in it.
|
||||
conversation, or under a `no_store` policy, is never answered from the
|
||||
cache nor kept in it.
|
||||
- **What a hit costs:** nothing. It is recorded as a request — status 200,
|
||||
the model, the key, the rate limit's count — with no tokens and no cost,
|
||||
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
|
||||
@@ -2507,46 +2045,8 @@ router:
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
### Similar questions
|
||||
|
||||
A help desk asked "how do I reset my password" in forty wordings asks the
|
||||
model forty times under the exact cache. `--similar` also gives a chat
|
||||
question the answer to one like it:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
|
||||
```
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
embedding_model: bge-m3 # an embedding model in the catalog
|
||||
```
|
||||
|
||||
- **What is compared:** the last user message, embedded by
|
||||
`router.cache.embedding_model`, against the questions already answered
|
||||
in the same conversation — the same system prompt, the same earlier turns,
|
||||
tools, `response_format` and sampling settings, the same model and
|
||||
project. A like question in another conversation, after a tool's result,
|
||||
or with a picture, a sound or a file in it, is not taken for this one. The exact cache is asked first, and an
|
||||
exact hit embeds nothing.
|
||||
- **How alike:** the cosine similarity of the two embeddings, at least the
|
||||
project's `--similar`. 0.95 is strict; below 0.9 different questions start
|
||||
to meet. A question that differs only in a number or a name ("2+2" and
|
||||
"2+3") can be closer than that, so a project whose questions turn on
|
||||
figures should not use it. A question over 2,000 characters is reused only
|
||||
when it is the same.
|
||||
- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
|
||||
and `Age`. Recorded and charged as a hit. The same rules as a hit decide
|
||||
which answers are reused and kept; guardrails judge the request before.
|
||||
- **The embedding** is asked of the model by the gateway on its own account,
|
||||
as a judge is, within 2 seconds; every miss waits for it. Without the
|
||||
model, or when it does not answer, only the same request is reused. The
|
||||
vectors are kept in memory with their answers and counted in
|
||||
`router.cache.max_bytes`.
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
|
||||
and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
@@ -2655,7 +2155,6 @@ triposr`. What that adds is this entry, and its workflow is built in:
|
||||
```yaml
|
||||
- name: triposr
|
||||
source: hf:stabilityai/TripoSR/model.ckpt
|
||||
allow_pickle: true # published as this checkpoint only
|
||||
comfyui:
|
||||
workflow: triposr # built in; a model of your own names a JSON file on every control plane
|
||||
output: meshes
|
||||
@@ -2803,62 +2302,6 @@ Without authentication, `threads.store` and `threads.retain_hours` in
|
||||
`server.yaml` keep both conversations and notes in one shared pool. That is for
|
||||
one trusted machine only; see [development.md](development.md).
|
||||
|
||||
## Three machines, each a control plane
|
||||
|
||||
The screens' three machines are high availability for the gateway as well as
|
||||
the store: the two database machines are made control planes too, standing
|
||||
by while the first leads — what `server add`, below, does for one machine.
|
||||
By command, `--standbys` with `--patroni`, and `--vip` for the floating
|
||||
address:
|
||||
|
||||
```
|
||||
sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --admin you@example.com \
|
||||
--patroni ubuntu@10.0.0.6 --patroni ubuntu@10.0.0.7 \
|
||||
--standbys --vip 10.0.0.160/24 --no-node --no-model
|
||||
```
|
||||
|
||||
| Machine | Runs |
|
||||
|---|---|
|
||||
| the first | control plane, leading at first; model mirror; etcd |
|
||||
| the other two | control plane, standing by; PostgreSQL under Patroni; etcd |
|
||||
|
||||
etcd votes on all three and PostgreSQL is on two, so any one machine can be
|
||||
lost: a standby leads within seconds, Patroni promotes the standby database
|
||||
if it was the primary, and the floating address follows whichever leads. The
|
||||
mirror's management API answers the three control planes' addresses, not
|
||||
only its own machine's, so a standby that comes to lead can still fetch and
|
||||
add models.
|
||||
|
||||
**A second mirror.** The mirror is on the first machine, and without a
|
||||
second one, models the nodes already have keep serving while that machine
|
||||
is down, and new ones wait for it. With a second, the cluster carries on.
|
||||
- **Setting it up:** the screens offer one on either database machine
|
||||
("A second mirror"); by command, `--second-mirror 1` (or `2`). For an
|
||||
install that has one mirror, run this on the mirror's machine:
|
||||
|
||||
```
|
||||
sudo llmgw mirror add-peer ubuntu@10.0.0.7 --control-plane 10.0.0.5 --control-plane 10.0.0.6 --control-plane 10.0.0.7
|
||||
```
|
||||
|
||||
It logs in with root's ssh key on the mirror's machine, so that key must
|
||||
be accepted for the user named on the second machine. When the mirror has
|
||||
a machine of its own, without the control plane, the command cannot change
|
||||
the cluster's settings itself: it prints the `llmgw ctl config set
|
||||
mirror.peers …` to run on a control plane.
|
||||
- **Kept in step:** each mirror reads the other every thirty seconds and
|
||||
copies what it lacks, keeping a copy only if its checksum matches.
|
||||
- A model removed from one is removed from the other, even if that one
|
||||
was down at the time and comes back later.
|
||||
- A model added again after it was removed stands on both.
|
||||
- Neither mirror leads.
|
||||
- **Control planes:** they name both (`mirror.peers` in the cluster's
|
||||
settings) and send the mirror's management calls to the first that
|
||||
answers.
|
||||
- **Nodes:** while the mirror a model's address names is down, a node is
|
||||
given the same file on the other.
|
||||
- **Disk:** the second machine needs room for every model, as the first
|
||||
does.
|
||||
|
||||
## Two control planes
|
||||
|
||||
Once the store is replicated (`server init --patroni`, above), one command on
|
||||
@@ -2875,12 +2318,8 @@ and lets it into the store; it starts there as a standby. Every node is told
|
||||
of it within half a minute and fails over to it without being touched.
|
||||
`--vip` keeps a floating address on whichever control plane leads, with
|
||||
keepalived on both: that is the address to give clients, or a proxy in front.
|
||||
Without it, put a proxy in front that sends to whichever answers `/leader`
|
||||
with 200 — not `/health`, which is 200 on a standby too, and not `/ready`,
|
||||
which is 503 on a leader with no model yet and would take the portal away
|
||||
with it. The end of [deploy/proxy/Caddyfile](../deploy/proxy/Caddyfile) has
|
||||
Caddy doing that for three control planes, and
|
||||
[deploy/ha/](../deploy/ha/README.md#check-leader-not-health) why.
|
||||
Without it, put a load balancer in front that checks `/ready` — not
|
||||
`/health`, which is 200 on a standby that would refuse every request.
|
||||
|
||||
- **What a failover costs.** Measured on the reference cluster, stopping the
|
||||
leader: the standby led within five seconds, every node had moved to it
|
||||
@@ -2906,16 +2345,9 @@ Caddy doing that for three control planes, and
|
||||
`server add` checks, and says what to add to the mirror's unit if it
|
||||
refuses the new control plane — which would otherwise find out the first
|
||||
time it leads and fetches a model.
|
||||
- **Standbys can answer clients too.** With `router.active_active: true`
|
||||
(`llmgw ctl config set router.active_active true`) a standby answers `/v1`
|
||||
itself, routing to the leader's replicas, instead of passing it on: more
|
||||
machines carry the traffic. Point clients at every control plane — DNS,
|
||||
or your own load balancer checking `/ready` — since the floating address
|
||||
stays on the leader. Rate limits stay the cluster's: each router's counts
|
||||
reach the others every two seconds, and a standby that becomes the leader
|
||||
keeps the minute it was counting. Without the setting, rate limits are
|
||||
counted in the leader's memory and a failover starts each key's minute
|
||||
afresh.
|
||||
- **Limits are per process.** Rate limits are counted in memory, and clients
|
||||
reach the leader through the floating address, so a failover starts each
|
||||
key's minute afresh.
|
||||
|
||||
[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
|
||||
their own load balancer in front.
|
||||
@@ -2948,89 +2380,6 @@ the same files and keys as `locales/en/`, and its `_meta.*` entries saying how
|
||||
it writes numbers and dates. The tests refuse a catalogue that lacks a key a
|
||||
page uses.
|
||||
|
||||
## The company's MCP servers
|
||||
|
||||
llmgw's `/mcp` can serve the company's own MCP servers beside its own tools.
|
||||
Each server, a ticket system, a file server or a database, is registered once.
|
||||
Its credentials stay on the control planes, and people's agents are given only
|
||||
`/mcp` and their llmgw key.
|
||||
|
||||
```
|
||||
llmgw ctl mcp add jira --url https://jira.internal/mcp \
|
||||
--description "Acme Jira" --secret-file Authorization=/root/jira-token
|
||||
llmgw ctl mcp test jira # its tools, asked with its credentials
|
||||
llmgw keys limits --scope team --id Acme/support --tools 'jira.*'
|
||||
```
|
||||
|
||||
- **Credentials:**
|
||||
- `--secret-file NAME=FILE` reads a header's value once and keeps it in
|
||||
the store, sealed by the master key, so no file has to be on each
|
||||
control plane.
|
||||
- `--header-ref NAME=REF` reads it at each call from a file, `env:`,
|
||||
`vault:` or `store:` instead.
|
||||
- `--header NAME=VALUE` is for what is not secret.
|
||||
- **Who may call what:** tools are named `server.tool` and granted as
|
||||
patterns (`jira.*`, `db.query`) at any level of the tenancy tree, as
|
||||
models are.
|
||||
- A level that names any narrows what is above it: a tool is seen where
|
||||
every level above that names tools names it too. A team under an
|
||||
organization that names `hr.*` sees no `jira.*` whatever the team is
|
||||
given, until the organization names it as well.
|
||||
- Unlike models, **none granted anywhere is none**: a tool reaches into a
|
||||
company system, so it is granted, never assumed.
|
||||
- `keys limits --tools` writes a level's list whole, every server's
|
||||
together; an empty list (`--tools ''`) closes the level to every tool.
|
||||
The console's **MCP servers** screen and the `/admin/mcp` page grant
|
||||
and take back one server's tools: a level given its first list starts
|
||||
from the nearest list above it, so it keeps what it saw; one left
|
||||
naming none goes back to naming none, or stays closed when a level
|
||||
above names tools, so taking one server away opens it to no other. They
|
||||
say when a level above keeps the granted tools out of reach.
|
||||
- **The server is told who calls:** `X-Llmgw-Key`, and `X-Llmgw-User` and
|
||||
`X-Llmgw-Project` where there are some. It is called with llmgw's
|
||||
credentials, never the caller's key.
|
||||
- **Bounds:** a call has 60 seconds (`--timeout`) and an answer 4 MB. A
|
||||
server's progress is passed on to a client that takes a stream. `--rate N`
|
||||
allows each caller N calls a minute to a server's tools.
|
||||
- **Guardrails, both ways:** the caller's policies read a tool's answer as
|
||||
the prompt it becomes.
|
||||
- An instruction planted in a ticket is caught as a pasted one would be
|
||||
(`injection: block`).
|
||||
- Personal data is redacted or refused as the policy says. A refused
|
||||
answer does not reach the agent.
|
||||
- The arguments are read before they leave, so a secret pasted into one
|
||||
does not reach the company system.
|
||||
- **A server that keeps failing:** five calls failing in a row take a server
|
||||
out for a minute. Its tools leave the list, and a call is answered at once.
|
||||
- **Record:** every call is in `llmgw ctl audit`, with its arguments' size
|
||||
and a digest, never their text. `llmgw_mcp_tool_calls_total` and
|
||||
`llmgw_mcp_tool_call_seconds` count them by server, tool and outcome.
|
||||
- **Changes:** a server added, changed or removed on one control plane is
|
||||
served by every one within half a minute; its tool list is read again
|
||||
every five minutes. `--disabled` keeps a server but serves none of its
|
||||
tools.
|
||||
|
||||
**This cluster's own, read only.** `llmgw ctl mcp add llmgw --url
|
||||
llmgw:admin` registers the cluster itself as a server: tools that say what
|
||||
`llmgw ctl` does, and change nothing. It is answered in the control plane's
|
||||
own process, with no address and no token to give.
|
||||
|
||||
| Tool | Says |
|
||||
|---|---|
|
||||
| `llmgw.cluster_status` | control planes, store, nodes, models against what they asked for, GPUs, what is unhealthy |
|
||||
| `llmgw.list_nodes` | each node's state, GPUs, load, fabrics, agent version and engines |
|
||||
| `llmgw.list_models` | each model's replicas ready against wanted, profiles, why one is parked |
|
||||
| `llmgw.list_replicas` | each replica's shards: node, GPUs, state (`model` for one model's) |
|
||||
| `llmgw.explain_placement` | why a model is not placed, node by node |
|
||||
| `llmgw.node_logs` | a node's last agent log lines, or an engine's (`source`) |
|
||||
|
||||
Grant it as any other: `llmgw keys limits --scope key --id KEY-ID --tools
|
||||
'llmgw.*'`. It is the same as an administrator's view, so grant it to the
|
||||
people who would have `ctl`.
|
||||
|
||||
Servers over HTTP only for now (the streamable HTTP transport). A server
|
||||
that runs as a local process (stdio) is planned.
|
||||
|
||||
## Single sign-on
|
||||
|
||||
People can sign in through your identity provider — Keycloak, Entra ID,
|
||||
@@ -3038,10 +2387,8 @@ Google, Okta, Authentik, anything that speaks OpenID Connect.
|
||||
|
||||
1. Register a client there with the redirect address
|
||||
`https://<gateway>/login/oidc/callback`.
|
||||
2. Put its secret in a file the llmgw user reads, on every control plane.
|
||||
3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
|
||||
cluster with a store, `auth` is the cluster's: the same block in
|
||||
`server.yaml` is not read, and the log says so.
|
||||
2. Put its secret in a file only root reads.
|
||||
3. Add to `server.yaml`:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
@@ -3090,99 +2437,6 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
|
||||
join and leave them. Membership in a group is removed when someone leaves
|
||||
it.
|
||||
|
||||
### Entra ID
|
||||
|
||||
The console sets this up: `sudo llmgw`, People → Sign-in, **E**. What
|
||||
follows is what it writes, and what to do in Entra.
|
||||
|
||||
Entra ID is an OpenID Connect provider like the others, with two things to
|
||||
set in the app registration:
|
||||
|
||||
1. **Redirect URI:** under Authentication, add a Web platform with
|
||||
`https://<gateway>/login/oidc/callback`. The issuer is
|
||||
`https://login.microsoftonline.com/<tenant-id>/v2.0`.
|
||||
2. **Groups in the token:** under Token configuration, add a groups claim.
|
||||
Choose **Groups assigned to the application**, and assign the groups you
|
||||
map in `roles` and `memberships` to the enterprise application. Entra
|
||||
sends groups by object id unless you choose to emit names
|
||||
(`sAMAccountName` for groups synced from AD), so write them the same way
|
||||
in the mapping.
|
||||
|
||||
Entra leaves the groups out of the token for someone in more than 200 of
|
||||
them and sends a pointer to Graph instead. llmgw then keeps that person's
|
||||
role and memberships as they were, rather than reading it as no groups, and
|
||||
logs an error saying so. "Groups assigned to the application" avoids this.
|
||||
So does SCIM below, which Entra also speaks: in the enterprise application
|
||||
under Provisioning, set the tenant URL to `https://<gateway>/scim/v2` and
|
||||
the secret token to the contents of `auth.scim.token_file`.
|
||||
|
||||
## Active Directory
|
||||
|
||||
With an on-premises Active Directory, people sign in on the gateway's own
|
||||
form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
|
||||
password. The gateway checks them with the domain controllers over LDAPS.
|
||||
|
||||
The console sets this up and tries it first: `sudo llmgw`, People →
|
||||
Sign-in, **D**. It keeps the service account's password in the store
|
||||
(`bind_password_file: store:auth/ldap-bind`) and writes the CA into the
|
||||
settings (`ca:`), so nothing has to be copied to each control plane.
|
||||
By hand it is this block in the cluster's settings (`llmgw ctl config
|
||||
edit`), where files it names must then be on every control plane:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
enabled: true
|
||||
ldap:
|
||||
urls: [ldaps://dc1.corp.example, ldaps://dc2.corp.example] # tried in order
|
||||
ca_file: /etc/llmgw/corp-ca.pem # the domain's CA, when not in the system's
|
||||
bind_dn: CN=svc-llmgw,OU=Service,DC=corp,DC=example
|
||||
bind_password_file: /etc/llmgw/secrets/ldap-bind # root-only
|
||||
base_dn: OU=Staff,DC=corp,DC=example # where people are
|
||||
user_filter: (memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)
|
||||
label: Acme # "Sign in with your Acme account"
|
||||
create_accounts: true
|
||||
roles: # by group CN or full DN; nested groups count
|
||||
admin: [llmgw-admins]
|
||||
operator: [llmgw-ops]
|
||||
memberships:
|
||||
ml-engineers: "team:Acme/ml"
|
||||
sync_interval: 15m # negative turns the sync off
|
||||
```
|
||||
|
||||
- **Service account:** an ordinary domain user is enough: it only reads.
|
||||
It finds the person and reads their groups. The person's password is
|
||||
checked by binding as them, and is never stored.
|
||||
- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
|
||||
Plaintext LDAP is refused, since passwords would cross the network.
|
||||
- **Certificate:** the domain controller's certificate must name it in
|
||||
its subject alternative names: a name in the CN only is refused, as are
|
||||
certificates with a negative serial number. Samba's own autogenerated
|
||||
certificate is both, so a Samba DC needs one from a CA (`tls certfile`
|
||||
in smb.conf). A CA file a control plane cannot read turns directory
|
||||
sign-in off on that control plane, said in its log. Local accounts still
|
||||
sign in.
|
||||
- **Who may sign in:** enabled user objects under `base_dn` that match
|
||||
`user_filter`. The example admits members of `llmgw-users`, nested groups
|
||||
included.
|
||||
- **Accounts:** an account is tied to the person's `objectGUID`, so a changed
|
||||
mail or name is still the same account. An account that already exists
|
||||
with the person's mail is taken over. Without `create_accounts`, only
|
||||
people who already have an account can sign in. Local accounts, such as
|
||||
the administrator `server init` made, keep signing in with their own
|
||||
password.
|
||||
- **Sync:** every `sync_interval` the leader reads the directory.
|
||||
- Someone disabled or deleted in AD is suspended and their sessions
|
||||
end.
|
||||
- When they are enabled again they are reinstated. An account an
|
||||
administrator suspended by hand stays suspended.
|
||||
- Roles follow groups, and memberships the directory gave are taken away
|
||||
when the person leaves the group.
|
||||
- A sync that would suspend more than half of the directory's accounts is
|
||||
refused and logged, rather than locking everyone out over a wrong
|
||||
`base_dn` or filter.
|
||||
- **Passwords:** a person's password is changed in AD. The gateway sends no
|
||||
reset link for these accounts.
|
||||
|
||||
## Secrets in Vault
|
||||
|
||||
Every setting that names a file holding a secret — the store's `@address`,
|
||||
@@ -3230,7 +2484,7 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
@@ -3754,8 +3008,6 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
|
||||
- `docs/choosing-models.md` — which models to run and how many, if the answer
|
||||
to "we have GPUs, now what" is not already obvious
|
||||
- `docs/large-models.md` — a model on several cards or several nodes: how it
|
||||
is split, what each way needs of a card, and bringing one up on a fabric
|
||||
- `docs/data.md` — what is stored and what leaves the machine, for the review
|
||||
that happens before this is allowed near anyone's documents
|
||||
- `docs/reference.md` — every configuration key, generated from the source, so
|
||||
|
||||
+14
-16
@@ -3,9 +3,9 @@
|
||||
A knowledge base is a collection of a company's documents that the cluster
|
||||
can answer from. People put their documents into it from their own machines
|
||||
with `llmgw-kb`. The tool reads the files on that machine, cuts them into
|
||||
passages ("chunks") and sends their text; the gateway embeds each passage
|
||||
with the person's own key and keeps the text and the vectors. The files
|
||||
themselves never leave the machine.
|
||||
passages ("chunks"), has the gateway embed each passage with the person's own
|
||||
key, and sends the text and the vectors. The files themselves never leave
|
||||
the machine.
|
||||
|
||||
## Why retrieval, and not training
|
||||
|
||||
@@ -159,12 +159,10 @@ llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, satura
|
||||
llmgw keys create import --project Acme/docs --band bulk
|
||||
```
|
||||
|
||||
The people's questions then go first, and the import waits behind them. The
|
||||
gateway embeds a document's passages 32 at a time, four batches at once,
|
||||
enough to fill an embedding replica's slots; a batch told to wait (`Retry-After`)
|
||||
waits up to five minutes before the document is answered the same, and
|
||||
`llmgw-kb` sends it again rather than counting it as failed. It gives up
|
||||
after 15 minutes.
|
||||
The people's questions then go first, and the import waits behind them. A
|
||||
request that waits as long as its band allows is answered 504 with
|
||||
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
|
||||
passage as failed. It gives up after 15 minutes.
|
||||
|
||||
## For everyone: `llmgw-kb`
|
||||
|
||||
@@ -227,7 +225,6 @@ not changed.
|
||||
| `--include`, `--exclude` | Glob patterns. `*.pdf` matches a file name; `reports/**/*.xlsx` matches a path. |
|
||||
| `--hidden` | Reads hidden files and folders too (left out by default). |
|
||||
| `-v` | Lists unchanged files too. |
|
||||
| `--embed-here` | Embeds the passages from this machine and sends the vectors. With a gateway that does not embed them itself it does so anyway. |
|
||||
|
||||
Documents are named by their path under the folder, with forward slashes,
|
||||
on every system. With several folders, each folder's own name comes first.
|
||||
@@ -262,12 +259,13 @@ scanned page with no OCR.
|
||||
|
||||
### What leaves your machine
|
||||
|
||||
For each new or changed document, one request leaves the machine, over the
|
||||
gateway's TLS connection: its name, its file's SHA-256 digest and size, and
|
||||
its passages' text. The gateway embeds the passages through `/v1/embeddings`
|
||||
as you, so your key's limits, budget and guardrails apply as if you had
|
||||
asked. That is about a kilobyte a passage; embedding here (`--embed-here`)
|
||||
brings each vector down and sends it up again, some thirty times more.
|
||||
For each new or changed document, two things leave the machine, both over
|
||||
the gateway's TLS connection:
|
||||
|
||||
- **The passages, to be embedded.** They go through `/v1/embeddings` like
|
||||
any request of yours. Your key's limits, budget and guardrails apply.
|
||||
- **The document itself.** Its name, its file's SHA-256 digest and size,
|
||||
and the passages with their vectors.
|
||||
|
||||
The file itself is never sent. `--dry-run` shows exactly what would be.
|
||||
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
# llmgw and Kong
|
||||
|
||||
For a site that already runs Kong. The two meet over plain HTTP: Kong in
|
||||
front of llmgw, as one more API it governs, or Kong behind llmgw, as an
|
||||
OpenAI-compatible provider. Nothing of either is built into the other.
|
||||
|
||||
Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
|
||||
and later need a Kong licence) in DB-less mode, in front of llmgw.
|
||||
|
||||
## Kong in front of llmgw
|
||||
|
||||
Two ways, both tried.
|
||||
|
||||
**A plain route.** Kong forwards the request as it came. Tried with chat
|
||||
completions, streamed and not; llmgw's other APIs — Anthropic's
|
||||
`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
|
||||
since Kong does not read them. Kong adds its own authentication, rate limits
|
||||
and logging; llmgw still checks the key, the budget and the guardrails.
|
||||
|
||||
```yaml
|
||||
_format_version: "3.0"
|
||||
services:
|
||||
- name: llmgw
|
||||
url: https://llm.example.com # llmgw's client port
|
||||
routes:
|
||||
- name: llmgw
|
||||
paths: [/llmgw] # /llmgw/v1/chat/completions → /v1/chat/completions
|
||||
```
|
||||
|
||||
**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
|
||||
route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
- name: llmgw-chat
|
||||
url: https://llm.example.com
|
||||
routes:
|
||||
- name: chat
|
||||
paths: [/chat]
|
||||
plugins:
|
||||
- name: ai-proxy
|
||||
config:
|
||||
route_type: llm/v1/chat
|
||||
logging: {log_statistics: true, log_payloads: false}
|
||||
model:
|
||||
provider: openai
|
||||
name: qwen-coder # a model llmgw serves
|
||||
options:
|
||||
upstream_url: https://llm.example.com/v1/chat/completions
|
||||
```
|
||||
|
||||
**Keys.** Without an `auth` block, `ai-proxy` passes the client's
|
||||
`Authorization` header on: each team keeps its own llmgw key, and llmgw's
|
||||
budgets, limits, guardrails and usage stay per team. With
|
||||
`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
|
||||
sends one key of its own instead, and llmgw sees every caller as that key —
|
||||
then budgets per team are Kong's to keep.
|
||||
|
||||
**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
|
||||
records prompt and completion tokens as llmgw did. For a streamed answer
|
||||
Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
|
||||
even with `stream_options.include_usage` set — the usage chunk reaches it
|
||||
and is not read. llmgw's own accounting counts both; use its usage export
|
||||
(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
|
||||
|
||||
**Timeouts.** A long answer from a model still loading outlasts Kong's
|
||||
60-second defaults: set the service's `read_timeout` to the time a model may
|
||||
take to start, as for nginx in [install.md](install.md).
|
||||
|
||||
## Kong behind llmgw
|
||||
|
||||
Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
|
||||
it reaches any provider: a model whose source is `external`.
|
||||
|
||||
```yaml
|
||||
- name: kong-gpt
|
||||
source: external
|
||||
external:
|
||||
url: https://kong.example.com/openai # Kong's route, without /v1
|
||||
model: gpt-4o-mini
|
||||
api_key_file: /etc/llmgw/secrets/kong-consumer.key
|
||||
```
|
||||
|
||||
The key is the one Kong's own authentication asks of a consumer. As a
|
||||
fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
|
||||
room.
|
||||
|
||||
## Which to choose
|
||||
|
||||
Kong in front when the site's API policy says every API goes through Kong:
|
||||
llmgw is then one more upstream, and its per-team accounting still holds
|
||||
when each team keeps its own key. Kong behind when the site already reaches
|
||||
outside providers through Kong and llmgw should too. llmgw does not need
|
||||
Kong for anything it does itself — keys, budgets, guardrails, routing to
|
||||
its own replicas.
|
||||
@@ -1,145 +0,0 @@
|
||||
# Large models: several cards, several nodes
|
||||
|
||||
A model too big for one card runs on several. llmgw chooses how from the
|
||||
model's own figures and the cluster as it is; this page is the rules it
|
||||
follows, the commands, and what to look at when a first one is brought up
|
||||
on new hardware.
|
||||
|
||||
## How a model is split
|
||||
|
||||
In order, the first that fits:
|
||||
|
||||
1. **One card.** Nothing to split.
|
||||
2. **Several cards of one node, tensor parallel.** Every layer is cut
|
||||
across the cards, so they talk for every token: they must be on one
|
||||
NVLink domain. The agent reads `nvidia-smi topo -m` and gives a replica
|
||||
the cards with the most NVLink between them
|
||||
([install.md](install.md#nvlink-if-your-gpus-have-it)).
|
||||
3. **Several nodes, pipeline parallel between them.** Each node holds a
|
||||
run of layers on all its cards (tensor parallel inside it), and only a
|
||||
layer's output crosses to the next node. The nodes must share one
|
||||
InfiniBand or RoCE fabric: a replica never spans two, and never a node
|
||||
with none. Any number of nodes, not only two or four.
|
||||
|
||||
The plan takes the fewest cards, then the fewest nodes, then the better
|
||||
precision up to eight bits.
|
||||
|
||||
**Tensor width and the KV heads.** The cards split the model's KV heads
|
||||
among them, or past their number each card holds a copy of one. So the
|
||||
width divides the KV heads or is a multiple of them: a model with 8 KV
|
||||
heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8.
|
||||
|
||||
**Mixture of experts.** A model of experts (Qwen3-235B-A22B, DeepSeek-V3,
|
||||
Llama 4, Mixtral) on more than one card is served with vLLM's
|
||||
`--enable-expert-parallel`: each card holds whole experts rather than a
|
||||
slice of every one, which is less traffic per token. It is read from
|
||||
`config.json` (`num_experts`, `n_routed_experts`, `num_local_experts`).
|
||||
|
||||
**When it does not fit.** A smaller precision first: an FP8 or AWQ
|
||||
repository of the same model, or a GGUF quantisation on llama.cpp if one
|
||||
node's cards hold it. Then more nodes. `llmgw ctl model plan REPO
|
||||
--precision bf16,fp8` shows each precision's fit from the model's config.
|
||||
|
||||
## What a way needs
|
||||
|
||||
Each way the plan offers says what a replica needs of each card, part by
|
||||
part, and of all its cards together:
|
||||
|
||||
```
|
||||
$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768
|
||||
FILE SIZE BEST WAY A CARD OF ALL CARDS NOW
|
||||
original 131 GB 4 × h100-80g 37.4 GB 72 GB 150 GB free
|
||||
|
||||
Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards.
|
||||
The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another).
|
||||
```
|
||||
|
||||
- **weights:** the files, divided over the cards.
|
||||
- **context:** the KV cache for the context planned, divided likewise.
|
||||
- **engine:** the engine's own: 2 GB and a fiftieth of the weights for vLLM.
|
||||
- **usable:** 90% of a card; the rest is the driver's and fragmentation's.
|
||||
|
||||
The console's **Serve a model** shows the same for every way a file runs.
|
||||
A context longer than planned is checked against it before anything is
|
||||
fetched.
|
||||
|
||||
## Serving one
|
||||
|
||||
Without flags the plan's best way is taken:
|
||||
|
||||
```
|
||||
llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768
|
||||
```
|
||||
|
||||
Or a way given by hand, for instance two nodes of eight:
|
||||
|
||||
```
|
||||
llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \
|
||||
--gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO
|
||||
```
|
||||
|
||||
`--gpus` is the cards on each node and `--nodes` the nodes. The pipeline
|
||||
is the nodes and the tensor width the cards of each.
|
||||
|
||||
`--env KEY=VALUE` goes into the engine's environment on every node:
|
||||
`NCCL_DEBUG=INFO` while bringing a cluster up, `NCCL_IB_*` to steer it.
|
||||
Only the variables engines are tuned with are taken. llmgw already sets
|
||||
these:
|
||||
- `NCCL_IB_HCA` to the fabric's adapters;
|
||||
- `NCCL_IB_GID_INDEX` on RoCE;
|
||||
- `NCCL_SOCKET_IFNAME` to the management network;
|
||||
- for an NVLink domain spanning nodes, `NCCL_MNNVL_ENABLE`.
|
||||
|
||||
The job's first step says what it needs of each card. The weights then
|
||||
come to the mirror once and from it to each node. The startup timeout (20
|
||||
minutes unless the profile's `startup_timeout` says otherwise) runs from
|
||||
when a node's download last moved. So a model of hundreds of gigabytes is
|
||||
not failed while it is still arriving, and one whose download stops is
|
||||
failed that long after.
|
||||
|
||||
## Watching it come up
|
||||
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| The replica and its shards, one per node, each with its download | `llmgw ctl replicas` |
|
||||
| Why it is not placed | `llmgw ctl explain-placement MODEL` |
|
||||
| A node's engine output (NCCL's lines with `NCCL_DEBUG=INFO`) | `llmgw ctl logs NODE --source SHARD` |
|
||||
| The bandwidth between two nodes, rail by rail | `llmgw ctl fabric-test NODE NODE` |
|
||||
| Each node's GPUs, NVLink and adapters | `llmgw doctor` on the node; its page under /admin |
|
||||
| Every step of a start, across nodes | the `replica start` trace ([install.md](install.md)) |
|
||||
|
||||
With `NCCL_DEBUG=INFO`, NCCL says which transport it took between nodes:
|
||||
`NET/IB` is the fabric. `NET/Socket` means it fell back to TCP, and the
|
||||
model will run, slowly.
|
||||
|
||||
## Bringing it up on new hardware
|
||||
|
||||
A first run on a fabric, in order. Each step is worth its output if
|
||||
something goes wrong.
|
||||
|
||||
1. **The nodes.** Add each with its fabric:
|
||||
`llmgw node add host --fabric ib-a:mlx5_0,mlx5_1`. Then
|
||||
`llmgw ctl nodes` shows every node ready, on `ib-a`.
|
||||
2. **The fabric.** `llmgw ctl fabric-test node1 node2`: each rail at its
|
||||
rated speed (a 400 Gb/s rail near 45 GB/s).
|
||||
3. **One node, several cards.** A dense 70B at bf16 on four or eight
|
||||
cards of one node, by the plan:
|
||||
`llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors`.
|
||||
Ready, then a question answered.
|
||||
4. **Two nodes.** The same model forced over two:
|
||||
`--gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO`, under
|
||||
another `--name`. Ready, answering, and `NET/IB` in both shards' logs.
|
||||
5. **Experts.** A mixture-of-experts model on one node's eight cards,
|
||||
for instance `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`: its profile
|
||||
carries `--enable-expert-parallel` (`llmgw ctl models`, or its
|
||||
definition from the admin API).
|
||||
6. **A node lost.** With the two-node replica serving, stop one node's
|
||||
agent (`systemctl stop llmgw-agent`): the replica fails and is placed
|
||||
again once the node is back or another has room. Nothing is left
|
||||
holding cards.
|
||||
|
||||
For each, worth keeping:
|
||||
- the time from serve to ready;
|
||||
- tokens a second from `llmgw ctl bench`;
|
||||
- the plan's per-card figure against what `nvidia-smi` shows once loaded;
|
||||
- anything it said that was wrong.
|
||||
Binary file not shown.
+14
-45
@@ -54,20 +54,10 @@ on the page that creates it and never again.
|
||||
|
||||
### Connecting an editor
|
||||
|
||||
The account page's **Connect your editor** makes a key for VS Code, Continue,
|
||||
Codex or Cline and shows what the tool needs with that key, this gateway's
|
||||
address and a model already in it:
|
||||
The account page's **Connect your editor** makes a key for Continue, Codex or
|
||||
Cline and shows what the tool needs with that key, this gateway's address and a
|
||||
model already in it:
|
||||
|
||||
- **VS Code** — llmgw's own extension, downloaded from the gateway at
|
||||
`/setup/vscode/llmgw.vsix` with this gateway's address already in it. It
|
||||
offers every chat model the key may use to VS Code itself: the chat's model
|
||||
picker, agent mode, and any extension that asks VS Code for a language model
|
||||
(Cline's "VS Code LM API" provider among them), with no setup of their own.
|
||||
**llmgw: Sign In** in VS Code opens this page with VS Code chosen; the page's
|
||||
**Open in VS Code** link hands the key back, and the extension keeps it in
|
||||
VS Code's secret storage. A link the extension did not ask for is confirmed
|
||||
first, with the gateway's address shown, since it decides where code is sent.
|
||||
**llmgw: Enter API Key** takes a pasted one instead.
|
||||
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
|
||||
so their `config.yaml` is not touched.
|
||||
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
|
||||
@@ -192,41 +182,22 @@ auth:
|
||||
A personal key has no project, so it keeps no conversations or agent notes;
|
||||
those are kept per project.
|
||||
|
||||
### Organization managers and admins
|
||||
### Organization managers
|
||||
|
||||
A member of an organization can be made its manager, who sees it, or its
|
||||
admin, who also changes it:
|
||||
A member of an organization can be made its manager:
|
||||
|
||||
```
|
||||
llmgw users join boss@acme.com --org Acme --role manager
|
||||
llmgw users join lead@acme.com --org Acme --role admin
|
||||
```
|
||||
|
||||
or with the role chosen on their organization membership on `/admin/users`.
|
||||
Both get an **Organization** link and a page at `/org`: usage over time by team,
|
||||
or with the star on their organization membership on `/admin/users`. A manager
|
||||
gets an **Organization** link and a page at `/org`: usage over time by team,
|
||||
project or person, the teams and projects with their limits and spend, and every
|
||||
member. Only what was spent in that organization's projects is on it; what its
|
||||
people do elsewhere is not.
|
||||
|
||||
A manager's page is read-only. An organization's admin also changes the
|
||||
organization from it:
|
||||
|
||||
- adds and renames its teams and projects, and removes an empty one;
|
||||
- adds a person by their sign-in address to the organization, a team or a
|
||||
project, and takes them out (their keys there are revoked with it);
|
||||
- makes a member of the organization a manager or an admin, or neither;
|
||||
- revokes a key spent in its projects.
|
||||
|
||||
What reaches past the organization stays with the cluster's administrators:
|
||||
its limits and budgets, approving or suspending an account (someone not yet
|
||||
approved cannot be added), the organization's own name, and anything else on
|
||||
the cluster. An admin cannot take away their own role. A group from single
|
||||
sign-on or Active Directory that puts someone in the organization does not, by
|
||||
saying so again at the next sign-in, take a manager's or admin's role away.
|
||||
|
||||
An administrator of the cluster following `/org` lands on the same
|
||||
organization's panel on the people page instead, where all of that can be
|
||||
changed.
|
||||
people do elsewhere is not. The page is read-only — a manager is not an
|
||||
administrator of the cluster, and limits and memberships stay with those who
|
||||
are. An administrator following `/org` lands on the same organization's panel
|
||||
on the people page instead, where all of that can be changed.
|
||||
|
||||
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
|
||||
will no longer move a team or project that already exists under another parent;
|
||||
@@ -536,10 +507,8 @@ included: `llmgw ctl audit`, or **Audit** in the console.
|
||||
suspends accounts from it; see [install.md](install.md#single-sign-on).
|
||||
Without them accounts are local, with local passwords.
|
||||
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
|
||||
(`llmgw users role`) decide what someone may see and do under /admin, and a
|
||||
page draws only the forms that person may send: a viewer sees no forms. An
|
||||
organization's manager can view it; its admin also changes its teams,
|
||||
projects, people and keys. Every change is in the audit log (`llmgw ctl
|
||||
audit`).
|
||||
(`llmgw users role`) decide what someone may see and do under /admin; an
|
||||
organization's manager can view it but not change it. Every change is in
|
||||
the audit log (`llmgw ctl audit`).
|
||||
- **Rate limits are per gateway instance.** With two instances each counts its own,
|
||||
so the effective limit doubles. v1 is single-instance by design.
|
||||
+2
-30
@@ -8,7 +8,7 @@ explains every choice.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 12 GB RAM when it is also a node running the starter model and the two judges, which setup checks before it installs anything; four cores or more, or the judges on machines of their own, keep a judgment within its wait. |
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
|
||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
||||
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
||||
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
||||
@@ -146,35 +146,7 @@ agent is stopped.
|
||||
## 4. Add a model
|
||||
|
||||
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
||||
Hugging Face, showing only what your nodes can run: judged by the cluster as it
|
||||
is now — the largest card and its compute capability, the most memory on one
|
||||
node — so a model too big for them, in a form those cards lack (FP8 needs 8.9),
|
||||
or in neither GGUF nor safetensors is left out, and the page says how many.
|
||||
Without GPUs it lists GGUF only. Untick **only what these nodes can run** to
|
||||
see everything. On a model's page a file too big for the cluster is shown
|
||||
without a **Fetch** button, with what it would need.
|
||||
|
||||
The **CPU tok/s** column is about how fast one reply would come on the CPU of
|
||||
the fastest node whose memory holds the model at Q4_K_M; a model's page gives
|
||||
it for each GGUF file by its own size. Every node's agent measures how fast its
|
||||
memory reads when it starts (it is in the agent's log as `memory_gbs`), and a
|
||||
token on a CPU reads every weight once: the estimate is that speed over the
|
||||
model's size — for a model of experts named like `30B-A3B`, its active part.
|
||||
On the test cluster it came within 15% of what llama.cpp measured. Several
|
||||
replies at once share the reads and each is slower.
|
||||
|
||||
**GPU tok/s** is the same on the cards, for the layout this cluster would
|
||||
give the model — the fewest cards that hold it, of each class, the fastest of
|
||||
those; its tooltip says which ("2× h100-80g, NVLink, vLLM", "2 nodes × 1×
|
||||
a100-80g over ib0 (InfiniBand 200 Gb/s), vLLM"). A card's memory bandwidth is
|
||||
known from its model name (an H100 SXM and an H100 PCIe differ by half); a card
|
||||
not in the table gets no figure. Tensor parallel across a node's cards divides
|
||||
what each reads, and adds two all-reduces a layer, short over NVLink and
|
||||
longer over PCIe; llama.cpp across cards splits by layer, so the cards take
|
||||
turns and a reply is no faster than on one; a pipeline across nodes takes turns
|
||||
too, and adds a hop over the fabric for each node. One card's figures were
|
||||
measured on the test cluster's RTX 6000; tensor parallel and the fabric are
|
||||
from what NCCL is known to take, not measured here.
|
||||
Hugging Face, showing only what your nodes can run.
|
||||
|
||||
1. Click a model to read its card.
|
||||
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
||||
|
||||
+1
-577
@@ -37,14 +37,11 @@ rather than repeated.
|
||||
| `default_tpm` | `int` | [Auth](#auth) |
|
||||
| `signup_open` | `bool` | [Auth](#auth) |
|
||||
| `oidc` | `OIDC` | [Auth](#auth) |
|
||||
| `ldap` | `LDAP` | [Auth](#auth) |
|
||||
| `scim` | `SCIM` | [Auth](#auth) |
|
||||
| `max_key_ttl` | `time.Duration` | [Auth](#auth) |
|
||||
| `personal` | `Personal` | [Auth](#auth) |
|
||||
| `pricing` | `map[string]Price` | [Auth](#auth) |
|
||||
| `default_price` | `Price` | [Auth](#auth) |
|
||||
| `gpu_hour_cost` | `float64` | [Auth](#auth) |
|
||||
| `core_hour_cost` | `float64` | [Auth](#auth) |
|
||||
| `flush_interval` | `time.Duration` | [Auth](#auth) |
|
||||
| `batch_size` | `int` | [Auth](#auth) |
|
||||
| `buffer_size` | `int` | [Auth](#auth) |
|
||||
@@ -55,7 +52,6 @@ rather than repeated.
|
||||
| `retain_days` | `int` | [Batch](#batch) |
|
||||
| `max_bytes` | `int64` | [Cache](#cache) |
|
||||
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
||||
| `embedding_model` | `string` | [Cache](#cache) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
@@ -109,13 +105,10 @@ rather than repeated.
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `api_key_files` | `[]ExternalKey` | [External](#external) |
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `output` | `string` | [External](#external) |
|
||||
| `file` | `string` | [ExternalKey](#externalkey) |
|
||||
| `weight` | `float64` | [ExternalKey](#externalkey) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -128,7 +121,6 @@ rather than repeated.
|
||||
| `name` | `string` | [Federation](#federation) |
|
||||
| `interval` | `time.Duration` | [Federation](#federation) |
|
||||
| `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
|
||||
| `binary` | `string` | [FreeTokenConfig](#freetokenconfig) |
|
||||
| `min_compute_capability` | `float64` | [GPUNeed](#gpuneed) |
|
||||
| `min_memory_gb` | `int` | [GPUNeed](#gpuneed) |
|
||||
| `format` | `string` | [GuardModel](#guardmodel) |
|
||||
@@ -152,21 +144,6 @@ rather than repeated.
|
||||
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
||||
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `max_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `urls` | `[]string` | [LDAP](#ldap) |
|
||||
| `start_tls` | `bool` | [LDAP](#ldap) |
|
||||
| `ca_file` | `string` | [LDAP](#ldap) |
|
||||
| `ca` | `string` | [LDAP](#ldap) |
|
||||
| `bind_dn` | `string` | [LDAP](#ldap) |
|
||||
| `bind_password_file` | `string` | [LDAP](#ldap) |
|
||||
| `base_dn` | `string` | [LDAP](#ldap) |
|
||||
| `group_base_dn` | `string` | [LDAP](#ldap) |
|
||||
| `user_filter` | `string` | [LDAP](#ldap) |
|
||||
| `label` | `string` | [LDAP](#ldap) |
|
||||
| `create_accounts` | `bool` | [LDAP](#ldap) |
|
||||
| `roles` | `map[string][]string` | [LDAP](#ldap) |
|
||||
| `memberships` | `map[string]string` | [LDAP](#ldap) |
|
||||
| `sync_interval` | `time.Duration` | [LDAP](#ldap) |
|
||||
| `timeout` | `time.Duration` | [LDAP](#ldap) |
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `language` | `string` | [Landing](#landing) |
|
||||
@@ -183,7 +160,6 @@ rather than repeated.
|
||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `api` | `string` | [Mirror](#mirror) |
|
||||
| `peers` | `[]string` | [Mirror](#mirror) |
|
||||
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
|
||||
| `registry` | `string` | [Mirror](#mirror) |
|
||||
| `timeout` | `time.Duration` | [Mirror](#mirror) |
|
||||
@@ -192,10 +168,8 @@ rather than repeated.
|
||||
| `aliases` | `[]string` | [Model](#model) |
|
||||
| `description` | `string` | [Model](#model) |
|
||||
| `source` | `string` | [Model](#model) |
|
||||
| `allow_pickle` | `bool` | [Model](#model) |
|
||||
| `profiles` | `[]Profile` | [Model](#model) |
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `stopped` | `*Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
@@ -231,13 +205,10 @@ rather than repeated.
|
||||
| `no_passwords` | `bool` | [OIDC](#oidc) |
|
||||
| `mirror` | `string` | [Offline](#offline) |
|
||||
| `checksums` | `string` | [Offline](#offline) |
|
||||
| `evict_after_idle` | `time.Duration` | [OnDemandSettings](#ondemandsettings) |
|
||||
| `min_lifetime` | `time.Duration` | [OnDemandSettings](#ondemandsettings) |
|
||||
| `nvidia_smi` | `string` | [Options](#options) |
|
||||
| `ib_root` | `string` | [Options](#options) |
|
||||
| `prefer_nvml` | `bool` | [Options](#options) |
|
||||
| `skip_gpus` | `bool` | [Options](#options) |
|
||||
| `dcgm_exporter` | `string` | [Options](#options) |
|
||||
| `repo` | `string` | [Origin](#origin) |
|
||||
| `revision` | `string` | [Origin](#origin) |
|
||||
| `files` | `[]OriginFile` | [Origin](#origin) |
|
||||
@@ -265,7 +236,6 @@ rather than repeated.
|
||||
| `require_capabilities` | `bool` | [Policy](#policy) |
|
||||
| `validate_structured` | `bool` | [Policy](#policy) |
|
||||
| `judge` | `*JudgePolicy` | [Policy](#policy) |
|
||||
| `services` | `[]Service` | [Policy](#policy) |
|
||||
| `stream` | `string` | [Policy](#policy) |
|
||||
| `enabled` | `bool` | [Prefix](#prefix) |
|
||||
| `max_nodes` | `int` | [Prefix](#prefix) |
|
||||
@@ -297,7 +267,6 @@ rather than repeated.
|
||||
| `keep_within` | `string` | [Profile](#profile) |
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
| `priority` | `int` | [Profile](#profile) |
|
||||
| `replicas` | `int` | [Profile](#profile) |
|
||||
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
|
||||
| `kv_events` | `bool` | [Profile](#profile) |
|
||||
| `gpu` | `*GPUNeed` | [Profile](#profile) |
|
||||
@@ -323,7 +292,6 @@ rather than repeated.
|
||||
| `batch` | `Batch` | [Router](#router) |
|
||||
| `cache` | `Cache` | [Router](#router) |
|
||||
| `trusted_proxies` | `[]string` | [Router](#router) |
|
||||
| `active_active` | `bool` | [Router](#router) |
|
||||
| `timeouts` | `Timeouts` | [Router](#router) |
|
||||
| `scoring` | `Scoring` | [Router](#router) |
|
||||
| `admission` | `Admission` | [Router](#router) |
|
||||
@@ -331,7 +299,6 @@ rather than repeated.
|
||||
| `injection` | `string` | [Rules](#rules) |
|
||||
| `deny` | `[]Pattern` | [Rules](#rules) |
|
||||
| `max_chars` | `int` | [Rules](#rules) |
|
||||
| `restore` | `bool` | [Rules](#rules) |
|
||||
| `token_file` | `string` | [SCIM](#scim) |
|
||||
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
||||
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
||||
@@ -378,9 +345,7 @@ rather than repeated.
|
||||
| `mail` | `mail.Config` | [Server](#server) |
|
||||
| `serving` | `ServingSettings` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `slurm` | `SlurmSettings` | [Server](#server) |
|
||||
| `presets` | `PresetSettings` | [Server](#server) |
|
||||
| `on_demand` | `OnDemandSettings` | [Server](#server) |
|
||||
| `agents` | `AgentSettings` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
@@ -393,33 +358,8 @@ rather than repeated.
|
||||
| `facility` | `Facility` | [Server](#server) |
|
||||
| `federation` | `Federation` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `name` | `string` | [Service](#service) |
|
||||
| `url` | `string` | [Service](#service) |
|
||||
| `header_files` | `map[string]string` | [Service](#service) |
|
||||
| `body` | `string` | [Service](#service) |
|
||||
| `blocked` | `string` | [Service](#service) |
|
||||
| `reason` | `string` | [Service](#service) |
|
||||
| `on` | `string` | [Service](#service) |
|
||||
| `action` | `string` | [Service](#service) |
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `path` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `server` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
|
||||
| `users` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
| `store` | `string` | [Threads](#threads) |
|
||||
| `retain_hours` | `int` | [Threads](#threads) |
|
||||
@@ -451,7 +391,6 @@ rather than repeated.
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `cached_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
@@ -490,7 +429,6 @@ rather than repeated.
|
||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `cached_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
@@ -500,8 +438,6 @@ rather than repeated.
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
||||
| `strata` | `StrataConfig` | [engines.Config](#enginesconfig) |
|
||||
| `freetoken` | `FreeTokenConfig` | [engines.Config](#enginesconfig) |
|
||||
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -677,15 +613,6 @@ OIDC is sign-in through an identity provider.
|
||||
|
||||
Takes a [OIDC](#oidc) block.
|
||||
|
||||
### `ldap`
|
||||
|
||||
`LDAP`
|
||||
|
||||
LDAP is sign-in against a directory, Active Directory above all, and
|
||||
the accounts kept in step with it.
|
||||
|
||||
Takes a [LDAP](#ldap) block.
|
||||
|
||||
### `scim`
|
||||
|
||||
`SCIM`
|
||||
@@ -724,21 +651,6 @@ table costs whatever DefaultPrice says, which is nothing unless set.
|
||||
|
||||
Takes a [config.Price](#configprice) block.
|
||||
|
||||
### `gpu_hour_cost`
|
||||
|
||||
`float64`
|
||||
|
||||
GPUHourCost is what an hour of one GPU costs the operator, in the
|
||||
units budgets are in: the hardware written off over its years, its
|
||||
power and room, and the people who run it, over the GPUs and the
|
||||
hours of a month. With CoreHourCost for a CPU core, it is what
|
||||
`llmgw ctl prices suggest` prices each model from: the hours its
|
||||
replicas held, idle ones too, over the tokens it served.
|
||||
|
||||
### `core_hour_cost`
|
||||
|
||||
`float64`
|
||||
|
||||
### `flush_interval`
|
||||
|
||||
`time.Duration`
|
||||
@@ -813,15 +725,6 @@ Past it, the answers used longest ago go first.
|
||||
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
unless set.
|
||||
|
||||
### `embedding_model`
|
||||
|
||||
`string`
|
||||
|
||||
EmbeddingModel embeds the questions of projects that reuse answers to
|
||||
similar ones (llmgw keys cache --similar): an embedding model in the
|
||||
catalog. Unset, those projects reuse only the same request's answer.
|
||||
The gateway asks it on its own account, as it asks a judge.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
@@ -1042,7 +945,7 @@ work stands without being renewed.
|
||||
Only meaningful with a shared store, where a second control plane can
|
||||
see the claim. Long enough to survive a slow query or a brief network
|
||||
stall, short enough that a standby takes over before anyone has finished
|
||||
reading the alert. Zero takes the default, 10s.
|
||||
reading the alert. Zero takes the default.
|
||||
|
||||
## CustomNode
|
||||
|
||||
@@ -1197,15 +1100,6 @@ APIKeyFile holds the provider's key, on the control plane. A file
|
||||
rather than the key itself, so the key is in neither the catalog, the
|
||||
store nor the audit log; empty for a provider that asks for none.
|
||||
|
||||
### `api_key_files`
|
||||
|
||||
`[]ExternalKey`
|
||||
|
||||
APIKeyFiles are several keys for the provider, in place of
|
||||
APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
|
||||
answers 429 or 401 is passed over while its breaker is open and the
|
||||
request goes to another. A provider's limits are per key.
|
||||
|
||||
### `peer`
|
||||
|
||||
`string`
|
||||
@@ -1238,20 +1132,6 @@ plane.
|
||||
|
||||
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
|
||||
## ExternalKey
|
||||
|
||||
### `file`
|
||||
|
||||
`string`
|
||||
|
||||
File holds the key, on the control plane, as APIKeyFile does.
|
||||
|
||||
### `weight`
|
||||
|
||||
`float64`
|
||||
|
||||
Weight is its share of the requests against the others'; zero is 1.
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
@@ -1352,16 +1232,6 @@ site whose own firewall does that job; the rules are kept, and true
|
||||
puts them back. Changed in the cluster's settings, it is in force on
|
||||
every node and control plane within seconds.
|
||||
|
||||
## FreeTokenConfig
|
||||
|
||||
### `binary`
|
||||
|
||||
`string`
|
||||
|
||||
Binary is its ft command, in the environment it was installed into.
|
||||
The agent installs one itself on a node whose GPUs it runs on; this
|
||||
is for one installed by hand.
|
||||
|
||||
## GPUNeed
|
||||
|
||||
### `min_compute_capability`
|
||||
@@ -1528,114 +1398,6 @@ MaxBlocks bounds how many of a prompt's blocks are hashed and looked
|
||||
up. Beyond it a longer match still helps, but the difference no longer
|
||||
decides anything. Zero takes 4096.
|
||||
|
||||
## LDAP
|
||||
|
||||
### `urls`
|
||||
|
||||
`[]string`
|
||||
|
||||
URLs are the directory's servers, tried in turn: ldaps:// (636), or
|
||||
ldap:// (389) with StartTLS.
|
||||
|
||||
### `start_tls`
|
||||
|
||||
`bool`
|
||||
|
||||
StartTLS upgrades an ldap:// connection before anything is sent. A
|
||||
password never crosses a connection without TLS: ldap:// without it
|
||||
is refused.
|
||||
|
||||
### `ca_file`
|
||||
|
||||
`string`
|
||||
|
||||
CAFile is the authority the servers' certificates are checked against,
|
||||
when it is not the system's: an AD Certificate Services root.
|
||||
|
||||
### `ca`
|
||||
|
||||
`string`
|
||||
|
||||
CA is the same authority written out, PEM: kept in the cluster's
|
||||
settings, so no file has to be put on every control plane. Used in
|
||||
place of CAFile when both are set.
|
||||
|
||||
### `bind_dn`
|
||||
|
||||
`string`
|
||||
|
||||
BindDN and the file holding its password: a service account that may
|
||||
read users and groups. Nothing is written to the directory.
|
||||
|
||||
### `bind_password_file`
|
||||
|
||||
`string`
|
||||
|
||||
### `base_dn`
|
||||
|
||||
`string`
|
||||
|
||||
BaseDN is where people are searched for, subtree included.
|
||||
|
||||
### `group_base_dn`
|
||||
|
||||
`string`
|
||||
|
||||
GroupBaseDN is where their groups are; empty is BaseDN's domain root.
|
||||
|
||||
### `user_filter`
|
||||
|
||||
`string`
|
||||
|
||||
UserFilter narrows who may sign in, ANDed with the login match: a
|
||||
group (memberOf=...) or an OU's people only. Empty is every enabled
|
||||
user object.
|
||||
|
||||
### `label`
|
||||
|
||||
`string`
|
||||
|
||||
Label is what the form says to sign in with: "Sign in with your
|
||||
<label> account". Empty is "Active Directory".
|
||||
|
||||
### `create_accounts`
|
||||
|
||||
`bool`
|
||||
|
||||
CreateAccounts makes an account, active, for a person the directory
|
||||
vouches for and the gateway does not know. Off, only existing accounts
|
||||
sign in this way.
|
||||
|
||||
### `roles`
|
||||
|
||||
`map[string][]string`
|
||||
|
||||
Roles maps a staff role (viewer, operator, billing, admin) to the
|
||||
groups that give it, by name (CN) or whole DN. Set at every sign-in
|
||||
and sync: someone taken out of the group loses the role.
|
||||
|
||||
### `memberships`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Memberships maps a group, by CN or DN, to a place in the tree —
|
||||
org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in
|
||||
and kept by the sync, which also takes away a place whose group the
|
||||
person has left.
|
||||
|
||||
### `sync_interval`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
SyncInterval is how often the leader reads the directory. Zero is
|
||||
15m; negative turns the sync off, leaving only sign-in.
|
||||
|
||||
### `timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Timeout bounds each round trip to a server. Zero is 10s.
|
||||
|
||||
## Landing
|
||||
|
||||
### `contact`
|
||||
@@ -1774,14 +1536,6 @@ The management port, not the one nodes fetch bytes from. Those are two
|
||||
listeners on the mirror precisely so this one can be locked down without
|
||||
re-certifying every agent's fetcher.
|
||||
|
||||
### `peers`
|
||||
|
||||
`[]string`
|
||||
|
||||
Peers are the management addresses of mirrors kept in step with this
|
||||
one (llmgw mirror serve --peer): asked when it cannot be reached, and
|
||||
a node given the same file on one of them while it is down.
|
||||
|
||||
### `tls`
|
||||
|
||||
`tlsconf.Config`
|
||||
@@ -1851,16 +1605,6 @@ choice by guesswork.
|
||||
|
||||
Source is the weights path or a HuggingFace id.
|
||||
|
||||
### `allow_pickle`
|
||||
|
||||
`bool`
|
||||
|
||||
AllowPickle lets the nodes fetch this model's weights, or its
|
||||
adapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code
|
||||
as they are loaded. Off, such a source is refused when the model is
|
||||
added and by the node before it fetches anything. For a model that has
|
||||
no safetensors or GGUF form and whose files were checked.
|
||||
|
||||
### `profiles`
|
||||
|
||||
`[]Profile`
|
||||
@@ -1871,16 +1615,6 @@ no safetensors or GGUF form and whose files were checked.
|
||||
|
||||
Takes a [Scaling](#scaling) block.
|
||||
|
||||
### `stopped`
|
||||
|
||||
`*Scaling`
|
||||
|
||||
Stopped is the scaling a model had when an administrator stopped it:
|
||||
kept in the catalog with none running (Scaling 0/0) until started
|
||||
again with this. Nil for a model not stopped.
|
||||
|
||||
Takes a [Scaling](#scaling) block.
|
||||
|
||||
### `class`
|
||||
|
||||
`string`
|
||||
@@ -2178,25 +1912,6 @@ kept by llmgw bundle import and llmgw bundle trust; every engine file
|
||||
a node fetches from the mirror is checked against it. Read again
|
||||
whenever it changes. Empty is /etc/llmgw/offline.sha256.
|
||||
|
||||
## OnDemandSettings
|
||||
|
||||
### `evict_after_idle`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
EvictAfterIdle is how long an on-demand replica must go unasked-for
|
||||
before a model that wants its card may have it. Zero keeps 2m.
|
||||
|
||||
### `min_lifetime`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
MinLifetime is how long a replica that has just come up is safe from
|
||||
that, so two models do not take turns paying for loads. Zero keeps
|
||||
5m. On one card shared by a picture model and a 3D model, 1m each
|
||||
brings the wait between a picture and its mesh to about a minute.
|
||||
Both are in force on the leader within seconds.
|
||||
|
||||
## Options
|
||||
|
||||
### `nvidia_smi`
|
||||
@@ -2228,15 +1943,6 @@ is used, is the more exact source of the peers.
|
||||
SkipGPUs omits GPU discovery. A node without nvidia-smi is already
|
||||
found to have none, so this is for one whose driver is broken.
|
||||
|
||||
### `dcgm_exporter`
|
||||
|
||||
`string`
|
||||
|
||||
DCGMExporter is the dcgm-exporter metrics URL read for how much of
|
||||
each card's chip is in use, on cards NVML's GPM does not cover (before
|
||||
Hopper). Empty is http://127.0.0.1:9400/metrics; "off" reads none. A
|
||||
node without one costs a refused connection every five minutes.
|
||||
|
||||
## Origin
|
||||
|
||||
### `repo`
|
||||
@@ -2408,12 +2114,6 @@ Judge has a judge model on the cluster read the text too.
|
||||
|
||||
Takes a [JudgePolicy](#judgepolicy) block.
|
||||
|
||||
### `services`
|
||||
|
||||
`[]Service`
|
||||
|
||||
Services are outside guardrails asked over HTTP, beside the judge.
|
||||
|
||||
### `stream`
|
||||
|
||||
`string`
|
||||
@@ -2643,16 +2343,6 @@ rather than a guess (plan §6.2.1).
|
||||
|
||||
Priority orders profiles during placement; highest that fits wins.
|
||||
|
||||
### `replicas`
|
||||
|
||||
`int`
|
||||
|
||||
Replicas, set on any of a model's profiles, is how many of its
|
||||
replicas run on this one: seven on cards and four on CPUs, all
|
||||
answering under the model's name, rather than each replica on the
|
||||
best profile that fits. The model's scaling is then the sum, min and
|
||||
max alike; a profile left at zero runs none.
|
||||
|
||||
### `startup_timeout`
|
||||
|
||||
`time.Duration`
|
||||
@@ -2853,22 +2543,6 @@ as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
|
||||
are believed; anyone else's are not, since they decide the address
|
||||
written into the setup scripts and whose attempts are counted where.
|
||||
|
||||
### `active_active`
|
||||
|
||||
`bool`
|
||||
|
||||
ActiveActive has standby control planes answer the client API
|
||||
themselves, routing to the leader's replicas, rather than passing
|
||||
every request to the leader. More machines carry the traffic, which
|
||||
is the point. Rate limits are the cluster's: each router's counts
|
||||
reach the others through the leader every two seconds, so a key
|
||||
spread over N control planes may pass its limit by what they allow
|
||||
in those two seconds. A budget is checked against the store every
|
||||
half minute; cached answers and prefix affinity are split between
|
||||
the routers. Off by
|
||||
default. Which machine a client reaches is the load balancer's or
|
||||
DNS's to say; the floating address stays on the leader.
|
||||
|
||||
### `timeouts`
|
||||
|
||||
`Timeouts`
|
||||
@@ -2914,14 +2588,6 @@ Deny are patterns, matched against the text as a model reads it.
|
||||
|
||||
MaxChars refuses text longer than this; zero is no limit.
|
||||
|
||||
### `restore`
|
||||
|
||||
`bool`
|
||||
|
||||
Restore, on input, puts what redact rewrote as placeholders back into
|
||||
the answer: the model reads "[EMAIL_1]", the client reads its own
|
||||
address. Only the values that request sent, only into its answer.
|
||||
|
||||
## SCIM
|
||||
|
||||
### `token_file`
|
||||
@@ -3310,15 +2976,6 @@ Firewall turns llmgw's firewall on every machine on or off.
|
||||
|
||||
Takes a [FirewallSettings](#firewallsettings) block.
|
||||
|
||||
### `slurm`
|
||||
|
||||
`SlurmSettings`
|
||||
|
||||
Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
|
||||
which of them it may use now.
|
||||
|
||||
Takes a [SlurmSettings](#slurmsettings) block.
|
||||
|
||||
### `presets`
|
||||
|
||||
`PresetSettings`
|
||||
@@ -3327,15 +2984,6 @@ Presets is how the models that come with llmgw reach the catalog.
|
||||
|
||||
Takes a [PresetSettings](#presetsettings) block.
|
||||
|
||||
### `on_demand`
|
||||
|
||||
`OnDemandSettings`
|
||||
|
||||
OnDemand is how models that load when asked for take turns on the
|
||||
cards.
|
||||
|
||||
Takes a [OnDemandSettings](#ondemandsettings) block.
|
||||
|
||||
### `agents`
|
||||
|
||||
`AgentSettings`
|
||||
@@ -3438,67 +3086,6 @@ Endpoints is the phase-1 static replica list. It is only consulted when
|
||||
no model declares a profile, which is what makes a router-only smoke test
|
||||
against sim-engines possible without a control plane.
|
||||
|
||||
## Service
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name says which service decided, in the verdict's rule.
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
URL is POSTed the body.
|
||||
|
||||
### `header_files`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
HeaderFiles are headers whose values are read from files each time,
|
||||
so a rotated key needs no restart: {Authorization: /etc/…/lakera}.
|
||||
|
||||
### `body`
|
||||
|
||||
`string`
|
||||
|
||||
Body is the request, a template given .Text; {{json .Text}} writes it
|
||||
as a JSON string. Empty is {"input": <text>}.
|
||||
|
||||
### `blocked`
|
||||
|
||||
`string`
|
||||
|
||||
Blocked is where in the answer the service says the text is unsafe:
|
||||
a dotted path, numbers for list items (results.0.flagged). True, a
|
||||
number above zero, a non-empty list or string other than "false" is
|
||||
unsafe.
|
||||
|
||||
### `reason`
|
||||
|
||||
`string`
|
||||
|
||||
Reason is where it says why, for the verdict; optional.
|
||||
|
||||
### `on`
|
||||
|
||||
`string`
|
||||
|
||||
On: input, output or both. Empty is input.
|
||||
|
||||
### `action`
|
||||
|
||||
`string`
|
||||
|
||||
Action on unsafe: block or flag. Empty is block.
|
||||
|
||||
### `timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Timeout bounds one call. Empty is 3s; past it the policy's fail says.
|
||||
|
||||
## ServingSettings
|
||||
|
||||
### `stopped`
|
||||
@@ -3511,30 +3098,6 @@ running and every model's definition as it is: llmgw stop sets it,
|
||||
llmgw start clears it, and the models come back as their catalog
|
||||
entries ask. In force on every control plane within seconds.
|
||||
|
||||
## SharedStorage
|
||||
|
||||
### `path`
|
||||
|
||||
`string`
|
||||
|
||||
Path is where every machine sees it, as /shared.
|
||||
|
||||
### `nfs`
|
||||
|
||||
`string`
|
||||
|
||||
NFS is the site's export, host:/path, mounted at Path on every
|
||||
machine of the Slurm cluster.
|
||||
|
||||
### `server`
|
||||
|
||||
`string`
|
||||
|
||||
Server is the llmgw node, the controller or one of the nodes, that
|
||||
exports Path over NFS to the others: for a site with no storage of
|
||||
its own. One machine: not highly available, and no faster than its
|
||||
disk and network.
|
||||
|
||||
## SimConfig
|
||||
|
||||
### `bin`
|
||||
@@ -3544,109 +3107,6 @@ disk and network.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## SlurmSettings
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
|
||||
Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
|
||||
what was installed is left installed, and every node is llmgw's.
|
||||
|
||||
### `cluster_name`
|
||||
|
||||
`string`
|
||||
|
||||
ClusterName is Slurm's ClusterName. Empty is "llmgw".
|
||||
|
||||
### `controller`
|
||||
|
||||
`string`
|
||||
|
||||
Controller is the llmgw node whose agent runs slurmctld. It needs no
|
||||
GPU, and need not be one of Nodes.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
|
||||
in Slurm, until it is also in Lent.
|
||||
|
||||
### `all_nodes`
|
||||
|
||||
`bool`
|
||||
|
||||
AllNodes runs slurmd on every llmgw node, those that join later too,
|
||||
besides any named in Nodes. Each is llmgw's until it is lent, as any.
|
||||
|
||||
### `lent`
|
||||
|
||||
`[]string`
|
||||
|
||||
Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
|
||||
is drained of llmgw's replicas first, as `ctl drain` does, and resumed
|
||||
in Slurm once none is left; a node taken out is drained in Slurm and
|
||||
given back to llmgw when its last job ends.
|
||||
|
||||
### `partition`
|
||||
|
||||
`string`
|
||||
|
||||
Partition is the one partition every node is in. Empty is "gpu".
|
||||
|
||||
### `shared`
|
||||
|
||||
`SharedStorage`
|
||||
|
||||
Shared is the file system every machine of the Slurm cluster sees at
|
||||
the same path: home directories, datasets, the jobs' output and
|
||||
checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
node it ran on.
|
||||
|
||||
Takes a [SharedStorage](#sharedstorage) block.
|
||||
|
||||
### `users`
|
||||
|
||||
`string`
|
||||
|
||||
Users is who keeps the Linux users that run jobs, the same on every
|
||||
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
account (its login from the e-mail address, a UID of llmgw's from
|
||||
200000 up, its SSH keys from the portal), let in to the controller,
|
||||
and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
empty leaves users to the site (a directory through sssd, or by hand).
|
||||
|
||||
### `containers`
|
||||
|
||||
`bool`
|
||||
|
||||
Containers lets jobs run in containers: `srun --container-image
|
||||
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
the GPUs and the shared path inside. The agents install both on
|
||||
every Slurm machine; jobs take it once every machine has them.
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
`string`
|
||||
|
||||
Dir is Strata's source directory, where setup.py and serve/ are.
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
|
||||
### `engine`
|
||||
|
||||
`string`
|
||||
|
||||
Engine is the directory of its prebuilt engine, where the strata
|
||||
binary and the CUDA libraries it ships are; Dir's engine/ when unset.
|
||||
|
||||
## TRTLLMConfig
|
||||
|
||||
### `python`
|
||||
@@ -3864,15 +3324,6 @@ is sha256=<hex HMAC of the body>.
|
||||
|
||||
`int64`
|
||||
|
||||
### `cached_per_1k`
|
||||
|
||||
`int64`
|
||||
|
||||
CachedPer1k is what a prompt token costs that the engine or provider
|
||||
read from its cache instead of computing: the part of prompt_tokens
|
||||
the answer counts as cached_tokens. Zero charges those at
|
||||
PromptPer1k, as before there was a rate for them.
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
@@ -4147,14 +3598,6 @@ had rather wait behind people than be turned away.
|
||||
|
||||
`int64`
|
||||
|
||||
### `cached_per_1k`
|
||||
|
||||
`int64`
|
||||
|
||||
CachedPer1k is the cost of a prompt token the engine or provider read
|
||||
from its cache (the answer's cached_tokens, counted within the
|
||||
prompt's). Zero charges them at prompt_per_1k.
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
@@ -4219,25 +3662,6 @@ ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
|
||||
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
||||
|
||||
### `strata`
|
||||
|
||||
`StrataConfig`
|
||||
|
||||
Strata serves a few mixture-of-experts models from RAM and one card;
|
||||
offered by a node whose Strata is named here, or that installed it.
|
||||
|
||||
Takes a [StrataConfig](#strataconfig) block.
|
||||
|
||||
### `freetoken`
|
||||
|
||||
`FreeTokenConfig`
|
||||
|
||||
FreeToken serves mixture-of-experts models larger than a card from
|
||||
RAM and one card of 8.0 or newer; installed by the agent where the
|
||||
cards allow it, or named here.
|
||||
|
||||
Takes a [FreeTokenConfig](#freetokenconfig) block.
|
||||
|
||||
## guard.Config
|
||||
|
||||
### `enabled`
|
||||
|
||||
@@ -12,7 +12,6 @@ require (
|
||||
github.com/charmbracelet/x/ansi v0.11.6
|
||||
github.com/coreos/go-oidc/v3 v3.21.0
|
||||
github.com/go-jose/go-jose/v4 v4.1.4
|
||||
github.com/go-ldap/ldap/v3 v3.4.15
|
||||
github.com/go-zeromq/zmq4 v0.17.0
|
||||
github.com/jackc/pgx/v5 v5.11.0
|
||||
github.com/knadh/koanf/parsers/yaml v1.1.1
|
||||
@@ -36,7 +35,6 @@ require (
|
||||
golang.org/x/net v0.59.0
|
||||
golang.org/x/oauth2 v0.37.0
|
||||
golang.org/x/sync v0.23.0
|
||||
golang.org/x/sys v0.48.0
|
||||
golang.org/x/term v0.46.0
|
||||
golang.org/x/text v0.42.0
|
||||
google.golang.org/protobuf v1.36.12
|
||||
@@ -46,7 +44,6 @@ require (
|
||||
require (
|
||||
filippo.io/edwards25519 v1.2.0 // indirect
|
||||
filippo.io/hpke v0.4.0 // indirect
|
||||
github.com/Azure/go-ntlmssp v0.1.1 // indirect
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||
@@ -59,7 +56,6 @@ require (
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
||||
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
||||
github.com/go-asn1-ber/asn1-ber v1.5.8 // indirect
|
||||
github.com/go-logr/logr v1.4.4 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
||||
@@ -92,6 +88,7 @@ require (
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.46.0 // indirect
|
||||
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
|
||||
golang.org/x/sys v0.48.0 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||
google.golang.org/grpc v1.83.1 // indirect
|
||||
|
||||
@@ -8,12 +8,8 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
||||
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
|
||||
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
|
||||
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktpoUAgOJK3OTFc/xug0PCXYCqU0FgDKI=
|
||||
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
||||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||
@@ -50,12 +46,8 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
|
||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||
github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
|
||||
github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
||||
github.com/go-ldap/ldap/v3 v3.4.15 h1:eRB4f4Ecn1POL1mO86+uUDlecmyWsdiJwh60bAUS4UY=
|
||||
github.com/go-ldap/ldap/v3 v3.4.15/go.mod h1:jeD0c98EdZQCKKKKe4kTWs5aCf6y+j8KuYsYLznZSQk=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
|
||||
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
@@ -77,8 +69,6 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
|
||||
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
|
||||
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||
@@ -91,18 +81,6 @@ github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
|
||||
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
|
||||
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
|
||||
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
|
||||
github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
|
||||
github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
|
||||
github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
|
||||
github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
|
||||
github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
|
||||
github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
|
||||
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
|
||||
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
|
||||
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=
|
||||
|
||||
@@ -1,63 +0,0 @@
|
||||
package account
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeDirectory struct {
|
||||
asked []string
|
||||
u store.User
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeDirectory) SignIn(_ context.Context, login, _ string) (store.User, error) {
|
||||
f.asked = append(f.asked, login)
|
||||
return f.u, f.err
|
||||
}
|
||||
|
||||
// With a directory, a name the store has no password for is the
|
||||
// directory's to check; an account with a password here — the
|
||||
// administrator init made — is still checked here, so a directory that is
|
||||
// down locks nobody out.
|
||||
func TestPasswordsKeptElsewhereAreTheDirectorysToCheck(t *testing.T) {
|
||||
svc, st := newService(t)
|
||||
ctx := t.Context()
|
||||
hash, _ := HashPassword("local-secret")
|
||||
admin := store.User{ID: "usr-admin", Email: "admin@example.com", Name: "a", PasswordHash: hash,
|
||||
State: store.UserActive, CreatedAt: time.Now()}
|
||||
ada := store.User{ID: "usr-ada", Email: "ada@corp.example", Name: "Ada", PasswordHash: "!ldap",
|
||||
State: store.UserActive, CreatedAt: time.Now()}
|
||||
for _, u := range []store.User{admin, ada} {
|
||||
if err := st.CreateUser(ctx, u); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
dir := &fakeDirectory{u: ada}
|
||||
svc.SetDirectory(dir)
|
||||
|
||||
if _, _, err := svc.SignIn(ctx, "admin@example.com", "local-secret"); err != nil || len(dir.asked) != 0 {
|
||||
t.Fatalf("the local administrator: %v, directory asked %v", err, dir.asked)
|
||||
}
|
||||
// A name, not an address, and an account kept by the directory.
|
||||
for _, login := range []string{`CORP\ada`, "ada@corp.example"} {
|
||||
if tok, _, err := svc.SignIn(ctx, login, "ad-secret"); err != nil || tok == "" {
|
||||
t.Errorf("%s: %v", login, err)
|
||||
}
|
||||
}
|
||||
if len(dir.asked) != 2 || dir.asked[0] != `CORP\ada` {
|
||||
t.Errorf("the directory was asked %v", dir.asked)
|
||||
}
|
||||
dir.err = ErrBadCredentials
|
||||
if _, _, err := svc.SignIn(ctx, "ada", "wrong"); !errors.Is(err, ErrBadCredentials) {
|
||||
t.Errorf("a password the directory refused: %v", err)
|
||||
}
|
||||
// A reset link would give a directory's account a password here.
|
||||
if _, _, err := svc.RequestReset(ctx, st, "ada@corp.example"); !errors.Is(err, ErrNoReset) {
|
||||
t.Errorf("a directory's account was sent a reset link: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -80,26 +80,8 @@ type Service struct {
|
||||
|
||||
mu sync.Mutex
|
||||
attempts map[string]*attemptWindow
|
||||
|
||||
// dir checks the passwords of accounts this store keeps none for.
|
||||
dir Directory
|
||||
}
|
||||
|
||||
// Directory checks a password against an outside directory — Active
|
||||
// Directory over LDAP — and answers the account to sign in: made or kept in
|
||||
// step with the directory. Its errors are this package's ErrBadCredentials
|
||||
// and ErrSuspended for the person's own, and anything else for the
|
||||
// directory's.
|
||||
type Directory interface {
|
||||
SignIn(ctx context.Context, login, password string) (store.User, error)
|
||||
}
|
||||
|
||||
// SetDirectory has passwords for accounts with none here, and names that are
|
||||
// not addresses, checked by the directory. An account with a password here
|
||||
// — the administrator init made — is still checked here, so a directory that
|
||||
// is down locks nobody out.
|
||||
func (s *Service) SetDirectory(d Directory) { s.dir = d }
|
||||
|
||||
type attemptWindow struct {
|
||||
count int
|
||||
since time.Time
|
||||
@@ -155,7 +137,6 @@ func (s *Service) SignUp(ctx context.Context, email, name, password string) erro
|
||||
// SignIn checks a password and opens a session, returning the session token and
|
||||
// the CSRF token that must accompany the forms in it.
|
||||
func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
|
||||
login := strings.TrimSpace(email)
|
||||
email = store.NormalizeEmail(email)
|
||||
// Counted per address and the client trying it, so somebody guessing
|
||||
// from elsewhere cannot lock the owner out of their own account; and per
|
||||
@@ -170,15 +151,6 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
|
||||
}
|
||||
|
||||
u, err := s.store.UserByEmail(ctx, email)
|
||||
// No password kept here — the directory's, or nobody's — and a
|
||||
// directory to ask: it decides.
|
||||
if s.dir != nil && (errors.Is(err, store.ErrNotFound) || (err == nil && strings.HasPrefix(u.PasswordHash, "!"))) {
|
||||
if u, err = s.dir.SignIn(ctx, login, password); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
s.clearAttempts(byClient)
|
||||
return s.open(ctx, u)
|
||||
}
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
// Verify anyway, against a hash that cannot match. Returning early here
|
||||
// would let a stopwatch enumerate every account on the system.
|
||||
@@ -208,11 +180,7 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
|
||||
case store.UserSuspended:
|
||||
return "", "", ErrSuspended
|
||||
}
|
||||
return s.open(ctx, u)
|
||||
}
|
||||
|
||||
// open makes a session for a person whose password was right.
|
||||
func (s *Service) open(ctx context.Context, u store.User) (token, csrf string, err error) {
|
||||
if token, err = store.NewSessionToken(); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
@@ -434,10 +402,7 @@ func (s *Service) RequestReset(ctx context.Context, rs ResetStore, email string)
|
||||
return "", store.User{}, ErrTooManyTries
|
||||
}
|
||||
u, err := rs.UserByEmail(ctx, email)
|
||||
// An account whose password is kept elsewhere — "!oidc", "!scim",
|
||||
// "!ldap" — gets no link: setting one here would give it a second way
|
||||
// in that its directory knows nothing of, and outlives its leaving.
|
||||
if err != nil || u.PasswordHash == "" || strings.HasPrefix(u.PasswordHash, "!") || u.State != store.UserActive {
|
||||
if err != nil || u.PasswordHash == "" || u.State != store.UserActive {
|
||||
return "", store.User{}, ErrNoReset
|
||||
}
|
||||
token, err := rs.CreatePasswordReset(ctx, u.ID, ResetLinkTTL)
|
||||
|
||||
@@ -78,40 +78,9 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
|
||||
if start.IsZero() {
|
||||
return nil
|
||||
}
|
||||
spend, keys, err := storedSpend(ctx, s, start)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restoring budgets: %w", err)
|
||||
}
|
||||
a.budgets.Restore(period, spend)
|
||||
a.log.Info("restored budgets from the store",
|
||||
"keys", keys, "counters", len(spend), "since", start)
|
||||
return nil
|
||||
}
|
||||
|
||||
// CatchUpBudgets raises each counter to what the store has recorded, where
|
||||
// that is more. Run every so often by a router that is one of several taking
|
||||
// requests: the others' spend reaches it only through the store. Never
|
||||
// lowers a counter, which leads the store by whatever this process has not
|
||||
// written yet.
|
||||
func (a *Accountant) CatchUpBudgets(ctx context.Context, s store.Store, period store.Period) error {
|
||||
start := PeriodStart(period, time.Now())
|
||||
if start.IsZero() {
|
||||
return nil
|
||||
}
|
||||
spend, _, err := storedSpend(ctx, s, start)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading budgets: %w", err)
|
||||
}
|
||||
a.budgets.Raise(period, spend)
|
||||
return nil
|
||||
}
|
||||
|
||||
// storedSpend is the spend since start the store has recorded, by counter,
|
||||
// and how many keys it came from.
|
||||
func storedSpend(ctx context.Context, s store.Store, start time.Time) (map[string]int64, int, error) {
|
||||
rows, err := s.SpendSinceByScope(ctx, start)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
return fmt.Errorf("restoring budgets: %w", err)
|
||||
}
|
||||
// Every scope a key sits under, not just the key. A budget owned by a team
|
||||
// is one counter fed by every key beneath it, and restoring only per-key
|
||||
@@ -144,7 +113,10 @@ func storedSpend(ctx context.Context, s store.Store, start time.Time) (map[strin
|
||||
}
|
||||
}
|
||||
}
|
||||
return spend, len(rows), nil
|
||||
a.budgets.Restore(period, spend)
|
||||
a.log.Info("restored budgets from the store",
|
||||
"keys", len(rows), "counters", len(spend), "since", start)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Reservation is what Check returns: the estimate it charged, so Record can
|
||||
@@ -351,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CachedTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,37 +39,20 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0, 0, 0, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 0, 1000, 0, 0); got != 4000 {
|
||||
if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0, 0, 0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACachedPromptCostsTheCachedRate(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, CachedPer1k: 100, CompletionPer1k: 3000}
|
||||
// 10,000 prompt tokens of which 9,000 were read from the cache: 1,000 at
|
||||
// the prompt rate and 9,000 at the cached one.
|
||||
if got, want := p.Cost(10000, 9000, 0, 0, 0), int64(1000+900); got != want {
|
||||
t.Errorf("a 90%% cached prompt cost %d, want %d", got, want)
|
||||
}
|
||||
// More cached than prompt is an engine's miscount, not a refund.
|
||||
if got := p.Cost(100, 500, 0, 0, 0); got != 10 {
|
||||
t.Errorf("cached beyond the prompt cost %d, want 10", got)
|
||||
}
|
||||
// No cached rate: cached tokens cost what any prompt token does.
|
||||
if got := (Price{PromptPer1k: 1000}).Cost(10000, 9000, 0, 0, 0); got != 10000 {
|
||||
t.Errorf("without a cached rate a cached prompt cost %d, want 10000", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 0, 3, 0); got != 20+150000 {
|
||||
if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
@@ -216,22 +199,6 @@ func TestBudgetSurvivesARestart(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCatchingUpRaisesButNeverLowers(t *testing.T) {
|
||||
// Another router's spend arrives through the store; this one's own, not
|
||||
// written yet, must not be wiped by a store that is behind.
|
||||
b := NewBudgets()
|
||||
b.Add("mine", store.PeriodMonth, 900)
|
||||
b.Add("shared", store.PeriodMonth, 100)
|
||||
|
||||
b.Raise(store.PeriodMonth, map[string]int64{"mine": 400, "shared": 700, "theirs": 50})
|
||||
|
||||
for key, want := range map[string]int64{"mine": 900, "shared": 700, "theirs": 50} {
|
||||
if got := b.Spent(key, store.PeriodMonth); got != want {
|
||||
t.Errorf("%s: spent %d, want %d", key, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBudgetsResetAtThePeriodBoundary(t *testing.T) {
|
||||
b := NewBudgets()
|
||||
now := time.Date(2026, 1, 31, 23, 59, 0, 0, time.UTC)
|
||||
|
||||
@@ -61,19 +61,6 @@ func (b *Budgets) Restore(period store.Period, spend map[string]int64) {
|
||||
}
|
||||
}
|
||||
|
||||
// Raise lifts counters to the spend given, where it is more.
|
||||
func (b *Budgets) Raise(period store.Period, spend map[string]int64) {
|
||||
start := PeriodStart(period, b.now())
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
for key, micros := range spend {
|
||||
if c := b.currentLocked(key, start); micros > c.micros {
|
||||
c.micros = micros
|
||||
b.spend[key] = c
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Spent returns a key's spend in the current period, resetting it if the period
|
||||
// has rolled over.
|
||||
func (b *Budgets) Spent(keyID string, period store.Period) int64 {
|
||||
|
||||
@@ -18,11 +18,6 @@ import (
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// CachedPer1k is what a prompt token costs that the engine or provider
|
||||
// read from its cache instead of computing: the part of prompt_tokens
|
||||
// the answer counts as cached_tokens. Zero charges those at
|
||||
// PromptPer1k, as before there was a rate for them.
|
||||
CachedPer1k int64 `koanf:"cached_per_1k"`
|
||||
// PerImage is what one picture costs, for a model that makes them: a
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
@@ -38,24 +33,15 @@ type Price struct {
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool {
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.CachedPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
}
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
//
|
||||
// cachedTokens are counted within promptTokens, as OpenAI's usage counts
|
||||
// them, and are charged at the cached rate instead of the prompt rate.
|
||||
func (p Price) Cost(promptTokens, cachedTokens, completionTokens, images, meshes int64) int64 {
|
||||
cachedTokens = min(max(cachedTokens, 0), max(promptTokens, 0))
|
||||
cachedRate := p.CachedPer1k
|
||||
if cachedRate == 0 {
|
||||
cachedRate = p.PromptPer1k
|
||||
}
|
||||
return divUp((promptTokens-cachedTokens)*p.PromptPer1k+cachedTokens*cachedRate, 1000) +
|
||||
divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
||||
}
|
||||
|
||||
|
||||
@@ -25,11 +25,7 @@ type Limiter struct {
|
||||
type window struct {
|
||||
// current and previous are counts in this minute and the one before.
|
||||
current, previous int64
|
||||
// othersCurrent and othersPrevious are the same counts on the other
|
||||
// control planes taking requests, as last heard: what makes a limit the
|
||||
// cluster's rather than each process's.
|
||||
othersCurrent, othersPrevious int64
|
||||
start time.Time
|
||||
start time.Time
|
||||
}
|
||||
|
||||
// NewLimiter builds an empty limiter.
|
||||
@@ -108,24 +104,21 @@ func (w *window) roll(now time.Time) {
|
||||
return
|
||||
case start.Sub(w.start) == windowLen:
|
||||
w.previous, w.current = w.current, 0
|
||||
w.othersPrevious, w.othersCurrent = w.othersCurrent, 0
|
||||
default:
|
||||
// More than one window has passed with no traffic.
|
||||
w.previous, w.current = 0, 0
|
||||
w.othersPrevious, w.othersCurrent = 0, 0
|
||||
}
|
||||
w.start = start
|
||||
}
|
||||
|
||||
// rate weights the previous window by how much of it is still in view.
|
||||
func (w *window) rate(now time.Time) float64 {
|
||||
cur, prev := w.current+w.othersCurrent, w.previous+w.othersPrevious
|
||||
elapsed := now.Sub(w.start)
|
||||
if elapsed >= windowLen {
|
||||
return float64(cur)
|
||||
return float64(w.current)
|
||||
}
|
||||
remaining := 1 - float64(elapsed)/float64(windowLen)
|
||||
return float64(prev)*remaining + float64(cur)
|
||||
return float64(w.previous)*remaining + float64(w.current)
|
||||
}
|
||||
|
||||
// Forget drops a key's window. Used when a key is revoked; keeping it would
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
package accounting
|
||||
|
||||
import "time"
|
||||
|
||||
// Counts are one limiter's counts by counter: this minute's and the one
|
||||
// before, as absolute figures, so reporting them twice changes nothing.
|
||||
type Counts struct {
|
||||
Minute time.Time `json:"minute"`
|
||||
Current map[string]int64 `json:"current,omitempty"`
|
||||
Previous map[string]int64 `json:"previous,omitempty"`
|
||||
}
|
||||
|
||||
// Add sums other into c, aligned to c's minute: counts of a minute that is
|
||||
// c's previous one land there, older ones are dropped. Clocks a second apart
|
||||
// at a minute's turn are why.
|
||||
func (c *Counts) Add(other Counts) {
|
||||
switch {
|
||||
case other.Minute.Equal(c.Minute):
|
||||
c.Current = addInto(c.Current, other.Current)
|
||||
c.Previous = addInto(c.Previous, other.Previous)
|
||||
case other.Minute.Add(windowLen).Equal(c.Minute):
|
||||
c.Previous = addInto(c.Previous, other.Current)
|
||||
}
|
||||
}
|
||||
|
||||
func addInto(dst, src map[string]int64) map[string]int64 {
|
||||
if len(src) == 0 {
|
||||
return dst
|
||||
}
|
||||
if dst == nil {
|
||||
dst = make(map[string]int64, len(src))
|
||||
}
|
||||
for k, v := range src {
|
||||
dst[k] += v
|
||||
}
|
||||
return dst
|
||||
}
|
||||
|
||||
// Counts reads this limiter's own counts, without the others'.
|
||||
func (l *Limiter) Counts() Counts {
|
||||
now := l.now()
|
||||
out := Counts{Minute: now.Truncate(windowLen), Current: map[string]int64{}, Previous: map[string]int64{}}
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
for k, w := range l.keys {
|
||||
w.roll(now)
|
||||
if w.current > 0 {
|
||||
out.Current[k] = w.current
|
||||
}
|
||||
if w.previous > 0 {
|
||||
out.Previous[k] = w.previous
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SetOthers replaces what the other control planes have counted. A counter
|
||||
// they did not mention has nothing from them.
|
||||
func (l *Limiter) SetOthers(c Counts) {
|
||||
now := l.now()
|
||||
aligned := Counts{Minute: now.Truncate(windowLen)}
|
||||
aligned.Add(c)
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
for k, w := range l.keys {
|
||||
w.roll(now)
|
||||
w.othersCurrent, w.othersPrevious = aligned.Current[k], aligned.Previous[k]
|
||||
delete(aligned.Current, k)
|
||||
delete(aligned.Previous, k)
|
||||
}
|
||||
// Counters only others have used yet: a key spread over the cluster is
|
||||
// limited here from its first request.
|
||||
for k := range aligned.Current {
|
||||
w := l.windowLocked(k, now)
|
||||
w.othersCurrent, w.othersPrevious = aligned.Current[k], aligned.Previous[k]
|
||||
delete(aligned.Previous, k)
|
||||
}
|
||||
for k, v := range aligned.Previous {
|
||||
l.windowLocked(k, now).othersPrevious = v
|
||||
}
|
||||
}
|
||||
|
||||
// Rates are an accountant's two limiters' counts.
|
||||
type Rates struct {
|
||||
Requests Counts `json:"requests"`
|
||||
Tokens Counts `json:"tokens"`
|
||||
}
|
||||
|
||||
// Add sums other into r.
|
||||
func (r *Rates) Add(other Rates) {
|
||||
r.Requests.Add(other.Requests)
|
||||
r.Tokens.Add(other.Tokens)
|
||||
}
|
||||
|
||||
// NewRates is an empty sum for the minute now is in.
|
||||
func NewRates(now time.Time) Rates {
|
||||
m := now.Truncate(windowLen)
|
||||
return Rates{Requests: Counts{Minute: m}, Tokens: Counts{Minute: m}}
|
||||
}
|
||||
|
||||
// Rates reads this process's own request and token counts.
|
||||
func (a *Accountant) Rates() Rates {
|
||||
return Rates{Requests: a.requests.Counts(), Tokens: a.tokens.Counts()}
|
||||
}
|
||||
|
||||
// SetOtherRates puts the other control planes' counts in force here.
|
||||
func (a *Accountant) SetOtherRates(r Rates) {
|
||||
a.requests.SetOthers(r.Requests)
|
||||
a.tokens.SetOthers(r.Tokens)
|
||||
}
|
||||
@@ -1,71 +0,0 @@
|
||||
package accounting
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Two control planes each allowing a key six of its ten a minute: once each
|
||||
// has heard the other's count, neither allows another.
|
||||
func TestALimitIsTheClustersOnceTheCountsAreShared(t *testing.T) {
|
||||
now := time.Date(2026, 11, 1, 12, 0, 10, 0, time.UTC)
|
||||
a, b := NewLimiter(), NewLimiter()
|
||||
a.now, b.now = func() time.Time { return now }, func() time.Time { return now }
|
||||
for range 6 {
|
||||
if !a.Allow("k", 10, 1) || !b.Allow("k", 10, 1) {
|
||||
t.Fatal("refused within the limit")
|
||||
}
|
||||
}
|
||||
a.SetOthers(b.Counts())
|
||||
b.SetOthers(a.Counts())
|
||||
if a.Allow("k", 10, 1) || b.Allow("k", 10, 1) {
|
||||
t.Error("twelve of ten allowed once both knew")
|
||||
}
|
||||
// Reporting the same counts again changes nothing: they are absolute.
|
||||
a.SetOthers(b.Counts())
|
||||
if got := a.Rate("k"); got != 12 {
|
||||
t.Errorf("rate %v after the same counts twice", got)
|
||||
}
|
||||
// What the others counted is never charged back to them.
|
||||
if c := a.Counts(); c.Current["k"] != 6 {
|
||||
t.Errorf("own count %d", c.Current["k"])
|
||||
}
|
||||
}
|
||||
|
||||
// A key only the others have used is limited here from its first request,
|
||||
// and their counts move into the previous minute with the clock.
|
||||
func TestOthersCountsFollowTheMinute(t *testing.T) {
|
||||
now := time.Date(2026, 11, 1, 12, 0, 50, 0, time.UTC)
|
||||
a, b := NewLimiter(), NewLimiter()
|
||||
a.now, b.now = func() time.Time { return now }, func() time.Time { return now }
|
||||
for range 10 {
|
||||
b.Allow("k", 10, 1)
|
||||
}
|
||||
a.SetOthers(b.Counts())
|
||||
if a.Allow("k", 10, 1) {
|
||||
t.Error("a key the cluster has used up was allowed where it had not been seen")
|
||||
}
|
||||
now = now.Add(20 * time.Second) // 12:01:10, a sixth into the next minute
|
||||
if got := a.Rate("k"); got < 8 || got > 8.5 {
|
||||
t.Errorf("rate %v: the others' last minute should weigh five sixths", got)
|
||||
}
|
||||
// Counts of a minute ago, sent late, land in the previous minute.
|
||||
old := b.Counts()
|
||||
old.Minute = now.Truncate(time.Minute).Add(-time.Minute)
|
||||
old.Current, old.Previous = map[string]int64{"k": 3}, nil
|
||||
a.SetOthers(old)
|
||||
if got := a.Rate("k"); got < 2 || got > 3 {
|
||||
t.Errorf("rate %v from late counts", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRatesAddAlignsMinutes(t *testing.T) {
|
||||
m := time.Date(2026, 11, 1, 12, 1, 0, 0, time.UTC)
|
||||
sum := NewRates(m)
|
||||
sum.Add(Rates{Requests: Counts{Minute: m, Current: map[string]int64{"k": 2}, Previous: map[string]int64{"k": 1}}})
|
||||
sum.Add(Rates{Requests: Counts{Minute: m.Add(-time.Minute), Current: map[string]int64{"k": 5}}})
|
||||
sum.Add(Rates{Requests: Counts{Minute: m.Add(-2 * time.Minute), Current: map[string]int64{"k": 100}}})
|
||||
if sum.Requests.Current["k"] != 2 || sum.Requests.Previous["k"] != 6 {
|
||||
t.Errorf("sum %+v", sum.Requests)
|
||||
}
|
||||
}
|
||||
@@ -1,91 +0,0 @@
|
||||
package accounting
|
||||
|
||||
import (
|
||||
"math"
|
||||
"slices"
|
||||
)
|
||||
|
||||
// Prices from what the cluster costs: what each model held of it (its
|
||||
// replicas' GPU- and core-hours) at what an hour of a GPU or a core costs
|
||||
// the operator, spread over the tokens it served. Idle time is in it: a
|
||||
// model held all month and used an hour costs what it held, not the hour.
|
||||
//
|
||||
// An output token is priced OutputPerInput input tokens, as producing one
|
||||
// is that much dearer (one step a token, against a prompt read in one
|
||||
// pass); a cached prompt token a tenth of an input one.
|
||||
const (
|
||||
OutputPerInput = 4
|
||||
CachedPerInput = 0.1
|
||||
)
|
||||
|
||||
// Basis is what an hour costs, in the units budgets are in.
|
||||
type Basis struct {
|
||||
GPUHour float64
|
||||
CoreHour float64
|
||||
}
|
||||
|
||||
// Held is what a model held, and Served what it answered with.
|
||||
type Held struct {
|
||||
GPUSeconds, CoreSeconds int64
|
||||
}
|
||||
|
||||
// Served is a model's tokens.
|
||||
type Served struct {
|
||||
Prompt, Completion int64
|
||||
}
|
||||
|
||||
// Suggestion is one model's price from its cost.
|
||||
type Suggestion struct {
|
||||
Model string `json:"model"`
|
||||
GPUHours float64 `json:"gpu_hours"`
|
||||
CoreHours float64 `json:"core_hours"`
|
||||
// Cost is what it held cost, in budget units.
|
||||
Cost float64 `json:"cost"`
|
||||
Prompt int64 `json:"prompt_tokens"`
|
||||
Completion int64 `json:"completion_tokens"`
|
||||
// Price is the suggestion; zero with a Note saying why there is none.
|
||||
Price Price `json:"price"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// Suggest prices every model that held something or served something.
|
||||
func Suggest(held map[string]Held, served map[string]Served, b Basis) []Suggestion {
|
||||
var names []string
|
||||
for m := range held {
|
||||
names = append(names, m)
|
||||
}
|
||||
for m := range served {
|
||||
if _, ok := held[m]; !ok {
|
||||
names = append(names, m)
|
||||
}
|
||||
}
|
||||
slices.Sort(names)
|
||||
out := make([]Suggestion, 0, len(names))
|
||||
for _, m := range names {
|
||||
h, s := held[m], served[m]
|
||||
g := Suggestion{Model: m, GPUHours: float64(h.GPUSeconds) / 3600, CoreHours: float64(h.CoreSeconds) / 3600,
|
||||
Prompt: s.Prompt, Completion: s.Completion}
|
||||
g.Cost = g.GPUHours*b.GPUHour + g.CoreHours*b.CoreHour
|
||||
// Tokens counted as output ones: an input token is a fraction of one.
|
||||
units := float64(s.Completion) + float64(s.Prompt)/OutputPerInput
|
||||
switch {
|
||||
case g.Cost == 0 && units > 0:
|
||||
g.Note = "held nothing recorded here: an outside provider's, or served before the hours were kept"
|
||||
case g.Cost > 0 && units == 0:
|
||||
g.Note = "held the cluster and was billed no tokens (a guard's judge, or a model priced by the picture or 3D model): no one pays what it held"
|
||||
case g.Cost > 0:
|
||||
// Micro-units per thousand output tokens, rounded up so a
|
||||
// cheap model is never free.
|
||||
out1k := int64(math.Ceil(g.Cost / units * 1000 * 1e6))
|
||||
g.Price = Price{
|
||||
CompletionPer1k: out1k,
|
||||
PromptPer1k: ceilDiv(out1k, OutputPerInput),
|
||||
CachedPer1k: max(int64(math.Ceil(float64(ceilDiv(out1k, OutputPerInput))*CachedPerInput)), 1),
|
||||
}
|
||||
}
|
||||
out = append(out, g)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func ceilDiv(a, b int64) int64 { return (a + b - 1) / b }
|
||||
@@ -1,45 +0,0 @@
|
||||
package accounting
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSuggestSpreadsWhatAModelHeldOverItsTokens(t *testing.T) {
|
||||
// The worked example: two cards for 1000 hours at 1.35 an hour is 2700;
|
||||
// 3.0 billion output tokens and 4.0 billion input ones, counted as 1.0
|
||||
// billion output ones, are 4.0 billion: 0.675 a million output tokens.
|
||||
held := map[string]Held{
|
||||
"llama-70b": {GPUSeconds: 2 * 1000 * 3600},
|
||||
"idle": {GPUSeconds: 3600},
|
||||
"small-cpu": {CoreSeconds: 8 * 100 * 3600},
|
||||
}
|
||||
served := map[string]Served{
|
||||
"llama-70b": {Prompt: 4_000_000_000, Completion: 3_000_000_000},
|
||||
"small-cpu": {Prompt: 1_000_000, Completion: 1_000_000},
|
||||
"outside": {Prompt: 10, Completion: 10},
|
||||
}
|
||||
got := Suggest(held, served, Basis{GPUHour: 1.35, CoreHour: 0.01})
|
||||
by := map[string]Suggestion{}
|
||||
for _, s := range got {
|
||||
by[s.Model] = s
|
||||
}
|
||||
l := by["llama-70b"]
|
||||
if l.Cost != 2700 || l.Price.CompletionPer1k != 675 || l.Price.PromptPer1k != 169 || l.Price.CachedPer1k != 17 {
|
||||
t.Errorf("llama: %+v", l)
|
||||
}
|
||||
// What it costs back at that price is what it held, within the rounding.
|
||||
back := float64(l.Price.Cost(l.Prompt, 0, l.Completion, 0, 0)) / 1e6
|
||||
if back < l.Cost || back > l.Cost*1.01 {
|
||||
t.Errorf("charged %.2f for a cost of %.2f", back, l.Cost)
|
||||
}
|
||||
if s := by["small-cpu"]; s.Cost != 8 || s.Price.CompletionPer1k == 0 {
|
||||
t.Errorf("cpu: %+v", s)
|
||||
}
|
||||
if s := by["idle"]; !s.Price.Zero() || !strings.Contains(s.Note, "billed no tokens") {
|
||||
t.Errorf("idle: %+v", s)
|
||||
}
|
||||
if s := by["outside"]; !s.Price.Zero() || !strings.Contains(s.Note, "held nothing") {
|
||||
t.Errorf("outside: %+v", s)
|
||||
}
|
||||
}
|
||||
+20
-165
@@ -6,13 +6,11 @@ import (
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"math"
|
||||
"math/rand/v2"
|
||||
"net"
|
||||
"net/http"
|
||||
@@ -35,7 +33,6 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
@@ -48,9 +45,7 @@ import (
|
||||
|
||||
// Agent runs on a GPU node.
|
||||
type Agent struct {
|
||||
// upgrading is true while an upgrade downloads.
|
||||
upgrading atomic.Bool
|
||||
cfg Config
|
||||
cfg Config
|
||||
// answered says the current session heard from its control plane, which
|
||||
// starts the reconnect backoff again when it ends.
|
||||
answered atomic.Bool
|
||||
@@ -98,10 +93,7 @@ type Agent struct {
|
||||
// (nil: this machine's shell).
|
||||
fwMu sync.Mutex
|
||||
fwRunner firewall.Runner
|
||||
// fwLast is the last firewall message, applied again when the node
|
||||
// is given a part in Slurm. Under fwMu.
|
||||
fwLast *llmgwv1.Firewall
|
||||
inv discovery.Inventory
|
||||
inv discovery.Inventory
|
||||
// shardReplica remembers which replica a shard belongs to, so a status
|
||||
// report can name both. The control plane keys on the replica.
|
||||
shardReplica map[string]string
|
||||
@@ -152,10 +144,6 @@ type Agent struct {
|
||||
|
||||
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
|
||||
trainer *train.Runner
|
||||
// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
|
||||
// its status goes, under mu.
|
||||
slurm *slurmnode.Keeper
|
||||
slurmOut chan<- *llmgwv1.AgentMessage
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -195,7 +183,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
logRing: ring,
|
||||
provKick: make(chan struct{}, 1),
|
||||
load: discovery.NewLoadSampler(),
|
||||
sensors: discovery.NewSensorSampler().WithProfile(discovery.NewProfSampler(cfg.Discovery.DCGMExporter)),
|
||||
sensors: discovery.NewSensorSampler(),
|
||||
wear: discovery.NewHealthSampler(),
|
||||
serverList: loadControlPlanes(cfg.Servers, cfg.TLS.Enabled()),
|
||||
kv: kvevents.NewMirror(),
|
||||
@@ -214,7 +202,6 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
}
|
||||
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
|
||||
RunAs: runAs, Log: log})
|
||||
a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
|
||||
return a, nil
|
||||
}
|
||||
|
||||
@@ -241,7 +228,6 @@ func (a *Agent) Run(ctx context.Context) error {
|
||||
}
|
||||
go a.provisionLoop(ctx)
|
||||
go a.ensureDiags(ctx)
|
||||
go a.slurm.Run(ctx, a.sendSlurm)
|
||||
|
||||
attempt := 0
|
||||
next := 0
|
||||
@@ -405,11 +391,8 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
_ = a.sup.Stop(ctx, o.ShardID)
|
||||
return
|
||||
}
|
||||
// What it computes on, for one started before the agent kept that — or
|
||||
// kept as the CPU for a GPU placement by an agent that did not follow
|
||||
// an engine into a process group of its own, which the check above has
|
||||
// just found on its GPUs.
|
||||
if c := a.sup.ComputeOf(o.ShardID); c == nil || (len(o.Spec.GPUIndices) > 0 && c.Backend == "CPU") {
|
||||
// What it computes on, for one started before the agent kept that.
|
||||
if a.sup.ComputeOf(o.ShardID) == nil {
|
||||
if c, ok := a.gpuCompute(ctx, o.Spec.GPUIndices, sh.Status().PID); ok {
|
||||
a.sup.SetCompute(o.ShardID, c)
|
||||
}
|
||||
@@ -529,8 +512,6 @@ func (a *Agent) scan(ctx context.Context) error {
|
||||
inv, err := discovery.Discover(ctx, opts, a.log)
|
||||
host := discovery.DiscoverHost()
|
||||
host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
|
||||
bw := a.memBandwidth()
|
||||
host.MemBandwidthGBs, host.MemBandwidthCoreGBs = bw.All, bw.Core
|
||||
a.mu.Lock()
|
||||
a.inv = inv
|
||||
a.prov.Host = host
|
||||
@@ -540,18 +521,6 @@ func (a *Agent) scan(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// forgetSent is a new stream: a control plane that knows nothing of where
|
||||
// installs stand, nor which files are here beyond what registration says —
|
||||
// and registration says what the startup scan found. So the next heartbeat
|
||||
// tells it both again. Without it, files deleted after the agent started came
|
||||
// back on a control plane that restarted, and stayed until the next change.
|
||||
func (a *Agent) forgetSent() {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
a.provSent = ""
|
||||
a.lastWeights = ""
|
||||
}
|
||||
|
||||
// session runs one connection until it breaks.
|
||||
func (a *Agent) session(ctx context.Context, server string) error {
|
||||
client := llmgwv1connect.NewAgentServiceClient(a.http, server)
|
||||
@@ -588,19 +557,17 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest, FeatureSlurm}
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
h := a.prov.Host
|
||||
// A new stream is a control plane that knows nothing of where installs
|
||||
// stand, so the next heartbeat tells it again.
|
||||
a.provSent = ""
|
||||
a.mu.Unlock()
|
||||
a.forgetSent()
|
||||
reg.Host = &llmgwv1.Host{
|
||||
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
|
||||
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
|
||||
// What is measured by now, not what the startup scan found before
|
||||
// the measurement ended: a reconnect would otherwise say 0.
|
||||
MemBandwidthGbs: a.memBandwidth().All,
|
||||
MemBandwidthCoreGbs: a.memBandwidth().Core,
|
||||
}
|
||||
|
||||
if err := stream.Send(&llmgwv1.AgentMessage{
|
||||
@@ -638,8 +605,6 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
go a.watchExits(ctx, out, unready(reg.RunningShards))
|
||||
go a.forwardKV(ctx, out)
|
||||
go a.reportCompute(ctx, out)
|
||||
a.setSlurmOut(out)
|
||||
defer a.setSlurmOut(nil)
|
||||
|
||||
for {
|
||||
msg, err := stream.Receive()
|
||||
@@ -700,8 +665,6 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
||||
// without anybody restarting anything.
|
||||
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
|
||||
hb.CpuCount = int32(runtime.NumCPU())
|
||||
bw := a.memBandwidth()
|
||||
hb.MemBandwidthGbs, hb.MemBandwidthCoreGbs = bw.All, bw.Core
|
||||
|
||||
a.reprobeEngines()
|
||||
a.mu.Lock()
|
||||
@@ -823,11 +786,6 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
|
||||
MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
|
||||
SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
|
||||
}
|
||||
if p := g.Profile; p != nil {
|
||||
gs.HasProfile, gs.ProfileSource = true, p.Source
|
||||
gs.SmActivePercent, gs.SmOccupancyPercent = p.SMActive, p.SMOccupancy
|
||||
gs.TensorActivePercent, gs.DramActivePercent = p.TensorActive, p.DRAMActive
|
||||
}
|
||||
if w, ok := wear[g.Index]; ok {
|
||||
gs.HasWear = true
|
||||
gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
|
||||
@@ -906,27 +864,10 @@ func (a *Agent) removeWeights(paths []string) {
|
||||
a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
|
||||
continue
|
||||
}
|
||||
// A set — every file of one version, fetched into a directory of its
|
||||
// own — goes whole: os.Remove refused it as not empty, and nothing
|
||||
// was freed of what a model of several files left. Lstat, so a link
|
||||
// is taken away and never what it points at.
|
||||
info, err := os.Lstat(abs)
|
||||
switch {
|
||||
case os.IsNotExist(err):
|
||||
continue
|
||||
case err != nil:
|
||||
a.log.Warn("could not look at cached weights", "path", p, "err", err)
|
||||
continue
|
||||
case info.IsDir():
|
||||
err = os.RemoveAll(abs)
|
||||
default:
|
||||
err = os.Remove(abs)
|
||||
}
|
||||
if err != nil {
|
||||
if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
|
||||
a.log.Warn("could not remove cached weights", "path", p, "err", err)
|
||||
continue
|
||||
}
|
||||
_ = os.Remove(weights.ChecksumPath(abs))
|
||||
a.log.Info("removed cached weights no model asks for", "path", p)
|
||||
}
|
||||
}
|
||||
@@ -963,8 +904,6 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
go a.applySettings(m.AgentSettings)
|
||||
case *llmgwv1.ControlMessage_TrainStart:
|
||||
go a.trainJob(ctx, m.TrainStart, out)
|
||||
case *llmgwv1.ControlMessage_Slurm:
|
||||
a.slurmMessage(m.Slurm)
|
||||
case *llmgwv1.ControlMessage_TrainCancel:
|
||||
a.trainer.Cancel(m.TrainCancel.GetJobId())
|
||||
case *llmgwv1.ControlMessage_RestartAgent:
|
||||
@@ -1180,11 +1119,11 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
send(&llmgwv1.ShardStatus{
|
||||
State: agentsvc.StateToWire(supervisor.Starting),
|
||||
Fetch: &llmgwv1.FetchProgress{
|
||||
Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal, Phase: p.Phase,
|
||||
Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal,
|
||||
},
|
||||
})
|
||||
})
|
||||
if path, err := a.weights().EnsureChecked(ctx, spec.Profile.ModelPath, spec.Profile.AllowPickle, spec.Profile.ModelSHA256); err != nil {
|
||||
if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath); err != nil {
|
||||
stop()
|
||||
report(supervisor.Failed, err.Error(), nil)
|
||||
return
|
||||
@@ -1192,37 +1131,11 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
stop()
|
||||
spec.Profile.ModelPath = path
|
||||
}
|
||||
// A shared ComfyUI replica's members' weights, each fetched like the
|
||||
// model's, and handed to the gate where they landed.
|
||||
if raw := spec.Profile.Env[engines.SharedModelsEnv]; raw != "" {
|
||||
var ms []map[string]string
|
||||
if err := json.Unmarshal([]byte(raw), &ms); err != nil {
|
||||
report(supervisor.Failed, "the shared replica's models are not readable: "+err.Error(), nil)
|
||||
return
|
||||
}
|
||||
for i, m := range ms {
|
||||
stop := a.watchFetch(m["source"], func(p weights.Progress) {
|
||||
send(&llmgwv1.ShardStatus{State: agentsvc.StateToWire(supervisor.Starting),
|
||||
Fetch: &llmgwv1.FetchProgress{Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal,
|
||||
Phase: p.Phase}})
|
||||
})
|
||||
path, err := a.weights().Ensure(ctx, m["source"], m["allow_pickle"] == "true")
|
||||
stop()
|
||||
if err != nil {
|
||||
report(supervisor.Failed, fmt.Sprintf("model %s: %v", m["name"], err), nil)
|
||||
return
|
||||
}
|
||||
ms[i]["path"] = path
|
||||
}
|
||||
b, _ := json.Marshal(ms)
|
||||
spec.Profile.Env = maps.Clone(spec.Profile.Env)
|
||||
spec.Profile.Env[engines.SharedModelsEnv] = string(b)
|
||||
}
|
||||
// Adapters are fetched like the model, and before the engine starts: it
|
||||
// loads them at start, and one missing is a replica that answers for the
|
||||
// base model under the adapter's name.
|
||||
for i, ad := range spec.Profile.Adapters {
|
||||
path, err := a.weights().Ensure(ctx, ad.Path, spec.Profile.AllowPickle)
|
||||
path, err := a.weights().Ensure(ctx, ad.Path)
|
||||
if err != nil {
|
||||
report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil)
|
||||
return
|
||||
@@ -1478,14 +1391,11 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
|
||||
// off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
|
||||
// in the socket's buffer until the kernel gives up retransmitting, fifteen
|
||||
// minutes later. Found by switching off a leader: the standby took over in 14
|
||||
// seconds and the node stayed with the dead one. An HTTP/2 ping after five
|
||||
// quiet seconds, given three to answer, finds it in eight: by the time the
|
||||
// standby leads, as it does within the lease's ten seconds, the node is on
|
||||
// its way to it. At fifteen, with the lease at fifteen too, the new leader
|
||||
// went without its nodes, and so its judges, for half a minute.
|
||||
// seconds and the node stayed with the dead one. An HTTP/2 ping after ten
|
||||
// quiet seconds, given five to answer, finds it in a quarter of a minute.
|
||||
const (
|
||||
streamIdle = 5 * time.Second
|
||||
streamPing = 3 * time.Second
|
||||
streamIdle = 10 * time.Second
|
||||
streamPing = 5 * time.Second
|
||||
)
|
||||
|
||||
// localizeSpec fills in what only this node knows.
|
||||
@@ -1566,25 +1476,18 @@ func ResolveManagement(c Config) (addr, iface string) {
|
||||
// shard computes on as soon as that is known: an adopted engine's is found
|
||||
// after registration, which carried what was known then.
|
||||
func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMessage) {
|
||||
// What was sent for each shard, so a compute that changes is sent again:
|
||||
// an adopted shard's kept one is sent at once, and adopting it may then
|
||||
// find it was kept wrong.
|
||||
sent := map[string]string{}
|
||||
sent := map[string]bool{}
|
||||
t := time.NewTicker(5 * time.Second)
|
||||
defer t.Stop()
|
||||
for range 60 {
|
||||
for _, st := range a.sup.Statuses() {
|
||||
if st.State.Terminal() {
|
||||
if st.State.Terminal() || sent[st.ID] {
|
||||
continue
|
||||
}
|
||||
c := a.sup.ComputeOf(st.ID)
|
||||
if c == nil {
|
||||
continue
|
||||
}
|
||||
said := fmt.Sprint(*c)
|
||||
if sent[st.ID] == said {
|
||||
continue
|
||||
}
|
||||
sh, err := a.sup.Get(st.ID)
|
||||
if err != nil {
|
||||
continue
|
||||
@@ -1595,7 +1498,7 @@ func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMess
|
||||
}}}
|
||||
select {
|
||||
case out <- msg:
|
||||
sent[st.ID] = said
|
||||
sent[st.ID] = true
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
@@ -1607,51 +1510,3 @@ func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMess
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// memBandwidth is the node's memory read rate in GB/s, 0 until measured.
|
||||
// Measured in the background, the first time it is asked for: it reads half
|
||||
// a gigabyte, which must not hold up an agent reaching its control plane,
|
||||
// and the heartbeat carries it when it is known. Not at once but after a
|
||||
// minute or so, chosen at random, and again every hour, keeping the best:
|
||||
// VMs on one host that all restart together measure each other, and read a
|
||||
// third less than any of them gets alone. Once a process rather than an
|
||||
// agent, so tests running many agents in one read it once.
|
||||
func (a *Agent) memBandwidth() discovery.Bandwidth {
|
||||
bwOnce.Do(func() {
|
||||
go func() {
|
||||
time.Sleep(bandwidthDelay())
|
||||
for {
|
||||
// The best of each seen so far: an hourly pass on a node
|
||||
// busy serving reads less, and that is not the memory.
|
||||
bw := measureBandwidth()
|
||||
cur := a.memBandwidthNow()
|
||||
if bw.All > cur.All || bw.Core > cur.Core {
|
||||
bwBits.Store(math.Float64bits(max(bw.All, cur.All)))
|
||||
coreBits.Store(math.Float64bits(max(bw.Core, cur.Core)))
|
||||
a.log.Info("this node's memory", "read_gbs", math.Round(bw.All*10)/10,
|
||||
"one_core_gbs", math.Round(bw.Core*10)/10)
|
||||
}
|
||||
time.Sleep(time.Hour)
|
||||
}
|
||||
}()
|
||||
})
|
||||
return a.memBandwidthNow()
|
||||
}
|
||||
|
||||
// memBandwidthNow is what has been measured so far, without starting a
|
||||
// measurement.
|
||||
func (a *Agent) memBandwidthNow() discovery.Bandwidth {
|
||||
return discovery.Bandwidth{All: math.Float64frombits(bwBits.Load()), Core: math.Float64frombits(coreBits.Load())}
|
||||
}
|
||||
|
||||
// bandwidthDelay is how long an agent waits before measuring, so agents
|
||||
// started together measure apart.
|
||||
var bandwidthDelay = func() time.Duration { return 10*time.Second + rand.N(110*time.Second) }
|
||||
|
||||
var (
|
||||
bwOnce sync.Once
|
||||
bwBits atomic.Uint64
|
||||
coreBits atomic.Uint64
|
||||
// measureBandwidth is replaced in tests, which have no use for it.
|
||||
measureBandwidth = discovery.MeasureMemoryBandwidth
|
||||
)
|
||||
@@ -68,7 +68,7 @@ func (r *statusRecorder) OnShardCompute(replicaID, shardID string, c reconciler.
|
||||
r.computes = append(r.computes, c)
|
||||
}
|
||||
|
||||
func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source, phase string, done, total int64) {
|
||||
func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.fetches = append(r.fetches, fetchReport{replicaID, shardID, source, done, total})
|
||||
@@ -671,7 +671,7 @@ func TestAShardWithMissingWeightsFailsBeforeLaunching(t *testing.T) {
|
||||
// nobody is watching — so it is caught here, where the message can say
|
||||
// what is actually wrong.
|
||||
a := &Agent{cfg: Config{WeightsCache: t.TempDir()}, log: slog.New(slog.DiscardHandler)}
|
||||
_, err := a.weights().Ensure(t.Context(), filepath.Join(t.TempDir(), "not-there.gguf"), false)
|
||||
_, err := a.weights().Ensure(t.Context(), filepath.Join(t.TempDir(), "not-there.gguf"))
|
||||
if err == nil {
|
||||
t.Fatal("a missing weights file was accepted")
|
||||
}
|
||||
@@ -925,23 +925,3 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
|
||||
t.Errorf("reply %+v, ran %+v", r, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
|
||||
// Registration carries what the startup scan found; a file deleted since
|
||||
// came back on a control plane that restarted, because the next
|
||||
// heartbeat saw the list unchanged since it was last sent.
|
||||
dir := t.TempDir()
|
||||
h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
|
||||
if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.ag.heartbeat()
|
||||
if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
|
||||
t.Fatal("an unchanged list should not be sent again on the same stream")
|
||||
}
|
||||
h.ag.forgetSent()
|
||||
hb := h.ag.heartbeat().GetHeartbeat()
|
||||
if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
|
||||
t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
|
||||
}
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// CUDAToolkit is the root of an installed CUDA toolkit of this major version
|
||||
// — the directory whose bin holds nvcc — or "" when there is none. Engines
|
||||
// that compile their kernels where they run (FreeToken, ComfyUI's compiled
|
||||
// nodes) need one matching the CUDA their torch was built for, and a
|
||||
// toolkit of another major version compiles code that torch cannot load.
|
||||
//
|
||||
// CUDA_HOME is looked at first, then /usr/local/cuda, then every
|
||||
// /usr/local/cuda-<version>, newest first.
|
||||
func CUDAToolkit(major int) string {
|
||||
return cudaToolkitIn(major, os.Getenv("CUDA_HOME"), "/usr/local")
|
||||
}
|
||||
|
||||
func cudaToolkitIn(major int, home, root string) string {
|
||||
var dirs []string
|
||||
if home != "" {
|
||||
dirs = append(dirs, home)
|
||||
}
|
||||
dirs = append(dirs, filepath.Join(root, "cuda"))
|
||||
versioned, _ := filepath.Glob(filepath.Join(root, "cuda-*"))
|
||||
slices.SortFunc(versioned, func(a, b string) int { return versionCompare(b, a) })
|
||||
dirs = append(dirs, versioned...)
|
||||
for _, d := range dirs {
|
||||
if _, err := os.Stat(filepath.Join(d, "bin", "nvcc")); err != nil {
|
||||
continue
|
||||
}
|
||||
if toolkitMajor(d) == major {
|
||||
return d
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// toolkitMajor is a toolkit's major version, from the version.json every
|
||||
// toolkit since 11.1 carries, or from nvcc where it has none; 0 when neither
|
||||
// says.
|
||||
func toolkitMajor(dir string) int {
|
||||
if b, err := os.ReadFile(filepath.Join(dir, "version.json")); err == nil { //nolint:gosec // a toolkit's own file
|
||||
var v struct {
|
||||
CUDA struct {
|
||||
Version string `json:"version"`
|
||||
} `json:"cuda"`
|
||||
}
|
||||
if json.Unmarshal(b, &v) == nil && v.CUDA.Version != "" {
|
||||
n, _ := strconv.Atoi(strings.SplitN(v.CUDA.Version, ".", 2)[0])
|
||||
return n
|
||||
}
|
||||
}
|
||||
//nolint:gosec // nvcc of a toolkit found on disk
|
||||
out, err := exec.Command(filepath.Join(dir, "bin", "nvcc"), "--version").Output()
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
if m := nvccRelease.FindSubmatch(out); m != nil {
|
||||
n, _ := strconv.Atoi(string(m[1]))
|
||||
return n
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
var nvccRelease = regexp.MustCompile(`release (\d+)\.`)
|
||||
|
||||
// versionCompare orders two paths ending in dotted versions: cuda-13.10 is
|
||||
// newer than cuda-13.4, which comparing them as text gets wrong.
|
||||
func versionCompare(a, b string) int {
|
||||
pa := strings.Split(strings.TrimPrefix(filepath.Base(a), "cuda-"), ".")
|
||||
pb := strings.Split(strings.TrimPrefix(filepath.Base(b), "cuda-"), ".")
|
||||
for i := range max(len(pa), len(pb)) {
|
||||
var x, y int
|
||||
if i < len(pa) {
|
||||
x, _ = strconv.Atoi(pa[i])
|
||||
}
|
||||
if i < len(pb) {
|
||||
y, _ = strconv.Atoi(pb[i])
|
||||
}
|
||||
if x != y {
|
||||
return x - y
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func toolkit(t *testing.T, root, name, version string) string {
|
||||
t.Helper()
|
||||
d := filepath.Join(root, name)
|
||||
put(t, d, "bin/nvcc", "#!/bin/sh")
|
||||
if version != "" {
|
||||
put(t, d, "version.json", `{"cuda":{"name":"CUDA SDK","version":"`+version+`"}}`)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func TestTheNewestToolkitOfTheMajorVersionAskedForIsFound(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
toolkit(t, root, "cuda-12.8", "12.8.1")
|
||||
toolkit(t, root, "cuda-13.4", "13.4.0")
|
||||
want := toolkit(t, root, "cuda-13.10", "13.10.0")
|
||||
// /usr/local/cuda is a 12: a toolkit of the wrong major version is not
|
||||
// one torch built for 13 can load what it compiles.
|
||||
toolkit(t, root, "cuda", "12.8.1")
|
||||
if got := cudaToolkitIn(13, "", root); got != want {
|
||||
t.Errorf("got %q, want %q", got, want)
|
||||
}
|
||||
if got := cudaToolkitIn(12, "", root); got != filepath.Join(root, "cuda") {
|
||||
t.Errorf("12: got %q", got)
|
||||
}
|
||||
// CUDA_HOME first, when it is one.
|
||||
home := toolkit(t, t.TempDir(), "mine", "13.0.2")
|
||||
if got := cudaToolkitIn(13, home, root); got != home {
|
||||
t.Errorf("CUDA_HOME: got %q", got)
|
||||
}
|
||||
if got := cudaToolkitIn(14, "", root); got != "" {
|
||||
t.Errorf("14: got %q", got)
|
||||
}
|
||||
// A directory without nvcc is not a toolkit.
|
||||
if err := os.MkdirAll(filepath.Join(root, "cuda-13.99"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := cudaToolkitIn(13, "", root); got != want {
|
||||
t.Errorf("with an empty cuda-13.99: got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -28,11 +28,6 @@ type Options struct {
|
||||
// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
|
||||
// found to have none, so this is for one whose driver is broken.
|
||||
SkipGPUs bool `koanf:"skip_gpus"`
|
||||
// DCGMExporter is the dcgm-exporter metrics URL read for how much of
|
||||
// each card's chip is in use, on cards NVML's GPM does not cover (before
|
||||
// Hopper). Empty is http://127.0.0.1:9400/metrics; "off" reads none. A
|
||||
// node without one costs a refused connection every five minutes.
|
||||
DCGMExporter string `koanf:"dcgm_exporter"`
|
||||
// ModelDirs are scanned for weights. Empty means do not look.
|
||||
ModelDirs []string `koanf:"-"`
|
||||
}
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
//go:build !nvml
|
||||
|
||||
package discovery
|
||||
|
||||
// newGPM has nothing to read GPM with in a build without the nvml tag.
|
||||
func newGPM() gpmSource { return nil }
|
||||
@@ -1,124 +0,0 @@
|
||||
//go:build nvml
|
||||
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"sync"
|
||||
|
||||
"github.com/NVIDIA/go-nvml/pkg/nvml"
|
||||
)
|
||||
|
||||
// nvmlGPM reads GPM, NVML's own profiling counters, on the cards that have
|
||||
// them: Hopper and later. Each read takes a sample and works out the metrics
|
||||
// against the one before, so NVML stays initialised between reads.
|
||||
type nvmlGPM struct {
|
||||
mu sync.Mutex
|
||||
opened bool
|
||||
failed bool
|
||||
cards map[int]*gpmCard
|
||||
}
|
||||
|
||||
type gpmCard struct {
|
||||
dev nvml.Device
|
||||
prev, cur nvml.GpmSample
|
||||
havePrevious bool
|
||||
}
|
||||
|
||||
func newGPM() gpmSource { return &nvmlGPM{} }
|
||||
|
||||
// The GPM metrics read, in the order they are asked for.
|
||||
var gpmMetrics = []struct {
|
||||
id nvml.GpmMetricId
|
||||
set func(*GPUProfile, float64)
|
||||
}{
|
||||
{nvml.GPM_METRIC_SM_UTIL, func(p *GPUProfile, v float64) { p.SMActive = v }},
|
||||
{nvml.GPM_METRIC_SM_OCCUPANCY, func(p *GPUProfile, v float64) { p.SMOccupancy = v }},
|
||||
{nvml.GPM_METRIC_ANY_TENSOR_UTIL, func(p *GPUProfile, v float64) { p.TensorActive = v }},
|
||||
{nvml.GPM_METRIC_DRAM_BW_UTIL, func(p *GPUProfile, v float64) { p.DRAMActive = v }},
|
||||
}
|
||||
|
||||
func (g *nvmlGPM) read() map[int]GPUProfile {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
if g.failed {
|
||||
return nil
|
||||
}
|
||||
if !g.opened && !g.open() {
|
||||
g.failed = true
|
||||
return nil
|
||||
}
|
||||
out := map[int]GPUProfile{}
|
||||
for idx, c := range g.cards {
|
||||
if c.cur.Get(c.dev) != nvml.SUCCESS {
|
||||
// A card in MIG mode, or one that has gone: no sample, and the
|
||||
// next must not be measured against an old one.
|
||||
c.havePrevious = false
|
||||
continue
|
||||
}
|
||||
if c.havePrevious {
|
||||
if p, ok := gpmBetween(c.prev, c.cur); ok {
|
||||
out[idx] = p
|
||||
}
|
||||
}
|
||||
c.prev, c.cur = c.cur, c.prev
|
||||
c.havePrevious = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// open finds the cards with GPM. False is NVML not starting, or no card
|
||||
// having it, and the reader then never tries again.
|
||||
func (g *nvmlGPM) open() bool {
|
||||
if nvml.Init() != nvml.SUCCESS {
|
||||
return false
|
||||
}
|
||||
g.cards = map[int]*gpmCard{}
|
||||
n, ret := nvml.DeviceGetCount()
|
||||
if ret != nvml.SUCCESS {
|
||||
n = 0
|
||||
}
|
||||
for i := range n {
|
||||
dev, ret := nvml.DeviceGetHandleByIndex(i)
|
||||
if ret != nvml.SUCCESS {
|
||||
continue
|
||||
}
|
||||
sup, ret := dev.GpmQueryDeviceSupport()
|
||||
if ret != nvml.SUCCESS || sup.IsSupportedDevice == 0 {
|
||||
continue
|
||||
}
|
||||
a, ret := nvml.GpmSampleAlloc()
|
||||
if ret != nvml.SUCCESS {
|
||||
continue
|
||||
}
|
||||
b, ret := nvml.GpmSampleAlloc()
|
||||
if ret != nvml.SUCCESS {
|
||||
_ = a.Free()
|
||||
continue
|
||||
}
|
||||
g.cards[i] = &gpmCard{dev: dev, prev: a, cur: b}
|
||||
}
|
||||
if len(g.cards) == 0 {
|
||||
_ = nvml.Shutdown()
|
||||
return false
|
||||
}
|
||||
g.opened = true
|
||||
return true
|
||||
}
|
||||
|
||||
// gpmBetween is the profile over the time between two samples.
|
||||
func gpmBetween(earlier, later nvml.GpmSample) (GPUProfile, bool) {
|
||||
q := nvml.GpmMetricsGetType{NumMetrics: uint32(len(gpmMetrics)), Sample1: earlier, Sample2: later}
|
||||
for i, m := range gpmMetrics {
|
||||
q.Metrics[i].MetricId = uint32(m.id)
|
||||
}
|
||||
if nvml.GpmMetricsGet(&q) != nvml.SUCCESS {
|
||||
return GPUProfile{}, false
|
||||
}
|
||||
p := GPUProfile{SMActive: -1, SMOccupancy: -1, TensorActive: -1, DRAMActive: -1, Source: "gpm"}
|
||||
for i, m := range gpmMetrics {
|
||||
if r := q.Metrics[i]; r.NvmlReturn == uint32(nvml.SUCCESS) && r.Value >= 0 {
|
||||
m.set(&p, r.Value)
|
||||
}
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
@@ -52,8 +52,6 @@ type GPU struct {
|
||||
// memory the instance's share. Utilisation stays unreported, because no
|
||||
// driver measures it per instance.
|
||||
MIG *MIGDevice `json:"mig,omitempty"`
|
||||
// PCIe is how the card is linked to its host, zero when not read.
|
||||
PCIe PCIeLink `json:"pcie,omitempty"`
|
||||
}
|
||||
|
||||
// GPUHealth records conditions that make a GPU unsafe to schedule onto.
|
||||
|
||||
@@ -67,23 +67,10 @@ func DiscoverGPUsNVML(_ context.Context) ([]GPU, error) {
|
||||
if major, minor, ret := dev.GetCudaComputeCapability(); ret == nvml.SUCCESS {
|
||||
g.ComputeCapability = fmt.Sprintf("%d.%d", major, minor)
|
||||
}
|
||||
if v, ret := dev.GetMaxPcieLinkGeneration(); ret == nvml.SUCCESS {
|
||||
g.PCIe.Gen = v
|
||||
}
|
||||
if v, ret := dev.GetGpuMaxPcieLinkGeneration(); ret == nvml.SUCCESS {
|
||||
g.PCIe.GPUGen = v
|
||||
}
|
||||
if v, ret := dev.GetCurrPcieLinkWidth(); ret == nvml.SUCCESS {
|
||||
g.PCIe.Width = v
|
||||
}
|
||||
if v, ret := dev.GetMaxPcieLinkWidth(); ret == nvml.SUCCESS {
|
||||
g.PCIe.WidthMax = v
|
||||
}
|
||||
if pci, ret := dev.GetPciInfo(); ret == nvml.SUCCESS {
|
||||
g.PCIBusID = busIDString(pci.BusId[:])
|
||||
byBusID[normalizeBusID(g.PCIBusID)] = i
|
||||
}
|
||||
g.PCIe = withSysfs(g.PCIe, "/sys/bus/pci/devices", g.PCIBusID)
|
||||
if t, ret := dev.GetTemperature(nvml.TEMPERATURE_GPU); ret == nvml.SUCCESS {
|
||||
g.Health.TemperatureC = int(t)
|
||||
}
|
||||
|
||||
@@ -64,7 +64,6 @@ func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
|
||||
if err != nil || len(cards) == 0 {
|
||||
return cards, err
|
||||
}
|
||||
readPCIe(ctx, binary, cards)
|
||||
// The query lists whole cards, MIG mode or not. A card in MIG mode is
|
||||
// replaced by its instances, which only -L names and only the full
|
||||
// report sizes; asked only when some card could be in MIG mode, since
|
||||
|
||||
@@ -31,14 +31,6 @@ type Host struct {
|
||||
// MinComputeCapability is the oldest GPU's, 0 when there are none or it
|
||||
// is not known. Not read here but from the GPU inventory, by the agent.
|
||||
MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
|
||||
// MemBandwidthGBs is how fast the memory reads, in GB/s, 0 when not
|
||||
// measured: what decides a model's speed on the CPU. Not read here but
|
||||
// measured by the agent when it starts (MeasureMemoryBandwidth).
|
||||
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
|
||||
// MemBandwidthCoreGBs is what one core reads alone, in GB/s, 0 when not
|
||||
// measured: a replica given n threads reads at about n times this, up to
|
||||
// MemBandwidthGBs.
|
||||
MemBandwidthCoreGBs float64 `json:"mem_bandwidth_core_gbs,omitempty"`
|
||||
}
|
||||
|
||||
// Family is the packaging family: "deb", "rpm", or "" when neither is known.
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// How fast this machine reads its memory, which on a CPU is what decides how
|
||||
// fast a model answers: every token of a reply reads all the weights it uses
|
||||
// once, so a node that reads 40 GB/s writes a 4 GB model's tokens about ten
|
||||
// times a second, whatever its core count. Measured once when the agent
|
||||
// starts, in a fraction of a second, by every CPU summing a share of a buffer
|
||||
// too large for any cache — a VM sharing its host's memory with others gets
|
||||
// its share, which is the number that matters for it.
|
||||
|
||||
// bandwidthBuffer is what is read at most; the least is eight times the
|
||||
// processor's last cache, and never under minBuffer. Smaller, it would be
|
||||
// the cache that was measured — some hold a gigabyte — and the figure kept
|
||||
// would be too high.
|
||||
const (
|
||||
bandwidthBuffer = 2 << 30
|
||||
minBuffer = 256 << 20
|
||||
)
|
||||
|
||||
// Bandwidth is how fast this machine's memory reads, in GB/s: with every
|
||||
// CPU reading, and with one alone. A replica given a few threads of a large
|
||||
// machine reads at a few times the second, not at the first: on a server of
|
||||
// many memory channels one core cannot keep more than a fraction of them
|
||||
// busy.
|
||||
type Bandwidth struct {
|
||||
All, Core float64
|
||||
}
|
||||
|
||||
// coreBuffer is what the one-core passes read: past any cache, and short
|
||||
// enough that one core reads it in a fraction of a second.
|
||||
const coreBuffer = 512 << 20
|
||||
|
||||
// MeasureMemoryBandwidth is the memory's read rate, the best of a few passes
|
||||
// each way; zero when the machine has too little free memory to measure.
|
||||
func MeasureMemoryBandwidth() Bandwidth {
|
||||
size := int64(bandwidthBuffer)
|
||||
// Never more than a quarter of what is free: a node already holding
|
||||
// models must not be pushed into swap by a measurement. Without room
|
||||
// for a buffer past the cache, none is measured: the hourly retry
|
||||
// measures it when there is.
|
||||
if avail := memAvailable("/proc/meminfo"); avail > 0 {
|
||||
size = min(size, avail/4)
|
||||
}
|
||||
if size < max(minBuffer, 8*lastCacheBytes("/sys/devices/system/cpu")) {
|
||||
return Bandwidth{}
|
||||
}
|
||||
buf := make([]uint64, size/8)
|
||||
for i := range buf {
|
||||
buf[i] = uint64(i)
|
||||
}
|
||||
workers := runtime.NumCPU()
|
||||
chunk := (len(buf) + workers - 1) / workers
|
||||
best := 0.0
|
||||
for pass := 0; pass < 4; pass++ {
|
||||
var wg sync.WaitGroup
|
||||
start := time.Now()
|
||||
for w := 0; w < workers; w++ {
|
||||
lo, hi := w*chunk, min((w+1)*chunk, len(buf))
|
||||
if lo >= hi {
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(part []uint64) {
|
||||
defer wg.Done()
|
||||
sink(sum(part))
|
||||
}(buf[lo:hi])
|
||||
}
|
||||
wg.Wait()
|
||||
if s := time.Since(start).Seconds(); s > 0 {
|
||||
best = max(best, float64(size)/s/1e9)
|
||||
}
|
||||
}
|
||||
// And one core alone, over a part of the same buffer.
|
||||
part := buf[:min(len(buf), coreBuffer/8)]
|
||||
core := 0.0
|
||||
for pass := 0; pass < 3; pass++ {
|
||||
start := time.Now()
|
||||
sink(sum(part))
|
||||
if s := time.Since(start).Seconds(); s > 0 {
|
||||
core = max(core, float64(len(part)*8)/s/1e9)
|
||||
}
|
||||
}
|
||||
return Bandwidth{All: best, Core: min(core, best)}
|
||||
}
|
||||
|
||||
// sum reads every word, eight at a time so the loop is not what limits it.
|
||||
func sum(p []uint64) uint64 {
|
||||
var a, b, c, d, e, f, g, h uint64
|
||||
i := 0
|
||||
for ; i+8 <= len(p); i += 8 {
|
||||
a += p[i]
|
||||
b += p[i+1]
|
||||
c += p[i+2]
|
||||
d += p[i+3]
|
||||
e += p[i+4]
|
||||
f += p[i+5]
|
||||
g += p[i+6]
|
||||
h += p[i+7]
|
||||
}
|
||||
for ; i < len(p); i++ {
|
||||
a += p[i]
|
||||
}
|
||||
return a + b + c + d + e + f + g + h
|
||||
}
|
||||
|
||||
// sinkValue keeps the sums alive, so the reads are not optimised away.
|
||||
var sinkValue struct {
|
||||
sync.Mutex
|
||||
v uint64
|
||||
}
|
||||
|
||||
func sink(v uint64) {
|
||||
sinkValue.Lock()
|
||||
sinkValue.v += v
|
||||
sinkValue.Unlock()
|
||||
}
|
||||
|
||||
// memAvailable is MemAvailable in bytes, 0 when unknown.
|
||||
func memAvailable(path string) int64 {
|
||||
f, err := os.Open(path) //nolint:gosec // a fixed system path
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
fields := strings.Fields(sc.Text())
|
||||
if len(fields) >= 2 && fields[0] == "MemAvailable:" {
|
||||
kb, _ := strconv.ParseInt(fields[1], 10, 64)
|
||||
return kb << 10
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// lastCacheBytes is the largest cache any CPU reports, 0 when unknown.
|
||||
func lastCacheBytes(root string) int64 {
|
||||
paths, _ := filepath.Glob(filepath.Join(root, "cpu*", "cache", "index*", "size"))
|
||||
var most int64
|
||||
for _, p := range paths {
|
||||
b, err := os.ReadFile(p) //nolint:gosec // under /sys
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
v := strings.TrimSpace(string(b))
|
||||
mult := int64(1)
|
||||
switch {
|
||||
case strings.HasSuffix(v, "K"):
|
||||
mult, v = 1<<10, strings.TrimSuffix(v, "K")
|
||||
case strings.HasSuffix(v, "M"):
|
||||
mult, v = 1<<20, strings.TrimSuffix(v, "M")
|
||||
}
|
||||
if n, err := strconv.ParseInt(v, 10, 64); err == nil {
|
||||
most = max(most, n*mult)
|
||||
}
|
||||
}
|
||||
return most
|
||||
}
|
||||
@@ -1,32 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMemoryBandwidthIsMeasured(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("reads half a gigabyte")
|
||||
}
|
||||
bw := MeasureMemoryBandwidth()
|
||||
// Any machine running this reads more than a gigabyte a second and less
|
||||
// than a few terabytes, and one core no more than all of them.
|
||||
if bw.All < 1 || bw.All > 5000 || bw.Core < 0.5 || bw.Core > bw.All {
|
||||
t.Errorf("measured %.1f GB/s, %.1f on one core", bw.All, bw.Core)
|
||||
}
|
||||
t.Logf("%.1f GB/s, %.1f on one core", bw.All, bw.Core)
|
||||
}
|
||||
|
||||
func TestTheLastCacheIsReadFromSys(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
for path, size := range map[string]string{"cpu0/cache/index0/size": "48K", "cpu0/cache/index3/size": "96M"} {
|
||||
p := filepath.Join(root, path)
|
||||
_ = os.MkdirAll(filepath.Dir(p), 0o755)
|
||||
_ = os.WriteFile(p, []byte(size+"\n"), 0o644)
|
||||
}
|
||||
if got := lastCacheBytes(root); got != 96<<20 {
|
||||
t.Errorf("%d", got)
|
||||
}
|
||||
}
|
||||
@@ -1,181 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/csv"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// A card in the wrong slot: an x16 card in a slot wired x8 or x4, behind a
|
||||
// riser, or in a slot of an older PCIe generation than the card's, links
|
||||
// slower than it was built to, and nothing says so. A model on the card
|
||||
// still decodes at the card's own memory speed, but it loads slower, a
|
||||
// tensor-parallel replica over PCIe is slower, and so is anything that
|
||||
// spills to the host. The driver says how the link stands and what the
|
||||
// card could do; the difference is reported where the node's warnings are.
|
||||
|
||||
// PCIeLink is how a card is linked to its host.
|
||||
type PCIeLink struct {
|
||||
// Gen is the fastest generation the card and the slot together allow —
|
||||
// not the current one, which drops while the card idles, to save power.
|
||||
// GPUGen is the card's own fastest, 0 when the driver does not say.
|
||||
Gen int `json:"gen,omitempty"`
|
||||
GPUGen int `json:"gpu_gen,omitempty"`
|
||||
// Width is the lanes in use now, WidthMax the card's most.
|
||||
Width int `json:"width,omitempty"`
|
||||
WidthMax int `json:"width_max,omitempty"`
|
||||
}
|
||||
|
||||
// laneGBs is one lane's usable rate in GB/s, by generation.
|
||||
var laneGBs = map[int]float64{1: 0.25, 2: 0.5, 3: 0.985, 4: 1.969, 5: 3.938, 6: 7.877}
|
||||
|
||||
// GBs is what a link of gen and width carries, in GB/s each way.
|
||||
func pcieGBs(gen, width int) float64 { return laneGBs[gen] * float64(width) }
|
||||
|
||||
// Problem says how a card's link falls short of what the card could do,
|
||||
// empty when it does not or the driver did not say.
|
||||
func (p PCIeLink) Problem() string {
|
||||
gen := p.Gen
|
||||
if p.GPUGen > 0 && gen == 0 {
|
||||
gen = p.GPUGen
|
||||
}
|
||||
slowGen := p.GPUGen > 0 && p.Gen > 0 && p.Gen < p.GPUGen
|
||||
narrow := p.WidthMax > 0 && p.Width > 0 && p.Width < p.WidthMax
|
||||
if !slowGen && !narrow {
|
||||
return ""
|
||||
}
|
||||
could := pcieGBs(max(p.GPUGen, gen), p.WidthMax)
|
||||
has := pcieGBs(gen, p.Width)
|
||||
var why []string
|
||||
if narrow {
|
||||
why = append(why, fmt.Sprintf("x%d of its x%d lanes", p.Width, p.WidthMax))
|
||||
}
|
||||
if slowGen {
|
||||
why = append(why, fmt.Sprintf("PCIe Gen%d of its Gen%d", p.Gen, p.GPUGen))
|
||||
}
|
||||
return fmt.Sprintf("linked at %s: %.1f GB/s where it could have %.1f. "+
|
||||
"Check the slot (wired fewer lanes, or an older generation), a riser, or the BIOS's PCIe setting",
|
||||
strings.Join(why, " and "), has, could)
|
||||
}
|
||||
|
||||
// pcieFields are asked apart from the rest, and the card's own fastest
|
||||
// generation apart again: a driver that does not know a field refuses the
|
||||
// whole query, and these must not cost a node its cards.
|
||||
var pcieFields = [][]string{
|
||||
{"index", "pcie.link.gen.max", "pcie.link.width.current", "pcie.link.width.max", "pcie.link.gen.gpumax"},
|
||||
{"index", "pcie.link.gen.max", "pcie.link.width.current", "pcie.link.width.max"},
|
||||
}
|
||||
|
||||
// readPCIe fills each card's link from nvidia-smi, leaving it empty when
|
||||
// the driver says nothing of it.
|
||||
func readPCIe(ctx context.Context, binary string, cards []GPU) {
|
||||
for _, fields := range pcieFields {
|
||||
out, err := exec.CommandContext(ctx, binary, "--query-gpu="+strings.Join(fields, ","),
|
||||
"--format=csv,noheader,nounits").Output()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
links := parsePCIe(string(out))
|
||||
for i := range cards {
|
||||
if l, ok := links[cards[i].Index]; ok && cards[i].MIG == nil {
|
||||
cards[i].PCIe = l
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
for i := range cards {
|
||||
if cards[i].MIG == nil {
|
||||
cards[i].PCIe = withSysfs(cards[i].PCIe, "/sys/bus/pci/devices", cards[i].PCIBusID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// withSysfs takes the card's own lane count, and its own fastest
|
||||
// generation, from the kernel: the driver's width.max is what card and slot
|
||||
// allow together, so a x16 card in a slot wired x8 reads x8 of x8 there,
|
||||
// while the device's max_link_width is the card's, 16.
|
||||
func withSysfs(l PCIeLink, root, busID string) PCIeLink {
|
||||
dev := sysfsBusID(busID)
|
||||
if dev == "" {
|
||||
return l
|
||||
}
|
||||
read := func(name string) string {
|
||||
b, err := os.ReadFile(filepath.Join(root, dev, name)) //nolint:gosec // under /sys
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
if n, err := strconv.Atoi(read("max_link_width")); err == nil && n > 0 {
|
||||
l.WidthMax = n
|
||||
}
|
||||
if n, err := strconv.Atoi(read("current_link_width")); err == nil && n > 0 {
|
||||
l.Width = n
|
||||
}
|
||||
if g := genOfSpeed(read("max_link_speed")); g > 0 && l.GPUGen == 0 {
|
||||
l.GPUGen = g
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
// sysfsBusID is nvidia-smi's 00000000:3B:00.0 as the kernel names it,
|
||||
// 0000:3b:00.0.
|
||||
func sysfsBusID(id string) string {
|
||||
id = strings.ToLower(strings.TrimSpace(id))
|
||||
dom, rest, ok := strings.Cut(id, ":")
|
||||
if !ok || len(dom) < 4 {
|
||||
return ""
|
||||
}
|
||||
return dom[len(dom)-4:] + ":" + rest
|
||||
}
|
||||
|
||||
// genOfSpeed is the PCIe generation of a link speed as sysfs writes it.
|
||||
func genOfSpeed(s string) int {
|
||||
switch {
|
||||
case strings.HasPrefix(s, "2.5 GT/s"):
|
||||
return 1
|
||||
case strings.HasPrefix(s, "5.0 GT/s"), strings.HasPrefix(s, "5 GT/s"):
|
||||
return 2
|
||||
case strings.HasPrefix(s, "8.0 GT/s"), strings.HasPrefix(s, "8 GT/s"):
|
||||
return 3
|
||||
case strings.HasPrefix(s, "16.0 GT/s"), strings.HasPrefix(s, "16 GT/s"):
|
||||
return 4
|
||||
case strings.HasPrefix(s, "32.0 GT/s"), strings.HasPrefix(s, "32 GT/s"):
|
||||
return 5
|
||||
case strings.HasPrefix(s, "64.0 GT/s"), strings.HasPrefix(s, "64 GT/s"):
|
||||
return 6
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func parsePCIe(out string) map[int]PCIeLink {
|
||||
r := csv.NewReader(strings.NewReader(out))
|
||||
r.TrimLeadingSpace = true
|
||||
r.FieldsPerRecord = -1
|
||||
rows, _ := r.ReadAll()
|
||||
num := func(s string) int {
|
||||
n, _ := strconv.Atoi(strings.TrimSpace(s))
|
||||
return n
|
||||
}
|
||||
links := map[int]PCIeLink{}
|
||||
for _, row := range rows {
|
||||
if len(row) < 4 {
|
||||
continue
|
||||
}
|
||||
idx, err := strconv.Atoi(strings.TrimSpace(row[0]))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
l := PCIeLink{Gen: num(row[1]), Width: num(row[2]), WidthMax: num(row[3])}
|
||||
if len(row) > 4 {
|
||||
l.GPUGen = num(row[4])
|
||||
}
|
||||
links[idx] = l
|
||||
}
|
||||
return links
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestACardInTheWrongSlotIsSaidSo(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
link PCIeLink
|
||||
want string // empty: nothing to say
|
||||
}{
|
||||
// The PoC's Quadro RTX 6000, idle: current Gen1, but its fastest is 3,
|
||||
// as the card's is, on all 16 lanes. Nothing wrong.
|
||||
{PCIeLink{Gen: 3, GPUGen: 3, Width: 16, WidthMax: 16}, ""},
|
||||
// An x16 card in a slot wired x8.
|
||||
{PCIeLink{Gen: 4, GPUGen: 4, Width: 8, WidthMax: 16}, "x8 of its x16 lanes: 15.8 GB/s where it could have 31.5"},
|
||||
// A Gen5 card in a Gen4 slot.
|
||||
{PCIeLink{Gen: 4, GPUGen: 5, Width: 16, WidthMax: 16}, "PCIe Gen4 of its Gen5: 31.5 GB/s where it could have 63.0"},
|
||||
// Both, behind a riser.
|
||||
{PCIeLink{Gen: 3, GPUGen: 4, Width: 4, WidthMax: 16}, "x4 of its x16 lanes and PCIe Gen3 of its Gen4"},
|
||||
// A driver that says nothing.
|
||||
{PCIeLink{}, ""},
|
||||
} {
|
||||
got := c.link.Problem()
|
||||
if (c.want == "") != (got == "") || !strings.Contains(got, c.want) {
|
||||
t.Errorf("%+v: %q, want %q", c.link, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLinkIsReadFromNvidiaSMI(t *testing.T) {
|
||||
links := parsePCIe("0, 3, 16, 16, 3\n1, 4, 8, 16, [N/A]\n")
|
||||
if links[0] != (PCIeLink{Gen: 3, GPUGen: 3, Width: 16, WidthMax: 16}) || links[1] != (PCIeLink{Gen: 4, Width: 8, WidthMax: 16}) {
|
||||
t.Errorf("%+v", links)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCardsOwnLanesComeFromTheKernel(t *testing.T) {
|
||||
// The driver says x8 of x8 for a x16 card in a slot wired x8; the
|
||||
// device's own max_link_width says 16.
|
||||
root := t.TempDir()
|
||||
dev := filepath.Join(root, "0000:3b:00.0")
|
||||
_ = os.MkdirAll(dev, 0o755)
|
||||
for f, v := range map[string]string{"max_link_width": "16", "current_link_width": "8", "max_link_speed": "16.0 GT/s PCIe"} {
|
||||
_ = os.WriteFile(filepath.Join(dev, f), []byte(v+"\n"), 0o644)
|
||||
}
|
||||
l := withSysfs(PCIeLink{Gen: 4, Width: 8, WidthMax: 8}, root, "00000000:3B:00.0")
|
||||
if l.WidthMax != 16 || l.Width != 8 || l.GPUGen != 4 || !strings.Contains(l.Problem(), "x8 of its x16") {
|
||||
t.Errorf("%+v: %q", l, l.Problem())
|
||||
}
|
||||
}
|
||||
@@ -1,176 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// GPUProfile is how much of a card's chip the work on it uses. nvidia-smi's
|
||||
// utilization.gpu cannot say: it is the share of time any kernel ran, and a
|
||||
// kernel keeping one SM of a hundred busy reads 100%. Measured on a Quadro
|
||||
// RTX 6000 under decode, it read 92% while the SMs had work 69% of the time
|
||||
// and issued an instruction on 18% of their cycles.
|
||||
//
|
||||
// These come from the card's profiling counters, which only two things read
|
||||
// without stopping the work: NVML's GPM, on Hopper and later, and DCGM, which
|
||||
// a site may run as dcgm-exporter. Percentages; -1 is not reported.
|
||||
type GPUProfile struct {
|
||||
// SMActive is the share of time the SMs had work, averaged over all of
|
||||
// them; SMOccupancy how full those SMs were with warps.
|
||||
SMActive, SMOccupancy float64
|
||||
// TensorActive is the tensor cores. Decode at batch one hardly uses them
|
||||
// even when nothing is wrong: a quantised matrix-vector product runs on
|
||||
// the integer units. Prefill and batches are where they show.
|
||||
TensorActive float64
|
||||
// DRAMActive is the memory interface, which decode is bound by.
|
||||
DRAMActive float64
|
||||
// Source is "gpm" or "dcgm".
|
||||
Source string
|
||||
}
|
||||
|
||||
// DefaultDCGMExporter is where dcgm-exporter listens unless told otherwise.
|
||||
const DefaultDCGMExporter = "http://127.0.0.1:9400/metrics"
|
||||
|
||||
// dcgmRetry is how long a dcgm-exporter that did not answer is left alone.
|
||||
// Most nodes run none, and asking every ten seconds would be asking nobody.
|
||||
const dcgmRetry = 5 * time.Minute
|
||||
|
||||
// gpmSource reads GPM, where this build and the card have it.
|
||||
type gpmSource interface {
|
||||
read() map[int]GPUProfile
|
||||
}
|
||||
|
||||
// ProfSampler reads each card's GPUProfile: from GPM where the card has it,
|
||||
// otherwise from a dcgm-exporter on the node.
|
||||
type ProfSampler struct {
|
||||
gpm gpmSource
|
||||
dcgm string
|
||||
get func(ctx context.Context, url string) ([]byte, error)
|
||||
now func() time.Time
|
||||
|
||||
dcgmAfter time.Time
|
||||
}
|
||||
|
||||
// NewProfSampler reads this machine. dcgmURL is the dcgm-exporter to ask:
|
||||
// empty is DefaultDCGMExporter, "off" asks none.
|
||||
func NewProfSampler(dcgmURL string) *ProfSampler {
|
||||
switch dcgmURL {
|
||||
case "":
|
||||
dcgmURL = DefaultDCGMExporter
|
||||
case "off":
|
||||
dcgmURL = ""
|
||||
}
|
||||
return &ProfSampler{gpm: newGPM(), dcgm: dcgmURL, get: httpGet, now: time.Now}
|
||||
}
|
||||
|
||||
// Sample is each card's profile by index; a card missing has none. GPM needs
|
||||
// two samples, so its cards appear from the second call.
|
||||
func (p *ProfSampler) Sample(ctx context.Context) map[int]GPUProfile {
|
||||
out := map[int]GPUProfile{}
|
||||
if p.gpm != nil {
|
||||
for i, v := range p.gpm.read() {
|
||||
out[i] = v
|
||||
}
|
||||
}
|
||||
if p.dcgm == "" || p.now().Before(p.dcgmAfter) {
|
||||
return out
|
||||
}
|
||||
body, err := p.get(ctx, p.dcgm)
|
||||
if err != nil {
|
||||
p.dcgmAfter = p.now().Add(dcgmRetry)
|
||||
return out
|
||||
}
|
||||
for i, v := range parseDCGM(body) {
|
||||
if _, ok := out[i]; !ok {
|
||||
out[i] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func httpGet(ctx context.Context, url string) ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, io.ErrUnexpectedEOF
|
||||
}
|
||||
return io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
||||
}
|
||||
|
||||
// The dcgm-exporter fields read, as ratios of one. Its default field list
|
||||
// has them on every card DCGM can profile.
|
||||
var dcgmFields = map[string]func(*GPUProfile, float64){
|
||||
"DCGM_FI_PROF_SM_ACTIVE": func(p *GPUProfile, v float64) { p.SMActive = v },
|
||||
"DCGM_FI_PROF_SM_OCCUPANCY": func(p *GPUProfile, v float64) { p.SMOccupancy = v },
|
||||
"DCGM_FI_PROF_PIPE_TENSOR_ACTIVE": func(p *GPUProfile, v float64) { p.TensorActive = v },
|
||||
"DCGM_FI_PROF_DRAM_ACTIVE": func(p *GPUProfile, v float64) { p.DRAMActive = v },
|
||||
}
|
||||
|
||||
// parseDCGM reads dcgm-exporter's text exposition: a line per field and
|
||||
// card, the card in the gpu label. A card with none of the fields — DCGM
|
||||
// cannot profile it — is left out. A MIG instance's lines carry a
|
||||
// GPU_I_ID and are left out too: these are whole cards.
|
||||
func parseDCGM(body []byte) map[int]GPUProfile {
|
||||
out := map[int]GPUProfile{}
|
||||
for _, line := range strings.Split(string(body), "\n") {
|
||||
if line == "" || line[0] == '#' {
|
||||
continue
|
||||
}
|
||||
open := strings.IndexByte(line, '{')
|
||||
end := strings.LastIndexByte(line, '}')
|
||||
if open < 0 || end < open {
|
||||
continue
|
||||
}
|
||||
set, ok := dcgmFields[line[:open]]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
labels := line[open+1 : end]
|
||||
if strings.Contains(labels, `GPU_I_ID="`) && !strings.Contains(labels, `GPU_I_ID=""`) {
|
||||
continue
|
||||
}
|
||||
idx, err := strconv.Atoi(labelValue(labels, "gpu"))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
val := strings.Fields(line[end+1:])
|
||||
if len(val) == 0 {
|
||||
continue
|
||||
}
|
||||
v, err := strconv.ParseFloat(val[0], 64)
|
||||
if err != nil || v < 0 {
|
||||
continue
|
||||
}
|
||||
p, seen := out[idx]
|
||||
if !seen {
|
||||
p = GPUProfile{SMActive: -1, SMOccupancy: -1, TensorActive: -1, DRAMActive: -1, Source: "dcgm"}
|
||||
}
|
||||
set(&p, 100*v)
|
||||
out[idx] = p
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// labelValue is one label's value from a Prometheus label set.
|
||||
func labelValue(labels, name string) string {
|
||||
for _, kv := range strings.Split(labels, ",") {
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(kv), "=")
|
||||
if ok && k == name {
|
||||
return strings.Trim(v, `"`)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -1,107 +0,0 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// dcgmBody is dcgm-exporter's output for two cards and a MIG instance:
|
||||
// card 0 profiled, card 1 a model DCGM cannot profile, which has the device
|
||||
// fields and none of the PROF ones.
|
||||
const dcgmBody = `# HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).
|
||||
# TYPE DCGM_FI_DEV_GPU_UTIL gauge
|
||||
DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 100
|
||||
DCGM_FI_DEV_GPU_UTIL{gpu="1",UUID="GPU-b",device="nvidia1",modelName="Quadro RTX 6000",Hostname="n1"} 92
|
||||
# HELP DCGM_FI_PROF_SM_ACTIVE The ratio of cycles an SM has at least 1 warp assigned.
|
||||
# TYPE DCGM_FI_PROF_SM_ACTIVE gauge
|
||||
DCGM_FI_PROF_SM_ACTIVE{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 0.007042
|
||||
DCGM_FI_PROF_SM_ACTIVE{gpu="2",UUID="GPU-c",device="nvidia2",modelName="NVIDIA A100",GPU_I_PROFILE="3g.40gb",GPU_I_ID="1",Hostname="n1"} 0.5
|
||||
DCGM_FI_PROF_SM_OCCUPANCY{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 0.25
|
||||
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 0
|
||||
`
|
||||
|
||||
func TestDCGMExporterSaysHowMuchOfEachChipIsInUse(t *testing.T) {
|
||||
got := parseDCGM([]byte(dcgmBody))
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("want card 0 alone (card 1 unprofiled, card 2 a MIG instance): %+v", got)
|
||||
}
|
||||
p := got[0]
|
||||
// The LinkedIn L40S: 100% busy, one SM of 142 at work.
|
||||
if p.SMActive < 0.70 || p.SMActive > 0.71 || p.SMOccupancy != 25 || p.TensorActive != 0 || p.Source != "dcgm" {
|
||||
t.Errorf("card 0 %+v", p)
|
||||
}
|
||||
// A field the exporter does not send is unknown, not zero.
|
||||
if p.DRAMActive != -1 {
|
||||
t.Errorf("DRAM active not sent, read as %v", p.DRAMActive)
|
||||
}
|
||||
}
|
||||
|
||||
type fakeGPM map[int]GPUProfile
|
||||
|
||||
func (f fakeGPM) read() map[int]GPUProfile { return f }
|
||||
|
||||
func TestGPMIsPreferredAndDCGMFillsTheCardsItDoesNotCover(t *testing.T) {
|
||||
calls := 0
|
||||
p := &ProfSampler{
|
||||
gpm: fakeGPM{1: {SMActive: 40, Source: "gpm"}},
|
||||
dcgm: DefaultDCGMExporter,
|
||||
get: func(context.Context, string) ([]byte, error) {
|
||||
calls++
|
||||
return []byte(`DCGM_FI_PROF_SM_ACTIVE{gpu="0"} 0.5` + "\n" + `DCGM_FI_PROF_SM_ACTIVE{gpu="1"} 0.9` + "\n"), nil
|
||||
},
|
||||
now: time.Now,
|
||||
}
|
||||
got := p.Sample(t.Context())
|
||||
if got[0].Source != "dcgm" || got[0].SMActive != 50 {
|
||||
t.Errorf("card 0 %+v", got[0])
|
||||
}
|
||||
if got[1].Source != "gpm" || got[1].SMActive != 40 {
|
||||
t.Errorf("GPM's card was taken from DCGM: %+v", got[1])
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Errorf("%d calls", calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeWithoutDCGMIsNotAskedEverySample(t *testing.T) {
|
||||
now := time.Now()
|
||||
calls := 0
|
||||
p := &ProfSampler{dcgm: DefaultDCGMExporter, now: func() time.Time { return now },
|
||||
get: func(context.Context, string) ([]byte, error) { calls++; return nil, errors.New("connection refused") }}
|
||||
for range 5 {
|
||||
if got := p.Sample(t.Context()); len(got) != 0 {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
now = now.Add(10 * time.Second)
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Errorf("asked %d times in 50 s", calls)
|
||||
}
|
||||
now = now.Add(dcgmRetry)
|
||||
p.Sample(t.Context())
|
||||
if calls != 2 {
|
||||
t.Errorf("not asked again after %v", dcgmRetry)
|
||||
}
|
||||
if off := NewProfSampler("off"); off.dcgm != "" {
|
||||
t.Errorf("off still asks %q", off.dcgm)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachSampleCarriesTheProfileOfTheCardsThatHaveOne(t *testing.T) {
|
||||
smi := func(context.Context, string) ([]byte, error) {
|
||||
return []byte("0, 71, 239, 260, 0x0, 92, 30, 23000, 24576, 1878, 0, 0\n" +
|
||||
"1, 30, 20, 260, 0x0, 0, 0, 512, 24576, 300, 0, 0\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
s := sampler(t.TempDir(), smi, &now).WithProfile(&ProfSampler{
|
||||
gpm: fakeGPM{0: {SMActive: 69, SMOccupancy: 45, TensorActive: 1, DRAMActive: 33, Source: "gpm"}}, now: time.Now})
|
||||
got := s.Sample(t.Context()).GPUs
|
||||
if got[0].Profile == nil || got[0].Profile.SMActive != 69 || got[0].UtilPercent != 92 {
|
||||
t.Errorf("card 0 %+v", got[0])
|
||||
}
|
||||
if got[1].Profile != nil {
|
||||
t.Errorf("card 1 has a profile nobody read: %+v", got[1].Profile)
|
||||
}
|
||||
}
|
||||
@@ -49,9 +49,6 @@ type GPUSensor struct {
|
||||
MemoryUsedMB, MemoryTotalMB float64
|
||||
SMClockMHz float64
|
||||
ECCUncorrected float64
|
||||
// Profile is how much of the chip the work uses, where something on the
|
||||
// node can say: nil on most cards. UtilPercent is not that.
|
||||
Profile *GPUProfile
|
||||
}
|
||||
|
||||
// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
|
||||
@@ -64,8 +61,6 @@ type SensorSampler struct {
|
||||
sys string // /sys, or a test's copy of it
|
||||
smi func(ctx context.Context, fields string) ([]byte, error)
|
||||
now func() time.Time
|
||||
// prof reads the cards' profiling counters; nil reads none.
|
||||
prof *ProfSampler
|
||||
|
||||
mu sync.Mutex
|
||||
last Sensors
|
||||
@@ -79,19 +74,6 @@ func NewSensorSampler() *SensorSampler {
|
||||
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
||||
}
|
||||
|
||||
// WithProfile has each sample carry the cards' profiles as well.
|
||||
func (s *SensorSampler) WithProfile(p *ProfSampler) *SensorSampler {
|
||||
s.prof = p
|
||||
return s
|
||||
}
|
||||
|
||||
// SampleGPUs reads every card's heat, power and how busy it is now, without
|
||||
// a sampler's ten seconds of memory: for a screen that asks every few seconds
|
||||
// and wants what it asked for. Nil on a machine without nvidia-smi or a card.
|
||||
func SampleGPUs(ctx context.Context) []GPUSensor {
|
||||
return (&SensorSampler{smi: runSMI}).gpus(ctx)
|
||||
}
|
||||
|
||||
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
@@ -112,14 +94,6 @@ func (s *SensorSampler) Sample(ctx context.Context) Sensors {
|
||||
out.CPUTempC, out.InletTempC = s.hwmon()
|
||||
out.CPUPowerW = s.rapl(now)
|
||||
out.GPUs = s.gpus(ctx)
|
||||
if s.prof != nil {
|
||||
profs := s.prof.Sample(ctx)
|
||||
for i := range out.GPUs {
|
||||
if p, ok := profs[out.GPUs[i].Index]; ok {
|
||||
out.GPUs[i].Profile = &p
|
||||
}
|
||||
}
|
||||
}
|
||||
s.last, s.lastAt = out, now
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -129,7 +129,7 @@ func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
||||
// so taken before baseEnv drops them. Without the repository the gate
|
||||
// links no Hugging Face cache, and sf3d's node went to the network for
|
||||
// its gated repository and was refused with a 401.
|
||||
for _, k := range []string{comfyNodesEnv, comfyHFRepoEnv, SharedModelsEnv} {
|
||||
for _, k := range []string{comfyNodesEnv, comfyHFRepoEnv} {
|
||||
if v := p.Env[k]; v != "" {
|
||||
env[k] = v
|
||||
}
|
||||
@@ -144,8 +144,3 @@ const comfyNodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
// comfyHFRepoEnv is the repository a custom node loads the weights by name
|
||||
// from; catalog.HFRepoEnv and comfy.HFRepoEnv by the same name.
|
||||
const comfyHFRepoEnv = "LLMGW_COMFYUI_HF_REPO"
|
||||
|
||||
// SharedModelsEnv is a shared replica's members — name, source, and the
|
||||
// path the agent fetched it to — as JSON; catalog.SharedModelsEnv and
|
||||
// comfy.SharedModelsEnv by the same name.
|
||||
const SharedModelsEnv = "LLMGW_COMFYUI_MODELS"
|
||||
@@ -47,13 +47,6 @@ type Profile struct {
|
||||
// KVEvents has the engine publish its KV cache events to the agent,
|
||||
// for the router's precise index. vLLM only.
|
||||
KVEvents bool `json:"kv_events,omitempty"`
|
||||
// AllowPickle lets the node fetch weights in a pickle format, which run
|
||||
// code when loaded; the catalog's allow_pickle.
|
||||
AllowPickle bool `json:"allow_pickle,omitempty"`
|
||||
// ModelSHA256 is the sha256 the registry published for the file
|
||||
// ModelPath names; the node refuses a copy that is not it. Empty when
|
||||
// nothing is known to check against.
|
||||
ModelSHA256 string `json:"model_sha256,omitempty"`
|
||||
}
|
||||
|
||||
// ShardSpec is one shard of one replica on this node.
|
||||
@@ -222,8 +215,6 @@ func NewRegistry(cfg Config) *Registry {
|
||||
r.Add(&TRTLLM{cfg: cfg.TRTLLM})
|
||||
r.Add(&Sim{cfg: cfg.Sim})
|
||||
r.Add(&ComfyUI{cfg: cfg.ComfyUI})
|
||||
r.Add(&Strata{cfg: cfg.Strata})
|
||||
r.Add(&FreeToken{cfg: cfg.FreeToken})
|
||||
return r
|
||||
}
|
||||
|
||||
@@ -274,13 +265,6 @@ type Config struct {
|
||||
Sim SimConfig `koanf:"sim"`
|
||||
// ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
ComfyUI ComfyUIConfig `koanf:"comfyui"`
|
||||
// Strata serves a few mixture-of-experts models from RAM and one card;
|
||||
// offered by a node whose Strata is named here, or that installed it.
|
||||
Strata StrataConfig `koanf:"strata"`
|
||||
// FreeToken serves mixture-of-experts models larger than a card from
|
||||
// RAM and one card of 8.0 or newer; installed by the agent where the
|
||||
// cards allow it, or named here.
|
||||
FreeToken FreeTokenConfig `koanf:"freetoken"`
|
||||
}
|
||||
|
||||
// --- shared helpers ---
|
||||
|
||||
@@ -477,14 +477,14 @@ func TestValidateRequiresModelPath(t *testing.T) {
|
||||
|
||||
func TestRegistry(t *testing.T) {
|
||||
r := NewRegistry(Config{})
|
||||
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "freetoken", "llamacpp", "sglang", "sim", "strata", "trtllm", "vllm"}) {
|
||||
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
t.Errorf("registry contents: %v", got)
|
||||
}
|
||||
if _, err := r.Get("vllm"); err != nil {
|
||||
t.Errorf("vllm missing: %v", err)
|
||||
}
|
||||
err := func() error { _, err := r.Get("nope"); return err }()
|
||||
if err == nil || !strings.Contains(err.Error(), "llamacpp, sglang, sim, strata, trtllm, vllm") {
|
||||
if err == nil || !strings.Contains(err.Error(), "llamacpp, sglang, sim, trtllm, vllm") {
|
||||
t.Errorf("an unknown engine should list what is available: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,137 +0,0 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// FreeTokenConfig locates a FreeToken installation.
|
||||
type FreeTokenConfig struct {
|
||||
// Binary is its ft command, in the environment it was installed into.
|
||||
// The agent installs one itself on a node whose GPUs it runs on; this
|
||||
// is for one installed by hand.
|
||||
Binary string `koanf:"binary"`
|
||||
}
|
||||
|
||||
// freetokenToolkit finds the CUDA 13 toolkit; a variable for tests.
|
||||
var freetokenToolkit = func() string { return discovery.CUDAToolkit(13) }
|
||||
|
||||
// FreeToken runs FreeToken (github.com/FlashML-org/FreeToken): a
|
||||
// mixture-of-experts engine that keeps a model's experts in host RAM, the
|
||||
// ones used most in VRAM, and streams the rest over PCIe, so a model several
|
||||
// times a card's size serves from one card. It reads Hugging Face
|
||||
// safetensors checkpoints as they are published (FP8, NVFP4, MXFP4, BF16)
|
||||
// and serves the OpenAI and Anthropic APIs. One card of compute capability
|
||||
// 8.0 or newer; it compiles some of its kernels on first start, with the
|
||||
// CUDA 13 toolkit.
|
||||
type FreeToken struct {
|
||||
cfg FreeTokenConfig
|
||||
installed atomic.Pointer[string]
|
||||
}
|
||||
|
||||
// Name identifies the engine in the catalog.
|
||||
func (*FreeToken) Name() string { return "freetoken" }
|
||||
|
||||
// HealthPath answers before the model is loaded too, with its progress in
|
||||
// the body; the supervisor waits for the status code, so it counts as ready
|
||||
// only once FreeToken's own startup — loading, warm-up, CUDA graphs — is
|
||||
// over and the API server is accepting.
|
||||
func (*FreeToken) HealthPath() string { return "/health" }
|
||||
|
||||
// MetricsPath is empty: FreeToken serves its numbers as JSON (/v1/stats),
|
||||
// not in Prometheus' text, so the router counts its requests itself.
|
||||
func (*FreeToken) MetricsPath() string { return "" }
|
||||
|
||||
// ServesHTTP is true: a FreeToken replica is one process on one node.
|
||||
func (*FreeToken) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// UseInstalled points the engine at the ft the agent installed.
|
||||
func (f *FreeToken) UseInstalled(path string) { f.installed.Store(&path) }
|
||||
|
||||
// Configured reports whether the configuration names a FreeToken.
|
||||
func (f *FreeToken) Configured() bool { return f.cfg.Binary != "" }
|
||||
|
||||
func (f *FreeToken) binary() string {
|
||||
if p := f.installed.Load(); p != nil {
|
||||
return *p
|
||||
}
|
||||
return f.cfg.Binary
|
||||
}
|
||||
|
||||
// HelpCommand is ft serve's own help.
|
||||
func (f *FreeToken) HelpCommand(Command) (string, []string) {
|
||||
return f.binary(), []string{"serve", "--help"}
|
||||
}
|
||||
|
||||
// Probe reports the FreeToken a configuration names, by what it says its
|
||||
// version is.
|
||||
func (f *FreeToken) Probe(ctx context.Context) (string, error) {
|
||||
bin := f.binary()
|
||||
if bin == "" {
|
||||
return "", fmt.Errorf("no FreeToken is configured")
|
||||
}
|
||||
out, err := exec.CommandContext(ctx, bin, "--version").Output() //nolint:gosec // the configured engine
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%s does not run: %w", bin, err)
|
||||
}
|
||||
// "freetoken version 0.2.1"
|
||||
if fs := strings.Fields(string(out)); len(fs) > 0 {
|
||||
return fs[len(fs)-1], nil
|
||||
}
|
||||
return Unknown, nil
|
||||
}
|
||||
|
||||
// Validate checks the profile: one card of 8.0 or newer, on one node.
|
||||
// FreeToken has no tensor parallel; it picks one device.
|
||||
func (*FreeToken) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("profile %q: freetoken runs on one GPU of one node", p.Name)
|
||||
}
|
||||
if err := requireComputeCapability(p, gpus, 8.0, "freetoken"); err != nil {
|
||||
return err
|
||||
}
|
||||
return validateCommon(p, gpus)
|
||||
}
|
||||
|
||||
// Commands is ft serve. The card is the shard's, by CUDA_VISIBLE_DEVICES,
|
||||
// so it is device 0 to FreeToken; its key goes in the environment, where it
|
||||
// reads it from, rather than on a command line every user can read.
|
||||
func (f *FreeToken) Commands(sh ShardSpec) ([]Command, error) {
|
||||
bin := f.binary()
|
||||
if bin == "" {
|
||||
return nil, fmt.Errorf("freetoken: no FreeToken is installed or configured (engines.freetoken.binary)")
|
||||
}
|
||||
toolkit := freetokenToolkit()
|
||||
if toolkit == "" {
|
||||
return nil, fmt.Errorf("freetoken compiles its kernels on first start and this node has no CUDA 13 toolkit " +
|
||||
"(nvcc in CUDA_HOME, /usr/local/cuda or /usr/local/cuda-13.*)")
|
||||
}
|
||||
p := sh.Profile
|
||||
args := []string{"serve",
|
||||
"--model", p.ModelPath,
|
||||
"--host", sh.Host(),
|
||||
"--port", strconv.Itoa(sh.HTTPPort),
|
||||
"--gpu", "0",
|
||||
}
|
||||
if p.ServedModelName != "" {
|
||||
args = append(args, "--served-model-name", p.ServedModelName)
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
env := baseEnv(sh)
|
||||
env["CUDA_HOME"] = toolkit
|
||||
// nvcc for the kernels it compiles, and the environment's own bin
|
||||
// first, where ninja is.
|
||||
env["PATH"] = strings.Join([]string{filepath.Dir(bin), filepath.Join(toolkit, "bin"), os.Getenv("PATH")}, ":")
|
||||
if sh.APIKey != "" {
|
||||
env["FREETOKEN_API_KEY"] = sh.APIKey
|
||||
}
|
||||
return []Command{{Name: "freetoken", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||
}
|
||||
@@ -1,112 +0,0 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func withToolkit(t *testing.T, dir string) {
|
||||
t.Helper()
|
||||
old := freetokenToolkit
|
||||
freetokenToolkit = func() string { return dir }
|
||||
t.Cleanup(func() { freetokenToolkit = old })
|
||||
}
|
||||
|
||||
func TestFreeTokenServesOnTheShardsCardWithTheKeyInTheEnvironment(t *testing.T) {
|
||||
withToolkit(t, "/usr/local/cuda-13.4")
|
||||
f := &FreeToken{}
|
||||
f.UseInstalled("/opt/llmgw/engines/freetoken/0.2.1-cu13/bin/ft")
|
||||
spec := ShardSpec{
|
||||
ReplicaID: "r1", HTTPPort: 18091, BindAddr: "10.0.0.160", GPUIndices: []int{3},
|
||||
APIKey: "secret-key",
|
||||
Profile: Profile{Name: "p", Engine: "freetoken", TP: 1, ModelPath: "/var/lib/llmgw-weights/glm-5.2-nvfp4",
|
||||
ServedModelName: "glm-5.2", Args: []string{"--max-running-requests", "8"}},
|
||||
}
|
||||
cmds, err := f.Commands(spec)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := cmds[0]
|
||||
if c.Path != "/opt/llmgw/engines/freetoken/0.2.1-cu13/bin/ft" {
|
||||
t.Errorf("path %s", c.Path)
|
||||
}
|
||||
args := strings.Join(c.Args, " ")
|
||||
for _, want := range []string{
|
||||
"serve --model /var/lib/llmgw-weights/glm-5.2-nvfp4", "--host 10.0.0.160", "--port 18091",
|
||||
// The shard's card is device 0 once CUDA_VISIBLE_DEVICES pins it.
|
||||
"--gpu 0", "--served-model-name glm-5.2", "--max-running-requests 8",
|
||||
} {
|
||||
if !strings.Contains(args, want) {
|
||||
t.Errorf("args lack %q: %s", want, args)
|
||||
}
|
||||
}
|
||||
if strings.Contains(args, "secret-key") {
|
||||
t.Error("the key is on the command line, which every user on the node can read")
|
||||
}
|
||||
for _, want := range []string{"FREETOKEN_API_KEY=secret-key", "CUDA_VISIBLE_DEVICES=3", "CUDA_HOME=/usr/local/cuda-13.4"} {
|
||||
if !slices.Contains(c.Env, want) {
|
||||
t.Errorf("env lacks %s", want)
|
||||
}
|
||||
}
|
||||
var path string
|
||||
for _, e := range c.Env {
|
||||
if p, ok := strings.CutPrefix(e, "PATH="); ok {
|
||||
path = p
|
||||
}
|
||||
}
|
||||
if !strings.HasPrefix(path, "/opt/llmgw/engines/freetoken/0.2.1-cu13/bin:/usr/local/cuda-13.4/bin:") {
|
||||
t.Errorf("nvcc and the environment's tools are not first on PATH: %s", path)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreeTokenWithoutTheToolkitSaysSoInsteadOfFailingOnItsFirstModel(t *testing.T) {
|
||||
withToolkit(t, "")
|
||||
f := &FreeToken{cfg: FreeTokenConfig{Binary: "/x/bin/ft"}}
|
||||
_, err := f.Commands(ShardSpec{HTTPPort: 1, Profile: Profile{Name: "p", ModelPath: "/m"}})
|
||||
if err == nil || !strings.Contains(err.Error(), "CUDA 13 toolkit") {
|
||||
t.Errorf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreeTokenRunsOnOneAmpereOrNewerCard(t *testing.T) {
|
||||
f := &FreeToken{}
|
||||
p := Profile{Name: "p", Engine: "freetoken", TP: 1, ModelPath: "/m"}
|
||||
if err := f.Validate(p, []GPUInfo{{Name: "NVIDIA RTX A6000", ComputeCapability: "8.6", MemoryTotalMB: 49140}}); err != nil {
|
||||
t.Errorf("an Ampere card was refused: %v", err)
|
||||
}
|
||||
// Tried on cuda01: no kernel image for the card, then an illegal memory
|
||||
// access in fp16.
|
||||
if err := f.Validate(p, []GPUInfo{{Name: "Quadro RTX 6000", ComputeCapability: "7.5", MemoryTotalMB: 24576}}); err == nil {
|
||||
t.Error("a Turing card was accepted")
|
||||
}
|
||||
p.TP = 2
|
||||
if err := f.Validate(p, h100(2)); err == nil {
|
||||
t.Error("a two-card profile was accepted; FreeToken has no tensor parallel")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreeTokenFlagsThatReadFilesOrMoveWhatTheAgentSetsAreRefused(t *testing.T) {
|
||||
for _, a := range []string{
|
||||
"--gpu", "--served-model-name=x", "--mmproj", "--allowed-local-media-path",
|
||||
// argparse takes a beginning for the whole flag.
|
||||
"--allowed-local", "--api-key", "--port", "--hf-overrides",
|
||||
} {
|
||||
if UnsafeArgs("freetoken", []string{a, "v"}) == nil {
|
||||
t.Errorf("%s was allowed", a)
|
||||
}
|
||||
}
|
||||
for _, a := range []string{"--max-running-requests", "--moe-strategy", "--dtype", "--max-seq-len-override", "--moe-cache-auto"} {
|
||||
if err := UnsafeArgs("freetoken", []string{a, "1"}); err != nil {
|
||||
t.Errorf("%s was refused: %v", a, err)
|
||||
}
|
||||
}
|
||||
for k, refuse := range map[string]bool{
|
||||
"FREETOKEN_PIN_BUDGET_GB": false, "FREETOKEN_API_KEY": true, "FREETOKEN_API_LOG_DIR": true,
|
||||
"FREETOKEN_MMPROJ": true, "FREETOKEN_HOST": true,
|
||||
} {
|
||||
if got, why := UnsafeEnv(k); got != refuse {
|
||||
t.Errorf("%s: refused %v (%s), want %v", k, got, why, refuse)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,58 +0,0 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAModelDownloadedByTheEngineIsReadAsSafetensorsOnly(t *testing.T) {
|
||||
for path, want := range map[string]bool{
|
||||
"Qwen/Qwen2.5-7B-Instruct": true,
|
||||
"/var/lib/llmgw/weights/abc/": false, "./local": false,
|
||||
"http://m/q/v/": false, "q.gguf": false,
|
||||
} {
|
||||
got := slices.Contains(safetensorsOnly(Profile{ModelPath: path}), "safetensors")
|
||||
if got != want {
|
||||
t.Errorf("%q: safetensors only = %v, want %v", path, got, want)
|
||||
}
|
||||
}
|
||||
if len(safetensorsOnly(Profile{ModelPath: "Qwen/Qwen2.5-7B-Instruct", AllowPickle: true})) != 0 {
|
||||
t.Error("allow_pickle still forced safetensors")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAForbiddenFlagIsKnownHoweverItIsSpelled(t *testing.T) {
|
||||
for engine, args := range map[string][]string{
|
||||
"vllm": {"--trust_remote_code"},
|
||||
"sglang": {"--load-format", "pt"},
|
||||
} {
|
||||
if UnsafeArgs(engine, args) == nil {
|
||||
t.Errorf("%s %v was let through", engine, args)
|
||||
}
|
||||
}
|
||||
for _, a := range [][]string{{"--load_format", "pt"}, {"--trust-remote"}, {"--load-form=pt"}} {
|
||||
if UnsafeArgs("vllm", a) == nil {
|
||||
t.Errorf("vllm %v was let through", a)
|
||||
}
|
||||
}
|
||||
// What is not one of them, nor the start of one, still passes.
|
||||
for _, a := range [][]string{{"--max-model-len", "8192"}, {"--tokenizer-mode", "mistral"}, {"--gpu-memory-utilization=0.9"}} {
|
||||
if err := UnsafeArgs("vllm", a); err != nil {
|
||||
t.Errorf("vllm %v was refused: %v", a, err)
|
||||
}
|
||||
}
|
||||
// Short flags are matched as written; llama.cpp takes no abbreviations,
|
||||
// so its --chat-template is not taken for --chat-template-file.
|
||||
for _, a := range [][]string{{"-m", "x"}, {"-hf", "o/r"}, {"-mu", "http://x"}} {
|
||||
if UnsafeArgs("llamacpp", a) == nil {
|
||||
t.Errorf("llamacpp %v was let through", a)
|
||||
}
|
||||
}
|
||||
if err := UnsafeArgs("llamacpp", []string{"--chat-template", "chatml"}); err != nil {
|
||||
t.Errorf("llama.cpp's own --chat-template was refused: %v", err)
|
||||
}
|
||||
// ComfyUI's argparse reads --li as --listen.
|
||||
if UnsafeArgs("comfyui", []string{"--li", "0.0.0.0"}) == nil {
|
||||
t.Error("comfyui --li was let through")
|
||||
}
|
||||
}
|
||||
@@ -102,7 +102,6 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
|
||||
if p.ServedModelName != "" {
|
||||
args = append(args, "--served-model-name", p.ServedModelName)
|
||||
}
|
||||
args = append(args, safetensorsOnly(p)...)
|
||||
if s.NNodes > 1 {
|
||||
args = append(args,
|
||||
"--nnodes", strconv.Itoa(s.NNodes),
|
||||
|
||||
@@ -1,184 +0,0 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"sync/atomic"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// StrataConfig locates a Strata installation.
|
||||
type StrataConfig struct {
|
||||
// Dir is Strata's source directory, where setup.py and serve/ are.
|
||||
Dir string `koanf:"dir"`
|
||||
// Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
Python string `koanf:"python"`
|
||||
// Engine is the directory of its prebuilt engine, where the strata
|
||||
// binary and the CUDA libraries it ships are; Dir's engine/ when unset.
|
||||
Engine string `koanf:"engine"`
|
||||
}
|
||||
|
||||
// Strata runs Strata (github.com/Niko1221/Strata): a mixture-of-experts
|
||||
// engine that keeps a model's experts in RAM and the ones used most in VRAM,
|
||||
// so a 125B model serves from one 24 GB card. It runs a few model families
|
||||
// only, each prepared from its GGUF files by Strata's own setup, so the agent
|
||||
// starts llmgw strata-serve, which prepares the model and then becomes
|
||||
// Strata's server.
|
||||
type Strata struct {
|
||||
cfg StrataConfig
|
||||
installed atomic.Pointer[string]
|
||||
}
|
||||
|
||||
// Name identifies the engine in the catalog.
|
||||
func (*Strata) Name() string { return "strata" }
|
||||
|
||||
// HealthPath answers 200 only once the model is loaded.
|
||||
func (*Strata) HealthPath() string { return "/health" }
|
||||
|
||||
// MetricsPath is Strata's Prometheus text, in vLLM's names; its /metrics is
|
||||
// JSON for its own monitor unless asked for this format.
|
||||
func (*Strata) MetricsPath() string { return "/metrics?format=prometheus" }
|
||||
|
||||
// ServesHTTP is true: a Strata replica is one process on one node.
|
||||
func (*Strata) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// UseInstalled points the engine at the environment the agent installed:
|
||||
// its interpreter, with Strata's code beside its bin in strata/ and the
|
||||
// engine in strata-engine/.
|
||||
func (s *Strata) UseInstalled(python string) { s.installed.Store(&python) }
|
||||
|
||||
// Configured reports whether the configuration names a Strata.
|
||||
func (s *Strata) Configured() bool { return s.cfg.Dir != "" }
|
||||
|
||||
// StrataSourceDir and StrataEngineDir are where an installed environment
|
||||
// keeps Strata's code and its prebuilt engine, beside bin/.
|
||||
const (
|
||||
StrataSourceDir = "strata"
|
||||
StrataEngineDir = "strata-engine"
|
||||
)
|
||||
|
||||
func (s *Strata) env() string {
|
||||
if p := s.installed.Load(); p != nil {
|
||||
return filepath.Dir(filepath.Dir(*p))
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (s *Strata) dir() string {
|
||||
if e := s.env(); e != "" {
|
||||
return filepath.Join(e, StrataSourceDir)
|
||||
}
|
||||
return s.cfg.Dir
|
||||
}
|
||||
|
||||
func (s *Strata) engineDir() string {
|
||||
if e := s.env(); e != "" {
|
||||
return filepath.Join(e, StrataEngineDir)
|
||||
}
|
||||
if s.cfg.Engine != "" {
|
||||
return s.cfg.Engine
|
||||
}
|
||||
return filepath.Join(s.cfg.Dir, "engine")
|
||||
}
|
||||
|
||||
func (s *Strata) python() string {
|
||||
if p := s.installed.Load(); p != nil {
|
||||
return *p
|
||||
}
|
||||
if s.cfg.Python != "" {
|
||||
return s.cfg.Python
|
||||
}
|
||||
return filepath.Join(s.cfg.Dir, ".venv", "bin", "python")
|
||||
}
|
||||
|
||||
var strataVersion = regexp.MustCompile(`PRODUCT_VERSION\s*=\s*"([^"]+)"`)
|
||||
|
||||
// Probe reports the Strata a configuration names, by the version its source
|
||||
// states: a directory with its setup and server, an interpreter, and the
|
||||
// engine binary. Without it a Strata named by hand was never reported, and
|
||||
// so never placed on.
|
||||
func (s *Strata) Probe(context.Context) (string, error) {
|
||||
dir := s.dir()
|
||||
if dir == "" {
|
||||
return "", fmt.Errorf("no Strata directory is configured")
|
||||
}
|
||||
for _, f := range []string{filepath.Join(dir, "setup.py"), filepath.Join(dir, "serve", "server.py"),
|
||||
s.python(), filepath.Join(s.engineDir(), "strata")} {
|
||||
if _, err := os.Stat(f); err != nil {
|
||||
return "", fmt.Errorf("%s is not a whole Strata: %w", dir, err)
|
||||
}
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "strata_cli", "__init__.py"))
|
||||
if err != nil {
|
||||
return Unknown, nil //nolint:nilerr // whole, of a version it does not say
|
||||
}
|
||||
if m := strataVersion.FindSubmatch(b); m != nil {
|
||||
return string(m[1]), nil
|
||||
}
|
||||
return Unknown, nil
|
||||
}
|
||||
|
||||
// Validate checks the profile: one card, on one node. Strata splits a model
|
||||
// over a node's cards by layer, but that is not driven from here yet.
|
||||
func (*Strata) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("profile %q: strata runs on one GPU of one node", p.Name)
|
||||
}
|
||||
// Pascal and Volta run its CUDA 12 build, which the install picks for
|
||||
// them.
|
||||
if err := requireComputeCapability(p, gpus, 6.0, "strata"); err != nil {
|
||||
return err
|
||||
}
|
||||
// A family whose engine needs more: said now, rather than by Strata's
|
||||
// setup after the replica was placed here.
|
||||
if fam, ok := strata.FamilyNamed(strata.ArgValue(p.Args, "--family")); ok && fam.MinCC > 0 {
|
||||
if err := requireComputeCapability(p, gpus, fam.MinCC, "strata's "+fam.Title+" engine"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return validateCommon(p, gpus)
|
||||
}
|
||||
|
||||
// Commands is llmgw strata-serve, which prepares the model with Strata's
|
||||
// setup and then execs Strata's server.
|
||||
func (s *Strata) Commands(sh ShardSpec) ([]Command, error) {
|
||||
dir := s.dir()
|
||||
if dir == "" {
|
||||
return nil, fmt.Errorf("strata: no Strata directory is configured (engines.strata.dir)")
|
||||
}
|
||||
bin, err := os.Executable()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p := sh.Profile
|
||||
args := []string{"strata-serve",
|
||||
"--listen", net.JoinHostPort(sh.Host(), strconv.Itoa(sh.HTTPPort)),
|
||||
"--model", p.ServedModelName,
|
||||
"--weights", p.ModelPath,
|
||||
// Relative to where engines run — their user's home — so they are
|
||||
// theirs to write. The data is shared by the node's replicas: a
|
||||
// prepared model and its draft layer are gigabytes, the same for
|
||||
// every replica of it.
|
||||
"--state", filepath.Join("strata", sh.ReplicaID),
|
||||
"--data", filepath.Join("strata", "data"),
|
||||
"--python", s.python(),
|
||||
"--strata", dir,
|
||||
"--engine", s.engineDir(),
|
||||
"--",
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
env := baseEnv(sh)
|
||||
if sh.APIKey != "" {
|
||||
// Strata's server reads its key from here, and strata-serve passes
|
||||
// it on in the environment: not on a command line every user on the
|
||||
// node can read.
|
||||
env["STRATA_API_KEY"] = sh.APIKey
|
||||
}
|
||||
return []Command{{Name: "strata", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestStrataStartsStrataServeWithTheKeyInTheEnvironment(t *testing.T) {
|
||||
s := &Strata{}
|
||||
s.UseInstalled("/opt/llmgw/engines/strata/0.2.7-cuda13/bin/python")
|
||||
spec := ShardSpec{
|
||||
ReplicaID: "r1", HTTPPort: 18090, BindAddr: "10.0.0.147", GPUIndices: []int{0},
|
||||
APIKey: "secret-key",
|
||||
Profile: Profile{Name: "p", Engine: "strata", TP: 1, ModelPath: "/var/lib/llmgw/models/q38",
|
||||
ServedModelName: "qwen3.8-flash-next", Args: []string{"--family", "qwen", "--size", "IQ2_XS"}},
|
||||
}
|
||||
cmds, err := s.Commands(spec)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := cmds[0]
|
||||
args := strings.Join(c.Args, " ")
|
||||
env := "/opt/llmgw/engines/strata/0.2.7-cuda13"
|
||||
for _, want := range []string{
|
||||
"strata-serve", "--listen 10.0.0.147:18090", "--model qwen3.8-flash-next",
|
||||
"--weights /var/lib/llmgw/models/q38", "--state " + filepath.Join("strata", "r1"),
|
||||
"--python " + env + "/bin/python", "--strata " + env + "/strata", "--engine " + env + "/strata-engine",
|
||||
"-- --family qwen --size IQ2_XS",
|
||||
} {
|
||||
if !strings.Contains(args, want) {
|
||||
t.Errorf("args lack %q: %s", want, args)
|
||||
}
|
||||
}
|
||||
if strings.Contains(args, "secret-key") {
|
||||
t.Error("the key is on the command line, which every user on the node can read")
|
||||
}
|
||||
if !slices.Contains(c.Env, "STRATA_API_KEY=secret-key") {
|
||||
t.Error("the key is not in the environment, where Strata's server reads it")
|
||||
}
|
||||
if !slices.Contains(c.Env, "CUDA_VISIBLE_DEVICES=0") {
|
||||
t.Error("the shard is not pinned to its GPU")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrataRunsOnOneCard(t *testing.T) {
|
||||
s := &Strata{}
|
||||
p := Profile{Name: "p", Engine: "strata", TP: 1, GPUsPerNode: 2, ModelPath: "/m"}
|
||||
if err := s.Validate(p, nil); err == nil {
|
||||
t.Error("a two-card profile was accepted")
|
||||
}
|
||||
p.GPUsPerNode = 1
|
||||
if err := s.Validate(p, []GPUInfo{{Name: "Quadro RTX 6000", ComputeCapability: "7.5"}}); err != nil {
|
||||
t.Errorf("one Turing card was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrataSetupFlagsThatMoveFilesAreRefused(t *testing.T) {
|
||||
for _, bad := range []string{"--gguf-dir", "--gguf", "--data-dir=/etc", "--model", "--mod", "--feed", "--host", "--gpus"} {
|
||||
if err := UnsafeArgs("strata", []string{bad, "x"}); err == nil {
|
||||
t.Errorf("%s was accepted", bad)
|
||||
}
|
||||
}
|
||||
if err := UnsafeArgs("strata", []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768",
|
||||
"--parallel", "2", "--vram-reserve-mib", "2048", "--vision", "no"}); err != nil {
|
||||
t.Errorf("an ordinary profile was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStrataNamedByHandIsReportedWithItsVersion(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for _, f := range []string{"setup.py", "serve/server.py", ".venv/bin/python", "engine/strata"} {
|
||||
p := filepath.Join(dir, f)
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, nil, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
s := &Strata{cfg: StrataConfig{Dir: dir}}
|
||||
if v, err := s.Probe(t.Context()); err != nil || v != Unknown {
|
||||
t.Fatalf("%q %v", v, err)
|
||||
}
|
||||
_ = os.MkdirAll(filepath.Join(dir, "strata_cli"), 0o755)
|
||||
_ = os.WriteFile(filepath.Join(dir, "strata_cli", "__init__.py"), []byte("PRODUCT_VERSION = \"0.2.7\"\n"), 0o644)
|
||||
if v, _ := s.Probe(t.Context()); v != "0.2.7" {
|
||||
t.Errorf("version %q", v)
|
||||
}
|
||||
r := NewRegistry(Config{Strata: StrataConfig{Dir: dir}})
|
||||
if got := r.ProbeConfigured(t.Context()); got["strata"] != "0.2.7" {
|
||||
t.Errorf("not reported: %v", got)
|
||||
}
|
||||
_ = os.Remove(filepath.Join(dir, "engine", "strata"))
|
||||
if _, err := s.Probe(t.Context()); err == nil {
|
||||
t.Error("reported without its engine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrataRefusesDeepSeekOnACardItsEngineWasNotBuiltFor(t *testing.T) {
|
||||
s := &Strata{}
|
||||
p := Profile{Name: "p", Engine: "strata", ModelPath: "/m", TP: 1, GPUsPerNode: 1, Args: []string{"--family", "deepseek41", "--size", "IST-DAS-1.75bpw"}}
|
||||
gpus := []GPUInfo{{Index: 0, Name: "Quadro RTX 6000", ComputeCapability: "7.5"}}
|
||||
if err := s.Validate(p, gpus); err == nil || !strings.Contains(err.Error(), "8.6") {
|
||||
t.Errorf("DeepSeek on 7.5: %v", err)
|
||||
}
|
||||
p.Args = []string{"--family", "qwen", "--size", "IQ2_XS"}
|
||||
if err := s.Validate(p, gpus); err != nil {
|
||||
t.Errorf("Qwen on 7.5: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,6 @@ package engines
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
@@ -25,7 +24,7 @@ var envFamilies = []string{
|
||||
"CUDA_", "NCCL_", "NVIDIA_", "NVTE_", "GLOO_", "UCX_", "TORCH_", "PYTORCH_",
|
||||
"VLLM_", "SGLANG_", "GGML_", "LLAMA_", "OMP_", "MKL_", "OPENBLAS_", "KMP_",
|
||||
"HF_", "TRANSFORMERS_", "TOKENIZERS_", "TRITON_", "FLASHINFER_",
|
||||
"TRTLLM_", "TLLM_", "FREETOKEN_",
|
||||
"TRTLLM_", "TLLM_",
|
||||
}
|
||||
|
||||
// envExact are single variables that are tuning too.
|
||||
@@ -33,14 +32,6 @@ var envExact = map[string]bool{
|
||||
"PYTHONUNBUFFERED": true, "PYTHONHASHSEED": true, "TZ": true,
|
||||
}
|
||||
|
||||
// envNeverExact are variables in an allowed family that no profile may set,
|
||||
// whose names the patterns below do not catch: what the agent sets itself,
|
||||
// and FreeToken's way of loading a projector file.
|
||||
var envNeverExact = map[string]string{
|
||||
"FREETOKEN_API_KEY": "is set by the agent", "FREETOKEN_HOST": "is set by the agent",
|
||||
"FREETOKEN_MMPROJ": "loads a file",
|
||||
}
|
||||
|
||||
// envNever are names in an allowed family that load code or move where the
|
||||
// engine reads and writes.
|
||||
var envNever = []string{
|
||||
@@ -67,9 +58,6 @@ func UnsafeEnv(key string) (bool, string) {
|
||||
return true, "only the variables engines are tuned with are allowed (" +
|
||||
strings.Join(envFamilies, "*, ") + "*)"
|
||||
}
|
||||
if why, ok := envNeverExact[k]; ok {
|
||||
return true, "it " + why
|
||||
}
|
||||
for _, n := range envNever {
|
||||
if strings.Contains(k, n) && !envExact[k] {
|
||||
return true, "it names a path, a plugin, a library or an address, which is a way to make the engine " +
|
||||
@@ -113,13 +101,6 @@ var flagsNever = map[string]map[string]string{
|
||||
"--hf-token": "is a credential", "--alias": "is set by the agent",
|
||||
"--model-draft": "loads a file", "-md": "loads a file",
|
||||
"--prompt-cache": "writes a file", "--logdir": "writes files",
|
||||
// The short spellings of the same.
|
||||
"-mm": "loads a file; the projector is fetched with the model", "--mmproj-url": "downloads a file",
|
||||
"-mmu": "downloads a file", "-hfr": "downloads a model", "-hff": "downloads a model",
|
||||
"--hf-repo-draft": "downloads a model", "-hfd": "downloads a model", "-hfrd": "downloads a model",
|
||||
"--hf-repo-v": "downloads a model", "-hfv": "downloads a model", "--hf-file-v": "downloads a model",
|
||||
"-hffv": "downloads a model", "-hft": "is a credential", "-jf": "reads a file", "-a": "is set by the agent",
|
||||
"--model-vocoder": "loads a file", "-mv": "loads a file",
|
||||
},
|
||||
"vllm": {
|
||||
"--download-dir": "writes files", "--tokenizer": "reads another repository or path",
|
||||
@@ -137,7 +118,6 @@ var flagsNever = map[string]map[string]string{
|
||||
"--file-storage-path": "writes files", "--log-requests": "logs prompts",
|
||||
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
||||
"--lora-paths": "loads files", "--download-dir": "writes files",
|
||||
"--load-format": "changes how weights are loaded, some ways running code",
|
||||
},
|
||||
// ComfyUI's main.py, behind the gate that sets where it listens and
|
||||
// where its files are.
|
||||
@@ -152,32 +132,6 @@ var flagsNever = map[string]map[string]string{
|
||||
"--tls-keyfile": "reads a file", "--tls-certfile": "reads a file",
|
||||
"--comfy-api-base": "sends work to another service",
|
||||
},
|
||||
// Strata's setup.py, which a profile's arguments go to: llmgw
|
||||
// strata-serve sets where the model is, where its files go and where
|
||||
// the server listens, and the engine is the one the agent installed.
|
||||
// Its own --model, the size, is spelt --size in a profile, since
|
||||
// --model is the source everywhere else.
|
||||
"strata": {
|
||||
"--gguf-dir": "is the model's source, set by the agent", "--models-dir": "is set by the agent",
|
||||
"--data-dir": "is set by the agent", "--move-data": "moves files", "--set-default-data-dir": "writes settings",
|
||||
"--feed": "downloads an engine", "--prebuilt": "loads an engine", "--build": "compiles an engine",
|
||||
"--update": "downloads an engine", "--rollback-engine": "changes the engine",
|
||||
"--dsv41-package": "loads a file", "--inspect": "reads another repository or path",
|
||||
"--source": "downloads from elsewhere", "--setup": "is not for a server",
|
||||
"--browser": "opens a browser", "--gpu": "is set by the agent", "--gpus": "is set by the agent",
|
||||
"--gpu-roles": "is set by the agent", "--backend": "is set by the agent",
|
||||
"--experimental-speed-projection": "changes what the model refuses",
|
||||
},
|
||||
// ft serve: where it listens, its key, its model and its card are the
|
||||
// agent's, and what reads files or fetches is refused.
|
||||
"freetoken": {
|
||||
"--gpu": "is set by the agent", "--served-model-name": "is set by the agent",
|
||||
"--mmproj": "loads a file", "--mm-encoder-weights": "loads a file",
|
||||
"--allowed-local-media-path": "serves local files to requests",
|
||||
"--allowed-media-domains": "lets requests have the engine fetch from those sites",
|
||||
"--moe-cache-split": "reads a profile file",
|
||||
"--hf-overrides": "rewrites the checkpoint's configuration, which can name code to load",
|
||||
},
|
||||
// trtllm-serve spells its flags with underscores, so the shared list's
|
||||
// hyphenated names do not reach them.
|
||||
"trtllm": {
|
||||
@@ -209,43 +163,14 @@ var flagsNever = map[string]map[string]string{
|
||||
// unsafeFlag is why an engine may not be given a flag, or "".
|
||||
func unsafeFlag(engine, flag, _ string) string {
|
||||
name, _, _ := strings.Cut(flag, "=")
|
||||
if !strings.HasPrefix(name, "-") {
|
||||
return ""
|
||||
}
|
||||
// Written as the engine would read it: vLLM takes --load_format for
|
||||
// --load-format, TensorRT-LLM the other way round. Short flags (-m,
|
||||
// -hf) are matched as written.
|
||||
norm := func(f string) string {
|
||||
if strings.HasPrefix(f, "--") {
|
||||
return "--" + strings.ReplaceAll(f[2:], "_", "-")
|
||||
}
|
||||
return f
|
||||
}
|
||||
given := norm(name)
|
||||
// argparse also takes any unambiguous beginning of a long flag for the
|
||||
// flag — --trust-remote is --trust-remote-code, --li is ComfyUI's
|
||||
// --listen — so for the engines that parse with it, a beginning of a
|
||||
// refused flag is refused. llama.cpp and TensorRT-LLM take no such
|
||||
// thing, and --chat-template there is a flag of its own.
|
||||
abbreviates := argparseEngines[engine] && strings.HasPrefix(given, "--") && len(given) >= 4
|
||||
for _, set := range []string{"", engine} {
|
||||
for never, why := range flagsNever[set] {
|
||||
if why == "" {
|
||||
continue
|
||||
}
|
||||
n := norm(never)
|
||||
if n == given || (abbreviates && strings.HasPrefix(n, "--") && strings.HasPrefix(n, given)) {
|
||||
return why
|
||||
}
|
||||
if why, ok := flagsNever[set][name]; ok && why != "" {
|
||||
return why
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// argparseEngines read their flags with Python's argparse, which takes a
|
||||
// long flag's beginning for the flag.
|
||||
var argparseEngines = map[string]bool{"vllm": true, "sglang": true, "comfyui": true, "strata": true, "freetoken": true}
|
||||
|
||||
// UnsafeArgs refuses the first of a profile's arguments an engine may not be
|
||||
// given, saying why.
|
||||
func UnsafeArgs(engine string, args []string) error {
|
||||
@@ -260,29 +185,3 @@ func UnsafeArgs(engine string, args []string) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// safetensorsOnly has vLLM or SGLang read only safetensors from a model it
|
||||
// downloads itself, by its Hugging Face id: left to choose, either takes
|
||||
// the repository's pickle files (pytorch_model.bin) when it has no
|
||||
// safetensors, and loading those runs whatever they hold. A path was
|
||||
// fetched by the agent, which checked it, or put there by the operator.
|
||||
func safetensorsOnly(p Profile) []string {
|
||||
if p.AllowPickle || !hubID(p.ModelPath) {
|
||||
return nil
|
||||
}
|
||||
return []string{"--load-format", "safetensors"}
|
||||
}
|
||||
|
||||
// hubID reports whether a model path is a Hugging Face id, org/name, rather
|
||||
// than a file or directory.
|
||||
func hubID(s string) bool {
|
||||
if s == "" || strings.HasPrefix(s, "/") || strings.HasPrefix(s, ".") || strings.Contains(s, "://") {
|
||||
return false
|
||||
}
|
||||
// org/name:Q4_K_M is a GGUF in a repository, which vLLM reads as one.
|
||||
if strings.Count(s, "/") != 1 || strings.Contains(s, ":") {
|
||||
return false
|
||||
}
|
||||
_, err := os.Stat(s)
|
||||
return err != nil
|
||||
}
|
||||
@@ -119,7 +119,6 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
|
||||
if p.ServedModelName != "" {
|
||||
args = append(args, "--served-model-name", p.ServedModelName)
|
||||
}
|
||||
args = append(args, safetensorsOnly(p)...)
|
||||
|
||||
env := baseEnv(s)
|
||||
|
||||
|
||||
@@ -10,20 +10,8 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
|
||||
// slurmd's, and srun's range for the steps it starts.
|
||||
var slurmPorts = []firewall.Port{
|
||||
{From: slurm.CtldPort, To: slurm.DbdPort, What: "slurm: slurmctld, slurmd and slurmdbd"},
|
||||
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
|
||||
}
|
||||
|
||||
// nfsPort is NFSv4's, on the machine that exports the Slurm cluster's
|
||||
// shared files.
|
||||
var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs: the Slurm cluster's shared files"}
|
||||
|
||||
// applyFirewall keeps llmgw's firewall on this node as the cluster's
|
||||
// settings say: the engines' ports and the node exporter's for the networks
|
||||
// the node reaches its control planes on, and those named, or no table at
|
||||
@@ -31,7 +19,6 @@ var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs:
|
||||
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
a.fwMu.Lock()
|
||||
defer a.fwMu.Unlock()
|
||||
a.fwLast = m
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
r := a.fwRunner
|
||||
@@ -68,14 +55,6 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
if from > 0 && to >= from {
|
||||
ports = append(ports, firewall.Engines(from, to))
|
||||
}
|
||||
// Once there, they stay: the ruleset keeps every port it held, so a
|
||||
// node that leaves Slurm keeps them closed to everyone but the cluster.
|
||||
if a.slurm != nil && a.slurm.HasPart() {
|
||||
ports = append(ports, slurmPorts...)
|
||||
}
|
||||
if a.slurm != nil && a.slurm.Serves() {
|
||||
ports = append(ports, nfsPort)
|
||||
}
|
||||
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
|
||||
switch {
|
||||
case err != nil:
|
||||
@@ -85,23 +64,6 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
}
|
||||
}
|
||||
|
||||
// slurmMessage is the leader's word on this node's part in Slurm. A node
|
||||
// that gets one, or starts or stops exporting the shared files, has the
|
||||
// firewall applied again, with Slurm's ports and NFS's.
|
||||
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
|
||||
had, served := a.slurm.HasPart(), a.slurm.Serves()
|
||||
a.slurm.Apply(m)
|
||||
if (had || !a.slurm.HasPart()) && served == a.slurm.Serves() {
|
||||
return
|
||||
}
|
||||
a.fwMu.Lock()
|
||||
last := a.fwLast
|
||||
a.fwMu.Unlock()
|
||||
if last != nil && last.GetEnabled() {
|
||||
go a.applyFirewall(last)
|
||||
}
|
||||
}
|
||||
|
||||
// controlPlaneIPs are the addresses of the control planes an agent dials.
|
||||
func controlPlaneIPs(ctx context.Context, servers []string) []string {
|
||||
var out []string
|
||||
|
||||
@@ -6,10 +6,8 @@ import (
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
)
|
||||
|
||||
// fakeShell answers the firewall package's scripts and records what was
|
||||
@@ -57,28 +55,3 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
|
||||
t.Error("turned off, the table stayed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
|
||||
sh := &fakeShell{}
|
||||
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
|
||||
slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
|
||||
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
|
||||
if strings.Contains(sh.installed, "6817") {
|
||||
t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
|
||||
}
|
||||
|
||||
a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
a.fwMu.Lock()
|
||||
got := sh.installed
|
||||
a.fwMu.Unlock()
|
||||
if strings.Contains(got, "6817-6819") && strings.Contains(got, "60001-63000") {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -20,7 +20,6 @@ var (
|
||||
gpuProcesses = discovery.GPUProcesses
|
||||
gpuMemory = discovery.GPUMemoryByProcess
|
||||
processGroup = pgidOf
|
||||
parentOf = ppidOf
|
||||
// gpuCheckWait is how long a shard has, once it answers, to show up on
|
||||
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
|
||||
gpuCheckWait = 15 * time.Second
|
||||
@@ -117,44 +116,19 @@ func (a *Agent) foreignGPUMemory(ctx context.Context, shards []int) (foreign, us
|
||||
return foreign, used
|
||||
}
|
||||
|
||||
// holds reports whether any of pids is the shard's process, in its group,
|
||||
// or a process it started: an engine that computes in a child process is
|
||||
// still the shard, and Strata's server starts its engine in a group of the
|
||||
// engine's own.
|
||||
// holds reports whether any of pids is the shard's process or in its group:
|
||||
// an engine that computes in a child process is still the shard.
|
||||
func holds(pids []int, pid, group int) bool {
|
||||
for _, p := range pids {
|
||||
if p == pid || (group > 0 && processGroup(p) == group) || descends(p, pid) {
|
||||
if p == pid || (group > 0 && processGroup(p) == group) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// descends reports whether process p was started, at any remove, by pid.
|
||||
func descends(p, pid int) bool {
|
||||
// Bounded: a chain this deep is no engine's, and a /proc read racing an
|
||||
// exit must not loop.
|
||||
for range 16 {
|
||||
p = parentOf(p)
|
||||
switch {
|
||||
case p == pid:
|
||||
return true
|
||||
case p <= 1:
|
||||
return false
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// pgidOf reads a process's group from /proc; zero when it cannot.
|
||||
func pgidOf(pid int) int { return statField(pid, 2) }
|
||||
|
||||
// ppidOf reads a process's parent from /proc; zero when it cannot.
|
||||
func ppidOf(pid int) int { return statField(pid, 1) }
|
||||
|
||||
// statField is one of the numbers after the command name in /proc's stat
|
||||
// line: 1 is the parent, 2 the group.
|
||||
func statField(pid, field int) int {
|
||||
func pgidOf(pid int) int {
|
||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||
if err != nil {
|
||||
return 0
|
||||
@@ -167,10 +141,10 @@ func statField(pid, field int) int {
|
||||
return 0
|
||||
}
|
||||
f := strings.Fields(s[i+1:])
|
||||
if len(f) <= field {
|
||||
if len(f) < 3 {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.Atoi(f[field])
|
||||
n, _ := strconv.Atoi(f[2])
|
||||
return n
|
||||
}
|
||||
|
||||
|
||||
@@ -123,30 +123,6 @@ func TestForeignGPUMemoryLeavesOutTheAgentsOwnShards(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineStartedInAGroupOfItsOwnIsStillTheShards(t *testing.T) {
|
||||
// cuda01: Strata's server (259709) starts its engine (261072) in a
|
||||
// group of the engine's own, and the engine holds the card. It read as
|
||||
// a shard on the CPU, and as 22 GB of somebody else's memory.
|
||||
oldMem, oldGroup, oldParent := gpuMemory, processGroup, parentOf
|
||||
t.Cleanup(func() { gpuMemory, processGroup, parentOf = oldMem, oldGroup, oldParent })
|
||||
processGroup = func(pid int) int { return pid }
|
||||
parentOf = func(pid int) int {
|
||||
return map[int]int{261072: 259709, 259709: 258734, 258734: 1, 999: 1}[pid]
|
||||
}
|
||||
gpuMemory = func(context.Context, string) ([]discovery.GPUMemory, error) {
|
||||
return []discovery.GPUMemory{{Index: 0, UsedMB: 23000,
|
||||
Procs: []discovery.GPUProcess{{PID: 261072, UsedMB: 22226}, {PID: 999, UsedMB: 700}}}}, nil
|
||||
}
|
||||
a := &Agent{log: slog.Default()}
|
||||
if c, ok := a.gpuCompute(context.Background(), []int{0}, 259709); !ok || c.Backend != "CUDA" ||
|
||||
len(c.Devices) != 1 || c.Devices[0] != "GPU 0: 21.7 GB" {
|
||||
t.Errorf("the engine's child on GPU 0: %+v %v", c, ok)
|
||||
}
|
||||
if foreign, _ := a.foreignGPUMemory(context.Background(), []int{259709}); !slices.Equal(foreign, []int32{774}) {
|
||||
t.Errorf("foreign %v; want only what the other process and nobody holds, [774]", foreign)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLlamaCppsOwnWordOnTheCPUIsEnoughToRefuse(t *testing.T) {
|
||||
eng := fallbackEngine{}
|
||||
// Without nvidia-smi to ask, the engine's own output says it.
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
// Half a gigabyte read per test binary, and contending with the tests
|
||||
// for the CPUs: these never look at the figure.
|
||||
measureBandwidth = func() discovery.Bandwidth { return discovery.Bandwidth{} }
|
||||
bandwidthDelay = func() time.Duration { return 0 }
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
@@ -164,13 +164,6 @@ func (a *Agent) provisionLoop(ctx context.Context) {
|
||||
|
||||
// provisionStatus records a change in one engine's installation.
|
||||
func (a *Agent) provisionStatus(st provision.Status) {
|
||||
if st == (provision.Status{}) {
|
||||
// A package install that was no engine's has ended.
|
||||
a.mu.Lock()
|
||||
delete(a.provStatus, "")
|
||||
a.mu.Unlock()
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
prev, had := a.provStatus[st.Engine]
|
||||
a.provStatus[st.Engine] = st
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
func TestFreeTokenIsInstalledOnlyWhereItsKernelsRun(t *testing.T) {
|
||||
old := freetokenToolkit
|
||||
toolkit := "/usr/local/cuda-13.4"
|
||||
freetokenToolkit = func() string { return toolkit }
|
||||
t.Cleanup(func() { freetokenToolkit = old })
|
||||
|
||||
ampere := discovery.Host{Arch: "amd64", NVIDIADriver: "580.95.05", MinComputeCapability: 8.6}
|
||||
got, err := freetokenCandidates("0.2.1", "", ampere)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := got[0]
|
||||
if b.Binary != "ft" || !slices.Equal(b.Requirements, []string{"freetoken[accel]==0.2.1"}) || !b.GPUCheck || b.Python == "" {
|
||||
t.Errorf("build %+v", b)
|
||||
}
|
||||
|
||||
// Not meant for these, and not reported as a fault: no GPU, a Turing
|
||||
// card (cuda01, where 0.2.1 had no kernel image for it), an Arm node.
|
||||
for name, h := range map[string]discovery.Host{
|
||||
"no gpu": {Arch: "amd64"},
|
||||
"turing": {Arch: "amd64", NVIDIADriver: "615.71.09", MinComputeCapability: 7.5},
|
||||
"aarch64": {Arch: "arm64", NVIDIADriver: "580.95.05", MinComputeCapability: 9.0},
|
||||
} {
|
||||
_, err := freetokenCandidates("0.2.1", "", h)
|
||||
var nh NotHere
|
||||
if !errors.As(err, &nh) {
|
||||
t.Errorf("%s: %v, want NotHere", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Meant for these, and said why not.
|
||||
if _, err := freetokenCandidates("0.2.1", "", discovery.Host{Arch: "amd64", NVIDIADriver: "570.1", MinComputeCapability: 9.0}); err == nil ||
|
||||
!strings.Contains(err.Error(), "driver 580") {
|
||||
t.Errorf("old driver: %v", err)
|
||||
}
|
||||
toolkit = ""
|
||||
_, err = freetokenCandidates("0.2.1", "", ampere)
|
||||
var nh NotHere
|
||||
if err == nil || errors.As(err, &nh) || !strings.Contains(err.Error(), "CUDA 13 toolkit") {
|
||||
t.Errorf("no toolkit: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -174,7 +174,7 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
}
|
||||
return st
|
||||
}
|
||||
if w.Offline != nil && (recipe.Repo == "" || builds[0].Python != "") {
|
||||
if w.Offline != nil && recipe.Repo == "" {
|
||||
// Said, not attempted: uv would spend minutes failing to reach PyPI,
|
||||
// and the failure would read like a network fault rather than the
|
||||
// design of an offline site.
|
||||
@@ -483,8 +483,7 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
return fmt.Errorf("%s is missing and no package manager was found to install it",
|
||||
strings.Join(missing, ", "))
|
||||
}
|
||||
// The state already says installing.
|
||||
in.progress(st, "%s, with %s", strings.Join(missing, ", "), tool)
|
||||
in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
|
||||
for _, c := range cmds {
|
||||
if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
|
||||
return err
|
||||
@@ -498,13 +497,7 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
|
||||
func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
|
||||
var st Status
|
||||
err := in.packages(ctx, byFamily, &st)
|
||||
if st.State == Installing {
|
||||
// Its progress was no engine's: an empty status takes it off the
|
||||
// node's page again, done or failed (the caller says which).
|
||||
in.report(Status{})
|
||||
}
|
||||
return err
|
||||
return in.packages(ctx, byFamily, &st)
|
||||
}
|
||||
|
||||
// lockWait bounds how long a package command waits for another to finish.
|
||||
|
||||
@@ -1,36 +0,0 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// A package install that is no engine's says it is done, so its progress
|
||||
// does not stay on the node's page (found on the PoC installing Slurm).
|
||||
func TestInstallPackagesClearsItsProgress(t *testing.T) {
|
||||
var got []Status
|
||||
in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
|
||||
Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
|
||||
if name == "sh" { // the query: not installed
|
||||
return nil, context.Canceled
|
||||
}
|
||||
return nil, nil
|
||||
}}
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
|
||||
t.Fatalf("statuses %+v", got)
|
||||
}
|
||||
|
||||
got = nil
|
||||
in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("installed already, yet said %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -109,11 +109,6 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
reqs = append(reqs, more...)
|
||||
}
|
||||
if len(b.Assets) > 0 {
|
||||
if err := in.unpackAssets(ctx, r, w, b, tmp, st); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
@@ -377,48 +372,6 @@ func (in *Installer) unpackSource(ctx context.Context, w Want, b Build, env stri
|
||||
return requirementsIn(string(f), b.IndexPackages)
|
||||
}
|
||||
|
||||
// unpackAssets fetches a Python engine's release binaries, each checked
|
||||
// against the digest GitHub publishes for it, and unpacks them into the
|
||||
// environment's AssetDir.
|
||||
func (in *Installer) unpackAssets(ctx context.Context, r Recipe, w Want, b Build, env string, st *Status) error {
|
||||
tag := b.AssetTag
|
||||
if tag == "" {
|
||||
tag = w.Version
|
||||
}
|
||||
in.progress(st, "asking GitHub for the checksums of %s %s", r.Repo, tag)
|
||||
sums, err := in.checksums(ctx, r.Repo, tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
dst := filepath.Join(env, b.AssetDir)
|
||||
if err := os.MkdirAll(dst, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, asset := range b.Assets {
|
||||
want, ok := sums[asset]
|
||||
if !ok {
|
||||
// Refused rather than installed unchecked: this is a binary that
|
||||
// will run on every node.
|
||||
return fmt.Errorf("%s %s publishes no checksum for %s, so it cannot be verified", r.Repo, tag, asset)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + r.Repo + "/releases/download/" + tag + "/" + asset
|
||||
archive := filepath.Join(env+".source", asset)
|
||||
if err := os.MkdirAll(filepath.Dir(archive), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := in.download(ctx, url, asset, archive, want, st); err != nil {
|
||||
return err
|
||||
}
|
||||
in.progress(st, "unpacking %s", asset)
|
||||
err := untarGz(archive, dst)
|
||||
_ = os.RemoveAll(filepath.Dir(archive))
|
||||
if err != nil {
|
||||
return fmt.Errorf("unpacking %s: %w", asset, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// requirementsIn reads a requirements file's lines. Options (-r, -e, --index-url
|
||||
// and the like) are refused: they would fetch from somewhere other than PyPI,
|
||||
// or install what the lock never saw.
|
||||
|
||||
@@ -76,11 +76,6 @@ type Build struct {
|
||||
// was published, which PyPI cannot be asked for.
|
||||
Source *Source
|
||||
AsOf time.Time
|
||||
// AssetTag and AssetDir, for a Python engine that also ships a binary
|
||||
// in its releases, are the release tag Assets are taken from, checked
|
||||
// against the digests GitHub publishes, and the directory in the
|
||||
// environment they are unpacked into.
|
||||
AssetTag, AssetDir string
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
@@ -126,13 +121,11 @@ type Recipe struct {
|
||||
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
|
||||
// checked against its published checksum.
|
||||
var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"comfyui": {Candidates: comfyuiCandidates},
|
||||
"strata": {Repo: StrataRepo, Candidates: strataCandidates},
|
||||
"freetoken": {PyPI: "freetoken", Candidates: freetokenCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"comfyui": {Candidates: comfyuiCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
// Installable are the engines this agent can install, as it tells the
|
||||
@@ -179,9 +172,6 @@ func Builds(engine, version, arch string) ([]Build, error) {
|
||||
continue
|
||||
}
|
||||
for _, b := range bs {
|
||||
if b.Python != "" {
|
||||
return nil, fmt.Errorf("%s is a Python environment from PyPI, which an offline bundle does not carry", engine)
|
||||
}
|
||||
if !seen[b.Variant] {
|
||||
seen[b.Variant] = true
|
||||
out = append(out, b)
|
||||
@@ -682,133 +672,3 @@ print("comfyui version", "` + version + `", "on torch", torch.__version__)`},
|
||||
{cu126, gpuNeeds{minDriverForCUDA12, 5.0}, "ComfyUI " + version + " on CUDA 12.6's torch"},
|
||||
}, h)
|
||||
}
|
||||
|
||||
// --- Strata ---
|
||||
|
||||
// StrataRepo is where Strata is published: its source, and in each release
|
||||
// the prebuilt engine for each CUDA.
|
||||
const StrataRepo = "Niko1221/Strata"
|
||||
|
||||
// strataReleases are the Strata versions this agent installs: the source
|
||||
// archive's digest, taken when it was added here and confirmed from a second
|
||||
// machine, when the version was published, and the engine its release
|
||||
// carries, whose files GitHub publishes digests for.
|
||||
var strataReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
engine string
|
||||
}{
|
||||
"0.2.7": {"41ae861478f5e5d9506864f97dfb5dd7cfe6b83193300e2b0d4f04c0380e3af9", "2026-10-27T14:54:42Z", "0.1.52"},
|
||||
}
|
||||
|
||||
// strataCandidates are the Strata builds for a machine: its source, in a
|
||||
// Python environment, with the prebuilt engine for the node's driver and
|
||||
// cards — built for CUDA 13, Turing to Blackwell; or for CUDA 12.9, Pascal
|
||||
// to Ada, which is what a V100, or a driver older than 580, gets.
|
||||
func strataCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("Strata", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if h.Arch != "amd64" {
|
||||
return nil, NotHere{"Strata publishes its Linux CUDA engine for x86-64 only"}
|
||||
}
|
||||
rel, ok := strataReleases[version]
|
||||
if !ok {
|
||||
known := make([]string, 0, len(strataReleases))
|
||||
for v := range strataReleases {
|
||||
known = append(known, v)
|
||||
}
|
||||
slices.Sort(known)
|
||||
return nil, fmt.Errorf("this agent knows the source digest of Strata %s, not of %s; "+
|
||||
"install that one by hand and name it in agent.yaml (engines.strata.dir)",
|
||||
strings.Join(known, ", "), version)
|
||||
}
|
||||
published, err := time.Parse(time.RFC3339, rel.published)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
build := func(cuda string) Build {
|
||||
return Build{
|
||||
Variant: cuda,
|
||||
Python: pythonForEngines,
|
||||
Source: &Source{
|
||||
URL: "/" + StrataRepo + "/archive/refs/tags/v" + version + ".tar.gz",
|
||||
SHA256: rel.sha256, Top: "Strata-" + version, Dir: "strata",
|
||||
Requirements: "requirements.txt",
|
||||
},
|
||||
AsOf: published.Add(24 * time.Hour),
|
||||
Assets: []string{"strata-engine-" + rel.engine + "-linux-x86_64-cuda-" + cuda + ".tar.gz"},
|
||||
AssetTag: "v" + version,
|
||||
AssetDir: "strata-engine",
|
||||
// Strata is started as a script, so what the agent is given is
|
||||
// the environment's interpreter; the proof is importing what its
|
||||
// server needs, and finding its engine beside it.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", `import os, sys, numpy, jinja2, regex
|
||||
e = os.path.dirname(os.path.dirname(os.path.abspath(sys.executable)))
|
||||
for f in ("strata/setup.py", "strata/serve/server.py", "strata-engine/strata"):
|
||||
assert os.path.isfile(os.path.join(e, f)), f + " is missing"
|
||||
print("strata version", "` + version + `", "engine", "` + rel.engine + `")`},
|
||||
FreeGB: 5,
|
||||
}
|
||||
}
|
||||
return pick("Strata "+version, variant, []candidate{
|
||||
{build("cuda13"), gpuNeeds{minDriverForCUDA13, 7.5}, "Strata " + version + " built for CUDA 13"},
|
||||
// 12.9 rather than 12: its machine code is for 6.0 to 8.9 and its
|
||||
// PTX was compiled by 12.9, so the driver must be 575.
|
||||
{build("cuda12"), gpuNeeds{575, 6.0}, "Strata " + version + " built for CUDA 12.9"},
|
||||
}, h)
|
||||
}
|
||||
|
||||
// --- FreeToken ---
|
||||
|
||||
// FreeTokenRepo is where FreeToken is developed; its builds are on PyPI.
|
||||
const FreeTokenRepo = "FlashML-org/FreeToken"
|
||||
|
||||
// freetokenMinCompute is the oldest GPU FreeToken runs on. It says RTX 30
|
||||
// and newer, and its kernels agree: on a Quadro RTX 6000 (7.5) version 0.2.1
|
||||
// loaded gpt-oss-20b and failed its first prefill with no kernel image for
|
||||
// the card, and in fp16 with fused experts its first request hit an illegal
|
||||
// memory access. Its FP8 kernels need 8.9 and its NVFP4 ones 12.0, which
|
||||
// the model's quantisation decides.
|
||||
const freetokenMinCompute = 8.0
|
||||
|
||||
// freetokenToolkit finds the CUDA toolkit FreeToken compiles its kernels
|
||||
// with; a variable for tests.
|
||||
var freetokenToolkit = func() string { return discovery.CUDAToolkit(13) }
|
||||
|
||||
// freetokenCandidates are the FreeToken builds for a machine: its PyPI
|
||||
// wheels with the kernel extras (FlashInfer, SGLang's kernels), CUDA 13,
|
||||
// x86-64 only. A node of older cards is not one it is meant for, and is not
|
||||
// reported as a fault; one of new cards without the toolkit is.
|
||||
func freetokenCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("FreeToken", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if h.Arch != "amd64" {
|
||||
return nil, NotHere{"FreeToken publishes wheels for x86-64 only; this node is " + h.Arch}
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < freetokenMinCompute {
|
||||
return nil, NotHere{fmt.Sprintf("FreeToken runs on GPUs of compute capability %.1f and newer "+
|
||||
"(RTX 30, A100 and later); this node has %.1f", freetokenMinCompute, cc)}
|
||||
}
|
||||
// It compiles kernels the first time a model loads, with nvcc, as its
|
||||
// own install guide says; without a toolkit the installation looks
|
||||
// finished and the first replica fails.
|
||||
if freetokenToolkit() == "" {
|
||||
return nil, fmt.Errorf("FreeToken compiles its kernels where it runs and needs the CUDA 13 toolkit's nvcc " +
|
||||
"(in CUDA_HOME, /usr/local/cuda or /usr/local/cuda-13.*); install cuda-toolkit-13 from NVIDIA's repository")
|
||||
}
|
||||
b := Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"freetoken[accel]==" + version},
|
||||
Binary: "ft",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 20,
|
||||
GPUCheck: true,
|
||||
}
|
||||
return pick("FreeToken "+version, variant, []candidate{
|
||||
{b, gpuNeeds{minDriverForCUDA13, freetokenMinCompute}, "FreeToken " + version},
|
||||
}, h)
|
||||
}
|
||||
@@ -1,118 +0,0 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// strataRelease is Strata's source archive as GitHub makes one, and its
|
||||
// release's engine for CUDA 13, with the recipe pointed at the archive.
|
||||
func strataRelease(t *testing.T, rel *release) {
|
||||
t.Helper()
|
||||
archive := tarGz(t, map[string]string{
|
||||
"Strata-0.2.7/setup.py": "print('setup')\n",
|
||||
"Strata-0.2.7/serve/server.py": "print('server')\n",
|
||||
"Strata-0.2.7/requirements.txt": "numpy==2.5.3; python_version >= \"3.12\"\njinja2==3.1.6\n",
|
||||
}, nil)
|
||||
rel.assets["v0.2.7.tar.gz"] = archive
|
||||
sum := sha256.Sum256(archive)
|
||||
engine := tarGz(t, map[string]string{"strata": "\x7fELF", "libcudart.so.13": "lib"}, nil)
|
||||
name := "strata-engine-0.1.52-linux-x86_64-cuda-cuda13.tar.gz"
|
||||
rel.assets[name] = engine
|
||||
esum := sha256.Sum256(engine)
|
||||
rel.digests[name] = "sha256:" + hex.EncodeToString(esum[:])
|
||||
|
||||
old := strataReleases
|
||||
strataReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
engine string
|
||||
}{"0.2.7": {hex.EncodeToString(sum[:]), "2026-10-27T14:54:42Z", "0.1.52"}}
|
||||
t.Cleanup(func() { strataReleases = old })
|
||||
}
|
||||
|
||||
func TestStrataIsInstalledWithItsEngineBesideItsCode(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
strataRelease(t, rel)
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "strata", Version: "0.2.7"}})
|
||||
if got[0].State != Ready {
|
||||
t.Fatalf("not ready: %+v", got[0])
|
||||
}
|
||||
dir := filepath.Join(in.Root, "strata", "0.2.7-cuda13")
|
||||
if got[0].Binary != filepath.Join(dir, "bin", "python") {
|
||||
t.Errorf("binary %s, want the environment's interpreter", got[0].Binary)
|
||||
}
|
||||
for _, f := range []string{"strata/setup.py", "strata/serve/server.py", "strata-engine/strata", "strata-engine/libcudart.so.13"} {
|
||||
if _, err := os.Stat(filepath.Join(dir, f)); err != nil {
|
||||
t.Errorf("%s is not in the environment: %v", f, err)
|
||||
}
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
for _, want := range []string{"--exclude-newer 2026-10-28T14:54:42Z", "jinja2==3.1.6"} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("the installation never used %q:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrataRefusesAnEngineWhoseDigestDoesNotMatch(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
strataRelease(t, rel)
|
||||
rel.digests["strata-engine-0.1.52-linux-x86_64-cuda-cuda13.tar.gz"] = "sha256:" + strings.Repeat("0", 64)
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "strata", Version: "0.2.7", Variant: "cuda13"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "does not match the checksum") {
|
||||
t.Fatalf("an engine that does not match its digest was not refused: %+v", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrataPicksTheEngineForTheCardsAndDriver(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
host discovery.Host
|
||||
want []string
|
||||
err string
|
||||
}{
|
||||
{"turing on 580", turingNode, []string{"cuda13", "cuda12"}, ""},
|
||||
{"volta", discovery.Host{Arch: "amd64", NVIDIADriver: "580.82.07", MinComputeCapability: 7.0}, []string{"cuda12"}, ""},
|
||||
{"driver 575", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 8.6}, []string{"cuda12"}, ""},
|
||||
{"driver 570", discovery.Host{Arch: "amd64", NVIDIADriver: "570.86.10", MinComputeCapability: 8.6}, nil, "575"},
|
||||
{"no gpu", discovery.Host{Arch: "amd64"}, nil, "no"},
|
||||
{"arm", discovery.Host{Arch: "arm64", NVIDIADriver: "580.82.07", MinComputeCapability: 9.0}, nil, "x86-64"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
bs, err := strataCandidates("0.2.7", "", tc.host)
|
||||
if tc.err != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tc.err) {
|
||||
t.Fatalf("got %v, %v; want an error saying %q", bs, err, tc.err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got []string
|
||||
for _, b := range bs {
|
||||
got = append(got, b.Variant)
|
||||
}
|
||||
if strings.Join(got, ",") != strings.Join(tc.want, ",") {
|
||||
t.Errorf("builds %v, want %v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStrataIsNotInAnOfflineBundle(t *testing.T) {
|
||||
if _, err := Builds("strata", "0.2.7", "amd64"); err == nil {
|
||||
t.Fatal("an offline bundle would carry Strata, which needs PyPI")
|
||||
}
|
||||
}
|
||||
@@ -94,35 +94,3 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
|
||||
t.Errorf("an agent with no cache directory deleted something: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of several files is fetched into a directory of its own, and an
|
||||
// unused one goes whole: it was refused as not empty, and nothing was freed.
|
||||
func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
a := agentWithCache(t, dir)
|
||||
set := weights.CachePathFor(dir, "https://mirror/coder/v1")
|
||||
if err := os.MkdirAll(set, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
|
||||
write(t, filepath.Join(set, ".complete"))
|
||||
// A link in the cache is taken away, not what it points at.
|
||||
target := t.TempDir()
|
||||
write(t, filepath.Join(target, "keep.gguf"))
|
||||
link := weights.CachePathFor(dir, "https://mirror/linked/v1")
|
||||
if err := os.Symlink(target, link); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
a.removeWeights([]string{set, link})
|
||||
|
||||
if _, err := os.Stat(set); !os.IsNotExist(err) {
|
||||
t.Errorf("the set is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Lstat(link); !os.IsNotExist(err) {
|
||||
t.Errorf("the link is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
|
||||
t.Errorf("what the link pointed at was deleted: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
|
||||
const FeatureSlurm = "slurm"
|
||||
|
||||
// slurmMarker says llmgw started Slurm's daemons on this machine.
|
||||
var slurmMarker = "/var/lib/llmgw-agent/slurm"
|
||||
|
||||
// setSlurmOut is where Slurm's status goes now: this session's queue, or
|
||||
// nowhere between sessions.
|
||||
func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
|
||||
a.mu.Lock()
|
||||
a.slurmOut = out
|
||||
a.mu.Unlock()
|
||||
if out != nil {
|
||||
a.slurm.Resend()
|
||||
}
|
||||
}
|
||||
|
||||
// sendSlurm queues a status without waiting: a full queue or no session
|
||||
// drops it, and the next session is sent it again.
|
||||
func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
|
||||
a.mu.Lock()
|
||||
out := a.slurmOut
|
||||
a.mu.Unlock()
|
||||
if out == nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
|
||||
default:
|
||||
a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
|
||||
}
|
||||
}
|
||||
@@ -1,214 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// accountsMarker lists what llmgw made in Slurm's accounting, a line each:
|
||||
// "account NAME" and "user LOGIN NAME". Nothing else is changed or taken
|
||||
// away: an account a Slurm administrator made is theirs.
|
||||
func (k *Keeper) accountsMarker() string { return k.marker + "-accounts" }
|
||||
|
||||
// maxEnded is the most ended jobs sent at once: a day's on a busy cluster
|
||||
// would otherwise be a status message of megabytes.
|
||||
const maxEnded = 2000
|
||||
|
||||
// accounts keeps Slurm's accounts and their users as want says, on the
|
||||
// controller.
|
||||
func (k *Keeper) accounts(ctx context.Context, want *llmgwv1.Slurm) error {
|
||||
made := k.madeAccounts()
|
||||
if !want.GetManageAccounts() {
|
||||
if len(made) == 0 {
|
||||
return nil
|
||||
}
|
||||
return k.applyAccounts(ctx, nil, made)
|
||||
}
|
||||
return k.applyAccounts(ctx, want.GetAccounts(), made)
|
||||
}
|
||||
|
||||
func (k *Keeper) madeAccounts() []string {
|
||||
b, err := k.host.ReadFile(k.accountsMarker())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, l := range strings.Split(strings.TrimSpace(string(b)), "\n") {
|
||||
if l != "" {
|
||||
out = append(out, l)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (k *Keeper) sacctmgr(ctx context.Context, args ...string) error {
|
||||
out, err := k.host.Run(ctx, nil, "sacctmgr", append([]string{"-i"}, args...)...)
|
||||
if err != nil {
|
||||
return fmt.Errorf("sacctmgr %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (k *Keeper) applyAccounts(ctx context.Context, want []*llmgwv1.SlurmAccount, made []string) error {
|
||||
out, err := k.host.Run(ctx, nil, "sacctmgr", slurm.AssocFormat...)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading Slurm's accounts: %v: %s", err, lastLine(out))
|
||||
}
|
||||
have := map[string]bool{} // "account NAME", "user LOGIN NAME"
|
||||
for _, a := range slurm.ParseAssociations(string(out)) {
|
||||
if a.User == "" {
|
||||
have["account "+a.Account] = true
|
||||
} else {
|
||||
have["user "+a.User+" "+a.Account] = true
|
||||
}
|
||||
}
|
||||
keep := map[string]bool{}
|
||||
var errs []error
|
||||
ours := slices.Clone(made)
|
||||
note := func(line string) {
|
||||
if !slices.Contains(ours, line) {
|
||||
ours = append(ours, line)
|
||||
}
|
||||
}
|
||||
for _, a := range want {
|
||||
name, parent := a.GetName(), a.GetParent()
|
||||
if !validAccount(name) || (parent != "root" && !validAccount(parent)) {
|
||||
errs = append(errs, fmt.Errorf("the leader sent an account %q under %q", name, parent))
|
||||
continue
|
||||
}
|
||||
line := "account " + name
|
||||
keep[line] = true
|
||||
if !have[line] {
|
||||
k.log.Info("a Slurm account is made, as llmgw's projects say", "account", name, "parent", parent)
|
||||
if err := k.sacctmgr(ctx, "add", "account", name, "parent="+parent, "description=llmgw"); err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
note(line)
|
||||
}
|
||||
for _, u := range a.GetUsers() {
|
||||
if !slurm.ValidLogin(u) {
|
||||
continue
|
||||
}
|
||||
ul := "user " + u + " " + name
|
||||
keep[ul] = true
|
||||
if have[ul] {
|
||||
continue
|
||||
}
|
||||
if err := k.sacctmgr(ctx, "add", "user", u, "account="+name); err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
note(ul)
|
||||
}
|
||||
}
|
||||
// Taken away: what llmgw made and no longer wants, users before their
|
||||
// accounts, children before their parents (made later, listed later).
|
||||
var left []string
|
||||
for i := len(ours) - 1; i >= 0; i-- {
|
||||
line := ours[i]
|
||||
if keep[line] || !strings.HasPrefix(line, "user ") {
|
||||
continue
|
||||
}
|
||||
f := strings.Fields(line)
|
||||
if !have[line] {
|
||||
continue
|
||||
}
|
||||
if err := k.removeAssoc(ctx, f[1], f[2], have, keep); err != nil {
|
||||
errs = append(errs, err)
|
||||
left = append(left, line)
|
||||
}
|
||||
}
|
||||
for i := len(ours) - 1; i >= 0; i-- {
|
||||
line := ours[i]
|
||||
if keep[line] || !strings.HasPrefix(line, "account ") {
|
||||
continue
|
||||
}
|
||||
if have[line] {
|
||||
k.log.Info("a Slurm account llmgw made is taken away; its jobs' records stay", "account", line[len("account "):])
|
||||
if err := k.sacctmgr(ctx, "remove", "account", line[len("account "):]); err != nil {
|
||||
errs = append(errs, err)
|
||||
left = append(left, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
var lines []string
|
||||
for _, l := range ours {
|
||||
if keep[l] || slices.Contains(left, l) {
|
||||
lines = append(lines, l)
|
||||
}
|
||||
}
|
||||
if !slices.Equal(lines, made) {
|
||||
var err error
|
||||
if len(lines) == 0 {
|
||||
err = k.host.Remove(k.accountsMarker())
|
||||
} else {
|
||||
_, err = k.write(k.accountsMarker(), []byte(strings.Join(lines, "\n")+"\n"), 0o644, "")
|
||||
}
|
||||
errs = append(errs, err)
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
func validAccount(s string) bool {
|
||||
return s != "" && len(s) <= 64 && strings.Trim(s, "abcdefghijklmnopqrstuvwxyz0123456789_-") == ""
|
||||
}
|
||||
|
||||
// ended are the jobs that ended in the last day, newest last.
|
||||
func (k *Keeper) ended(ctx context.Context) ([]*llmgwv1.SlurmJob, error) {
|
||||
out, err := k.host.Run(ctx, nil, "sacct", slurm.SacctArgs...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("sacct: %v: %s", err, lastLine(out))
|
||||
}
|
||||
jobs, err := slurm.ParseJobs(string(out))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(jobs) > maxEnded {
|
||||
jobs = jobs[len(jobs)-maxEnded:]
|
||||
}
|
||||
ps := make([]*llmgwv1.SlurmJob, 0, len(jobs))
|
||||
for _, j := range jobs {
|
||||
ps = append(ps, &llmgwv1.SlurmJob{Id: j.ID, User: j.User, Account: j.Account, Partition: j.Partition,
|
||||
State: j.State, StartUnix: j.Start.Unix(), EndUnix: j.End.Unix(), ElapsedSeconds: int64(j.Elapsed.Seconds()),
|
||||
Cpus: int32(j.CPUs), Gpus: int32(j.GPUs)}) //nolint:gosec // counts of a node's
|
||||
}
|
||||
return ps, nil
|
||||
}
|
||||
|
||||
// removeAssoc takes login off account. Slurm refuses to take away a
|
||||
// user's default account, so another of theirs is made the default first,
|
||||
// one llmgw keeps if there is one; with none left the user goes from the
|
||||
// accounting entirely.
|
||||
func (k *Keeper) removeAssoc(ctx context.Context, login, account string, have, keep map[string]bool) error {
|
||||
var others []string
|
||||
for line := range have {
|
||||
f := strings.Fields(line)
|
||||
if len(f) == 3 && f[0] == "user" && f[1] == login && f[2] != account {
|
||||
others = append(others, f[2])
|
||||
}
|
||||
}
|
||||
if len(others) == 0 {
|
||||
return k.sacctmgr(ctx, "remove", "user", login)
|
||||
}
|
||||
slices.SortFunc(others, func(a, b string) int {
|
||||
ka, kb := keep["user "+login+" "+a], keep["user "+login+" "+b]
|
||||
switch {
|
||||
case ka && !kb:
|
||||
return -1
|
||||
case kb && !ka:
|
||||
return 1
|
||||
}
|
||||
return strings.Compare(a, b)
|
||||
})
|
||||
if err := k.sacctmgr(ctx, "modify", "user", login, "set", "defaultaccount="+others[0]); err != nil {
|
||||
return err
|
||||
}
|
||||
return k.sacctmgr(ctx, "remove", "user", login, "account="+account)
|
||||
}
|
||||
@@ -1,83 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func TestAccountsMadeAndTakenAwayOnlyIfLlmgwsOwn(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, nil, "/var/lib/llmgw-agent/slurm", nil)
|
||||
assoc := "sacctmgr " + strings.Join(slurm.AssocFormat, " ")
|
||||
// root, and an account a Slurm administrator made.
|
||||
h.outputs[assoc] = "root||\nroot|root|\nphysics||root\n"
|
||||
m := &llmgwv1.Slurm{ManageAccounts: true, Accounts: []*llmgwv1.SlurmAccount{
|
||||
{Name: "test", Parent: "root"}, {Name: "test-arge", Parent: "test"},
|
||||
{Name: "test-arge-deneme", Parent: "test-arge", Users: []string{"ada", "ayse"}},
|
||||
}}
|
||||
if err := k.accounts(context.Background(), m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{"sacctmgr -i add account test parent=root description=llmgw",
|
||||
"sacctmgr -i add account test-arge-deneme parent=test-arge description=llmgw",
|
||||
"sacctmgr -i add user ayse account=test-arge-deneme"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if !strings.Contains(string(h.files[k.accountsMarker()]), "user ada test-arge-deneme") {
|
||||
t.Errorf("marker %q", h.files[k.accountsMarker()])
|
||||
}
|
||||
|
||||
// Now there; ayse leaves the project, and the team goes.
|
||||
h.outputs[assoc] = "root||\nphysics||root\ntest||root\ntest-arge||test\ntest-arge-deneme||test-arge\n" +
|
||||
"test-arge-deneme|ada|test-arge\ntest-arge-deneme|ayse|test-arge\n"
|
||||
h.ran = nil
|
||||
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
|
||||
if err := k.accounts(context.Background(), m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Neither has another account: they go from the accounting.
|
||||
want := []string{"sacctmgr -i remove user ayse", "sacctmgr -i remove user ada",
|
||||
"sacctmgr -i remove account test-arge-deneme", "sacctmgr -i remove account test-arge"}
|
||||
var got []string
|
||||
for _, c := range h.ran {
|
||||
if strings.Contains(c, "remove") || strings.Contains(c, " add ") {
|
||||
got = append(got, c)
|
||||
}
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("ran:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "physics") }) {
|
||||
t.Error("touched an account llmgw did not make")
|
||||
}
|
||||
|
||||
// One still on another account: that one is made the default first.
|
||||
h.outputs[assoc] = "root||\nphysics||root\nphysics|ada|\ntest||root\nx||test\nx|ada|\n"
|
||||
h.files[k.accountsMarker()] = []byte("account test\naccount x\nuser ada x\n")
|
||||
h.ran = nil
|
||||
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
|
||||
if err := k.accounts(context.Background(), m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if i, j := slices.Index(h.ran, "sacctmgr -i modify user ada set defaultaccount=physics"),
|
||||
slices.Index(h.ran, "sacctmgr -i remove user ada account=x"); i < 0 || j < i {
|
||||
t.Errorf("ran:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
|
||||
// Off: what llmgw made goes.
|
||||
h.outputs[assoc] = "root||\nphysics||root\ntest||root\n"
|
||||
h.ran = nil
|
||||
if err := k.accounts(context.Background(), &llmgwv1.Slurm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !slices.Contains(h.ran, "sacctmgr -i remove account test") || h.files[k.accountsMarker()] != nil {
|
||||
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
@@ -1,198 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// cacheDir holds what is fetched for Slurm: enroot's packages, pyxis's
|
||||
// source and its build.
|
||||
const cacheDir = "/var/cache/llmgw-agent/slurm"
|
||||
|
||||
// pyxisMarker is the pyxis version built here, against this Slurm.
|
||||
func (k *Keeper) pyxisMarker() string { return k.marker + "-pyxis" }
|
||||
|
||||
// containers has enroot and pyxis here, set up, when want says so; what
|
||||
// is there is left when it no longer does (Slurm stops loading pyxis once
|
||||
// the controller's plugstack.conf goes). Answers the status line, empty
|
||||
// while either is missing.
|
||||
func (k *Keeper) containers(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
if !want.GetContainers() {
|
||||
return "", nil
|
||||
}
|
||||
family := k.family()
|
||||
if family == "" {
|
||||
return "", fmt.Errorf("neither dpkg nor rpm here: enroot and pyxis come as deb or rpm")
|
||||
}
|
||||
if err := k.enroot(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("enroot: %w", err)
|
||||
}
|
||||
if err := k.enrootConf(ctx, want.GetShared().GetPath()); err != nil {
|
||||
return "", fmt.Errorf("enroot's settings: %w", err)
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-smi"); err == nil {
|
||||
if err := k.nvidiaContainer(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("the GPUs in containers (nvidia-container-cli): %w", err)
|
||||
}
|
||||
}
|
||||
if err := k.pyxis(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("pyxis: %w", err)
|
||||
}
|
||||
return slurm.ContainersStatus(), nil
|
||||
}
|
||||
|
||||
func (k *Keeper) family() string {
|
||||
if _, err := k.host.LookPath("dpkg"); err == nil {
|
||||
return "deb"
|
||||
}
|
||||
if _, err := k.host.LookPath("rpm"); err == nil {
|
||||
return "rpm"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// enroot installs the pinned release when another, or none, is here.
|
||||
func (k *Keeper) enroot(ctx context.Context, family string) error {
|
||||
if k.installed["enroot"] {
|
||||
return nil
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "enroot", "version"); err == nil && strings.TrimSpace(string(out)) == slurm.EnrootVersion {
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
pkgs, err := slurm.EnrootPackages(family, runtime.GOARCH)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("enroot missing, and this agent installs nothing")
|
||||
}
|
||||
var files []string
|
||||
for _, p := range pkgs {
|
||||
f := filepath.Join(cacheDir, p.Name)
|
||||
if err := k.host.Download(ctx, p.URL(), f, p.SHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
files = append(files, f)
|
||||
}
|
||||
k.log.Info("installing enroot, for Slurm's jobs in containers", "version", slurm.EnrootVersion)
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: files}); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "enroot", "version")
|
||||
if v := strings.TrimSpace(string(out)); err != nil || v != slurm.EnrootVersion {
|
||||
return fmt.Errorf("enroot says %q after installing %s: %v", v, slurm.EnrootVersion, err)
|
||||
}
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// enrootConf writes enroot's settings and makes its directories.
|
||||
func (k *Keeper) enrootConf(ctx context.Context, shared string) error {
|
||||
if _, err := k.write(slurm.EnrootConfPath, []byte(slurm.EnrootConf), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
changed, err := k.write(slurm.EnrootTmpfiles, []byte(slurm.EnrootTmpfilesConf), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "systemd-tmpfiles", "--create", slurm.EnrootTmpfiles); err != nil {
|
||||
return fmt.Errorf("systemd-tmpfiles: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
for _, d := range []string{slurm.EnrootLib + "/cache", slurm.EnrootLib + "/data"} {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
continue
|
||||
}
|
||||
// Each user makes their own beneath, as in /tmp.
|
||||
if out, err := k.host.Run(ctx, nil, "install", "-d", "-m", "1777", d); err != nil {
|
||||
return fmt.Errorf("making %s: %v: %s", d, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if shared == "" {
|
||||
return k.host.Remove(slurm.EnrootMounts)
|
||||
}
|
||||
_, err = k.write(slurm.EnrootMounts, []byte(slurm.EnrootSharedMount(shared)), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// nvidiaContainer installs nvidia-container-cli, which enroot's hook uses
|
||||
// to give a container the job's GPUs, from NVIDIA's repository.
|
||||
func (k *Keeper) nvidiaContainer(ctx context.Context, family string) error {
|
||||
if k.installed["nvidia-container-cli"] {
|
||||
return nil
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err == nil {
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("nvidia-container-cli missing, and this agent installs nothing")
|
||||
}
|
||||
switch family {
|
||||
case "deb":
|
||||
if err := k.host.Download(ctx, slurm.NvidiaContainerKeyURL, slurm.NvidiaContainerKeyPath,
|
||||
slurm.NvidiaContainerKeySHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.write(slurm.NvidiaContainerAptList, []byte(slurm.NvidiaContainerApt), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
case "rpm":
|
||||
if _, err := k.write(slurm.NvidiaContainerYumRepo, []byte(slurm.NvidiaContainerYum), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
k.log.Info("installing nvidia-container-cli, so containers get the job's GPUs")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.NvidiaContainerPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err != nil {
|
||||
return fmt.Errorf("still missing after installing libnvidia-container-tools")
|
||||
}
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// pyxis builds the pinned pyxis against this machine's Slurm and installs
|
||||
// it, when it has not been, or was for another version.
|
||||
func (k *Keeper) pyxis(ctx context.Context, family string) error {
|
||||
if b, err := k.host.ReadFile(k.pyxisMarker()); err == nil && strings.TrimSpace(string(b)) == slurm.PyxisVersion {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err == nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("pyxis missing, and this agent installs nothing")
|
||||
}
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: slurm.PyxisBuildPackages[family]}); err != nil {
|
||||
return fmt.Errorf("installing what builds it: %w", err)
|
||||
}
|
||||
tarball := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion+".tar.gz")
|
||||
if err := k.host.Download(ctx, slurm.PyxisSource, tarball, slurm.PyxisSHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
src := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion)
|
||||
k.log.Info("building pyxis against this machine's Slurm", "version", slurm.PyxisVersion)
|
||||
for _, c := range [][]string{
|
||||
{"rm", "-rf", src},
|
||||
{"tar", "-xzf", tarball, "-C", cacheDir},
|
||||
{"make", "-C", src, "install"},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err != nil {
|
||||
return fmt.Errorf("built, yet %s is not there", slurm.PyxisPlugin)
|
||||
}
|
||||
_, err := k.write(k.pyxisMarker(), []byte(slurm.PyxisVersion+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
@@ -1,92 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func TestContainersInstalledFromPinnedReleases(t *testing.T) {
|
||||
if _, err := slurm.EnrootPackages("deb", runtime.GOARCH); err != nil {
|
||||
t.Skip(err)
|
||||
}
|
||||
h := newFake()
|
||||
h.bins["dpkg"], h.bins["nvidia-smi"] = true, true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
// enroot is not here until installed, and then says its version.
|
||||
h.fail["enroot version"] = true
|
||||
h.fail["test -d "+slurm.EnrootLib+"/cache"] = true
|
||||
h.fail["test -f "+slurm.PyxisPlugin] = true
|
||||
m := nodeMsg(true)
|
||||
m.Containers = true
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "containers: enroot") {
|
||||
t.Fatalf("ready with enroot saying nothing: %v", st)
|
||||
}
|
||||
pkgs, _ := slurm.EnrootPackages("deb", runtime.GOARCH)
|
||||
for _, p := range pkgs {
|
||||
if !slices.Contains(h.ran, "download "+p.URL()+" "+cacheDir+"/"+p.Name+" "+p.SHA256) {
|
||||
t.Errorf("%s not fetched with its digest:\n%s", p.Name, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if len(inst.got["deb"]) != 2 || !strings.HasPrefix(inst.got["deb"][0], cacheDir) {
|
||||
t.Errorf("installed %v", inst.got)
|
||||
}
|
||||
|
||||
delete(h.fail, "enroot version")
|
||||
h.outputs["enroot version"] = slurm.EnrootVersion + "\n"
|
||||
h.bins["nvidia-container-cli"] = false
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if !strings.Contains(st.GetError(), "nvidia-container-cli") {
|
||||
t.Errorf("a GPU machine without nvidia-container-cli is ready: %v", st)
|
||||
}
|
||||
if string(h.files[slurm.NvidiaContainerAptList]) != slurm.NvidiaContainerApt ||
|
||||
!slices.Contains(h.ran, "download "+slurm.NvidiaContainerKeyURL+" "+slurm.NvidiaContainerKeyPath+" "+slurm.NvidiaContainerKeySHA256) {
|
||||
t.Errorf("NVIDIA's repository not added with its key checked")
|
||||
}
|
||||
|
||||
h.bins["nvidia-container-cli"] = true
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
// Built, but the plugin is still not there.
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "pyxis") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
for _, want := range []string{"tar -xzf " + cacheDir + "/pyxis-" + slurm.PyxisVersion + ".tar.gz -C " + cacheDir,
|
||||
"make -C " + cacheDir + "/pyxis-" + slurm.PyxisVersion + " install", "install -d -m 1777 " + slurm.EnrootLib + "/cache"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q", want)
|
||||
}
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.PyxisPlugin)
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != slurm.ContainersStatus() || st.GetError() != "" {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if string(h.files[slurm.EnrootConfPath]) != slurm.EnrootConf ||
|
||||
!strings.Contains(string(h.files[slurm.EnrootMounts]), "/shared /shared none") {
|
||||
t.Errorf("enroot's settings not written")
|
||||
}
|
||||
|
||||
// Done: nothing fetched or built again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "download") || strings.HasPrefix(c, "make") {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Off: nothing said, nothing taken away.
|
||||
m.Containers = false
|
||||
if st := k.Reconcile(context.Background(), m); st.GetContainers() != "" {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
}
|
||||
@@ -1,170 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// mariaDBConfDirs are where each family's MariaDB reads its settings: Debian
|
||||
// and Ubuntu's, then Red Hat's.
|
||||
var mariaDBConfDirs = []string{"/etc/mysql/mariadb.conf.d", "/etc/my.cnf.d"}
|
||||
|
||||
const mariaDBConfName = "90-llmgw-slurm.cnf"
|
||||
|
||||
// acctSQLPath is where the database's statements are put for the client to
|
||||
// read, never on its command line: they hold the password.
|
||||
const acctSQLPath = "/run/llmgw-slurm-acct.sql"
|
||||
|
||||
// accounting keeps slurmdbd on the controller: MariaDB on this machine
|
||||
// alone, the database and its user, slurmdbd on them, and the cluster in it.
|
||||
// Before slurmctld, which reports to it from its first start.
|
||||
func (k *Keeper) accounting(ctx context.Context, key []byte, files map[string]string) error {
|
||||
password := slurm.DbdPassword(key)
|
||||
changed, err := k.mariaDBSettings(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("MariaDB, for Slurm's accounting, starts on this machine alone")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "mariadb"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
client, err := k.mariaDBClient()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.waitFor(ctx, "MariaDB", client, "-e", "SELECT 1"); err != nil {
|
||||
return err
|
||||
}
|
||||
// The database and its user, once each time the agent starts or the
|
||||
// password is new: cheap, and a database restored from elsewhere gets
|
||||
// its user back.
|
||||
sum := sha256.Sum256([]byte(password))
|
||||
if mark := hex.EncodeToString(sum[:]); k.acctDone != mark {
|
||||
if err := k.host.WriteFile(acctSQLPath, []byte(slurm.AcctSQL(password)), 0o600, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, client, "-e", "source "+acctSQLPath)
|
||||
_ = k.host.Remove(acctSQLPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("making Slurm's database: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.acctDone = mark
|
||||
}
|
||||
|
||||
out, err := k.host.Run(ctx, nil, "hostname", "-s")
|
||||
host := strings.TrimSpace(string(out))
|
||||
if err != nil || host == "" {
|
||||
return fmt.Errorf("this machine's short hostname: %v: %s", err, lastLine(out))
|
||||
}
|
||||
changed, err = k.write(slurm.DbdConfPath, []byte(slurm.DbdConf(host, password)), 0o600, "slurm")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("slurmdbd starts again on new settings")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "slurmdbd"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := k.waitFor(ctx, "slurmdbd", "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster"); err != nil {
|
||||
return err
|
||||
}
|
||||
name := clusterName(files["slurm.conf"])
|
||||
if name == "" || k.clusterAdded == name {
|
||||
return nil
|
||||
}
|
||||
out, err = k.host.Run(ctx, nil, "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster")
|
||||
if err != nil {
|
||||
return fmt.Errorf("sacctmgr show cluster: %v: %s", err, lastLine(out))
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
if strings.TrimSpace(l) == name {
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
}
|
||||
k.log.Info("the cluster is added to Slurm's accounting", "cluster", name)
|
||||
if out, err := k.host.Run(ctx, nil, "sacctmgr", "-i", "add", "cluster", name); err != nil {
|
||||
return fmt.Errorf("sacctmgr add cluster %s: %v: %s", name, err, lastLine(out))
|
||||
}
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
|
||||
// mariaDBSettings keeps MariaDB to this machine, when llmgw installed it;
|
||||
// one that was here before llmgw keeps its own settings, and only gets the
|
||||
// database and its user.
|
||||
func (k *Keeper) mariaDBSettings(ctx context.Context) (bool, error) {
|
||||
if _, err := k.host.ReadFile(k.mariaDBMarker()); err != nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, d := range mariaDBConfDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
return k.write(d+"/"+mariaDBConfName, []byte(slurm.MariaDBConf), 0o644, "")
|
||||
}
|
||||
}
|
||||
return false, fmt.Errorf("MariaDB has none of %s to put its settings in", strings.Join(mariaDBConfDirs, ", "))
|
||||
}
|
||||
|
||||
// mariaDBMarker is there when llmgw installed MariaDB on this machine.
|
||||
func (k *Keeper) mariaDBMarker() string { return k.marker + "-mariadb" }
|
||||
|
||||
// mariaDBClient is MariaDB's command-line client, under either of its
|
||||
// names.
|
||||
func (k *Keeper) mariaDBClient() (string, error) {
|
||||
for _, c := range []string{"mariadb", "mysql"} {
|
||||
if _, err := k.host.LookPath(c); err == nil {
|
||||
return c, nil
|
||||
}
|
||||
}
|
||||
return "", errors.New("MariaDB's client (mariadb or mysql) is not installed")
|
||||
}
|
||||
|
||||
// waitFor runs a command until it succeeds, half a minute at most.
|
||||
func (k *Keeper) waitFor(ctx context.Context, what, name string, args ...string) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, name, args...)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("%s did not answer after starting: %s", what, lastLine(last))
|
||||
}
|
||||
|
||||
// clusterName is ClusterName in slurm.conf.
|
||||
func clusterName(conf string) string {
|
||||
sc := bufio.NewScanner(strings.NewReader(conf))
|
||||
for sc.Scan() {
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "ClusterName="); ok {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -1,150 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// What a machine says of the shared path.
|
||||
const (
|
||||
SharedMounted = "mounted"
|
||||
SharedExported = "exported"
|
||||
SharedMissing = "missing"
|
||||
)
|
||||
|
||||
// shared keeps the shared path as want says: exported from here, mounted
|
||||
// from an export, or looked for where the site mounts it. Answers how it
|
||||
// is.
|
||||
func (k *Keeper) shared(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
sh := want.GetShared()
|
||||
path := sh.GetPath()
|
||||
k.unshare(ctx, path, sh.GetServe(), sh.GetNfs())
|
||||
if path == "" {
|
||||
return "", nil
|
||||
}
|
||||
if !strings.HasPrefix(path, "/") || strings.Contains(path, "..") || strings.ContainsAny(path, " \n\t\"'\\$`") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent a path %q", path)
|
||||
}
|
||||
switch {
|
||||
case sh.GetServe():
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed, err := k.write(slurm.ExportsPath, []byte(slurm.Exports(path, sh.GetClients())), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if !k.active(ctx, "nfs-server") {
|
||||
k.log.Info("NFS starts on this machine, exporting the Slurm cluster's shared files", "path", path)
|
||||
if err := k.systemctl(ctx, "enable", "--now", "nfs-server"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "exportfs", "-ra"); err != nil {
|
||||
return SharedMissing, fmt.Errorf("exportfs -ra: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return SharedExported, nil
|
||||
case sh.GetNfs() != "":
|
||||
nfs := sh.GetNfs()
|
||||
if strings.ContainsAny(nfs, " \n\t\"'\\$`") || !strings.Contains(nfs, ":/") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent an export %q", nfs)
|
||||
}
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
unit := slurm.MountUnitName(path)
|
||||
changed, err := k.write(filepath.Join(slurm.MountUnitDir, unit), []byte(slurm.MountUnit(nfs, path)), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("the Slurm cluster's shared files are mounted here", "from", nfs, "at", path)
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
// From another export: a restart mounts the new one, and fails
|
||||
// while a job still has files open on the old.
|
||||
if err := k.systemctl(ctx, "restart", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
} else if !k.active(ctx, unit) {
|
||||
if err := k.systemctl(ctx, "start", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "mountpoint", "-q", path); err != nil {
|
||||
if sh.GetNfs() == "" {
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted here: the site mounts it (slurm.shared names no export)", path)
|
||||
}
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted from %s", path, sh.GetNfs())
|
||||
}
|
||||
return SharedMounted, nil
|
||||
}
|
||||
|
||||
// unshare takes away the export or the mount llmgw kept here that want no
|
||||
// longer has: path is where the files are wanted now, serve and nfs how.
|
||||
func (k *Keeper) unshare(ctx context.Context, path string, serve bool, nfs string) {
|
||||
if _, err := k.host.ReadFile(slurm.ExportsPath); err == nil && (!serve || path == "") {
|
||||
k.log.Info("this machine no longer exports the Slurm cluster's shared files")
|
||||
_ = k.host.Remove(slurm.ExportsPath)
|
||||
_, _ = k.host.Run(ctx, nil, "exportfs", "-ra")
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--type=mount")
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
keep := ""
|
||||
if path != "" && nfs != "" && !serve {
|
||||
keep = slurm.MountUnitName(path)
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
f := strings.Fields(l)
|
||||
if len(f) == 0 || f[0] == keep {
|
||||
continue
|
||||
}
|
||||
file := filepath.Join(slurm.MountUnitDir, f[0])
|
||||
b, err := k.host.ReadFile(file)
|
||||
if err != nil || !strings.HasPrefix(string(b), "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.") {
|
||||
continue
|
||||
}
|
||||
k.log.Info("the Slurm cluster's old shared mount is taken away", "unit", f[0])
|
||||
if err := k.systemctl(ctx, "disable", "--now", f[0]); err != nil {
|
||||
k.log.Warn("could not unmount the old shared files", "unit", f[0], "err", err)
|
||||
continue
|
||||
}
|
||||
_ = k.host.Remove(file)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
}
|
||||
}
|
||||
|
||||
// sharedPackages are what the shared path needs here, by the command that
|
||||
// shows each is installed.
|
||||
func sharedPackages(want *llmgwv1.Slurm, need map[string][2]string) {
|
||||
switch sh := want.GetShared(); {
|
||||
case sh.GetPath() == "":
|
||||
case sh.GetServe():
|
||||
need["exportfs"] = [2]string{"nfs-kernel-server", "nfs-utils"}
|
||||
case sh.GetNfs() != "":
|
||||
need["mount.nfs"] = [2]string{"nfs-common", "nfs-utils"}
|
||||
}
|
||||
}
|
||||
|
||||
// Serves is whether this machine exports the shared files, so NFS's port
|
||||
// is kept for the cluster's networks.
|
||||
func (k *Keeper) Serves() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetShared().GetServe() && k.want.GetShared().GetPath() != ""
|
||||
}
|
||||
@@ -1,171 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func ayse() *llmgwv1.SlurmUser {
|
||||
return &llmgwv1.SlurmUser{Login: "ayse", Uid: 200000, Name: "Ayşe Y.", Home: "/shared/home/ayse",
|
||||
SshKeys: []string{"ssh-ed25519 AAAA ayse@laptop"}}
|
||||
}
|
||||
|
||||
func TestTheServerExportsAndMakesUsersWithHomes(t *testing.T) {
|
||||
h := newFake()
|
||||
h.active["ssh"] = true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Serve: true, Clients: []string{"10.0.0.22", "10.0.0.21"}}
|
||||
m.ManageUsers, m.MakeHomes, m.Users = true, true, []*llmgwv1.SlurmUser{ayse()}
|
||||
h.fail["test -d /shared/home/ayse"] = true
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if st.GetShared() != SharedExported || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "nfs-kernel-server") {
|
||||
t.Errorf("packages %v", inst.got)
|
||||
}
|
||||
if e := string(h.files[slurm.ExportsPath]); !strings.Contains(e, `"/shared" 10.0.0.21(`) {
|
||||
t.Errorf("exports:\n%s", e)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable --now nfs-server", "exportfs -ra", "groupadd -g 200000 ayse",
|
||||
"useradd -u 200000 -g 200000 -M -d /shared/home/ayse -s /bin/bash -c Ayşe Y. ayse",
|
||||
"install -d -m 0700 -o 200000 -g 200000 /shared/home/ayse", "sshd -t", "systemctl reload ssh"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if string(h.files[slurm.AuthorizedKeysDir+"/ayse"]) != "ssh-ed25519 AAAA ayse@laptop\n" ||
|
||||
string(h.files[k.usersMarker()]) != "ayse\n" || h.files[slurm.SSHDropIn] == nil {
|
||||
t.Errorf("files: keys %q, marker %q", h.files[slurm.AuthorizedKeysDir+"/ayse"], h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Again, made: nothing is run again.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:200000:200000:Ayşe Y.:/shared/home/ayse:/bin/bash\n"
|
||||
delete(h.fail, "test -d /shared/home/ayse")
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "useradd") || strings.HasPrefix(c, "usermod") || strings.HasPrefix(c, "exportfs") ||
|
||||
strings.HasPrefix(c, "install") || c == "systemctl reload ssh" {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Gone from the accounts: taken away, its home left.
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "userdel ayse") || h.files[k.usersMarker()] != nil || h.files[slurm.SSHDropIn] != nil ||
|
||||
h.files[slurm.AuthorizedKeysDir+"/ayse"] != nil {
|
||||
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "rm") || strings.Contains(c, "userdel -r") }) {
|
||||
t.Error("a home was removed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientMountsAndLeavesOthersUsersAlone(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
bob := &llmgwv1.SlurmUser{Login: "bob", Uid: 200001, Home: "/shared/home/bob"}
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse(), bob}
|
||||
// The site's own ayse, with its own UID.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:1001:1001::/home/ayse:/bin/bash\n"
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetShared() != SharedMounted || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !strings.Contains(st.GetError(), "ayse is already a user here with UID 1001") {
|
||||
t.Errorf("error %q", st.GetError())
|
||||
}
|
||||
if u := string(h.files["/etc/systemd/system/shared.mount"]); !strings.Contains(u, "What=10.0.0.11:/shared") {
|
||||
t.Fatalf("unit:\n%s", u)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable shared.mount", "systemctl restart shared.mount", "mountpoint -q /shared",
|
||||
"useradd -u 200001 -g 200001 -M -d /shared/home/bob -s /bin/bash -c bob"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "ayse") && !strings.HasPrefix(c, "getent") }) ||
|
||||
slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "install -d") }) {
|
||||
t.Errorf("touched the site's ayse, or made a home:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if string(h.files[k.usersMarker()]) != "bob\n" {
|
||||
t.Errorf("marker %q", h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Not mounted: said.
|
||||
h.fail["mountpoint -q /shared"] = true
|
||||
if st := k.Reconcile(context.Background(), m); st.GetShared() != SharedMissing ||
|
||||
!strings.Contains(st.GetError(), "not mounted from 10.0.0.11:/shared") {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
|
||||
// No shared path any more: the mount is taken away.
|
||||
h.outputs["systemctl list-units --all --plain --no-legend --type=mount"] = "shared.mount loaded active mounted llmgw\n"
|
||||
m.Shared = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "systemctl disable --now shared.mount") || h.files["/etc/systemd/system/shared.mount"] != nil {
|
||||
t.Errorf("mount kept:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
const pamStack = "@include common-auth\naccount required pam_nologin.so\n@include common-account\n@include common-session\n"
|
||||
|
||||
func TestUsersReachANodeOnlyThroughTheirJobs(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
// The module is not there until its package is installed.
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
h.fail["test -f "+d+"/"+slurm.AdoptModule] = true
|
||||
}
|
||||
m := nodeMsg(true)
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse()}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(inst.got["deb"], "libpam-slurm-adopt") {
|
||||
t.Fatalf("not installed: %v", inst.got)
|
||||
}
|
||||
if got := string(h.files[slurm.PAMSSHD]); got != pamStack {
|
||||
t.Fatalf("lines put in with the module still missing:\n%s", got)
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.ModuleDirs[0]+"/"+slurm.AdoptModule)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
got := string(h.files[slurm.PAMSSHD])
|
||||
if !strings.Contains(got, "pam_listfile.so item=user sense=allow file="+k.usersMarker()) ||
|
||||
!strings.Contains(got, "account required "+slurm.AdoptModule) {
|
||||
t.Fatalf("not held to their jobs (%s):\n%s", st.GetError(), got)
|
||||
}
|
||||
|
||||
// The controller is where they log in: not held there.
|
||||
m.Controller = true
|
||||
m.Files = map[string]string{"slurm.conf": "x"}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("held on the controller:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
|
||||
// No users of llmgw's any more: the lines go.
|
||||
m.Controller, m.Files = false, nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("lines left with no users:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
}
|
||||
@@ -1,747 +0,0 @@
|
||||
// Package slurmnode keeps this machine's part in the Slurm cluster run beside
|
||||
// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
|
||||
// on a node, munge on both, installed from the machine's own package sources.
|
||||
//
|
||||
// What the leader sends is wanted state, kept by Keeper's loop: sent again,
|
||||
// or checked again a minute later, it changes nothing that is already so.
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// Host is what Keeper does to the machine; Local in the agent, a fake in
|
||||
// tests.
|
||||
type Host interface {
|
||||
Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
|
||||
LookPath(name string) (string, error)
|
||||
ReadFile(path string) ([]byte, error)
|
||||
// WriteFile writes b at path with mode, owned by owner ("" for root),
|
||||
// making its directory.
|
||||
WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
|
||||
// MkdirAll makes path, owned by owner.
|
||||
MkdirAll(path string, mode fs.FileMode, owner string) error
|
||||
Remove(path string) error
|
||||
// Download fetches url to path, kept only if its SHA-256 is sha; a file
|
||||
// already there with it is not fetched again.
|
||||
Download(ctx context.Context, url, path, sha string) error
|
||||
}
|
||||
|
||||
// Installer installs packages by family, "deb" and "rpm".
|
||||
type Installer interface {
|
||||
InstallPackages(ctx context.Context, byFamily map[string][]string) error
|
||||
}
|
||||
|
||||
// Reasons llmgw drains a node with; one drained for another reason, by a
|
||||
// Slurm administrator, is never resumed by llmgw.
|
||||
const (
|
||||
ReasonServing = "llmgw: serving models"
|
||||
reasonPrefix = "llmgw"
|
||||
)
|
||||
|
||||
// DropIn is the unit drop-in that starts slurmd configless.
|
||||
const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
|
||||
|
||||
// Keeper keeps the machine as the last Slurm message says.
|
||||
type Keeper struct {
|
||||
host Host
|
||||
inst Installer
|
||||
log *slog.Logger
|
||||
marker string
|
||||
|
||||
mu sync.Mutex
|
||||
want *llmgwv1.Slurm
|
||||
kick chan struct{}
|
||||
// resend has the next status sent even if unchanged: a new stream's
|
||||
// control plane may never have had it.
|
||||
resend bool
|
||||
// installed is true once the packages were found or put there, so a
|
||||
// minute's check does not look again.
|
||||
installed map[string]bool
|
||||
files string
|
||||
// acctDone is the digest of the password the database's user was last
|
||||
// given; clusterAdded the cluster last found in the accounting.
|
||||
acctDone string
|
||||
clusterAdded string
|
||||
}
|
||||
|
||||
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
|
||||
// an agent started again knows to stop what it started when told it has no
|
||||
// part any more.
|
||||
func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Keeper{host: host, inst: inst, log: log, marker: marker,
|
||||
kick: make(chan struct{}, 1), installed: map[string]bool{}}
|
||||
}
|
||||
|
||||
// Apply is a new message from the leader: kept from now on.
|
||||
func (k *Keeper) Apply(m *llmgwv1.Slurm) {
|
||||
k.mu.Lock()
|
||||
k.want = proto.CloneOf(m)
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// HasPart is whether the last message gives this machine a part in Slurm.
|
||||
func (k *Keeper) HasPart() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetController() || k.want.GetNode()
|
||||
}
|
||||
|
||||
// Resend has the status sent again on the next pass, which it starts now.
|
||||
func (k *Keeper) Resend() {
|
||||
k.mu.Lock()
|
||||
k.resend = true
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// Every is how often the machine is checked again without a new message.
|
||||
var Every = time.Minute
|
||||
|
||||
// resendAfter is how often an unchanged status is sent anyway.
|
||||
const resendAfter = 5 * time.Minute
|
||||
|
||||
// Run keeps the machine until ctx ends, sending its status when it changes
|
||||
// and every few minutes while it has a part.
|
||||
func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
var last *llmgwv1.SlurmStatus
|
||||
var sent time.Time
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-k.kick:
|
||||
case <-t.C:
|
||||
}
|
||||
k.mu.Lock()
|
||||
want, again := k.want, k.resend
|
||||
k.resend = false
|
||||
k.mu.Unlock()
|
||||
if want == nil {
|
||||
continue
|
||||
}
|
||||
st := k.Reconcile(ctx, want)
|
||||
if st == nil {
|
||||
last = nil
|
||||
continue
|
||||
}
|
||||
if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
|
||||
send(st)
|
||||
last, sent = st, time.Now()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile makes the machine what want says once, and answers its status;
|
||||
// nil when it has no part in Slurm.
|
||||
func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
|
||||
defer cancel()
|
||||
if !want.GetController() && !want.GetNode() {
|
||||
k.stop(ctx)
|
||||
return nil
|
||||
}
|
||||
st := &llmgwv1.SlurmStatus{}
|
||||
fail := func(what string, err error) *llmgwv1.SlurmStatus {
|
||||
st.Error = what + ": " + err.Error()
|
||||
k.log.Warn("keeping Slurm on this node: "+what, "err", err)
|
||||
return st
|
||||
}
|
||||
if err := k.ensurePackages(ctx, want); err != nil {
|
||||
return fail("installing Slurm", err)
|
||||
}
|
||||
// From the daemon, not scontrol: a client command looks for its files
|
||||
// even to say its version, and a configless node has none before slurmd
|
||||
// has fetched them.
|
||||
daemon := "slurmd"
|
||||
if !want.GetNode() {
|
||||
daemon = "slurmctld"
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, daemon, "-V")
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
|
||||
}
|
||||
v, err := slurm.ParseVersion(string(out))
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", err)
|
||||
}
|
||||
st.Version = strings.TrimSpace(string(out))
|
||||
if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
|
||||
st.Version = st.Version[i+1:]
|
||||
}
|
||||
if slurm.TooOld(v) {
|
||||
return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
|
||||
"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
|
||||
slurm.MinVersion[0], slurm.MinVersion[1]))
|
||||
}
|
||||
if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
|
||||
return fail("keeping its marker", err)
|
||||
}
|
||||
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
|
||||
return fail("munge", err)
|
||||
}
|
||||
// The shared files, then the users whose homes are on them. Neither
|
||||
// keeps Slurm from running: what goes wrong is said, and Slurm kept.
|
||||
var shErr, usErr error
|
||||
st.Shared, shErr = k.shared(ctx, want)
|
||||
if shErr != nil {
|
||||
k.log.Warn("keeping the Slurm cluster's shared files on this node", "err", shErr)
|
||||
}
|
||||
if shErr == nil || !want.GetMakeHomes() || want.GetShared().GetPath() == "" {
|
||||
st.Users, usErr = k.users(ctx, want)
|
||||
} else {
|
||||
// Homes made now would be made beside the shared files, not on them.
|
||||
usErr = errors.New("waiting for the shared files, where the homes are")
|
||||
}
|
||||
if usErr != nil {
|
||||
k.log.Warn("keeping the Slurm users on this node", "err", usErr)
|
||||
}
|
||||
// Jobs in containers; Slurm runs without them, and is kept anyway.
|
||||
var ctErr error
|
||||
if st.Containers, ctErr = k.containers(ctx, want); ctErr != nil {
|
||||
k.log.Warn("setting up jobs in containers on this node", "err", ctErr)
|
||||
}
|
||||
defer func() {
|
||||
switch {
|
||||
case st.Error != "":
|
||||
case shErr != nil:
|
||||
st.Error = "shared files: " + shErr.Error()
|
||||
case usErr != nil:
|
||||
st.Error = "users: " + usErr.Error()
|
||||
case ctErr != nil:
|
||||
st.Error = "containers: " + ctErr.Error()
|
||||
}
|
||||
}()
|
||||
if want.GetController() {
|
||||
// Accounting first: slurmctld reports to it from its first start.
|
||||
// Without it slurmctld still runs jobs, a pass of backfill late
|
||||
// each; so a failure here is said, and slurmctld is kept anyway.
|
||||
var acctErr error
|
||||
if acctErr = k.accounting(ctx, want.GetMungeKey(), want.GetFiles()); acctErr != nil {
|
||||
k.log.Warn("keeping Slurm's accounting on this node", "err", acctErr)
|
||||
}
|
||||
st.SlurmdbdRunning = k.active(ctx, "slurmdbd")
|
||||
defer func() {
|
||||
if acctErr != nil && st.Error == "" {
|
||||
st.Error = "slurmdbd: " + acctErr.Error()
|
||||
}
|
||||
}()
|
||||
sum, err := k.controller(ctx, want.GetFiles())
|
||||
st.FilesSha256 = sum
|
||||
if err != nil {
|
||||
return fail("slurmctld", err)
|
||||
}
|
||||
st.SlurmctldRunning = k.active(ctx, "slurmctld")
|
||||
// The accounts, and the jobs that ended, need the accounting.
|
||||
if st.SlurmdbdRunning && st.SlurmctldRunning {
|
||||
if err := k.accounts(ctx, want); err != nil {
|
||||
k.log.Warn("keeping Slurm's accounts", "err", err)
|
||||
if acctErr == nil {
|
||||
acctErr = fmt.Errorf("accounts: %w", err)
|
||||
}
|
||||
}
|
||||
if ended, err := k.ended(ctx); err != nil {
|
||||
k.log.Warn("reading the jobs that ended", "err", err)
|
||||
} else {
|
||||
st.Ended = ended
|
||||
}
|
||||
}
|
||||
}
|
||||
if want.GetNode() {
|
||||
if err := k.node(ctx, want, st); err != nil {
|
||||
return fail("slurmd", err)
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
|
||||
need := map[string][2]string{
|
||||
"munged": {"munge", "munge"},
|
||||
"scontrol": {"slurm-client", "slurm"},
|
||||
}
|
||||
if want.GetController() {
|
||||
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
|
||||
need["slurmdbd"] = [2]string{"slurmdbd", "slurm-slurmdbd"}
|
||||
need["mariadbd"] = [2]string{"mariadb-server", "mariadb-server"}
|
||||
}
|
||||
if want.GetNode() {
|
||||
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
|
||||
}
|
||||
sharedPackages(want, need)
|
||||
pkgs := map[string][]string{}
|
||||
for bin, p := range need {
|
||||
if k.installed[bin] {
|
||||
continue
|
||||
}
|
||||
if _, err := k.host.LookPath(bin); err == nil {
|
||||
k.installed[bin] = true
|
||||
continue
|
||||
}
|
||||
pkgs["deb"] = append(pkgs["deb"], p[0])
|
||||
pkgs["rpm"] = append(pkgs["rpm"], p[1])
|
||||
}
|
||||
if len(pkgs) == 0 {
|
||||
return nil
|
||||
}
|
||||
slices.Sort(pkgs["deb"])
|
||||
slices.Sort(pkgs["rpm"])
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
|
||||
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
|
||||
return err
|
||||
}
|
||||
if slices.Contains(pkgs["deb"], "mariadb-server") {
|
||||
// Installed by llmgw: its settings are llmgw's to keep. One that was
|
||||
// here before is another's, and is left as it is.
|
||||
if err := k.host.WriteFile(k.mariaDBMarker(), []byte("llmgw installed MariaDB here, for Slurm's accounting\n"),
|
||||
0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for bin := range need {
|
||||
if _, err := k.host.LookPath(bin); err != nil {
|
||||
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.installed[bin] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// munge writes the cluster's key and keeps munged running on it.
|
||||
func (k *Keeper) munge(ctx context.Context, key []byte) error {
|
||||
if len(key) < 32 {
|
||||
return errors.New("the leader sent no key")
|
||||
}
|
||||
changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("munge key written; munge starts again on it")
|
||||
return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
|
||||
}
|
||||
if !k.active(ctx, "munge") {
|
||||
return k.systemctl(ctx, "enable", "--now", "munge")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controller writes the files and keeps slurmctld running on them,
|
||||
// answering their digest.
|
||||
func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
|
||||
if files["slurm.conf"] == "" {
|
||||
return "", errors.New("the leader sent no slurm.conf")
|
||||
}
|
||||
for _, d := range []string{slurm.StateDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
names := make([]string, 0, len(files))
|
||||
for n := range files {
|
||||
names = append(names, n)
|
||||
}
|
||||
slices.Sort(names)
|
||||
h := sha256.New()
|
||||
changed := false
|
||||
for _, n := range names {
|
||||
if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
|
||||
return "", fmt.Errorf("the leader sent a file named %q", n)
|
||||
}
|
||||
fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
|
||||
c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
changed = changed || c
|
||||
}
|
||||
sum := hex.EncodeToString(h.Sum(nil))
|
||||
if changed || sum != k.files {
|
||||
if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Nodes added or taken away need a restart before 23.11 and a
|
||||
// reconfigure after it; a restart does both, and running jobs live
|
||||
// through it.
|
||||
k.log.Info("slurmctld starts again on new files")
|
||||
if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
if err := k.waitPing(ctx); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Hands the configless slurmd the new files.
|
||||
if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
|
||||
return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.files = sum
|
||||
return sum, nil
|
||||
}
|
||||
if !k.active(ctx, "slurmctld") {
|
||||
return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
|
||||
}
|
||||
return sum, nil
|
||||
}
|
||||
|
||||
func (k *Keeper) waitPing(ctx context.Context) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, "scontrol", "ping")
|
||||
if err == nil && bytes.Contains(out, []byte("UP")) {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
|
||||
}
|
||||
|
||||
var pingWait = time.Second
|
||||
|
||||
// node keeps slurmd running configless and puts the node in or out of the
|
||||
// controller's hands as lent says.
|
||||
func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
|
||||
name, server := want.GetName(), want.GetConfServer()
|
||||
if name == "" || server == "" {
|
||||
return errors.New("the leader sent no name or controller")
|
||||
}
|
||||
if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
|
||||
return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
|
||||
}
|
||||
for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
bin, err := k.host.LookPath("slurmd")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
|
||||
"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
|
||||
// --systemd as the packaged unit has it (Type=notify, since 22.05):
|
||||
// without it systemd waits for a readiness that never comes.
|
||||
"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
|
||||
// A node new to the controller is refused until the leader has
|
||||
// written it into slurm.conf; tried again until it is.
|
||||
"Restart=on-failure\nRestartSec=15\n"
|
||||
changed, err := k.write(DropIn, []byte(unit), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
|
||||
st.Hardware = strings.TrimSpace(string(out))
|
||||
}
|
||||
if changed {
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if !k.active(ctx, "slurmd") {
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
st.SlurmdRunning = k.active(ctx, "slurmd")
|
||||
|
||||
st.Jobs = k.localJobs(ctx)
|
||||
env := k.clientEnv()
|
||||
out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
|
||||
if err != nil {
|
||||
// Not in slurm.conf yet: the leader writes it once this node's
|
||||
// hardware is in.
|
||||
return nil
|
||||
}
|
||||
ns, err := slurm.ParseNodeState(string(out))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
|
||||
switch {
|
||||
case want.GetLent() && ns.Drained() && ours:
|
||||
k.log.Info("this node is lent to Slurm: resumed there")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
|
||||
return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
|
||||
k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
|
||||
"Reason="+ReasonServing); err != nil {
|
||||
return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
|
||||
if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
|
||||
ns = ns2
|
||||
}
|
||||
}
|
||||
st.State, st.Reason = ns.State, ns.Reason
|
||||
return nil
|
||||
}
|
||||
|
||||
// localJobs counts the job steps running here, from their slurmstepd, not
|
||||
// from the controller: a job outlives a controller that is down, and is
|
||||
// what decides whether llmgw may have the cards back. The one slurmstepd
|
||||
// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
|
||||
// is not counted. -1 when it cannot tell.
|
||||
func (k *Keeper) localJobs(ctx context.Context) int32 {
|
||||
// pgrep exits 1 with no output when there is none.
|
||||
out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
|
||||
if err != nil && len(bytes.TrimSpace(out)) > 0 {
|
||||
return -1
|
||||
}
|
||||
n := int32(0)
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 2 || f[len(f)-1] == "infinity" {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// clientEnv points scontrol at slurmd's copy of the files on a node that is
|
||||
// not the controller, which has none of its own.
|
||||
func (k *Keeper) clientEnv() []string {
|
||||
if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
|
||||
return nil
|
||||
}
|
||||
return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
|
||||
}
|
||||
|
||||
// stop stops what llmgw started here, if it did.
|
||||
func (k *Keeper) stop(ctx context.Context) {
|
||||
if _, err := k.host.ReadFile(k.marker); err != nil {
|
||||
return
|
||||
}
|
||||
k.log.Info("this node has no part in Slurm any more: slurmd, slurmctld and slurmdbd stopped; the packages and MariaDB stay")
|
||||
// MariaDB stays: it may have been the machine's before, and holds the
|
||||
// jobs' records.
|
||||
for _, u := range []string{"slurmd", "slurmctld", "slurmdbd"} {
|
||||
_ = k.systemctl(ctx, "disable", "--now", u)
|
||||
}
|
||||
_ = k.host.Remove(DropIn)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
k.unshare(ctx, "", false, "")
|
||||
if err := k.removeUsers(ctx, k.madeUsers(), nil); err != nil {
|
||||
k.log.Warn("could not take away every Slurm user llmgw made", "err", err)
|
||||
}
|
||||
_ = k.host.Remove(k.marker)
|
||||
k.files, k.acctDone, k.clusterAdded = "", "", ""
|
||||
}
|
||||
|
||||
// write writes a file when it differs, saying whether it did.
|
||||
func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
|
||||
if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
|
||||
return false, nil
|
||||
}
|
||||
return true, k.host.WriteFile(path, b, mode, owner)
|
||||
}
|
||||
|
||||
// systemctl runs systemctl; a last argument "&&restart" restarts the unit
|
||||
// after.
|
||||
func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
|
||||
restart := false
|
||||
if n := len(args); n > 0 && args[n-1] == "&&restart" {
|
||||
args, restart = args[:n-1], true
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
|
||||
return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
|
||||
}
|
||||
if restart {
|
||||
u := args[len(args)-1]
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
|
||||
return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (k *Keeper) active(ctx context.Context, unit string) bool {
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
|
||||
return err == nil && strings.TrimSpace(string(out)) == "active"
|
||||
}
|
||||
|
||||
func lastLine(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if i := strings.LastIndexByte(s, '\n'); i >= 0 {
|
||||
s = s[i+1:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Local is the machine the agent runs on.
|
||||
type Local struct{}
|
||||
|
||||
// Run runs a command, with env added to the agent's.
|
||||
func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
if len(env) > 0 {
|
||||
cmd.Env = append(os.Environ(), env...)
|
||||
}
|
||||
return cmd.CombinedOutput()
|
||||
}
|
||||
|
||||
// LookPath finds a command, in the sbin directories too: an agent started
|
||||
// by systemd has them, one started by hand may not.
|
||||
func (Local) LookPath(name string) (string, error) {
|
||||
if p, err := exec.LookPath(name); err == nil {
|
||||
return p, nil
|
||||
}
|
||||
// /usr/libexec: Red Hat's mariadbd.
|
||||
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin", "/usr/libexec"} {
|
||||
p := filepath.Join(d, name)
|
||||
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
|
||||
return p, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("%s is not installed", name)
|
||||
}
|
||||
|
||||
// ReadFile reads a file.
|
||||
func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
|
||||
|
||||
// WriteFile writes a file whole, through a temporary one beside it.
|
||||
func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
if err := os.WriteFile(tmp, b, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Chmod(tmp, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := chown(tmp, owner); err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// Download fetches url to path through a temporary file beside it, kept
|
||||
// only when its SHA-256 is sha.
|
||||
func (Local) Download(ctx context.Context, url, path, sha string) error {
|
||||
if f, err := os.Open(path); err == nil { //nolint:gosec // our paths
|
||||
h := sha256.New()
|
||||
_, cerr := io.Copy(h, f)
|
||||
_ = f.Close()
|
||||
if cerr == nil && hex.EncodeToString(h.Sum(nil)) == sha {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("fetching %s: %s", url, resp.Status)
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) //nolint:gosec // our paths
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
_, err = io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 1<<30))
|
||||
if cerr := f.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); got != sha {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("%s has SHA-256 %s, not the %s llmgw was built with: not used", url, got, sha)
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// MkdirAll makes a directory owned by owner.
|
||||
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(path, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
return chown(path, owner)
|
||||
}
|
||||
|
||||
// Remove removes a file, and is content when there is none.
|
||||
func (Local) Remove(path string) error {
|
||||
if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chown(path, owner string) error {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
u, err := user.Lookup(owner)
|
||||
if err != nil {
|
||||
return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
|
||||
}
|
||||
uid, _ := strconv.Atoi(u.Uid)
|
||||
gid, _ := strconv.Atoi(u.Gid)
|
||||
return os.Chown(path, uid, gid)
|
||||
}
|
||||
@@ -1,367 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
type fakeHost struct {
|
||||
files map[string][]byte
|
||||
owners map[string]string
|
||||
bins map[string]bool
|
||||
active map[string]bool
|
||||
ran []string
|
||||
outputs map[string]string
|
||||
fail map[string]bool
|
||||
state string
|
||||
reason string
|
||||
}
|
||||
|
||||
func newFake() *fakeHost {
|
||||
return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
|
||||
active: map[string]bool{}, outputs: map[string]string{
|
||||
"slurmd -V": "slurm-wlm 23.11.4\n",
|
||||
"slurmctld -V": "slurm-wlm 23.11.4\n",
|
||||
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
|
||||
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
|
||||
"hostname -s": "node06\n",
|
||||
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
|
||||
}
|
||||
|
||||
func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
c := strings.Join(append([]string{name}, args...), " ")
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] {
|
||||
return []byte("it failed"), errors.New("exit status 1")
|
||||
}
|
||||
switch {
|
||||
case name == "systemctl" && len(args) >= 1:
|
||||
u := args[len(args)-1]
|
||||
switch args[0] {
|
||||
case "is-active":
|
||||
if f.active[u] {
|
||||
return []byte("active\n"), nil
|
||||
}
|
||||
return []byte("inactive\n"), errors.New("exit status 3")
|
||||
case "restart", "start":
|
||||
f.active[u] = true
|
||||
case "enable":
|
||||
if slices.Contains(args, "--now") {
|
||||
f.active[u] = true
|
||||
}
|
||||
case "disable":
|
||||
f.active[u] = false
|
||||
}
|
||||
return []byte(f.outputs[c]), nil
|
||||
case strings.HasPrefix(c, "scontrol show node"):
|
||||
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
|
||||
case strings.HasPrefix(c, "scontrol update"):
|
||||
if slices.Contains(args, "State=RESUME") {
|
||||
f.state, f.reason = "IDLE", ""
|
||||
} else {
|
||||
f.state, f.reason = "IDLE+DRAIN", ReasonServing
|
||||
}
|
||||
return nil, nil
|
||||
case strings.HasPrefix(c, "pgrep"):
|
||||
out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
|
||||
return []byte(out), nil
|
||||
}
|
||||
return []byte(f.outputs[c]), nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) LookPath(name string) (string, error) {
|
||||
if f.bins[name] {
|
||||
return "/usr/sbin/" + name, nil
|
||||
}
|
||||
return "", errors.New("not found")
|
||||
}
|
||||
|
||||
func (f *fakeHost) ReadFile(path string) ([]byte, error) {
|
||||
b, ok := f.files[path]
|
||||
if !ok {
|
||||
return nil, fs.ErrNotExist
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
|
||||
f.files[path] = append([]byte(nil), b...)
|
||||
f.owners[path] = owner
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
|
||||
|
||||
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
|
||||
|
||||
func (f *fakeHost) Download(_ context.Context, url, path, sha string) error {
|
||||
c := "download " + url + " " + path + " " + sha
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] || f.fail["download "+url] {
|
||||
return errors.New("fetch failed")
|
||||
}
|
||||
f.files[path] = []byte("fetched " + url)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) count(prefix string) int {
|
||||
n := 0
|
||||
for _, c := range f.ran {
|
||||
if strings.HasPrefix(c, prefix) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
type fakeInst struct {
|
||||
host *fakeHost
|
||||
got map[string][]string
|
||||
}
|
||||
|
||||
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
|
||||
i.got = by
|
||||
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
i.host.bins[b] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var key = []byte(strings.Repeat("k", 1024))
|
||||
|
||||
func nodeMsg(lent bool) *llmgwv1.Slurm {
|
||||
return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
|
||||
}
|
||||
|
||||
func TestNodeInstallsAndStaysDrained(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
|
||||
!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
|
||||
t.Fatal("the munge key was not written for munge")
|
||||
}
|
||||
unit := string(h.files[DropIn])
|
||||
if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
|
||||
!strings.Contains(unit, "ConditionPathExists=\n") {
|
||||
t.Fatalf("drop-in:\n%s", unit)
|
||||
}
|
||||
if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Not lent: left drained, with llmgw's first reason.
|
||||
if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
|
||||
t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
|
||||
}
|
||||
if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
|
||||
t.Fatal("no marker")
|
||||
}
|
||||
|
||||
// Again, nothing changed: no install, no restart.
|
||||
h.ran = nil
|
||||
inst.got = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
|
||||
t.Fatalf("a second pass did work: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendAndReclaim(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
|
||||
t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
|
||||
}
|
||||
h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
|
||||
st = k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
|
||||
t.Fatalf("a node taken back was not drained: %v", h.ran)
|
||||
}
|
||||
if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Drained already: not drained again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("drained twice: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminsDrainIsLeft(t *testing.T) {
|
||||
h := newFake()
|
||||
h.reason = "bad DIMM, ticket 42"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
|
||||
t.Fatalf("env %v", env)
|
||||
}
|
||||
h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
|
||||
if env := k.clientEnv(); env != nil {
|
||||
t.Fatalf("env %v on the controller", env)
|
||||
}
|
||||
}
|
||||
|
||||
func TestController(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/m", nil)
|
||||
files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
|
||||
m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
|
||||
t.Fatal("slurm.conf not written")
|
||||
}
|
||||
if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
|
||||
t.Fatalf("slurmctld not started on its files: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Accounting: MariaDB on this machine alone, the database, slurmdbd on
|
||||
// it, the cluster added, all before slurmctld.
|
||||
if !slices.Contains(inst.got["deb"], "slurmdbd") || !slices.Contains(inst.got["deb"], "mariadb-server") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if !strings.Contains(string(h.files["/etc/mysql/mariadb.conf.d/90-llmgw-slurm.cnf"]), "bind-address=127.0.0.1") {
|
||||
t.Fatal("MariaDB, installed by llmgw, not kept to this machine")
|
||||
}
|
||||
dbd := string(h.files[slurm.DbdConfPath])
|
||||
if !strings.Contains(dbd, "DbdHost=node06\n") || !strings.Contains(dbd, "StoragePass="+slurm.DbdPassword(key)+"\n") ||
|
||||
h.owners[slurm.DbdConfPath] != "slurm" {
|
||||
t.Fatalf("slurmdbd.conf for %q:\n%s", h.owners[slurm.DbdConfPath], dbd)
|
||||
}
|
||||
if _, ok := h.files[acctSQLPath]; ok {
|
||||
t.Fatal("the database's statements, password and all, were left on disk")
|
||||
}
|
||||
if strings.Contains(strings.Join(h.ran, "\n"), slurm.DbdPassword(key)) {
|
||||
t.Fatal("the password was on a command line")
|
||||
}
|
||||
order := strings.Join(h.ran, "\n")
|
||||
if !strings.Contains(order, "sacctmgr -i add cluster llmgw") ||
|
||||
strings.Index(order, "slurmdbd") > strings.Index(order, "systemctl restart slurmctld") {
|
||||
t.Fatalf("accounting not before slurmctld: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
sum := st.GetFilesSha256()
|
||||
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart") != 0 {
|
||||
t.Fatalf("restarted on the same files: %v", h.ran)
|
||||
}
|
||||
|
||||
files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
|
||||
h.ran = nil
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
|
||||
t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
|
||||
}
|
||||
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
|
||||
t.Fatalf("a path in a file name was taken: %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTooOld(t *testing.T) {
|
||||
h := newFake()
|
||||
h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if !strings.Contains(st.GetError(), "older than 22.05") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if _, ok := h.files[slurm.MungeKeyPath]; ok {
|
||||
t.Fatal("went on with a Slurm too old")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStop(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
h.ran = nil
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
|
||||
t.Fatalf("status %v for no part", st)
|
||||
}
|
||||
if !slices.Contains(h.ran, "systemctl disable --now slurmd") {
|
||||
t.Fatalf("slurmd not stopped: %v", h.ran)
|
||||
}
|
||||
if _, ok := h.files[DropIn]; ok {
|
||||
t.Fatal("drop-in left")
|
||||
}
|
||||
// Never started here: nothing touched.
|
||||
h2 := newFake()
|
||||
k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
|
||||
k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
|
||||
if len(h2.ran) != 0 {
|
||||
t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBadNames(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Name = "cuda01 -c /etc/evil"
|
||||
if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMariaDBThatWasThereKeepsItsSettings(t *testing.T) {
|
||||
h := newFake()
|
||||
for _, b := range []string{"munged", "scontrol", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
h.bins[b] = true
|
||||
}
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "ClusterName=llmgw\n"}})
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
for p := range h.files {
|
||||
if strings.HasSuffix(p, ".cnf") {
|
||||
t.Fatalf("a MariaDB llmgw did not install had its settings changed: %s", p)
|
||||
}
|
||||
}
|
||||
if h.count("mariadb -e source") != 1 || !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("the database was not made in the MariaDB that was there: %v", h.ran)
|
||||
}
|
||||
}
|
||||
@@ -1,319 +0,0 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// usersMarker lists, a line each, the logins llmgw made here: the ones it
|
||||
// may change and take away. A user that was here before is never touched.
|
||||
func (k *Keeper) usersMarker() string { return k.marker + "-users" }
|
||||
|
||||
// users keeps the Linux users want has, the same login and UID as on every
|
||||
// other machine, their keys where sshd reads them, and their homes when
|
||||
// this machine makes them; and takes away those llmgw made that are gone.
|
||||
// Answers how many it has. A user that cannot be made is said, the others
|
||||
// are made anyway.
|
||||
func (k *Keeper) users(ctx context.Context, want *llmgwv1.Slurm) (int32, error) {
|
||||
made := k.madeUsers()
|
||||
if !want.GetManageUsers() {
|
||||
return 0, k.removeUsers(ctx, made, nil)
|
||||
}
|
||||
var errs []error
|
||||
var keep []string
|
||||
n := int32(0)
|
||||
for _, u := range want.GetUsers() {
|
||||
err := k.user(ctx, u, slices.Contains(made, u.GetLogin()), want.GetMakeHomes())
|
||||
if errors.Is(err, errNotMade) {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
keep = append(keep, u.GetLogin())
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
if err := k.keepMade(keep); err != nil {
|
||||
return n, err
|
||||
}
|
||||
if err := k.removeUsers(ctx, made, keep); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.sshConf(ctx, len(keep) > 0); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
// The controller is where they log in and submit from; a node only
|
||||
// while a job of theirs runs there.
|
||||
if err := k.nodeLogin(ctx, len(keep) > 0 && want.GetNode() && !want.GetController()); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
return n, errors.Join(errs...)
|
||||
}
|
||||
|
||||
// nodeLogin holds the users llmgw made here to their jobs when they log in
|
||||
// (pam_slurm_adopt), or no longer. Only they are: root and the site's
|
||||
// administrators log in as before, so a mistake here never locks out the
|
||||
// people who would put it right.
|
||||
func (k *Keeper) nodeLogin(ctx context.Context, on bool) error {
|
||||
old, err := k.host.ReadFile(slurm.PAMSSHD)
|
||||
if err != nil {
|
||||
if !on {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("reading %s: %w", slurm.PAMSSHD, err)
|
||||
}
|
||||
if on {
|
||||
if err := k.adoptModule(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
read := func(name string) (string, error) {
|
||||
b, err := k.host.ReadFile(filepath.Join(filepath.Dir(slurm.PAMSSHD), name))
|
||||
return string(b), err
|
||||
}
|
||||
stack, contained, err := slurm.PAMAdopt(string(old), k.usersMarker(), read, on)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if stack == string(old) {
|
||||
return nil
|
||||
}
|
||||
if on {
|
||||
k.log.Info("llmgw's Slurm users log in to this node only while a job of theirs runs here",
|
||||
"in_the_job", contained)
|
||||
if !contained {
|
||||
k.log.Warn("pam_systemd could not be skipped for them safely here: their ssh sessions are not held " +
|
||||
"to their job's cores and cards, only to having one")
|
||||
}
|
||||
} else {
|
||||
k.log.Info("llmgw's Slurm users are no longer held to their jobs when they log in here")
|
||||
}
|
||||
// Read by sshd for each login: nothing to reload.
|
||||
_, err = k.write(slurm.PAMSSHD, []byte(stack), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// adoptModule is there, or is installed.
|
||||
func (k *Keeper) adoptModule(ctx context.Context) error {
|
||||
if k.installed[slurm.AdoptModule] {
|
||||
return nil
|
||||
}
|
||||
find := func() bool {
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", filepath.Join(d, slurm.AdoptModule)); err == nil {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
if !find() {
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", slurm.AdoptModule)
|
||||
}
|
||||
k.log.Info("installing pam_slurm_adopt, which holds llmgw's Slurm users to their jobs here")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.AdoptPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if !find() {
|
||||
return fmt.Errorf("%s is still missing after installing %s", slurm.AdoptModule,
|
||||
strings.Join(slurm.AdoptPackages["deb"], ", "))
|
||||
}
|
||||
}
|
||||
k.installed[slurm.AdoptModule] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// errNotMade is a user llmgw could not make here, nor took for its own.
|
||||
var errNotMade = errors.New("not made")
|
||||
|
||||
type notMade struct{ msg string }
|
||||
|
||||
func (e notMade) Error() string { return e.msg }
|
||||
func (e notMade) Is(target error) bool { return target == errNotMade }
|
||||
|
||||
// user keeps one user; ours is whether llmgw made it here before.
|
||||
func (k *Keeper) user(ctx context.Context, u *llmgwv1.SlurmUser, ours, makeHome bool) error {
|
||||
login, uid, home := u.GetLogin(), u.GetUid(), u.GetHome()
|
||||
if !slurm.ValidLogin(login) || uid < slurm.MinUID || !filepath.IsAbs(home) || strings.Contains(home, "..") {
|
||||
return notMade{fmt.Sprintf("the leader sent a user %q, UID %d, home %q", login, uid, home)}
|
||||
}
|
||||
id := strconv.FormatUint(uint64(uid), 10)
|
||||
gecos := strings.Map(func(r rune) rune {
|
||||
if r == ':' || r == ',' || r == '\n' || r < ' ' {
|
||||
return -1
|
||||
}
|
||||
return r
|
||||
}, u.GetName())
|
||||
byName, okName := k.passwd(ctx, login)
|
||||
byID, okID := k.passwd(ctx, id)
|
||||
switch {
|
||||
case okName && byName[2] != id:
|
||||
return notMade{fmt.Sprintf("%s is already a user here with UID %s, not llmgw's %s", login, byName[2], id)}
|
||||
case !okName && okID:
|
||||
return notMade{fmt.Sprintf("UID %s is already %s's here, so %s is not made", id, byID[0], login)}
|
||||
case !okName:
|
||||
if _, ok := k.group(ctx, login); !ok {
|
||||
if out, err := k.host.Run(ctx, nil, "groupadd", "-g", id, login); err != nil {
|
||||
return notMade{fmt.Sprintf("groupadd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
}
|
||||
k.log.Info("a Slurm user is made here, as llmgw's accounts say", "login", login, "uid", uid)
|
||||
if out, err := k.host.Run(ctx, nil, "useradd", "-u", id, "-g", id, "-M", "-d", home, "-s", "/bin/bash",
|
||||
"-c", gecos, login); err != nil {
|
||||
return notMade{fmt.Sprintf("useradd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
case ours && (byName[4] != gecos || byName[5] != home):
|
||||
if out, err := k.host.Run(ctx, nil, "usermod", "-c", gecos, "-d", home, login); err != nil {
|
||||
return fmt.Errorf("usermod %s: %v: %s", login, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
keys := strings.Join(u.GetSshKeys(), "\n")
|
||||
file := filepath.Join(slurm.AuthorizedKeysDir, login)
|
||||
if keys == "" {
|
||||
_ = k.host.Remove(file)
|
||||
} else if _, err := k.write(file, []byte(keys+"\n"), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
if makeHome {
|
||||
return k.home(ctx, login, id, home)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// home makes a user's home directory, from /etc/skel, when it is not there.
|
||||
func (k *Keeper) home(ctx context.Context, login, id, home string) error {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", home); err == nil {
|
||||
return nil
|
||||
}
|
||||
if err := k.host.MkdirAll(filepath.Dir(home), 0o755, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
k.log.Info("a Slurm user's home is made", "login", login, "home", home)
|
||||
for _, c := range [][]string{
|
||||
{"install", "-d", "-m", "0700", "-o", id, "-g", id, home},
|
||||
{"cp", "-rT", "/etc/skel", home},
|
||||
{"chown", "-R", id + ":" + id, home},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("making %s's home: %s: %v: %s", login, c[0], err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// removeUsers takes away the users llmgw made here that are not in keep.
|
||||
// Their homes and files stay; their UID is never given to another.
|
||||
func (k *Keeper) removeUsers(ctx context.Context, made, keep []string) error {
|
||||
var errs []error
|
||||
left := slices.Clone(keep)
|
||||
for _, login := range made {
|
||||
if slices.Contains(keep, login) {
|
||||
continue
|
||||
}
|
||||
k.log.Info("a Slurm user llmgw made is taken away; its home stays", "login", login)
|
||||
_ = k.host.Remove(filepath.Join(slurm.AuthorizedKeysDir, login))
|
||||
if _, ok := k.passwd(ctx, login); ok {
|
||||
if out, err := k.host.Run(ctx, nil, "userdel", login); err != nil {
|
||||
// Still logged in, or running something: tried again.
|
||||
errs = append(errs, fmt.Errorf("userdel %s: %v: %s", login, err, lastLine(out)))
|
||||
left = append(left, login)
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
if !slices.Equal(left, made) {
|
||||
if err := k.keepMade(left); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
if len(left) == 0 {
|
||||
if err := k.sshConf(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.nodeLogin(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
func (k *Keeper) madeUsers() []string {
|
||||
b, err := k.host.ReadFile(k.usersMarker())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return strings.Fields(string(b))
|
||||
}
|
||||
|
||||
func (k *Keeper) keepMade(logins []string) error {
|
||||
s := slices.Clone(logins)
|
||||
slices.Sort(s)
|
||||
s = slices.Compact(s)
|
||||
if len(s) == 0 {
|
||||
return k.host.Remove(k.usersMarker())
|
||||
}
|
||||
_, err := k.write(k.usersMarker(), []byte(strings.Join(s, "\n")+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// sshConf has sshd read the users' keys from AuthorizedKeysDir, or no
|
||||
// longer.
|
||||
func (k *Keeper) sshConf(ctx context.Context, on bool) error {
|
||||
if !on {
|
||||
if _, err := k.host.ReadFile(slurm.SSHDropIn); err != nil {
|
||||
return nil
|
||||
}
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
changed, err := k.write(slurm.SSHDropIn, []byte(slurm.SSHConf), 0o644, "")
|
||||
if err != nil || !changed {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "sshd", "-t"); err != nil {
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return fmt.Errorf("sshd refuses llmgw's setting for the users' keys, so it is not kept: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.log.Info("sshd reads the Slurm users' keys from llmgw too", "dir", slurm.AuthorizedKeysDir)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
|
||||
// reloadSSH has sshd read its settings again, under Debian's name for it
|
||||
// or Red Hat's; one that is not running reads them when it starts.
|
||||
func (k *Keeper) reloadSSH(ctx context.Context) error {
|
||||
for _, u := range []string{"ssh", "sshd"} {
|
||||
if k.active(ctx, u) {
|
||||
return k.systemctl(ctx, "reload", u)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// passwd is getent's line for a user, by login or UID, in its fields.
|
||||
func (k *Keeper) passwd(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "passwd", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 7 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
|
||||
func (k *Keeper) group(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "group", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 3 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
@@ -54,13 +54,9 @@ func NewHTTPHealth(timeout time.Duration, opts ...HTTPHealthOption) *HTTPHealth
|
||||
|
||||
// Ready reports whether the engine at baseURL is serving model.
|
||||
func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model, apiKey string) error {
|
||||
var body healthBody
|
||||
if err := h.get(ctx, baseURL+h.healthPath, apiKey, &body); err != nil {
|
||||
if err := h.get(ctx, baseURL+h.healthPath, apiKey, nil); err != nil {
|
||||
return fmt.Errorf("health: %w", err)
|
||||
}
|
||||
if body.notReady() {
|
||||
return fmt.Errorf("health: the engine says %q", body.Status)
|
||||
}
|
||||
if !h.requireName || model == "" {
|
||||
return nil
|
||||
}
|
||||
@@ -102,29 +98,5 @@ func (h *HTTPHealth) get(ctx context.Context, url, apiKey string, out any) error
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
if hb, ok := out.(*healthBody); ok {
|
||||
// Most engines answer /health with nothing, or with text: only a
|
||||
// JSON status says anything, and a body that is not one is not a
|
||||
// failure.
|
||||
_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(hb)
|
||||
return nil
|
||||
}
|
||||
return json.NewDecoder(resp.Body).Decode(out)
|
||||
}
|
||||
|
||||
// healthBody is what an engine's /health may say in its body. Some answer
|
||||
// 200 from the moment their HTTP server is up and say in the body that the
|
||||
// model is still loading — FreeToken's {"status":"loading","phase":...} —
|
||||
// and a model that lists itself in /v1/models meanwhile would be routed to.
|
||||
type healthBody struct {
|
||||
Status string `json:"status"`
|
||||
}
|
||||
|
||||
// notReady is a body that says the engine cannot serve yet, or has failed.
|
||||
func (b healthBody) notReady() bool {
|
||||
switch b.Status {
|
||||
case "loading", "starting", "initializing", "warming_up", "error", "unhealthy":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -103,29 +103,3 @@ func TestTheHealthCheckSendsTheEnginesKey(t *testing.T) {
|
||||
t.Error("without the key the engine refused, and that read as ready")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotReadyWhileTheHealthBodySaysLoading(t *testing.T) {
|
||||
// FreeToken answers 200 from the moment its HTTP server is up and lists
|
||||
// its model meanwhile; only the body says the weights are still loading.
|
||||
body := `{"status":"loading","phase":"weights","progress":{"done_bytes":1,"total_bytes":9},"model":"glm"}`
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) { io.WriteString(w, body) })
|
||||
mux.HandleFunc("GET /v1/models", func(w http.ResponseWriter, _ *http.Request) {
|
||||
io.WriteString(w, `{"data":[{"id":"glm"}]}`)
|
||||
})
|
||||
srv := httptest.NewServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
h := NewHTTPHealth(time.Second)
|
||||
if err := h.Ready(t.Context(), srv.URL, "glm", ""); err == nil || !strings.Contains(err.Error(), "loading") {
|
||||
t.Errorf("a loading engine was reported ready, or not said to be loading: %v", err)
|
||||
}
|
||||
body = `{"status":"ok","model":"glm","uptime_s":3}`
|
||||
if err := h.Ready(t.Context(), srv.URL, "glm", ""); err != nil {
|
||||
t.Errorf("a serving engine was reported unready: %v", err)
|
||||
}
|
||||
// Text, as some engines answer: not a status, and not a failure.
|
||||
body = "OK"
|
||||
if err := h.Ready(t.Context(), srv.URL, "glm", ""); err != nil {
|
||||
t.Errorf("a plain-text health answer was taken for a failure: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"bytes"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
@@ -64,71 +63,3 @@ func isZombie(pid int) bool {
|
||||
}
|
||||
return b[i+2] == 'Z'
|
||||
}
|
||||
|
||||
// strays are a process's descendants that left its group, as an engine that
|
||||
// starts its worker in a session of its own does (Strata): signalling the
|
||||
// group misses them, and one left behind holds the card and the RAM the next
|
||||
// replica needs. Found before the group is signalled, while they are still
|
||||
// its descendants: after, they belong to init. Linux only; elsewhere none.
|
||||
func strays(pid int) []int {
|
||||
entries, err := os.ReadDir("/proc")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
parent := map[int]int{}
|
||||
group := map[int]int{}
|
||||
for _, e := range entries {
|
||||
p, err := strconv.Atoi(e.Name())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if pp, pg, ok := parentAndGroup(p); ok {
|
||||
parent[p], group[p] = pp, pg
|
||||
}
|
||||
}
|
||||
var out []int
|
||||
for p := range parent {
|
||||
if group[p] == pid || p == pid {
|
||||
continue
|
||||
}
|
||||
// Bounded: no engine's tree is this deep, and a pid reused while
|
||||
// /proc was read must not loop.
|
||||
for q, n := parent[p], 0; q > 1 && n < 32; q, n = parent[q], n+1 {
|
||||
if q == pid {
|
||||
out = append(out, p)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// parentAndGroup reads a process's parent and group from /proc.
|
||||
func parentAndGroup(pid int) (int, int, bool) {
|
||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||
if err != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
i := bytes.LastIndexByte(b, ')')
|
||||
if i < 0 {
|
||||
return 0, 0, false
|
||||
}
|
||||
f := strings.Fields(string(b[i+1:]))
|
||||
if len(f) < 3 {
|
||||
return 0, 0, false
|
||||
}
|
||||
pp, err1 := strconv.Atoi(f[1])
|
||||
pg, err2 := strconv.Atoi(f[2])
|
||||
return pp, pg, err1 == nil && err2 == nil
|
||||
}
|
||||
|
||||
// signalStrays signals each stray, and its own group when it leads one.
|
||||
func signalStrays(pids []int, sig syscall.Signal) {
|
||||
for _, p := range pids {
|
||||
if _, pg, ok := parentAndGroup(p); ok && pg == p {
|
||||
_ = syscall.Kill(-p, sig)
|
||||
continue
|
||||
}
|
||||
_ = syscall.Kill(p, sig)
|
||||
}
|
||||
}
|
||||
@@ -582,24 +582,10 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
|
||||
}
|
||||
|
||||
// Signal the group, not the process: engines fork workers that hold GPU
|
||||
// memory, and those outlive a signal sent only to the leader. And what
|
||||
// left the group, found while it is still the leader's.
|
||||
left := strays(pid)
|
||||
// memory, and those outlive a signal sent only to the leader.
|
||||
if err := killGroup(pid, syscall.SIGTERM); err != nil && !errors.Is(err, os.ErrProcessDone) {
|
||||
s.log.Warn("SIGTERM failed", "shard", id, "err", err)
|
||||
}
|
||||
signalStrays(left, syscall.SIGTERM)
|
||||
defer func() {
|
||||
// Given as long as the group to go; then killed.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) && slices.ContainsFunc(left, processAlive) {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
if slices.ContainsFunc(left, processAlive) {
|
||||
s.log.Warn("killing an engine's processes that left its group", "shard", id, "pids", left)
|
||||
signalStrays(left, syscall.SIGKILL)
|
||||
}
|
||||
}()
|
||||
|
||||
// force kills the group outright. exec.Cmd's cancel does this for shards we
|
||||
// started; an adopted shard has none, so signal its group directly.
|
||||
@@ -612,7 +598,6 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
|
||||
if err := killGroup(pid, syscall.SIGKILL); err != nil && !errors.Is(err, os.ErrProcessDone) {
|
||||
s.log.Warn("SIGKILL failed", "shard", id, "err", err)
|
||||
}
|
||||
signalStrays(left, syscall.SIGKILL)
|
||||
}
|
||||
|
||||
select {
|
||||
|
||||
Loaded 100 of 577 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user