Compare commits

..
22 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 2735c5a318 Mirror a many-file model whole, with its checksums, for vLLM
CI / test (push) Successful in 3m58s
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
The explorer fetches a safetensors set with its config and tokenizer;
the mirror gives out a version's manifest only once nothing is
downloading and every shard its index names is there, with each file's
sha256, which nodes check; mirror add takes several URLs; a repository
page is refused as a source.

Refs #2 (item 9).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:21:04 +02:00
Ilker ManapandClaude Opus 5.5 8957db30a2 Recognise a duplicate sign-up on PostgreSQL, and say what to do
A taken address on PostgreSQL fell through to a generic error, because
the store looked for SQLite's "UNIQUE". It checks code 23505 too. The
done page tells someone whose address had an account to sign in or ask
for a reset, without saying which case it was; other failures give a
logged reference.

Refs #2 (item 6).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:15:37 +02:00
Ilker ManapandClaude Opus 5.5 2ef4220643 Fail a node only for the fabrics it declares
A RoCE port without a v2 GID and a missing ib_uverbs failed doctor, and
node add, whatever fabric used them. They are now failures only for a
declared fabric and warnings otherwise; node add passes its --fabric
list to doctor so those are checked as declared.

Refs #2 (item 5).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:13:18 +02:00
Ilker ManapandClaude Opus 5.5 6014d1fee8 Say how a restart with no room beside it goes
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:11:46 +02:00
Ilker ManapandClaude Opus 5.5 a3ee48aced Restart in place at once when only the old replica is in the way
A restart with no room beside it waited the drain grace period before
stopping the old replica, and stopped it even when the replacement
would fit nowhere. It now stops at once when the replica alone blocks
its replacement, gives the restart up (saying why) when nothing would
fit, and ctl restart --in-place stops first on request.

Refs #2 (item 3).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:11:37 +02:00
Ilker ManapandClaude Opus 5.5 e313f32696 Decide engines per node, and let a node join without one it cannot run
engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.

Refs #2 (items 4 and 7).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:07:25 +02:00
Ilker ManapandClaude Opus 5.5 cba3e329fc Let the mirror's serve form place a model on a GPU
The form offered no placement and named every GGUF profile cpu. It now
lists the fleet's GPU classes (the freest chosen) and the CPU, a GPU
count and parallel requests, names the profile from the placement, shows
what the model will run as, and refuses the CPU while GPUs are free
unless that is ticked.

Refs #2 (item 1).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:59:22 +02:00
Ilker ManapandClaude Opus 5.5 cbd5521f4f Place llama.cpp across several GPUs of a node with gpus_per_node
tp on a llama.cpp profile was a width nothing placed. gpus_per_node is
now the count: placement reserves that many cards on one node and the
engine splits layers across them. tp, pp and nodes are refused with
what to write instead.

Refs #2 (item 8).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:56:34 +02:00
Ilker ManapandClaude Opus 5.5 057be16a92 Do not call a GPU shard ready while it computes on the CPU
llama.cpp whose CUDA backend cannot load runs on the CPU without an
error. The CUDA build is now asked for its devices when installed, a
shard on GPUs must hold memory on each before it is ready, and
engine-check reports the devices. Shards get CUDA_DEVICE_ORDER=PCI_BUS_ID
so CUDA numbers GPUs as placement does.

Refs #2 (item 2).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:54:39 +02:00
Ilker ManapandClaude Opus 5.5 0c7934ba96 v0.10.6
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m51s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 08:52:01 +02:00
Ilker ManapandClaude Opus 5.5 e40b9ac202 Load LoRA adapters at scale zero
Release / build (push) Successful in 1m17s
CI / test (push) Successful in 4m0s
Release / notes (push) Successful in 5s
--lora with --lora-init-without-apply leaves an adapter's scale at 1,
and llama-server applies it at that scale to any request without a
"lora" list of its own: every plain request for the model was served
with its adapters on. Found live on qwen2.5-coder-7b-instruct. Each
adapter is now loaded with --lora-scaled FILE:0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:39:12 +02:00
Ilker ManapandClaude Opus 5.5 323d172538 Attach LoRA adapters from the command line
Release / build (push) Successful in 1m14s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m52s
llmgw ctl model adapters lists a model's adapters and adds or removes
them through POST /api/v1/models/{name}/adapters, which reads each new
file first and refuses one that is not an adapter for the model's
architecture, as the mirror page's card does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:30:13 +02:00
Ilker ManapandClaude Opus 5.5 6b1fa65f59 Format kind_live_test.go
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:20:35 +02:00
Ilker ManapandClaude Opus 5.5 663770226e Find LoRA adapters on Hugging Face and attach them from the mirror page
CI / test (push) Failing after 33s
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:19:08 +02:00
Ilker ManapandClaude Opus 5.5 cf53904cc5 Serve LoRA adapters on llama.cpp models, asked for by name
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m11s
CI / test (push) Successful in 3m47s
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:40:47 +02:00
Ilker ManapandClaude Opus 5.5 94eec557ad Serve a model from an external OpenAI-compatible provider
CI / test (push) Successful in 3m47s
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:29:01 +02:00
Ilker ManapandClaude Opus 5.5 d5f02c0a4b Spill a saturated pool's requests to the next pool
CI / test (push) Successful in 3m40s
The pool-priority filter kept the highest-priority pool however full it
was. It now takes the first pool, by priority, that is not saturated by
admission's own measure, and the preferred one when all are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:23:32 +02:00
Ilker ManapandClaude Opus 5.5 3d3c96e422 Displace the newest lowest-band request when the gateway queue is full
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m9s
CI / test (push) Successful in 3m50s
A total bound across bands (admission.queue_capacity_total, default half of
all bands' capacity together); reached, a request is not refused while a
lower band has requests queued, and the newest of the lowest band gets a 503
with Retry-After instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:07:53 +02:00
Ilker ManapandClaude Opus 5.5 623cdb5ba3 Sign releases, and install only signed ones once a key is trusted
CI / test (push) Successful in 4m6s
ed25519 over checksums.txt: llmgw release keygen makes the pair, CI signs
with the RELEASE_SIGNING_KEY secret when it is set, and release.Install
refuses an unsigned or wrongly signed release when the build embeds a
trusted key (internal/release/trusted.pub, empty for now) or
LLMGW_RELEASE_KEY names one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:03:48 +02:00
Ilker ManapandClaude Opus 5.5 561667c9be Name only the replicas that would make room in headroom
CI / test (push) Successful in 3m58s
Candidates tried on the way to finding no room were listed as whose replicas
a placement would stop.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:01:40 +02:00
Ilker ManapandClaude Opus 5.5 cf0178bf05 Test that a restart borrows a lower class's room and returns it
CI / test (push) Successful in 3m41s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:47:12 +02:00
Ilker ManapandClaude Opus 5.5 f288f3c423 Set capacity quotas, class, owner and reservation from the admin pages
Release / build (push) Successful in 1m9s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m49s
The quotas page gains a capacity section over the same API methods the CLI
uses; a model's card sets its class, owner, memory and cores, applying only
what changed. Headroom counts the replicas a model could get by preempting
what its class outranks, and the card and ctl model target use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:40:45 +02:00
105 changed files with 5609 additions and 317 deletions

No files matched your search

+18
View File
@@ -93,6 +93,24 @@ jobs:
./dist/llmgw-${version}-linux-amd64 --version
cat dist/checksums.txt
# Signed when the repository has a signing key, so a machine that trusts
# it installs nothing the server alone could have put there. Without the
# secret nothing is signed and releases are checked on their checksums,
# as before; a build that trusts a key refuses those, so the secret has
# to be in place before trusted.pub is filled in.
- name: Sign the checksums
env:
RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
run: |
set -euo pipefail
version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
exit 0
fi
./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
echo "signed: dist/checksums.txt.sig"
- name: Attach to the release
env:
TOKEN: ${{ secrets.GITEA_TOKEN }}
+269
View File
@@ -20,6 +20,275 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
From the field report in issue #2, after a deployment on GPU nodes.
### A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:
- **Installing** the CUDA build asks it which devices it sees
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
own `ldd` could not say it: the backend is opened at run time.
- **Starting** a shard on GPUs is ready only when its process holds memory on
each of them (`nvidia-smi --query-compute-apps`, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.
### A model of many files, mirrored whole for vLLM
vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.
- **The explorer fetches a safetensors set with what the engine reads beside
it**: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a `.py` would be the engine
running what the repository ships, which is to be chosen on purpose.
- **A version is offered to nodes only whole.** The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its `model.safetensors.index.json` names, or a part of a
split (`-00002-of-00004`), is absent. A node reports "not complete on the
mirror yet" and is tried again.
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
- **`llmgw mirror add` takes several URLs** into one version.
- **A Hugging Face repository page is refused as a source**, saying to mirror
it or to give the repository id for vLLM to fetch itself.
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
such a model, as before.
### Signing up with an address that has an account
On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.
An error the page does not explain says the account was not created and
gives a reference (`err-…`), logged beside the error, for an operator to find
it; the database's own words stay in the log.
### An adapter no fabric uses does not keep a node out
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.
- Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
- Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; `node add` still warns
once that adapters are up with no fabric given.
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
written afterwards.
### A restart with no room beside it
`ctl restart` places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:
- **The replica is the only thing in the way**, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
- **The replacement fits nowhere even then**: the restart is given up and the
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
- **Room being made** — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
### Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly: `node add` refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and `--ignore-checks` passed over every other problem with it.
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
node joins without it and serves what it does run. `node add` checks the
vLLM the control plane installs, not the binary's own default.
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
or labels a different version of an engine, or none: a CUDA 12 vLLM on
Volta beside a CUDA 13 one on newer cards.
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
the threshold had been CUDA 13's for every version, so the doctor's advice
to set an older version could not work. A refusal now says which versions
would.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
### Serving from the mirror says where a model will run
The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by `gpu_class`, which the form never
set. Now:
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
- **GPUs per replica** sets how many cards a replica takes on one node
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
how many a llama.cpp replica serves at once.
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
typed.
- A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
anyway** is ticked.
### llama.cpp on several GPUs of one node
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
placement reserves N cards of its `gpu_class` on one node, the engine sees
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
(`--split-mode layer`, in proportion to each card's free memory, which for
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card — and is now refused
when the model is added, saying to write `gpus_per_node` instead; so is
`nodes` above one.
## v0.10.6 — 2026-10-01
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
format check and has no binaries.
Tried on the PoC cluster before release: LoRA adapters asked for by name
through the gateway, two on one model, each answering as trained and the model
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
requests on the preferred replica, 4 on the other); and a bronze request
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
live: a model served from an external provider, which needs a provider key.
### Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks as `llmgw ctl quota`.
- **A model's card** on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said.
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
### LoRA adapters
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (`--lora-scaled FILE:0`); a request that names an
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in `/v1/models`, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.
### Finding and attaching adapters from the mirror page
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
held model's card, *find adapters on Hugging Face* — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original.
- **Opening a repository** reads the first bytes of each GGUF file and marks it
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says.
- **Held adapter files** are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog.
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
lists and changes them from the command line, through
`POST /api/v1/models/{name}/adapters`, with the same checks.
### Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
model, api_key_file}` and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model's `fallbacks`, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md). `ctl models` and the cluster page mark it
external.
### Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.
### Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it. `admission.queue_capacity_total` now bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted as `displaced` in the admission rejection metric.
### Releases can be signed
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
public key in `internal/release/trusted.pub`, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.
### Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
## v0.10.5 — 2026-09-30
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
+96 -26
View File
@@ -2873,8 +2873,11 @@ type EngineProfile struct {
Env map[string]string `protobuf:"bytes,9,rep,name=env,proto3" json:"env,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
ModelPath string `protobuf:"bytes,10,opt,name=model_path,json=modelPath,proto3" json:"model_path,omitempty"`
ServedModelName string `protobuf:"bytes,11,opt,name=served_model_name,json=servedModelName,proto3" json:"served_model_name,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
// adapters are LoRA adapters loaded beside the model, in order: the index
// is the id a request applies one by.
Adapters []*LoraAdapter `protobuf:"bytes,12,rep,name=adapters,proto3" json:"adapters,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *EngineProfile) Reset() {
@@ -2984,6 +2987,13 @@ func (x *EngineProfile) GetServedModelName() string {
return ""
}
func (x *EngineProfile) GetAdapters() []*LoraAdapter {
if x != nil {
return x.Adapters
}
return nil
}
// RemoveWeights deletes cached model files a node no longer needs.
//
// The paths are computed by the control plane from the catalog, but the agent
@@ -3777,6 +3787,60 @@ func (x *UpgradeReply) GetError() string {
return ""
}
// LoraAdapter is one adapter: the name clients ask for it by, and its source,
// which the agent fetches as it does the model's weights.
type LoraAdapter struct {
state protoimpl.MessageState `protogen:"open.v1"`
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
Source string `protobuf:"bytes,2,opt,name=source,proto3" json:"source,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *LoraAdapter) Reset() {
*x = LoraAdapter{}
mi := &file_llmgw_v1_agent_proto_msgTypes[36]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *LoraAdapter) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*LoraAdapter) ProtoMessage() {}
func (x *LoraAdapter) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[36]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use LoraAdapter.ProtoReflect.Descriptor instead.
func (*LoraAdapter) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{36}
}
func (x *LoraAdapter) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *LoraAdapter) GetSource() string {
if x != nil {
return x.Source
}
return ""
}
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
const file_llmgw_v1_agent_proto_rawDesc = "" +
@@ -4066,7 +4130,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" +
"\vcpu_threads\x18\x11 \x01(\x05R\n" +
"cpuThreads\"\xfd\x02\n" +
"cpuThreads\"\xb0\x03\n" +
"\rEngineProfile\x12\x12\n" +
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
@@ -4080,7 +4144,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\n" +
"model_path\x18\n" +
" \x01(\tR\tmodelPath\x12*\n" +
"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x1a6\n" +
"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x121\n" +
"\badapters\x18\f \x03(\v2\x15.llmgw.v1.LoraAdapterR\badapters\x1a6\n" +
"\bEnvEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"%\n" +
@@ -4143,7 +4208,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\n" +
"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x14\n" +
"\x05error\x18\x03 \x01(\tR\x05error2O\n" +
"\x05error\x18\x03 \x01(\tR\x05error\"9\n" +
"\vLoraAdapter\x12\x12\n" +
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
"\x06source\x18\x02 \x01(\tR\x06source2O\n" +
"\fAgentService\x12?\n" +
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
"\fcom.llmgw.v1B\n" +
@@ -4162,7 +4230,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
}
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 42)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 43)
var file_llmgw_v1_agent_proto_goTypes = []any{
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
@@ -4201,12 +4269,13 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
(*FabricTestReply)(nil), // 34: llmgw.v1.FabricTestReply
(*UpgradeAgent)(nil), // 35: llmgw.v1.UpgradeAgent
(*UpgradeReply)(nil), // 36: llmgw.v1.UpgradeReply
nil, // 37: llmgw.v1.Register.LabelsEntry
nil, // 38: llmgw.v1.Register.EngineVersionsEntry
nil, // 39: llmgw.v1.GPU.NicPathsEntry
nil, // 40: llmgw.v1.IBPort.CountersEntry
nil, // 41: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 42: llmgw.v1.EngineProfile.EnvEntry
(*LoraAdapter)(nil), // 37: llmgw.v1.LoraAdapter
nil, // 38: llmgw.v1.Register.LabelsEntry
nil, // 39: llmgw.v1.Register.EngineVersionsEntry
nil, // 40: llmgw.v1.GPU.NicPathsEntry
nil, // 41: llmgw.v1.IBPort.CountersEntry
nil, // 42: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 43: llmgw.v1.EngineProfile.EnvEntry
}
var file_llmgw_v1_agent_proto_depIdxs = []int32{
3, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -4226,24 +4295,24 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
31, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
33, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
35, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
37, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
38, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
7, // 18: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
8, // 19: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
38, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
39, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
6, // 21: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
9, // 22: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
5, // 23: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
4, // 24: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
15, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 26: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
39, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
40, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
12, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
11, // 29: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
40, // 30: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
41, // 30: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
14, // 31: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
5, // 32: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
9, // 33: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
41, // 34: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
42, // 34: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
20, // 35: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
10, // 36: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
13, // 37: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
@@ -4254,14 +4323,15 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
0, // 42: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
22, // 43: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
24, // 44: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
42, // 45: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
1, // 46: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
2, // 47: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
47, // [47:48] is the sub-list for method output_type
46, // [46:47] is the sub-list for method input_type
46, // [46:46] is the sub-list for extension type_name
46, // [46:46] is the sub-list for extension extendee
0, // [0:46] is the sub-list for field type_name
43, // 45: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
37, // 46: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
1, // 47: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
2, // 48: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
48, // [48:49] is the sub-list for method output_type
47, // [47:48] is the sub-list for method input_type
47, // [47:47] is the sub-list for extension type_name
47, // [47:47] is the sub-list for extension extendee
0, // [0:47] is the sub-list for field type_name
}
func init() { file_llmgw_v1_agent_proto_init() }
@@ -4296,7 +4366,7 @@ func file_llmgw_v1_agent_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
NumEnums: 1,
NumMessages: 42,
NumMessages: 43,
NumExtensions: 0,
NumServices: 1,
},
+10
View File
@@ -572,6 +572,9 @@ message EngineProfile {
map<string, string> env = 9;
string model_path = 10;
string served_model_name = 11;
// adapters are LoRA adapters loaded beside the model, in order: the index
// is the id a request applies one by.
repeated LoraAdapter adapters = 12;
}
// RemoveWeights deletes cached model files a node no longer needs.
@@ -704,3 +707,10 @@ message UpgradeReply {
string version = 2;
string error = 3;
}
// LoraAdapter is one adapter: the name clients ask for it by, and its source,
// which the agent fetches as it does the model's weights.
message LoraAdapter {
string name = 1;
string source = 2;
}
+140 -8
View File
@@ -376,8 +376,10 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
vLLM is built for CUDA 13. A V100 cannot run it;
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
Volta (7.0);
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -394,8 +396,38 @@ engines:
```
A changed version reaches every node when it next connects. To use an engine
you installed yourself instead, set its version to `""` and name its path in
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
then installs none of its own beside it.
#### Different GPUs, different engines
A cluster of two GPU generations may need two builds of an engine, or one on
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
id, or by labels, every one given having to match — and lays engines over the
cluster's for them, in order:
```yaml
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
A node told nothing it can run of an engine joins without it: `llmgw doctor`
and `node add` warn, and the node serves what it does run. To keep an engine
off a node for good, whatever the control plane says, add the node with
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
than `--ignore-checks`, which passes over every other problem as well.
A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
asks it which devices it sees, and a replica placed on GPUs is not ready until
its process holds memory on each of them. A build missing a CUDA library fails
there, naming the library, rather than serving on the CPU.
After changing a version, check it still takes the flags llmgw builds for it,
which drift between releases:
@@ -671,6 +703,74 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
which names every candidate and why each was rejected, rather than leaving you
to infer it.
### LoRA adapters
A llama.cpp model can carry LoRA adapters, each asked for by its own name as
though it were a model:
```yaml
- name: qwen2.5-coder-7b-instruct
source: http://mirror:8081/qwen2.5-coder-7b-instruct/5d567d26/model.gguf
adapters:
- name: coder-sql
source: http://mirror:8081/coder-sql/v1/adapter.gguf
scale: 1.0 # optional; 1 when left out
profiles: [...]
```
Every replica of the model fetches every adapter and loads it at start, applied
to nothing. A request for `coder-sql` goes to the model's replicas with the
adapter applied to it alone; a request for the model gets the model. Adapter
names share the namespace of models and aliases, appear in `/v1/models`, and
follow key allowlists on their own: a key allowed `coder-sql` reaches that
adapter and not the model under it, while a key allowed the model reaches its
adapters too. Adding or removing an adapter is a change to the model; running
replicas load the new set when restarted (`llmgw ctl restart --model M`).
Besides the mirror page's model card, adapters are attached from the command
line, which reads the file first and refuses one that is not an adapter for
this model's architecture:
llmgw ctl model adapters coder --add coder-sql=http://mirror:8081/sql-lora/v1/sql.gguf@0.8
llmgw ctl model adapters coder --remove coder-sql
The adapter file must be a GGUF LoRA made for this model — llama.cpp's
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
another gateway — rather than replicas here. Nothing is placed for it; the
router sends its requests on, with the provider's key and the provider's name
for the model:
```yaml
- name: cloud-mini
source: external
external:
url: https://api.openai.com # without /v1
model: gpt-4o-mini # what the provider calls it
api_key_file: /etc/llmgw/secrets/openai.key
```
Named in another model's `fallbacks`, it is used only when the cluster has no
capacity left for that model:
```yaml
- name: qwen2.5-coder-7b-instruct
fallbacks: [cloud-mini]
...
```
The key is read from its file on the control plane, never stored in the
catalog, and a key rotated in the file is used within seconds. A model whose key
file cannot be read is not offered. Everything else applies as to any model:
prices and budgets, rate limits, usage records, and key allowlists.
**Its prompts leave the cluster.** Decide who may send them: a key or project
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
@@ -964,7 +1064,39 @@ A replica may come back marked **`ready (stale profile)`**. That means it is
running a profile that has since been edited — the reconciler leaves a serving
replica alone on purpose, because churning one costs more than the difference.
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
the new one before stopping the old.
the new one before stopping the old. Where the old one is all that stands in
the new one's way — a single GPU — it is stopped first, at once, and the model
is unserved while the new one loads; where the new profile fits nowhere even
then, the restart is refused and the old one keeps serving, which `ctl
replicas` shows as `restart refused:` with the reason. `--in-place` asks for
stop-then-start whatever room there is.
### Signed releases
Every install above checks a release against its published checksums, which
proves the download is what the server holds — not that the server holds what
was built. Signing closes that: CI signs the checksums, and a binary that
trusts a key installs only releases signed with it. Off until a key exists;
turning it on, once:
```
llmgw release keygen
```
1. Put the **private** key in the repository's Actions secret
`RELEASE_SIGNING_KEY`. From the next tag, releases carry
`checksums.txt.sig`.
2. Put the **public** key in `internal/release/trusted.pub` and release. Every
binary built from then on carries it and refuses releases that are not
signed with it — `ctl upgrade`, agents upgrading themselves, and
`llmgw upgrade` alike.
In that order: a build that trusts a key refuses unsigned releases, so the
secret has to be signing before the key is trusted. The first trusting release
is itself installed by a binary that trusts nothing; every one after is checked
by the one before. `LLMGW_RELEASE_KEY` overrides the built-in key, for rotating
it. The onboarding script fetches with `curl` and checks checksums only; a node
is checked from its first upgrade on.
## Backing up
@@ -1121,9 +1253,9 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
install a vLLM built for your CUDA yourself.
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
(step 4), or install one built for your CUDA yourself.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
+124
View File
@@ -15,12 +15,16 @@ rather than repeated.
| Key | Type | In |
|---|---|---|
| `name` | `string` | [Adapter](#adapter) |
| `source` | `string` | [Adapter](#adapter) |
| `scale` | `float64` | [Adapter](#adapter) |
| `enabled` | `bool` | [Admission](#admission) |
| `bands` | `[]Band` | [Admission](#admission) |
| `default_band` | `string` | [Admission](#admission) |
| `band_header` | `string` | [Admission](#admission) |
| `queue_ttl` | `time.Duration` | [Admission](#admission) |
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
| `queue_capacity_total` | `int` | [Admission](#admission) |
| `interval` | `time.Duration` | [Admission](#admission) |
| `enabled` | `bool` | [Auth](#auth) |
| `store` | `string` | [Auth](#auth) |
@@ -77,6 +81,13 @@ rather than repeated.
| `pool` | `string` | [Endpoint](#endpoint) |
| `weight` | `float64` | [Endpoint](#endpoint) |
| `priority` | `int` | [Endpoint](#endpoint) |
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
| `contact` | `string` | [Landing](#landing) |
| `notice` | `string` | [Landing](#landing) |
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
@@ -94,6 +105,8 @@ rather than repeated.
| `scaling` | `Scaling` | [Model](#model) |
| `class` | `string` | [Model](#model) |
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
| `adapters` | `[]Adapter` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
@@ -174,6 +187,7 @@ rather than repeated.
| `tracing` | `Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
@@ -195,6 +209,27 @@ rather than repeated.
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
## Adapter
### `name`
`string`
Name is what clients ask for, like a model name.
### `source`
`string`
Source is the adapter file (GGUF for llama.cpp), fetched by each node
as the model's weights are.
### `scale`
`float64`
Scale is how strongly it is applied; zero takes 1.
## Admission
### `enabled`
@@ -228,6 +263,15 @@ should not be able to promote themselves.
`int`
### `queue_capacity_total`
`int`
QueueCapacityTotal bounds everything queued across bands. Reached, a
request is not refused while a lower band has any queued: the newest of
the lowest band is turned away instead, with a 503 and Retry-After.
Zero takes half of all the bands' capacities together.
### `interval`
`time.Duration`
@@ -606,6 +650,54 @@ called. Empty means vLLM's names.
`int`
## EngineOverride
### `gpu_class`
`string`
GPUClass picks nodes with a GPU of this class ("v100-32g").
### `nodes`
`[]string`
Nodes picks nodes by id.
### `labels`
`map[string]string`
Labels picks nodes carrying all of these.
### `engines`
`map[string]string`
Engines are laid over the cluster's: a version, or "" for none.
## External
### `url`
`string`
URL is the provider's base address, without /v1: https://api.openai.com.
### `model`
`string`
Model is what the provider calls it; requests are rewritten to it.
### `api_key_file`
`string`
APIKeyFile holds the provider's key, on the control plane. A file
rather than the key itself, so the key is in neither the catalog, the
store nor the audit log; empty for a provider that asks for none.
## Landing
### `contact`
@@ -755,6 +847,28 @@ Owner is the part of the tenancy tree whose capacity quota this model
counts against, written org:ID, team:ID or project:ID. Empty is shared:
held by the cluster, counted against nobody.
### `external`
`*External`
External is a model served by someone else — an OpenAI-compatible API
outside the cluster — rather than by replicas placed here. Such a model
has no profiles and nothing is placed for it; the router sends its
requests to the provider. Most useful named in another model's
fallbacks: used only when the cluster has no capacity left. Its
prompts leave the cluster, so which keys may name it is worth an
allowlist.
Takes a [External](#external) block.
### `adapters`
`[]Adapter`
Adapters are LoRA adapters served on this model's replicas, each asked
for by its own name as though it were a model. Every replica loads
every adapter, applying one only to a request that names it.
### `fallbacks`
`[]string`
@@ -1329,6 +1443,16 @@ never trusted to already have what somebody once put there.
Unset keeps DefaultEngines. A version replaces the default; an empty
one leaves that engine to be installed by hand.
### `engine_overrides`
`[]EngineOverride`
EngineOverrides give some nodes different engines from the rest: a
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
all on nodes meant only for llama.cpp. Each picks nodes and lays its
engines over the cluster's, in order, later ones last; an empty version
removes that engine from the nodes it picks.
### `backup_dir`
`string`
+20 -1
View File
@@ -450,7 +450,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
// vLLM accepted a vLLM shard and failed it minutes later.
reg.EngineVersions = a.engines(parent)
reg.EnginesProbed = true
reg.InstallableEngines = provision.Installable()
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
reg.Features = []string{FeatureFabricTest}
reg.RunningShards = a.runningShards()
a.mu.Lock()
@@ -891,6 +891,17 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
stop()
spec.Profile.ModelPath = path
}
// Adapters are fetched like the model, and before the engine starts: it
// loads them at start, and one missing is a replica that answers for the
// base model under the adapter's name.
for i, ad := range spec.Profile.Adapters {
path, err := a.weights().Ensure(ctx, ad.Path)
if err != nil {
report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil)
return
}
spec.Profile.Adapters[i].Path = path
}
cmds, err := eng.Commands(spec)
if err != nil {
@@ -920,6 +931,14 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
} else {
err = a.sup.WaitAlive(ctx, sh)
}
if err == nil {
err = a.onItsGPUs(ctx, eng, spec, sh.Status().PID)
if err != nil {
// Serving, but not where it was placed: stopped, so the GPU it
// holds in the inventory is not taken by a CPU process.
_ = a.sup.Stop(ctx, shardID)
}
}
if err != nil {
report(supervisor.Failed, err.Error(), sh.Logs(20))
return
+32
View File
@@ -0,0 +1,32 @@
package agent
import (
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
)
func TestAnEngineDisabledOnTheNodeIsNeitherInstalledNorAdvertised(t *testing.T) {
a := &Agent{}
a.cfg.Engines.Disabled = []string{"vllm"}
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{
{Engine: "llamacpp", Version: "b1"}, {Engine: "vllm", Version: "0.30.0"},
}})
if len(a.provWants) != 1 || a.provWants[0].Engine != "llamacpp" {
t.Errorf("wants %+v: vLLM is disabled here and must not be installed", a.provWants)
}
a.configuredVers = map[string]string{"vllm": "0.25.0"}
a.installedVers = map[string]string{"llamacpp": "b1"}
if got := a.engineReportLocked(); got["vllm"] != "" || got["llamacpp"] != "b1" {
t.Errorf("advertised %v: a disabled engine found on disk is still not offered", got)
}
}
func TestAnEngineNamedByHandIsNotInstalledOverIt(t *testing.T) {
a := &Agent{reg: engines.NewRegistry(engines.Config{LlamaCpp: engines.LlamaCppConfig{Bin: "/opt/own/llama-server"}})}
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "llamacpp", Version: "b1"}}})
if len(a.provWants) != 0 {
t.Errorf("wants %+v: an installed build would be used over the operator's own", a.provWants)
}
}
+65
View File
@@ -0,0 +1,65 @@
package discovery
import (
"context"
"fmt"
"os/exec"
"strconv"
"strings"
)
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
// nvidia-smi index: what a process actually computes on, as opposed to what it
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
// backend did not load runs on the CPU without a word, and this is where that
// shows — the process is on no GPU's list.
//
// A node with no nvidia-smi returns nil and no error, as discovery does.
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
if binary == "" {
if _, err := exec.LookPath("nvidia-smi"); err != nil {
return nil, nil
}
binary = "nvidia-smi"
}
gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
if err != nil {
return nil, fmt.Errorf("running %s: %w", binary, err)
}
apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
if err != nil {
return nil, fmt.Errorf("running %s: %w", binary, err)
}
return parseGPUProcesses(string(gpus), string(apps)), nil
}
// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
func parseGPUProcesses(gpus, apps string) map[int][]int {
index := map[string]int{}
for _, line := range strings.Split(gpus, "\n") {
i, uuid, ok := strings.Cut(line, ",")
if !ok {
continue
}
n, err := strconv.Atoi(strings.TrimSpace(i))
if err != nil {
continue
}
index[strings.TrimSpace(uuid)] = n
}
out := map[int][]int{}
for _, line := range strings.Split(apps, "\n") {
uuid, p, ok := strings.Cut(line, ",")
if !ok {
continue
}
pid, err := strconv.Atoi(strings.TrimSpace(p))
i, known := index[strings.TrimSpace(uuid)]
if err != nil || !known {
continue
}
out[i] = append(out[i], pid)
}
return out
}
+21
View File
@@ -0,0 +1,21 @@
package discovery
import (
"reflect"
"testing"
)
func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
// Two processes on GPU 1, one on GPU 2, and a GPU this list has never
// heard of, which is dropped rather than guessed at.
apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
got := parseGPUProcesses(gpus, apps)
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
if !reflect.DeepEqual(got, want) {
t.Errorf("got %v, want %v", got, want)
}
if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
t.Errorf("no compute apps should be no GPU in use: %v", got)
}
}
+29
View File
@@ -9,6 +9,7 @@
package engines
import (
"context"
"fmt"
"os"
"slices"
@@ -40,6 +41,9 @@ type Profile struct {
ModelPath string
// ServedModelName is the name clients use, which need not match the path.
ServedModelName string
// Adapters are LoRA adapters to load beside the model, in order: an
// adapter's index is the id requests apply it by.
Adapters []Adapter
}
// ShardSpec is one shard of one replica on this node.
@@ -143,6 +147,15 @@ type Engine interface {
MetricsPath() string
}
// CPUFallback is an engine that can come up on the CPU when it was placed on
// GPUs, and keep serving there without an error. The agent checks such a shard
// is on its GPUs before calling it ready.
type CPUFallback interface {
FallsBackToCPU() bool
// Devices lists the GPUs the engine's build can see, for saying why.
Devices(ctx context.Context) ([]string, error)
}
// GPUInfo is the subset of GPU facts validation needs.
type GPUInfo struct {
Index int
@@ -200,6 +213,10 @@ func (r *Registry) Names() []string {
// Config configures the engine adapters.
type Config struct {
// Disabled are engines this node never installs or advertises, whatever
// the control plane asks: vLLM on a node of cards no build of it runs on,
// that is meant for llama.cpp alone.
Disabled []string `koanf:"disabled"`
VLLM VLLMConfig `koanf:"vllm"`
SGLang SGLangConfig `koanf:"sglang"`
LlamaCpp LlamaCppConfig `koanf:"llamacpp"`
@@ -229,6 +246,11 @@ func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
func baseEnv(s ShardSpec) map[string]string {
env := map[string]string{
"CUDA_VISIBLE_DEVICES": joinInts(s.GPUIndices),
// The indices above are nvidia-smi's, which counts by PCI bus. CUDA
// counts fastest first unless told otherwise, and on a node with two
// kinds of GPU the shard would get a different card than placement
// gave it — one another shard may hold.
"CUDA_DEVICE_ORDER": "PCI_BUS_ID",
}
if len(s.HCAs) > 0 {
@@ -373,3 +395,10 @@ func quantizationOf(args []string) string {
}
return ""
}
// Adapter is a LoRA adapter an engine loads beside its model. Path is a
// source until the agent has fetched it, a local file after.
type Adapter struct {
Name string `json:"name"`
Path string `json:"path"`
}
+66 -13
View File
@@ -1,8 +1,10 @@
package engines
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
"sort"
@@ -65,25 +67,25 @@ func (*LlamaCpp) ServesHTTP(ShardSpec) bool { return true }
// Validate checks the profile against the GPUs it would run on.
//
// The two refusals here are about not pretending: llama.cpp has no equivalent
// of a tensor-parallel group spanning machines, and its multi-GPU support is a
// different mechanism (--split-mode, --tensor-split) that this adapter does not
// drive. Accepting either would produce a replica that starts and then serves
// from one machine while the catalog believes it has several.
// llama.cpp spreads a model over a node's GPUs by layer (--split-mode layer),
// so gpus_per_node is how many it gets. It has no tensor parallelism and no
// group spanning machines, and accepting tp or nodes would produce a replica
// that serves from fewer cards, or one machine, while the catalog believes
// otherwise.
func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
if err := validateCommon(p, gpus); err != nil {
return err
}
if p.Nodes > 1 {
return fmt.Errorf("profile %q: llamacpp does not span nodes; use vllm or sglang for a %d-node replica",
p.Name, p.Nodes)
}
if p.TP > 1 {
return fmt.Errorf("profile %q: llamacpp does not do tensor parallelism; "+
"it splits a model across local GPUs with --tensor-split, which this adapter does not drive",
p.Name)
if p.TP > 1 || p.PP > 1 {
return fmt.Errorf("profile %q: llamacpp splits a model across a node's GPUs by layer, not by tp or pp; "+
"write gpus_per_node: %d instead", p.Name, max(p.TP, 1)*max(p.PP, 1))
}
return nil
// The common checks count a replica's GPUs as tp; for llama.cpp the
// count is gpus_per_node.
q := p
q.TP = max(p.GPUsPerNode, 1)
return validateCommon(q, gpus)
}
// Commands returns the processes for one shard, which is always one process.
@@ -132,6 +134,13 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
args = append(args, "--n-gpu-layers", "999")
}
// Several GPUs: the layers spread across all of them, which is the only
// split llama.cpp does without a matched interconnect. In proportion to
// each card's free memory, which for the whole cards placement gives is
// an even split.
if len(s.GPUIndices) > 1 && !hasFlag(p.Args, "--split-mode", "-sm") {
args = append(args, "--split-mode", "layer")
}
if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
// What placement reserved first: it is what the node's other
// replicas were placed around.
@@ -144,6 +153,15 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
}
args = append(args, "--threads", strconv.Itoa(n))
}
// Loaded at start, applied per request: a request that names an adapter
// carries "lora": [{"id": i, "scale": s}], and one that names none is the
// base model. Each is loaded at scale zero, not with --lora and
// --lora-init-without-apply: that leaves an adapter's scale at 1, and
// llama-server applies it at that scale to every request with no "lora"
// of its own — every request for the model itself.
for _, a := range p.Adapters {
args = append(args, "--lora-scaled", a.Path+":0")
}
args = append(args, p.Args...)
env := baseEnv(s)
@@ -225,3 +243,38 @@ func (l *LlamaCpp) bin() string {
}
return "llama-server"
}
// FallsBackToCPU says that llama.cpp keeps running when its GPU backend does
// not load: --n-gpu-layers asks for layers on a GPU it then has none of, and
// the server comes up and answers on the CPU, at a tenth of the speed, with no
// error anywhere. Its readiness on a GPU placement needs checking separately.
func (*LlamaCpp) FallsBackToCPU() bool { return true }
// Devices lists the GPUs this build can compute on, as llama-server's
// --list-devices names them ("CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)").
// Empty, on a machine with GPUs, is a build whose GPU backend did not load:
// the CPU build, or a CUDA build missing a library its backend links.
func (l *LlamaCpp) Devices(ctx context.Context) ([]string, error) {
out, err := exec.CommandContext(ctx, l.bin(), "--list-devices").CombinedOutput() //nolint:gosec // the engine's own binary
if err != nil {
return nil, fmt.Errorf("%s --list-devices: %v: %s", l.bin(), err, strings.TrimSpace(string(out)))
}
return parseDevices(string(out)), nil
}
// parseDevices reads --list-devices: the lines after "Available devices:",
// which says "(none)" when there are none. Log lines before it are skipped.
func parseDevices(out string) []string {
var devs []string
listing := false
for _, line := range strings.Split(out, "\n") {
line = strings.TrimSpace(line)
switch {
case strings.HasPrefix(line, "Available devices"):
listing = true
case listing && line != "" && line != "(none)":
devs = append(devs, line)
}
}
return devs
}
+55
View File
@@ -306,3 +306,58 @@ func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
}
}
func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
s := llamaSpec(nil)
s.Profile.Adapters = []Adapter{{Name: "sql", Path: "/w/sql.gguf"}, {Name: "docs", Path: "/w/docs.gguf"}}
cmds, err := (&LlamaCpp{}).Commands(s)
if err != nil {
t.Fatal(err)
}
args := strings.Join(cmds[0].Args, " ")
if !strings.Contains(args, "--lora-scaled /w/sql.gguf:0 --lora-scaled /w/docs.gguf:0") {
t.Errorf("args %s: both adapters, in order, loaded at scale zero", args)
}
plain, _ := (&LlamaCpp{}).Commands(llamaSpec(nil))
if strings.Contains(strings.Join(plain[0].Args, " "), "--lora") {
t.Error("a model with no adapters was given lora flags")
}
}
func TestDevicesAreReadFromListDevices(t *testing.T) {
cuda := "ggml_cuda_init: found 2 CUDA devices:\nAvailable devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n CUDA1: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
if got := parseDevices(cuda); len(got) != 2 || !strings.HasPrefix(got[0], "CUDA0: NVIDIA L4") {
t.Errorf("CUDA build: %q", got)
}
// A CUDA build whose backend did not load says what the CPU build says.
if got := parseDevices("Available devices:\n (none)\n"); len(got) != 0 {
t.Errorf("no devices: %q", got)
}
}
func TestLlamaCppSpreadsAReplicaOverItsNodesGPUsByLayer(t *testing.T) {
four := Profile{Name: "t4x4", Engine: "llamacpp", GPUClass: "t4", TP: 1, GPUsPerNode: 4, ModelPath: "/m"}
if err := (&LlamaCpp{}).Validate(four, h100(4)); err != nil {
t.Fatalf("four GPUs on a four-GPU node: %v", err)
}
if err := (&LlamaCpp{}).Validate(four, h100(2)); err == nil {
t.Error("four GPUs were accepted on a node with two")
}
s := llamaSpec([]int{0, 1, 2, 3})
s.Profile.GPUsPerNode = 4
cmds, err := (&LlamaCpp{}).Commands(s)
if err != nil {
t.Fatal(err)
}
if v, _ := flagValue(cmds[0].Args, "--split-mode"); v != "layer" {
t.Errorf("--split-mode %q over four GPUs", v)
}
if !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0,1,2,3") {
t.Errorf("env %v: the replica must see all four of its GPUs", cmds[0].Env)
}
one, _ := (&LlamaCpp{}).Commands(llamaSpec([]int{2}))
if hasFlag(one[0].Args, "--split-mode") {
t.Error("a one-GPU replica was given a split mode")
}
}
+116
View File
@@ -0,0 +1,116 @@
package agent
import (
"context"
"fmt"
"os"
"strconv"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
)
// gpuProcesses and processGroup are variables so tests can stand in for
// nvidia-smi and /proc.
var (
gpuProcesses = discovery.GPUProcesses
processGroup = pgidOf
// gpuCheckWait is how long a shard has, once it answers, to show up on
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
gpuCheckWait = 15 * time.Second
)
// onItsGPUs checks that a shard placed on GPUs computes on them.
//
// For an engine that falls back to the CPU when its GPU backend does not load,
// "it answers" is not "it is ready": llama.cpp built for CUDA but missing
// cuBLAS comes up on the CPU, answers at a tenth of the speed, and the GPU it
// was given sits idle and reserved. What it computes on is what nvidia-smi
// lists it on, so a shard none of whose processes hold memory on one of its
// GPUs is refused, with what the engine itself can see.
func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.ShardSpec, pid int) error {
fb, ok := eng.(engines.CPUFallback)
if !ok || !fb.FallsBackToCPU() || len(spec.GPUIndices) == 0 || pid <= 0 {
return nil
}
group := processGroup(pid)
deadline := time.Now().Add(gpuCheckWait)
var missing []int
for {
procs, err := gpuProcesses(ctx, a.cfg.Discovery.NvidiaSMI)
if err != nil || procs == nil {
// No nvidia-smi to ask is no evidence either way, and a node that
// was given GPUs has one; say so and let the shard serve.
a.log.Warn("could not check that a GPU shard runs on its GPUs", "shard", spec.ShardID, "err", err)
return nil
}
missing = missing[:0]
for _, i := range spec.GPUIndices {
if !holds(procs[i], pid, group) {
missing = append(missing, i)
}
}
if len(missing) == 0 {
return nil
}
if time.Now().After(deadline) {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(time.Second):
}
}
seen := "it lists no GPU at all"
if devs, err := fb.Devices(ctx); err != nil {
seen = "listing its devices failed: " + err.Error()
} else if len(devs) > 0 {
seen = "it lists " + strings.Join(devs, "; ")
}
return fmt.Errorf("placed on GPU %s but computing on the CPU: the engine holds no memory there. "+
"Its GPU backend did not load (%s). Check the build with `ldd` on its libggml-cuda.so, "+
"or run `llmgw doctor` on this node", joinIndices(missing), seen)
}
// holds reports whether any of pids is the shard's process or in its group:
// an engine that computes in a child process is still the shard.
func holds(pids []int, pid, group int) bool {
for _, p := range pids {
if p == pid || (group > 0 && processGroup(p) == group) {
return true
}
}
return false
}
// pgidOf reads a process's group from /proc; zero when it cannot.
func pgidOf(pid int) int {
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
if err != nil {
return 0
}
// The command name, in parentheses, may hold spaces; the fields after its
// closing parenthesis are state, ppid, pgrp.
s := string(b)
i := strings.LastIndexByte(s, ')')
if i < 0 {
return 0
}
f := strings.Fields(s[i+1:])
if len(f) < 3 {
return 0
}
n, _ := strconv.Atoi(f[2])
return n
}
func joinIndices(is []int) string {
s := make([]string, len(is))
for k, i := range is {
s[k] = strconv.Itoa(i)
}
return strings.Join(s, ", ")
}
+91
View File
@@ -0,0 +1,91 @@
package agent
import (
"context"
"log/slog"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
)
// fallbackEngine is an engine that comes up on the CPU when its GPU backend
// does not load, and lists the devices it can see.
type fallbackEngine struct {
engines.Engine
devices []string
}
func (fallbackEngine) FallsBackToCPU() bool { return true }
func (f fallbackEngine) Devices(context.Context) ([]string, error) {
return f.devices, nil
}
func TestAGPUShardComputingOnTheCPUIsNotReady(t *testing.T) {
procs := map[int][]int{}
oldProcs, oldGroup, oldWait := gpuProcesses, processGroup, gpuCheckWait
gpuProcesses = func(context.Context, string) (map[int][]int, error) { return procs, nil }
processGroup = func(pid int) int {
if pid >= 500 && pid < 600 {
return 500 // the shard's group: 500 and its children
}
return pid
}
gpuCheckWait = 0
t.Cleanup(func() { gpuProcesses, processGroup, gpuCheckWait = oldProcs, oldGroup, oldWait })
a := &Agent{log: slog.Default()}
spec := engines.ShardSpec{ShardID: "s", GPUIndices: []int{0, 1}}
eng := fallbackEngine{}
// On no GPU at all: refused, saying what the engine sees.
err := a.onItsGPUs(context.Background(), eng, spec, 500)
if err == nil || !strings.Contains(err.Error(), "computing on the CPU") || !strings.Contains(err.Error(), "lists no GPU") {
t.Fatalf("a shard on no GPU was let through: %v", err)
}
// On one of its two: refused, naming the one it is missing.
procs[0] = []int{500}
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil || !strings.Contains(err.Error(), "GPU 1 ") {
t.Fatalf("a shard on half its GPUs: %v", err)
}
// Another process on GPU 1 is not the shard.
procs[1] = []int{900}
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil {
t.Fatal("another process's memory was taken for the shard's")
}
// A child of the shard holding GPU 1 is the shard.
procs[1] = []int{900, 512}
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err != nil {
t.Fatalf("a shard on both its GPUs: %v", err)
}
// An engine that fails rather than falling back is not checked, nor is a
// shard with no GPUs.
procs = map[int][]int{}
if err := a.onItsGPUs(context.Background(), struct{ engines.Engine }{}, spec, 500); err != nil {
t.Errorf("checked an engine that does not fall back: %v", err)
}
if err := a.onItsGPUs(context.Background(), eng, engines.ShardSpec{}, 500); err != nil {
t.Errorf("checked a CPU shard: %v", err)
}
}
func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
calls := 0
oldProcs, oldWait := gpuProcesses, gpuCheckWait
gpuProcesses = func(context.Context, string) (map[int][]int, error) {
calls++
if calls < 2 {
return map[int][]int{}, nil
}
return map[int][]int{0: {77}}, nil
}
gpuCheckWait = 5 * time.Second
t.Cleanup(func() { gpuProcesses, gpuCheckWait = oldProcs, oldWait })
a := &Agent{log: slog.Default()}
if err := a.onItsGPUs(context.Background(), fallbackEngine{}, engines.ShardSpec{GPUIndices: []int{0}}, 77); err != nil {
t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
}
}
+30
View File
@@ -46,9 +46,30 @@ func (a *Agent) engineReportLocked() map[string]string {
out = map[string]string{}
}
maps.Copy(out, a.installedVers)
maps.DeleteFunc(out, func(e, _ string) bool { return a.disabled(e) })
return out
}
// byHand reports whether agent.yaml names this engine's binary, which opts
// it out of being installed.
func (a *Agent) byHand(engine string) bool {
if a.reg == nil {
return false
}
e, err := a.reg.Get(engine)
if err != nil {
return false
}
c, ok := e.(engines.Configured)
return ok && c.Configured()
}
// disabled reports whether agent.yaml says this node never runs an engine:
// not installed when asked, and not advertised when found.
func (a *Agent) disabled(engine string) bool {
return slices.Contains(a.cfg.Engines.Disabled, engine)
}
// provisionEvery is how often installed engines are checked again. A library
// removed by an unrelated upgrade is found here rather than by the next
// replica to start.
@@ -58,6 +79,15 @@ const provisionEvery = 10 * time.Minute
func (a *Agent) setProvision(p *llmgwv1.Provision) {
wants := []provision.Want{}
for _, e := range p.GetEngines() {
if a.disabled(e.GetEngine()) {
continue // a control plane older than the node's say in it
}
if a.byHand(e.GetEngine()) {
// The operator named a binary for it: a local build, a CUDA
// runtime of their own. Installing one beside it would win —
// the installed build is used first — and quietly replace theirs.
continue
}
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
}
a.mu.Lock()
+46
View File
@@ -299,9 +299,55 @@ func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, vers
if err != nil {
return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
}
if b.GPUBackend != "" {
if err := in.verifyGPUBackend(tctx, dir, bin, b); err != nil {
return "", "", err
}
}
return bin, firstVersionLine(out), nil
}
// verifyGPUBackend checks a GPU build can compute on this node's GPUs.
//
// The binary's own ldd says nothing about it: the GPU backend is a library
// the server opens at run time, and when it cannot — cuBLAS missing, a driver
// too old — the server carries on with its CPU backend. So the build is asked
// which devices it sees, and when it sees none, ldd on the backend says why.
func (in *Installer) verifyGPUBackend(ctx context.Context, dir, bin string, b Build) error {
out, err := in.run(ctx, nil, bin, "--list-devices")
if err != nil {
return fmt.Errorf("%s --list-devices failed: %v: %s", b.Binary, err, lastLine(out))
}
if listsDevice(string(out)) {
return nil
}
why := "its GPU backend did not load"
if lib, err := findFile(dir, b.GPUBackend); err != nil {
why = "the build has no " + b.GPUBackend
} else if out, err := in.run(ctx, nil, "ldd", lib); err == nil {
if missing := lddMissing(string(out)); len(missing) > 0 {
why = b.GPUBackend + " cannot load: " + strings.Join(missing, "; ")
}
}
return fmt.Errorf("the %s build of %s sees no GPU, and would run on the CPU: %s", b.Variant, b.Binary, why)
}
// listsDevice reads llama-server --list-devices: a device line after
// "Available devices:", which holds "(none)" when there is none.
func listsDevice(out string) bool {
listing := false
for _, line := range strings.Split(out, "\n") {
line = strings.TrimSpace(line)
switch {
case strings.HasPrefix(line, "Available devices"):
listing = true
case listing && line != "" && line != "(none)":
return true
}
}
return false
}
// packages installs the operating system packages a build needs.
//
// Only those missing, and from the node's own package sources: a package the
+38 -1
View File
@@ -89,7 +89,11 @@ type fakeSystem struct {
installed map[string]bool
tools map[string]bool
lddOut string
calls []string
// devices is what llama-server --list-devices prints, and backendLdd
// what ldd says of its GPU backend.
devices string
backendLdd string
calls []string
// locked is how many more package commands find the lock held.
locked int
}
@@ -127,8 +131,12 @@ func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...str
}
}
return nil, nil
case name == "ldd" && strings.HasSuffix(args[0], "libggml-cuda.so"):
return []byte(f.backendLdd), nil
case name == "ldd":
return []byte(f.lddOut), nil
case strings.HasSuffix(name, "llama-server") && args[0] == "--list-devices":
return []byte(f.devices), nil
case strings.HasSuffix(name, "llama-server"):
return []byte("version: 10991 (abc)\n"), nil
}
@@ -388,3 +396,32 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
t.Errorf("link %q %v", target, err)
}
}
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
// loader looks, and llama-server serving a GPU placement on the CPU.
cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
rel := newRelease(map[string][]byte{
"llama-b1-" + cuda: tarGz(t, map[string]string{
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
"cudart-llama-b1-" + cuda: tarGz(t, map[string]string{"libcudart.so.12": "x"}, nil),
})
gpu := ubuntu
gpu.NVIDIADriver = "570.86.15"
sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
devices: "Available devices:\n (none)\n",
backendLdd: "\tlibcublas.so.12 => not found\n\tlibcudart.so.12 => /x/libcudart.so.12\n"}
in, _ := installer(t, rel.serve(t), sys, gpu)
got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
if got[0].State != Failed || !strings.Contains(got[0].Detail, "sees no GPU") ||
!strings.Contains(got[0].Detail, "libcublas.so.12 is not installed") {
t.Fatalf("expected a failure naming the missing library: %+v", got[0])
}
// The same build seeing its GPU is installed.
sys.devices = "Available devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
if got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}}); got[0].State != Ready {
t.Fatalf("a CUDA build that sees its GPU: %+v", got[0])
}
}
+70 -15
View File
@@ -43,6 +43,11 @@ type Build struct {
// VersionArgs is what Binary is run with to prove the build works, when
// --version is not it.
VersionArgs []string
// GPUBackend, when set, is the shared library the build computes on GPUs
// with, loaded at run time rather than linked: "libggml-cuda.so". The
// build must list a device with it, or it is not installed — a llama.cpp
// whose CUDA backend cannot load runs on the CPU without complaint.
GPUBackend string
// FreeGB is the disk space the installation needs before it starts: a
// Python engine is several gigabytes, and one that fills the disk halfway
// leaves the node worse off than one that never began.
@@ -124,25 +129,72 @@ const minDriverForCUDA13 = 580
// Volta, so a V100 cannot run these builds at all.
const minComputeForCUDA13 = 7.5
// cudaGate says whether a node can run an engine built on CUDA 13.
func cudaGate(engine string, h discovery.Host) error {
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
// version compatibility, and on Volta.
const (
minDriverForCUDA12 = 525
minComputeForCUDA12 = 7.0
)
// cuda13Since is the first version of each Python engine built for CUDA 13.
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
// cudaGate says whether a node can run this version of an engine, and which
// CUDA it is built for.
//
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
// nodes the older build (engine_overrides in the server's configuration)
// rather than going without vLLM, or without the Volta nodes.
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
switch h.Arch {
case "amd64", "arm64":
default:
return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
return "", fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
}
if h.NVIDIADriver == "" {
return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
}
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
engine, minDriverForCUDA13, h.NVIDIADriver)
variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
if versionBefore(version, cuda13Since[engine]) {
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
}
if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
engine, minComputeForCUDA13, cc)
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
engine, version, cuda, minDriver, h.NVIDIADriver)
}
return nil
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
hint := ""
if cuda == "13" {
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
}
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, hint)
}
return variant, nil
}
// versionBefore reports whether version a is older than b, comparing the
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
// number counts as zero.
func versionBefore(a, b string) bool {
if b == "" {
return false
}
pa, pb := strings.Split(strings.TrimPrefix(a, "v"), "."), strings.Split(b, ".")
for i := 0; i < max(len(pa), len(pb)); i++ {
x, y := 0, 0
if i < len(pa) {
x = leadingInt(pa[i])
}
if i < len(pb) {
y = leadingInt(pb[i])
}
if x != y {
return x < y
}
}
return false
}
// Python engines need a C compiler at run time: Triton builds a small
@@ -151,11 +203,12 @@ func cudaGate(engine string, h discovery.Host) error {
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
func chooseVLLM(version string, h discovery.Host) (Build, error) {
if err := cudaGate("vLLM", h); err != nil {
variant, err := cudaGate("vLLM", version, h)
if err != nil {
return Build{}, err
}
return Build{
Variant: "cu13",
Variant: variant,
Python: pythonForEngines,
Requirements: []string{"vllm==" + version},
Binary: "vllm",
@@ -165,13 +218,14 @@ func chooseVLLM(version string, h discovery.Host) (Build, error) {
}
func chooseSGLang(version string, h discovery.Host) (Build, error) {
if err := cudaGate("SGLang", h); err != nil {
variant, err := cudaGate("SGLang", version, h)
if err != nil {
return Build{}, err
}
// SGLang is launched as a module, so what the agent is given is the
// environment's interpreter, and the proof is importing it.
return Build{
Variant: "cu13",
Variant: variant,
Python: pythonForEngines,
Requirements: []string{"sglang==" + version},
Binary: "python",
@@ -234,6 +288,7 @@ func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
if h.NVIDIADriver != "" && arch == "x64" {
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
b.Variant = "cuda-" + cudaForLlamaCpp
b.GPUBackend = "libggml-cuda.so"
b.Assets = []string{
fmt.Sprintf("llama-%s-%s", version, name),
fmt.Sprintf("cudart-llama-%s-%s", version, name),
+43 -2
View File
@@ -2,6 +2,8 @@ package weights
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
@@ -26,8 +28,9 @@ type manifest struct {
Model string `json:"model"`
Version string `json:"version"`
Files []struct {
Name string `json:"name"`
Bytes int64 `json:"bytes"`
Name string `json:"name"`
Bytes int64 `json:"bytes"`
SHA256 string `json:"sha256"`
} `json:"files"`
Bytes int64 `json:"bytes"`
}
@@ -87,6 +90,25 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error)
}
}
// Each file against the checksum the mirror took when it fetched it,
// before the set is called complete: a shard damaged on the way loads as
// a model that answers nonsense, not as an error.
for _, file := range m.Files {
if file.SHA256 == "" {
continue
}
target := filepath.Join(dir, file.Name)
got, err := fileSHA256(target)
if err != nil {
return "", err
}
if !strings.EqualFold(got, file.SHA256) {
_ = os.Remove(target)
return "", fmt.Errorf("%s%s: sha256 %s, the mirror has %s; removed, to be fetched again",
source, file.Name, got, file.SHA256)
}
}
if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
return "", err
}
@@ -108,6 +130,12 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusConflict {
// Still downloading on the mirror, or a shard missing: said in the
// mirror's words, which name what is not there yet.
why, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
return manifest{}, fmt.Errorf("%s is not complete on the mirror yet: %s", source, strings.TrimSpace(string(why)))
}
if resp.StatusCode != http.StatusOK {
return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
}
@@ -127,3 +155,16 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
}
return m, nil
}
func fileSHA256(path string) (string, error) {
f, err := os.Open(path) //nolint:gosec // a file this fetcher wrote
if err != nil {
return "", err
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
+39
View File
@@ -1,6 +1,8 @@
package weights
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"net/http"
"net/http/httptest"
@@ -165,3 +167,40 @@ func TestOnlyATrailingSlashMeansASet(t *testing.T) {
}
}
}
func TestASetFileThatDoesNotMatchTheMirrorsChecksumIsRefused(t *testing.T) {
good := "the shard as mirrored"
sum := sha256.Sum256([]byte(good))
served := "the shard damaged on the way"
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/") {
fmt.Fprintf(w, `{"model":"m","version":"v","files":[{"name":"s.safetensors","bytes":%d,"sha256":%q}]}`,
len(served), hex.EncodeToString(sum[:]))
return
}
_, _ = w.Write([]byte(served))
}))
t.Cleanup(srv.Close)
f := &Fetcher{CacheDir: t.TempDir()}
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err == nil || !strings.Contains(err.Error(), "sha256") {
t.Fatalf("a damaged shard was accepted: %v", err)
}
served = good
// The size differs, so it is fetched again; this time it matches.
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
t.Fatalf("the shard as mirrored: %v", err)
}
}
func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
http.Error(w, "m/v: this version is not complete: still downloading model-00002-of-00002.safetensors", http.StatusConflict)
}))
t.Cleanup(srv.Close)
f := &Fetcher{CacheDir: t.TempDir()}
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") ||
!strings.Contains(err.Error(), "model-00002-of-00002") {
t.Fatalf("got %v", err)
}
}
+2
View File
@@ -254,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
switch {
case m.Parked:
return "parked"
case m.External != "":
return "external: " + m.External
case m.HeldFor != "":
return "gave room to " + m.HeldFor
case m.Unplaceable:
+13 -7
View File
@@ -12,8 +12,9 @@ import (
func newRestartCmd(client func() *ctlClient) *cobra.Command {
var (
models []string
all bool
models []string
all bool
inPlace bool
)
c := &cobra.Command{
Use: "restart [REPLICA...]",
@@ -22,10 +23,14 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
"an edited profile, a new engine version.\n\n" +
"Each replacement is placed beside the replica it replaces, and the old one\n" +
"keeps serving until the new one is ready. Replicas of one model are\n" +
"replaced one at a time. Where nothing else has room for a second copy —\n" +
"the usual case on a small CPU fleet — the old one is stopped after the\n" +
"drain grace period (5 minutes by default) and the new one starts in its\n" +
"place; a model with only that replica is unserved while it loads.\n\n" +
"replaced one at a time.\n\n" +
"Where there is no room for a second copy and the old replica is all that\n" +
"is in the way — one GPU, every card that fits held — it is stopped at once\n" +
"and the new one starts in its place; a model with only that replica is\n" +
"unserved while it loads. Where the new one would not fit even then — a\n" +
"profile edited to need more than any node has — the restart is given up\n" +
"and the old replica keeps serving: llmgw ctl replicas says so, and why.\n\n" +
"--in-place stops each replica first, whatever room there is.\n\n" +
" llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
" llmgw ctl restart --model qwen2.5-7b\n" +
" llmgw ctl restart --all",
@@ -33,7 +38,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
if len(args) == 0 && len(models) == 0 && !all {
return errors.New("name replicas, --model, or --all")
}
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all, InPlace: inPlace})
var res adminapi.RestartResult
if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
return err
@@ -44,6 +49,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
}
c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before starting its replacement")
return c
}
+17 -2
View File
@@ -23,6 +23,7 @@ func newDoctorCmd() *cobra.Command {
opts doctor.Options
asJSON bool
verbose bool
fabrics []string
)
cmd := &cobra.Command{
Use: "doctor",
@@ -38,8 +39,21 @@ that file when the report is not enough to go on.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
doctor.SetConfigLoader(config.LoadAgent)
if opts.VLLMVersion == "" {
for _, spec := range fabrics {
name, hcas, _ := strings.Cut(spec, ":")
if name == "" || hcas == "" {
return fmt.Errorf("--fabric %q: give it as name:mlx5_0,mlx5_1", spec)
}
if opts.Fabrics == nil {
opts.Fabrics = map[string][]string{}
}
opts.Fabrics[name] = strings.Split(hcas, ",")
}
switch opts.VLLMVersion {
case "":
opts.VLLMVersion = config.DefaultEngines["vllm"]
case "none":
opts.VLLMVersion = ""
}
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
defer cancel()
@@ -66,7 +80,8 @@ that file when the report is not enough to go on.`,
f.StringVar(&opts.ConfigPath, "config", "", "agent configuration (default /etc/llmgw/agent.yaml when present)")
f.StringVar(&opts.IBRoot, "ib-root", discovery.SysfsRoot, "sysfs InfiniBand directory")
f.StringVar(&opts.NvidiaSMI, "nvidia-smi", "", "path to nvidia-smi")
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs)")
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs; none to skip)")
f.StringArrayVar(&fabrics, "fabric", nil, "a fabric about to be declared, name:mlx5_0,mlx5_1, checked as if it were; repeatable")
f.BoolVar(&asJSON, "json", false, "write every fact and finding as JSON, to send back")
f.BoolVar(&verbose, "verbose", false, "also print the raw nvidia-smi topology")
return cmd
+29 -1
View File
@@ -3,6 +3,7 @@ package cli
import (
"context"
"fmt"
"io"
"os/exec"
"path/filepath"
"sort"
@@ -126,6 +127,14 @@ func newEngineCheckCmd() *cobra.Command {
}
sort.Strings(unknown)
// Flags accepted is not the same as running where it is placed: a
// GPU build whose backend does not load accepts every flag and
// computes on the CPU. Asked of the engine here, not of a help.
var deviceErr error
if fb, ok := eng.(engines.CPUFallback); ok && help == "" {
deviceErr = checkDevices(cmd.Context(), name, fb, out)
}
fmt.Fprintf(out, "%s: %d flags checked", name, checked)
if skipped > 0 {
fmt.Fprintf(out, ", %d commands skipped", skipped)
@@ -137,7 +146,7 @@ func newEngineCheckCmd() *cobra.Command {
return fmt.Errorf("nothing was checked; install the engine here or pass --help-output")
}
fmt.Fprintln(out, "every flag llmgw builds is accepted")
return nil
return deviceErr
}
fmt.Fprintf(out, "\n%d flags not mentioned by the program they are passed to:\n", len(unknown))
for _, f := range unknown {
@@ -249,3 +258,22 @@ func isFlagChar(b byte) bool {
return b == '-' || b == '_' || (b >= 'a' && b <= 'z') ||
(b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9')
}
// checkDevices says which GPUs an engine that can fall back to the CPU sees on
// this machine, and fails when the machine has NVIDIA GPUs and it sees none.
func checkDevices(ctx context.Context, name string, fb engines.CPUFallback, out io.Writer) error {
if _, err := exec.LookPath("nvidia-smi"); err != nil {
fmt.Fprintln(out, "no NVIDIA GPU here: which devices it computes on was not checked")
return nil
}
devs, err := fb.Devices(ctx)
if err != nil {
return err
}
if len(devs) == 0 {
return fmt.Errorf("this machine has NVIDIA GPUs and %s sees none: it would serve a GPU placement on the CPU. "+
"Its GPU backend did not load; ldd on its libggml-cuda.so names the missing library", name)
}
fmt.Fprintf(out, "computes on: %s\n", strings.Join(devs, "; "))
return nil
}
+54 -29
View File
@@ -2,6 +2,7 @@ package cli
import (
"encoding/json"
"errors"
"fmt"
"log/slog"
"net"
@@ -56,52 +57,69 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
var model, version, as string
c := &cobra.Command{
Use: "add <url>",
Use: "add <url>...",
Short: "Download a model into the mirror",
Long: "Downloads to a temporary name and renames when complete, so an\n" +
"interrupted download never leaves something a catalog could be pointed\n" +
"at. Reports the checksum: a mirror is the one place in this system where\n" +
"a corrupted file is copied to every node rather than caught once.\n\n" +
"Several URLs go into one version, as the files of one model: the shards\n" +
"of a safetensors set with its config.json and tokenizer, for vLLM. The\n" +
"version is offered to nodes only once it is whole — nothing still\n" +
"downloading, every shard its index names present — and its source is\n" +
"the version, ending in a slash.\n\n" +
"Refuses to replace a version that is already here. Add a new one beside\n" +
"it instead — that is what makes the old one still available when the new\n" +
"one turns out to be worse.",
Args: cobra.ExactArgs(1),
Args: cobra.MinimumNArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
source := args[0]
file := as
if file == "" {
file = mirror.LastPathElement(source)
}
if model == "" || version == "" {
return fmt.Errorf("--model and --version are required: they are the layout")
}
e := mirror.Entry{Model: model, Version: version, File: file}
out := cmd.OutOrStdout()
fmt.Fprintf(out, "fetching %s\n", source)
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
if total > 0 {
fmt.Fprintf(out, " %s of %s (%d%%)\n",
fileSize(done), fileSize(total), done*100/total)
return
}
fmt.Fprintf(out, " %s so far\n", fileSize(done))
})
if err != nil {
return err
if as != "" && len(args) > 1 {
return fmt.Errorf("--as names one file; with %d URLs each keeps its own name", len(args))
}
fmt.Fprintf(out, "\n%s/%s/%s %s\n sha256 %s\n",
added.Entry.Model, added.Entry.Version, added.Entry.File,
fileSize(added.Entry.Bytes), added.SHA256)
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n",
added.Entry.URL("http://<this-host>:8081"))
out := cmd.OutOrStdout()
var last mirror.Added
for _, source := range args {
file := as
if file == "" {
file = mirror.LastPathElement(source)
}
e := mirror.Entry{Model: model, Version: version, File: file}
fmt.Fprintf(out, "fetching %s\n", source)
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
if total > 0 {
fmt.Fprintf(out, " %s of %s (%d%%)\n",
fileSize(done), fileSize(total), done*100/total)
return
}
fmt.Fprintf(out, " %s so far\n", fileSize(done))
})
if err != nil {
return err
}
fmt.Fprintf(out, "%s/%s/%s %s\n sha256 %s\n",
added.Entry.Model, added.Entry.Version, added.Entry.File,
fileSize(added.Entry.Bytes), added.SHA256)
last = added
}
src := last.Entry.URL("http://<this-host>:8081")
if len(args) > 1 {
src = strings.TrimSuffix(mirror.Entry{Model: model, Version: version}.URL("http://<this-host>:8081"), "/") + "/"
if m, err := mirror.ManifestFor(*dir, model, version); err != nil {
fmt.Fprintf(out, "\nNot yet offered to nodes: %v\n", err)
} else {
fmt.Fprintf(out, "\n%d files, %s, whole.\n", len(m.Files), fileSize(m.Bytes))
}
}
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n", src)
return nil
},
}
c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
c.Flags().StringVar(&as, "as", "", "the filename to save as; taken from the URL when omitted")
c.Flags().StringVar(&as, "as", "", "the filename to save as, for one URL; taken from the URL when omitted")
return c
}
@@ -405,7 +423,14 @@ func withManifest(root string, next http.Handler) http.Handler {
return
}
m, err := mirror.ManifestFor(root, model, version)
if err != nil {
switch {
case errors.Is(err, mirror.ErrIncomplete):
// Said, not hidden behind a 404: a node asking now is told why
// it cannot have the version yet, and tries again later.
w.Header().Set("Retry-After", "60")
http.Error(w, err.Error(), http.StatusConflict)
return
case err != nil:
http.NotFound(w, r)
return
}
+70 -1
View File
@@ -5,6 +5,9 @@ import (
"fmt"
"os"
"os/user"
"slices"
"strconv"
"strings"
"time"
"github.com/knadh/koanf/parsers/yaml"
@@ -31,7 +34,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
}
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
newTargetCmd(client, asJSON),
newTargetCmd(client, asJSON), newAdaptersCmd(client),
newModelServeCmd(client))
return c
}
@@ -264,3 +267,69 @@ func newClassCmd(client func() *ctlClient) *cobra.Command {
},
}
}
func newAdaptersCmd(client func() *ctlClient) *cobra.Command {
var add, remove []string
c := &cobra.Command{
Use: "adapters <model> [--add NAME=URL[@SCALE]] [--remove NAME]",
Short: "List, add or remove a model's LoRA adapters",
Long: "An adapter is asked for by its NAME as though it were a model, and is served\n" +
"by this model's replicas with it applied. URL is the adapter file, usually in\n" +
"the mirror; it must be a GGUF LoRA adapter for this model's architecture,\n" +
"which is checked before it is attached. Replicas load a changed set when\n" +
"restarted: llmgw ctl restart --model MODEL.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
name := args[0]
var models []adminapi.ModelView
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
return err
}
var current []catalog.Adapter
found := false
for _, m := range models {
if m.Name == name {
current, found = m.Adapters, true
}
}
if !found {
return fmt.Errorf("no model %q", name)
}
out := cmd.OutOrStdout()
if len(add) == 0 && len(remove) == 0 {
if len(current) == 0 {
fmt.Fprintf(out, "%s has no adapters\n", name)
}
for _, a := range current {
fmt.Fprintf(out, "%s\tscale %g\t%s\n", a.Name, a.EffectiveScale(), a.Source)
}
return nil
}
next := slices.DeleteFunc(slices.Clone(current), func(a catalog.Adapter) bool { return slices.Contains(remove, a.Name) })
for _, spec := range add {
n, rest, ok := strings.Cut(spec, "=")
if !ok || n == "" || rest == "" {
return fmt.Errorf("--add %q is not NAME=URL[@SCALE]", spec)
}
ad := catalog.Adapter{Name: n, Source: rest}
if i := strings.LastIndex(rest, "@"); i > 0 {
if s, err := strconv.ParseFloat(rest[i+1:], 64); err == nil {
ad.Source, ad.Scale = rest[:i], s
}
}
next = append(next, ad)
}
body, _ := json.Marshal(adminapi.AdaptersRequest{Adapters: next})
var res adminapi.AdaptersRequest
if err := client().post(cmd.Context(), "/api/v1/models/"+name+"/adapters", body, &res); err != nil {
return err
}
fmt.Fprintf(out, "%s serves %d adapter(s); restart its replicas to load them: llmgw ctl restart --model %s\n",
name, len(res.Adapters), name)
return nil
},
}
c.Flags().StringArrayVar(&add, "add", nil, "NAME=URL[@SCALE]; repeatable")
c.Flags().StringArrayVar(&remove, "remove", nil, "an adapter's name; repeatable")
return c
}
+20 -1
View File
@@ -12,6 +12,7 @@ import (
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
@@ -113,13 +114,14 @@ engines are kept.`,
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed, instead of the one the agent installs")
f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
@@ -148,6 +150,23 @@ func (na *nodeAdd) prepare() error {
}
}
}
for _, e := range o.DisableEngines {
if !slices.Contains(provision.Installable(), e) {
return fmt.Errorf("--disable-engine %s: the engines are %s", e, strings.Join(provision.Installable(), ", "))
}
}
// What the control plane will ask the node to install, so doctor checks
// that vLLM rather than its own default. Its overrides for a GPU class
// are not known until the node says what it has, and are not applied
// here; doctor's verdict on vLLM is a warning either way.
if o.VLLMVersion == "" {
if cfg, err := config.Load(na.serverCfg); err == nil {
o.VLLMVersion = cfg.WantedEngines()["vllm"]
if o.VLLMVersion == "" {
o.VLLMVersion = "none"
}
}
}
o.Version = version.Version
if o.Binary == "" {
exe, err := os.Executable()
+4
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"net/url"
"strconv"
"strings"
"github.com/spf13/cobra"
@@ -192,6 +193,9 @@ func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
switch {
case v.More == 0:
fmt.Fprintln(out, "It already has them.")
case v.Fits && len(v.Preempts) > 0:
fmt.Fprintf(out, "%d more; there is room once replicas of %s are stopped, which its class may do.\n",
v.More, strings.Join(v.Preempts, ", "))
case v.Fits:
fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
default:
+67
View File
@@ -0,0 +1,67 @@
package cli
import (
"errors"
"fmt"
"os"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
)
func newReleaseCmd() *cobra.Command {
c := &cobra.Command{
Use: "release",
Short: "Sign releases, and make the key they are signed with",
}
c.AddCommand(&cobra.Command{
Use: "keygen",
Short: "Make a release signing key pair",
Long: "Prints a public key and a private key. The private key goes into the CI\n" +
"secret RELEASE_SIGNING_KEY and nowhere else; the public key into\n" +
"internal/release/trusted.pub, which every build from then on carries\n" +
"and checks releases against. Run once, on a machine you trust.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
pub, priv, err := release.GenerateKey()
if err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "public: %s\nprivate: %s\n", pub, priv)
fmt.Fprintln(cmd.ErrOrStderr(), "\nThe private key signs every release: put it in the CI secret "+
"RELEASE_SIGNING_KEY, then clear this terminal's scrollback.")
return nil
},
})
c.AddCommand(&cobra.Command{
Use: "sign FILE",
Short: "Sign a release's checksums, writing FILE.sig",
Long: "Reads the private key from RELEASE_SIGNING_KEY, never from an argument,\n" +
"so it does not land in a shell history or a process list.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
key := os.Getenv("RELEASE_SIGNING_KEY")
if key == "" {
return errors.New("RELEASE_SIGNING_KEY is not set")
}
data, err := os.ReadFile(args[0])
if err != nil {
return err
}
sig, err := release.Sign(key, data)
if err != nil {
return err
}
if pub := release.TrustedKey(); pub != "" {
// Signed with a key this build would not accept is a release
// every upgraded machine refuses: stopped here instead.
if err := release.Verify(pub, data, sig); err != nil {
return fmt.Errorf("the signing key does not match this build's trusted key: %w", err)
}
}
return os.WriteFile(args[0]+".sig", []byte(sig+"\n"), 0o644) //nolint:gosec // public
},
})
return c
}
+1
View File
@@ -48,6 +48,7 @@ func Root() *cobra.Command {
newDoctorCmd(),
newNodeCmd(),
newSelfUpgradeCmd(),
newReleaseCmd(),
)
return root
}
+12 -6
View File
@@ -5,19 +5,23 @@ import (
"crypto/tls"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
"github.com/prometheus/client_golang/prometheus"
"log/slog"
"maps"
"net"
"net/http"
"os/signal"
"slices"
"strings"
"sync"
"syscall"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
"github.com/prometheus/client_golang/prometheus"
"github.com/spf13/cobra"
"golang.org/x/net/http2"
"golang.org/x/net/http2/h2c"
@@ -496,8 +500,10 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
}
}
svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
svc.SetEngines(cfg.WantedEngines())
log.Info("nodes install these engines", "engines", cfg.WantedEngines())
svc.SetEnginesFor(func(n inventory.Node) map[string]string {
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
})
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
// A standby must refuse agents as well as refusing to place. Commands
// travel down the stream the agent opened, so a standby holding one is a
// node the leader cannot command — and the leader's placements would fail
+74 -1
View File
@@ -8,6 +8,7 @@ package config
import (
"fmt"
"os"
"slices"
"strings"
"time"
@@ -85,6 +86,13 @@ type Server struct {
// one leaves that engine to be installed by hand.
Engines map[string]string `koanf:"engines"`
// EngineOverrides give some nodes different engines from the rest: a
// vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
// all on nodes meant only for llama.cpp. Each picks nodes and lays its
// engines over the cluster's, in order, later ones last; an empty version
// removes that engine from the nodes it picks.
EngineOverrides []EngineOverride `koanf:"engine_overrides"`
// BackupDir is where `llmgw server backup` writes, and where the newest
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
// Empty means /var/backups/llmgw-server.
@@ -378,7 +386,12 @@ type Admission struct {
QueueTTL time.Duration `koanf:"queue_ttl"`
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
Interval time.Duration `koanf:"interval"`
// QueueCapacityTotal bounds everything queued across bands. Reached, a
// request is not refused while a lower band has any queued: the newest of
// the lowest band is turned away instead, with a 503 and Retry-After.
// Zero takes half of all the bands' capacities together.
QueueCapacityTotal int `koanf:"queue_capacity_total"`
Interval time.Duration `koanf:"interval"`
}
// Prefix configures approximate prefix routing.
@@ -543,6 +556,14 @@ func Load(path string) (Server, error) {
// Validate rejects configurations that would otherwise fail confusingly at
// request time.
func (s Server) Validate() error {
for i, o := range s.EngineOverrides {
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
}
if len(o.Engines) == 0 {
return fmt.Errorf("engine_overrides[%d] changes no engine: give it engines, with \"\" for none", i)
}
}
if s.Listen == "" {
return fmt.Errorf("listen is empty")
}
@@ -621,6 +642,58 @@ var DefaultEngines = map[string]string{
"vllm": "0.30.0",
}
// EngineOverride gives the nodes it picks different engines. Every selector
// set must match; at least one must be set.
type EngineOverride struct {
// GPUClass picks nodes with a GPU of this class ("v100-32g").
GPUClass string `koanf:"gpu_class"`
// Nodes picks nodes by id.
Nodes []string `koanf:"nodes"`
// Labels picks nodes carrying all of these.
Labels map[string]string `koanf:"labels"`
// Engines are laid over the cluster's: a version, or "" for none.
Engines map[string]string `koanf:"engines"`
}
// Picks reports whether the override applies to a node with this id, these
// GPU classes and these labels.
func (o EngineOverride) Picks(node string, classes []string, labels map[string]string) bool {
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
return false
}
if o.GPUClass != "" && !slices.Contains(classes, o.GPUClass) {
return false
}
if len(o.Nodes) > 0 && !slices.Contains(o.Nodes, node) {
return false
}
for k, v := range o.Labels {
if labels[k] != v {
return false
}
}
return true
}
// EnginesFor is what one node installs: WantedEngines with every override
// that picks it laid over, in order.
func (s Server) EnginesFor(node string, classes []string, labels map[string]string) map[string]string {
out := s.WantedEngines()
for _, o := range s.EngineOverrides {
if !o.Picks(node, classes, labels) {
continue
}
for k, v := range o.Engines {
if v == "" {
delete(out, k)
continue
}
out[k] = v
}
}
return out
}
// WantedEngines is DefaultEngines with the configuration laid over it.
func (s Server) WantedEngines() map[string]string {
out := map[string]string{}
+38
View File
@@ -0,0 +1,38 @@
package config
import (
"strings"
"testing"
)
func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) {
s := Server{
Engines: map[string]string{"vllm": "0.30.0"},
EngineOverrides: []EngineOverride{
// Volta: CUDA 13 dropped it, CUDA 12 vLLM still runs.
{GPUClass: "v100-32g", Engines: map[string]string{"vllm": "0.25.1"}},
// Nodes meant for llama.cpp alone.
{Labels: map[string]string{"role": "gguf"}, Engines: map[string]string{"vllm": ""}},
},
}
if got := s.EnginesFor("h1", []string{"h100-80g"}, nil); got["vllm"] != "0.30.0" || got["llamacpp"] == "" {
t.Errorf("an H100 node: %v", got)
}
if got := s.EnginesFor("v1", []string{"v100-32g"}, nil); got["vllm"] != "0.25.1" {
t.Errorf("a V100 node: %v", got)
}
got := s.EnginesFor("v2", []string{"v100-32g"}, map[string]string{"role": "gguf"})
if _, has := got["vllm"]; has || got["llamacpp"] == "" {
t.Errorf("a V100 node for llama.cpp alone, the later override last: %v", got)
}
for _, bad := range []EngineOverride{
{Engines: map[string]string{"vllm": ""}},
{GPUClass: "v100-32g"},
} {
s := Server{Listen: ":8080", EngineOverrides: []EngineOverride{bad}}
if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "engine_overrides[0]") {
t.Errorf("%+v: %v", bad, err)
}
}
}
@@ -0,0 +1,71 @@
package adminapi
import (
"bytes"
"encoding/binary"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
// ggufStart is the start of a GGUF file with these string keys.
func ggufStart(kv ...string) []byte {
var b bytes.Buffer
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
b.WriteString("GGUF")
w(uint32(3))
w(uint64(0))
w(uint64(len(kv) / 2))
for i := 0; i < len(kv); i += 2 {
str(kv[i])
w(uint32(8)) // string
str(kv[i+1])
}
b.Write(make([]byte, 4096))
return b.Bytes()
}
func TestOnlyAnAdapterForTheModelsArchitectureIsAttached(t *testing.T) {
files := map[string][]byte{
"/base.gguf": ggufStart("general.architecture", "qwen2"),
"/model.gguf": ggufStart("general.architecture", "qwen2", "general.type", "model"),
"/llama.gguf": ggufStart("general.architecture", "llama", "general.type", "adapter"),
"/lora.gguf": ggufStart("general.architecture", "qwen2", "general.type", "adapter"),
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusPartialContent)
_, _ = w.Write(files[r.URL.Path])
}))
defer srv.Close()
models := []catalog.Model{{
Name: "coder", Source: srv.URL + "/base.gguf",
Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 8}},
}}
for _, tc := range []struct {
file, want string
code int
}{
{"/model.gguf", "not a LoRA adapter", http.StatusBadRequest},
{"/llama.gguf", "for llama models", http.StatusBadRequest},
{"/lora.gguf", "", http.StatusOK},
} {
a, _ := setup(t, models)
fm := &fakeModels{}
a.SetModels(fm)
mux := http.NewServeMux()
a.Register(mux)
req := httptest.NewRequest(http.MethodPost, "/api/v1/models/coder/adapters",
strings.NewReader(`{"adapters":[{"name":"fix","source":"`+srv.URL+tc.file+`"}]}`))
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != tc.code || !strings.Contains(rec.Body.String(), tc.want) {
t.Errorf("%s: %d %s; want %d mentioning %q", tc.file, rec.Code, rec.Body.String(), tc.code, tc.want)
}
}
}
+107 -13
View File
@@ -10,16 +10,19 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"io"
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"time"
"unicode"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
@@ -60,6 +63,7 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
a.registerQuotas(mux)
mux.HandleFunc("POST /api/v1/models/{name}/adapters", a.setAdapters)
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
@@ -204,10 +208,13 @@ type NodeView struct {
// GPUView is one card as a node's page shows it.
type GPUView struct {
Index int `json:"index"`
Name string `json:"name"`
NUMA int `json:"numa"`
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
Index int `json:"index"`
Name string `json:"name"`
// ComputeCapability is what decides which engine builds run on the card:
// CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones.
ComputeCapability string `json:"compute_capability,omitempty"`
NUMA int `json:"numa"`
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
// NearestNIC is the adapter closest to the card, and how close.
NearestNIC string `json:"nearest_nic,omitempty"`
NICPath string `json:"nic_path,omitempty"`
@@ -312,10 +319,15 @@ type ModelView struct {
Warnings []string `json:"warnings,omitempty"`
// Class is its capacity class; Preempted the replicas it lost to a higher
// class; HeldFor the model it is giving room to now, if any.
Class string `json:"class"`
Owner string `json:"owner,omitempty"`
Preempted int `json:"preempted,omitempty"`
HeldFor string `json:"held_for,omitempty"`
Class string `json:"class"`
Owner string `json:"owner,omitempty"`
// External is where a model served outside the cluster goes, provider
// host and its name for the model; empty for one served here.
External string `json:"external,omitempty"`
// Adapters are the LoRA adapters it serves.
Adapters []catalog.Adapter `json:"adapters,omitempty"`
Preempted int `json:"preempted,omitempty"`
HeldFor string `json:"held_for,omitempty"`
}
// ReplicaView is a replica as `ctl replicas` shows it.
@@ -408,6 +420,10 @@ type HeadroomView struct {
Blocked []string `json:"blocked,omitempty"`
// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
TokPerSec float64 `json:"tok_per_sec,omitempty"`
// ByPreempting is how many more would fit by stopping replicas this
// model's class may take; Preempts the models those belong to.
ByPreempting int `json:"by_preempting,omitempty"`
Preempts []string `json:"preempts,omitempty"`
}
// RejectionView is one rejected candidate.
@@ -484,7 +500,8 @@ func (a *API) Nodes() []NodeView {
v.Load = loadView(n.Load, time.Now())
v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
for _, g := range n.GPUs {
gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability,
NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
for hca, rel := range g.NICPaths {
if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
@@ -565,6 +582,7 @@ func (a *API) Models() []ModelView {
Desired: h.Desired, Failures: h.Failures,
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
External: externalLabel(m), Adapters: m.Adapters,
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
@@ -647,6 +665,7 @@ type Models interface {
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
}
// ResourcesRequest sets what a model's profiles reserve of a node.
@@ -967,7 +986,8 @@ func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
return HeadroomView{}, false
}
return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec}, true
Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec,
ByPreempting: h.ByPreempting, Preempts: h.Preempts}, true
}
func (a *API) Explain(model string) ExplainView {
@@ -1222,6 +1242,9 @@ type RestartRequest struct {
Replicas []string `json:"replicas,omitempty"`
Models []string `json:"models,omitempty"`
All bool `json:"all,omitempty"`
// InPlace stops each replica before starting its replacement, rather
// than starting the replacement beside it first.
InPlace bool `json:"in_place,omitempty"`
}
// RestartResult lists the replicas queued.
@@ -1245,7 +1268,11 @@ func (a *API) restart(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
return
}
if err := a.ctl.Reconciler().Restart(ids...); err != nil {
restart := a.ctl.Reconciler().Restart
if req.InPlace {
restart = a.ctl.Reconciler().RestartInPlace
}
if err := restart(ids...); err != nil {
writeError(w, http.StatusNotFound, err.Error())
return
}
@@ -1438,3 +1465,70 @@ func reservationWarnings(m catalog.Model, measuredGB float64) []string {
}
return out
}
// externalLabel is "api.openai.com · gpt-4o-mini" for a model served elsewhere.
func externalLabel(m catalog.Model) string {
if !m.IsExternal() {
return ""
}
host := m.External.URL
if u, err := url.Parse(host); err == nil && u.Host != "" {
host = u.Host
}
return host + " · " + m.External.Model
}
// AdaptersRequest replaces a model's LoRA adapters with these.
type AdaptersRequest struct {
Adapters []catalog.Adapter `json:"adapters"`
}
// setAdapters replaces a model's adapters, after reading each new file to
// check it is an adapter for the model's architecture: one that is not is a
// replica that fails to start, found minutes later on a node.
func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req AdaptersRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
name := r.PathValue("name")
m, ok := a.ctl.Catalog().Resolve(name)
if !ok {
writeError(w, http.StatusBadRequest, "no model "+name)
return
}
base, baseErr := gguf.KindFromURL(r.Context(), nil, m.Source)
have := map[string]bool{}
for _, ad := range m.Adapters {
have[ad.Source] = true
}
for _, ad := range req.Adapters {
if have[ad.Source] {
continue // already checked when it was attached
}
h, err := gguf.KindFromURL(r.Context(), nil, ad.Source)
switch {
case err != nil:
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
return
case !h.IsAdapter():
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
return
case baseErr == nil && h.Architecture != base.Architecture:
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
ad.Name, h.Architecture, m.Name, base.Architecture))
return
}
}
out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
}
+4
View File
@@ -258,6 +258,10 @@ type fakeModels struct {
err error
}
func (f *fakeModels) SetAdapters(_ context.Context, name string, as []catalog.Adapter, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Adapters: as}, f.err
}
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Owner: owner}, f.err
}
+66 -18
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"net/http"
"sort"
"strings"
"time"
@@ -72,38 +73,24 @@ func (a *API) registerQuotas(mux *http.ServeMux) {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
scope, id, _, err := a.resolveScope(r.Context(), req.Scope)
key, err := a.SetQuota(r.Context(), req, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if !req.Guarantee.Any() && !req.Limit.Any() {
writeError(w, http.StatusBadRequest, "a quota needs a guarantee or a limit; to remove one, DELETE it")
return
}
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit,
UpdatedBy: actorOf(r)}
if err := a.quotas.PutCapacityQuota(r.Context(), q); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"scope": quota.Key(scope, id)})
writeJSON(w, http.StatusOK, map[string]string{"scope": key})
})
mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
if a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
scope, id, _, err := a.resolveScope(r.Context(), r.PathValue("scope"))
key, err := a.ClearQuota(r.Context(), r.PathValue("scope"))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if err := a.quotas.DeleteCapacityQuota(r.Context(), scope, id); err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"removed": quota.Key(scope, id)})
writeJSON(w, http.StatusOK, map[string]string{"removed": key})
})
mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
if a.models == nil || a.quotas == nil {
@@ -137,6 +124,9 @@ func (a *API) registerQuotas(mux *http.ServeMux) {
// QuotaViews lists every quota with what its scope holds now.
func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
if a.quotas == nil {
return nil, nil
}
qs, err := a.quotas.CapacityQuotas(ctx)
if err != nil {
return nil, err
@@ -235,3 +225,61 @@ func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string,
}
return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
}
// SetQuota sets a scope's capacity quota, named by path or ID, and returns
// the scope as org:ID.
func (a *API) SetQuota(ctx context.Context, req QuotaRequest, by string) (string, error) {
if a.quotas == nil {
return "", fmt.Errorf("this control plane has no store")
}
scope, id, _, err := a.resolveScope(ctx, req.Scope)
if err != nil {
return "", err
}
if !req.Guarantee.Any() && !req.Limit.Any() {
return "", fmt.Errorf("a quota needs a guarantee or a limit; to remove one, clear it")
}
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit, UpdatedBy: by}
if err := a.quotas.PutCapacityQuota(ctx, q); err != nil {
return "", err
}
return quota.Key(scope, id), nil
}
// ClearQuota removes a scope's capacity quota.
func (a *API) ClearQuota(ctx context.Context, scope string) (string, error) {
if a.quotas == nil {
return "", fmt.Errorf("this control plane has no store")
}
s, id, _, err := a.resolveScope(ctx, scope)
if err != nil {
return "", err
}
if err := a.quotas.DeleteCapacityQuota(ctx, s, id); err != nil {
return "", err
}
return quota.Key(s, id), nil
}
// ScopeName is a part of the tenancy tree: org:ID and its path.
type ScopeName struct {
Key string `json:"scope"`
Name string `json:"name"`
}
// Scopes lists every organization, team and project, by path.
func (a *API) Scopes(ctx context.Context) ([]ScopeName, error) {
if a.quotas == nil {
return nil, nil
}
names, err := a.scopeNames(ctx)
if err != nil {
return nil, err
}
out := make([]ScopeName, 0, len(names))
for k, n := range names {
out = append(out, ScopeName{Key: k, Name: n})
}
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
return out, nil
}
+8 -2
View File
@@ -24,6 +24,9 @@ type TargetView struct {
Room int `json:"room"`
Fits bool `json:"fits"`
Blocked []string `json:"blocked,omitempty"`
// Preempts is whose replicas would be stopped to make the room, when
// free room alone is not enough; Room counts that room too.
Preempts []string `json:"preempts,omitempty"`
}
// Target sizes a model for a throughput.
@@ -40,8 +43,11 @@ func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
}
v.Current = h.Current
v.More = max(v.Needed-h.Current, 0)
v.Room = h.More
v.Fits = v.More <= h.More || h.Limited
v.Room = h.More + h.ByPreempting
v.Fits = v.More <= v.Room || h.Limited
if v.More > h.More {
v.Preempts = h.Preempts
}
if !v.Fits {
v.Blocked = h.Blocked
}
+5
View File
@@ -31,6 +31,11 @@ func (a *API) Orphans() []OrphanView {
var wanted []string
for _, m := range a.ctl.Catalog().Models() {
wanted = append(wanted, m.Source)
// A model's adapters are its weights too; collecting them would
// leave replicas that cannot restart.
for _, ad := range m.Adapters {
wanted = append(wanted, ad.Source)
}
}
out := make([]OrphanView, 0)
+18
View File
@@ -189,6 +189,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
GpusPerNode: int32(p.GPUsPerNode), Nodes: int32(p.Nodes),
Args: p.Args, Env: p.Env,
ModelPath: p.ModelPath, ServedModelName: p.ServedModelName,
Adapters: adaptersToWire(p.Adapters),
},
StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(),
Pool: cmd.Pool,
@@ -224,6 +225,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
GPUsPerNode: int(p.GetGpusPerNode()), Nodes: int(p.GetNodes()),
Args: p.GetArgs(), Env: p.GetEnv(),
ModelPath: p.GetModelPath(), ServedModelName: p.GetServedModelName(),
Adapters: adaptersFromWire(p.GetAdapters()),
},
}
return spec, time.Duration(m.GetStartupTimeoutMs()) * time.Millisecond
@@ -292,3 +294,19 @@ func fabricFrom(state, status, cluster, clique, bandwidth string) *discovery.GPU
}
return &discovery.GPUFabric{State: state, Status: status, ClusterUUID: cluster, CliqueID: clique, Bandwidth: bandwidth}
}
func adaptersToWire(as []engines.Adapter) []*llmgwv1.LoraAdapter {
var out []*llmgwv1.LoraAdapter
for _, a := range as {
out = append(out, &llmgwv1.LoraAdapter{Name: a.Name, Source: a.Path})
}
return out
}
func adaptersFromWire(as []*llmgwv1.LoraAdapter) []engines.Adapter {
var out []engines.Adapter
for _, a := range as {
out = append(out, engines.Adapter{Name: a.GetName(), Path: a.GetSource()})
}
return out
}
+20
View File
@@ -0,0 +1,20 @@
package agentsvc
import (
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
)
func TestAdaptersTravelToTheAgentInOrder(t *testing.T) {
cmd := reconciler.ShardCommand{ReplicaID: "r", ShardID: "s", Spec: engines.ShardSpec{
Profile: engines.Profile{Name: "cpu", Engine: "llamacpp", ModelPath: "/m.gguf",
Adapters: []engines.Adapter{{Name: "sql", Path: "http://mirror/sql.gguf"}, {Name: "docs", Path: "http://mirror/docs.gguf"}}},
}}
spec, _ := ShardSpecFromStart(StartShardFromCommand(cmd))
got := spec.Profile.Adapters
if len(got) != 2 || got[0].Name != "sql" || got[1].Path != "http://mirror/docs.gguf" {
t.Errorf("adapters %+v", got)
}
}
@@ -6,6 +6,7 @@ import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
@@ -25,3 +26,31 @@ func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
t.Error("uv, a tool for the others, is offered as an engine")
}
}
func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
s := &Service{}
s.SetEnginesFor(func(n inventory.Node) map[string]string {
if n.ID == "volta" {
return map[string]string{"llamacpp": "b1", "vllm": "0.25.1"}
}
return map[string]string{"llamacpp": "b1", "vllm": "0.30.0"}
})
reg := &llmgwv1.Register{InstallableEngines: []string{"llamacpp", "vllm"}}
want := func(p *llmgwv1.Provision) map[string]string {
out := map[string]string{}
for _, e := range p.GetEngines() {
out[e.GetEngine()] = e.GetVersion()
}
return out
}
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, reg)); got["vllm"] != "0.25.1" {
t.Errorf("the Volta node: %v", got)
}
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, reg)); got["vllm"] != "0.30.0" {
t.Errorf("the Hopper node: %v", got)
}
// A node that disabled vLLM does not offer it, and is not asked for it.
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, &llmgwv1.Register{InstallableEngines: []string{"llamacpp"}})); got["vllm"] != "" {
t.Errorf("asked a node for an engine it does not install: %v", got)
}
}
+28 -12
View File
@@ -53,8 +53,9 @@ type Service struct {
mu sync.RWMutex
conns map[string]*conn
// engines is what every node is told to install, set once at start.
engines map[string]string
// enginesFor is what a node is told to install, set once at start: the
// cluster's engines, with any override that picks the node laid over.
enginesFor func(node inventory.Node) map[string]string
// logWaits are the log requests awaiting a reply, by request id.
logMu sync.Mutex
@@ -68,7 +69,14 @@ type Service struct {
}
// SetEngines sets the engines every node installs, by name and version.
func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
func (s *Service) SetEngines(e map[string]string) {
e = maps.Clone(e)
s.enginesFor = func(inventory.Node) map[string]string { return e }
}
// SetEnginesFor sets what each node installs, decided per node: a cluster
// of two GPU generations needs two vLLM builds, or vLLM on only one.
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
// conn is one agent's open stream.
type conn struct {
@@ -179,15 +187,8 @@ func (s *Service) Connect(
// What this node should have installed. Sent on every registration, so a
// node rebuilt from nothing is told the same as one that has been here a
// month, and a changed version reaches every node on its next connect.
if s.engines != nil {
p := &llmgwv1.Provision{}
can := installable(reg)
for _, e := range slices.Sorted(maps.Keys(s.engines)) {
if !slices.Contains(can, e) {
continue
}
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
}
if s.enginesFor != nil {
p := s.provisionFor(node, reg)
if err := stream.Send(&llmgwv1.ControlMessage{
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
}); err != nil {
@@ -853,6 +854,21 @@ func rate(before, now uint64, secs float64) float64 {
return float64(now-before) / secs
}
// provisionFor is what a node is told to install: the engines decided for
// it, those it can install.
func (s *Service) provisionFor(node inventory.Node, reg *llmgwv1.Register) *llmgwv1.Provision {
p := &llmgwv1.Provision{}
can := installable(reg)
wants := s.enginesFor(node)
for _, e := range slices.Sorted(maps.Keys(wants)) {
if !slices.Contains(can, e) {
continue
}
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e]})
}
return p
}
// installable are the engines a node can be asked to install. An agent older
// than the list said nothing, and could install llama.cpp alone: asked for
// vLLM too, it reported it unsupported, and a cluster half way through an
+186 -6
View File
@@ -7,6 +7,7 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"net/url"
"slices"
"strings"
"time"
@@ -160,8 +161,20 @@ type Model struct {
// Owner is the part of the tenancy tree whose capacity quota this model
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
// held by the cluster, counted against nobody.
Owner string `json:"owner,omitempty" koanf:"owner"`
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
Owner string `json:"owner,omitempty" koanf:"owner"`
// External is a model served by someone else — an OpenAI-compatible API
// outside the cluster — rather than by replicas placed here. Such a model
// has no profiles and nothing is placed for it; the router sends its
// requests to the provider. Most useful named in another model's
// fallbacks: used only when the cluster has no capacity left. Its
// prompts leave the cluster, so which keys may name it is worth an
// allowlist.
External *External `json:"external,omitempty" koanf:"external"`
// Adapters are LoRA adapters served on this model's replicas, each asked
// for by its own name as though it were a model. Every replica loads
// every adapter, applying one only to a request that names it.
Adapters []Adapter `json:"adapters,omitempty" koanf:"adapters"`
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
// RoutingProfile selects how the router scores endpoints for this model.
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
SLO SLO `json:"slo,omitempty" koanf:"slo"`
@@ -204,14 +217,17 @@ type Capabilities struct {
type Catalog struct {
models map[string]Model
aliases map[string]string
order []string
// adapters maps an adapter's name to the model it runs on.
adapters map[string]string
order []string
}
// New builds a catalog, rejecting definitions that would fail confusingly later.
func New(models []Model) (*Catalog, error) {
c := &Catalog{
models: make(map[string]Model, len(models)),
aliases: map[string]string{},
models: make(map[string]Model, len(models)),
aliases: map[string]string{},
adapters: map[string]string{},
}
for i := range models {
@@ -240,6 +256,18 @@ func New(models []Model) (*Catalog, error) {
}
}
// Adapters are named like models, so they share the namespace too.
for _, name := range c.order {
for _, a := range c.models[name].Adapters {
_, isModel := c.models[a.Name]
_, isAlias := c.aliases[a.Name]
if _, isAdapter := c.adapters[a.Name]; isModel || isAlias || isAdapter {
return nil, fmt.Errorf("model %q: adapter %q collides with a model, an alias or another adapter", name, a.Name)
}
c.adapters[a.Name] = name
}
}
// Fallbacks are resolved now rather than at request time: a typo in a
// fallback should stop a deployment, not surface during an outage when the
// fallback is finally needed.
@@ -305,7 +333,21 @@ func (m Model) validate() error {
if m.Source == "" {
return fmt.Errorf("model %q: source is required", m.Name)
}
if len(m.Profiles) == 0 {
if repo, ok := repositoryPage(m.Source); ok {
// A page, not weights: a node fetches the HTML as one file and the
// engine fails at start, on a path that looks right.
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
m.Name, m.Source, repo)
}
if err := validateAdapters(m); err != nil {
return err
}
if m.External != nil {
if err := m.External.validate(m.Name, len(m.Profiles)); err != nil {
return err
}
} else if len(m.Profiles) == 0 {
return fmt.Errorf("model %q: at least one profile is required", m.Name)
}
if m.Owner != "" && !validOwner(m.Owner) {
@@ -346,6 +388,21 @@ func (m Model) validate() error {
}
continue
}
// llama.cpp on GPUs splits a model across one node's cards by layer,
// and how many is gpus_per_node. Not tp: llama.cpp has no tensor
// parallelism, and a tp it was given was a width nothing placed — the
// replica stayed unplaced, or ran on one card while four were meant.
if p.Engine == "llamacpp" {
switch {
case p.Nodes > 1:
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
m.Name, p.Name, p.Nodes)
case p.TP > 1 || p.PP > 1:
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
}
continue
}
// The arithmetic has to add up, or the engine is handed a shard spec it
// cannot satisfy and fails minutes later during model loading.
if want := p.GPUsPerNode * p.Nodes; want != p.TP*max(p.PP, 1) {
@@ -357,6 +414,23 @@ func (m Model) validate() error {
}
// Resolve maps a client-supplied name or alias onto a model.
// ResolveAdapter finds an adapter by its name: the model it runs on, the
// adapter, and its index among the model's adapters — which is the id the
// engine knows it by.
func (c *Catalog) ResolveAdapter(name string) (Model, Adapter, int, bool) {
base, ok := c.adapters[name]
if !ok {
return Model{}, Adapter{}, 0, false
}
m := c.models[base]
for i, a := range m.Adapters {
if a.Name == name {
return m, a, i, true
}
}
return Model{}, Adapter{}, 0, false
}
func (c *Catalog) Resolve(name string) (Model, bool) {
if m, ok := c.models[name]; ok {
return m, true
@@ -433,6 +507,7 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
Env: p.Env,
ModelPath: m.Source,
ServedModelName: m.Name,
Adapters: m.engineAdapters(),
}, true
}
return engines.Profile{}, false
@@ -564,3 +639,108 @@ func validOwner(o string) bool {
}
return false
}
// External is a provider outside the cluster that serves a model.
type External struct {
// URL is the provider's base address, without /v1: https://api.openai.com.
URL string `json:"url" koanf:"url"`
// Model is what the provider calls it; requests are rewritten to it.
Model string `json:"model" koanf:"model"`
// APIKeyFile holds the provider's key, on the control plane. A file
// rather than the key itself, so the key is in neither the catalog, the
// store nor the audit log; empty for a provider that asks for none.
APIKeyFile string `json:"api_key_file,omitempty" koanf:"api_key_file"`
}
func (e *External) validate(model string, profiles int) error {
if profiles > 0 {
return fmt.Errorf("model %q: an external model is served elsewhere and has no profiles", model)
}
u, err := url.Parse(e.URL)
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
return fmt.Errorf("model %q: external url %q is not an http(s) address", model, e.URL)
}
if strings.TrimSpace(e.Model) == "" {
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
}
return nil
}
// IsExternal reports whether a model is served outside the cluster.
func (m Model) IsExternal() bool { return m.External != nil }
// Adapter is a LoRA adapter served on a model's replicas.
type Adapter struct {
// Name is what clients ask for, like a model name.
Name string `json:"name" koanf:"name"`
// Source is the adapter file (GGUF for llama.cpp), fetched by each node
// as the model's weights are.
Source string `json:"source" koanf:"source"`
// Scale is how strongly it is applied; zero takes 1.
Scale float64 `json:"scale,omitempty" koanf:"scale"`
}
// EffectiveScale is the scale an adapter is applied at.
func (a Adapter) EffectiveScale() float64 {
if a.Scale <= 0 {
return 1
}
return a.Scale
}
func validateAdapters(m Model) error {
if len(m.Adapters) == 0 {
return nil
}
if m.External != nil {
return fmt.Errorf("model %q: an external model cannot carry adapters", m.Name)
}
for _, p := range m.Profiles {
// llama.cpp loads adapters at start and applies them per request;
// the other engines are not driven that way yet.
if p.Engine != "llamacpp" && p.Engine != "sim" {
return fmt.Errorf("model %q profile %q: adapters are served by llamacpp profiles only", m.Name, p.Name)
}
}
seen := map[string]bool{}
for _, a := range m.Adapters {
switch {
case a.Name == "" || a.Source == "":
return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)
case a.Name == m.Name:
return fmt.Errorf("model %q: an adapter cannot have the model's own name", m.Name)
case seen[a.Name]:
return fmt.Errorf("model %q: adapter %q is declared twice", m.Name, a.Name)
case a.Scale < 0:
return fmt.Errorf("model %q: adapter %q has a negative scale", m.Name, a.Name)
}
seen[a.Name] = true
}
return nil
}
func (m Model) engineAdapters() []engines.Adapter {
var out []engines.Adapter
for _, a := range m.Adapters {
out = append(out, engines.Adapter{Name: a.Name, Path: a.Source})
}
return out
}
// repositoryPage reports whether source is a Hugging Face repository's page
// (https://huggingface.co/org/repo, or one of its tree views) rather than a
// file under /resolve/, and the repository's id.
func repositoryPage(source string) (string, bool) {
for _, host := range []string{"https://huggingface.co/", "http://huggingface.co/", "https://hf.co/"} {
rest, ok := strings.CutPrefix(source, host)
if !ok {
continue
}
parts := strings.Split(strings.Trim(rest, "/"), "/")
if len(parts) < 2 || (len(parts) > 2 && parts[2] == "resolve") {
return "", false
}
return parts[0] + "/" + parts[1], true
}
return "", false
}
+45
View File
@@ -321,3 +321,48 @@ func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
t.Errorf("ordinary tuning refused: %v", err)
}
}
func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
four := model("big", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", GPUsPerNode: 4})
c, err := New([]Model{four})
if err != nil {
t.Fatalf("gpus_per_node 4 on llama.cpp: %v", err)
}
m, _ := c.Resolve("big")
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
t.Errorf("placement was not asked for four GPUs: %+v", pl)
}
// tp is a width llama.cpp has no way to use; refused, saying what to
// write instead.
for _, p := range []Profile{
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
} {
_, err := New([]Model{model("x", p)})
if err == nil {
t.Errorf("%s: accepted", p.Name)
}
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
}
}
}
func TestARepositoryPageIsNotASource(t *testing.T) {
for _, src := range []string{
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
} {
m := model("awq")
m.Source = src
_, err := New([]Model{m})
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
t.Errorf("%s: %v", src, err)
}
}
m := model("file")
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
if _, err := New([]Model{m}); err != nil {
t.Errorf("a file under /resolve/ is a source: %v", err)
}
}
+31 -1
View File
@@ -9,6 +9,7 @@ import (
"encoding/hex"
"fmt"
"log/slog"
"os"
"strings"
"sync"
"time"
@@ -338,6 +339,34 @@ func (c *Controller) endpoints() []endpoint.Info {
Priority: prof.Priority,
})
}
return append(out, c.externalEndpoints()...)
}
// externalEndpoints are the models served outside the cluster, as endpoints:
// one each, with the provider's key read from its file every time, so a key
// rotated on disk is used from the next publish on.
func (c *Controller) externalEndpoints() []endpoint.Info {
var out []endpoint.Info
for _, m := range c.cat.Load().Models() {
if !m.IsExternal() {
continue
}
key := ""
if f := m.External.APIKeyFile; f != "" {
b, err := os.ReadFile(f) //nolint:gosec // the operator's own path
if err != nil {
c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
continue
}
key = strings.TrimSpace(string(b))
}
out = append(out, endpoint.Info{
ID: "external:" + m.Name, URL: strings.TrimRight(m.External.URL, "/"),
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
UpstreamModel: m.External.Model,
})
}
return out
}
@@ -345,7 +374,8 @@ func (c *Controller) endpoints() []endpoint.Info {
func fingerprint(infos []endpoint.Info) string {
var b strings.Builder
for _, i := range infos {
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority)
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority,
i.UpstreamModel, sha256.Sum256([]byte(i.APIKey)))
}
return b.String()
}
+75
View File
@@ -0,0 +1,75 @@
package control
import (
"context"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
type recordSink struct {
mu sync.Mutex
infos []endpoint.Info
times int
}
func (s *recordSink) SetEndpoints(_ context.Context, infos []endpoint.Info) {
s.mu.Lock()
defer s.mu.Unlock()
s.infos, s.times = infos, s.times+1
}
type noDispatch struct{}
func (noDispatch) StartShard(context.Context, reconciler.ShardCommand) error { return nil }
func (noDispatch) StopShard(context.Context, string, string) error { return nil }
func (noDispatch) DrainShard(context.Context, string, string) error { return nil }
func TestAnExternalModelIsPublishedWithItsKeyAndNothingIsPlacedForIt(t *testing.T) {
dir := t.TempDir()
keyFile := filepath.Join(dir, "provider.key")
if err := os.WriteFile(keyFile, []byte("sk-provider-1\n"), 0o600); err != nil {
t.Fatal(err)
}
models := []catalog.Model{
{Name: "cloud", Source: "external", Aliases: []string{"gpt"}, Scaling: catalog.Scaling{Min: 3, Max: 3},
External: &catalog.External{URL: "https://api.example.com/", Model: "gpt-4o-mini", APIKeyFile: keyFile}},
{Name: "gone", Source: "external",
External: &catalog.External{URL: "https://api.example.com", Model: "m", APIKeyFile: filepath.Join(dir, "missing")}},
}
cat, err := catalog.New(models)
if err != nil {
t.Fatal(err)
}
sink := &recordSink{}
c := New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler))
c.Publish(t.Context())
if len(sink.infos) != 1 {
t.Fatalf("published %+v; the model whose key is missing must not be offered", sink.infos)
}
e := sink.infos[0]
if e.ID != "external:cloud" || e.URL != "https://api.example.com" || e.APIKey != "sk-provider-1" ||
e.UpstreamModel != "gpt-4o-mini" || e.Engine != endpoint.EngineExternal || len(e.Models) != 2 {
t.Errorf("endpoint %+v", e)
}
// A key rotated on disk is published on the next pass.
_ = os.WriteFile(keyFile, []byte("sk-provider-2"), 0o600)
c.Publish(t.Context())
if sink.times != 2 || sink.infos[0].APIKey != "sk-provider-2" {
t.Errorf("the rotated key was not published: %d pushes, key %q", sink.times, sink.infos[0].APIKey)
}
// Scaling of 3 or not, nothing is wanted of the cluster for a model
// served elsewhere, and so it never reads as short or unplaceable.
for _, h := range c.Reconciler().ModelHealth() {
if h.Model == "cloud" && (h.Desired != 0 || h.Unplaceable) {
t.Errorf("an external model wants replicas: %+v", h)
}
}
}
+20
View File
@@ -87,6 +87,9 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
if m.Source != "" {
out[m.Source] = append(out[m.Source], m)
}
for _, ad := range m.Adapters {
out[ad.Source] = append(out[ad.Source], m)
}
}
return out, nil
}
@@ -268,6 +271,23 @@ func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog
return final, err
}
// SetAdapters replaces the LoRA adapters a model serves. Replicas load the
// new set when they next start.
func (s *Service) SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error) {
final, err := s.update(ctx, name, by, "change its adapters", func(m *catalog.Model) error {
m.Adapters = adapters
return nil
})
if err == nil {
names := make([]string, 0, len(adapters))
for _, a := range adapters {
names = append(names, a.Name)
}
s.log.Info("model adapters set", "model", name, "adapters", names, "by", by)
}
return final, err
}
// update changes one model added at runtime and keeps the change.
//
// Built before anything is written, like Add: catalog.New is what applies the
+3
View File
@@ -94,6 +94,9 @@ func (r *Reconciler) desiredFor(m catalog.Model) int {
// baseFor is what a model runs without load: its minimum, or for an on-demand
// model one while somebody is asking and none once they stop.
func (r *Reconciler) baseFor(m catalog.Model) int {
if m.IsExternal() {
return 0 // served elsewhere: nothing to place, whatever its scaling says
}
if !m.Scaling.OnDemand() {
return m.Scaling.Min
}
+40 -3
View File
@@ -28,6 +28,11 @@ type Headroom struct {
// TokPerSec is what one more replica was measured to serve, on the
// profile placement would give it; zero when it was never measured.
TokPerSec float64
// ByPreempting is how many more, beyond More, would fit by stopping
// replicas the model's class may take, and Preempts the models those
// replicas belong to: what "room" means for a model that outranks others.
ByPreempting int
Preempts []string
}
// Headroom tries placing up to limit more replicas of a model on a copy of
@@ -61,12 +66,38 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
if pol != nil {
usage = pol.Usage(r.Holdings())
}
for h.More < limit {
place := func() placement.Result {
res := sched.Place(r.orderedProfiles(m), avoid)
if res.Placement == nil && len(avoid) > 0 {
// As scaleUp does: spreading is a preference, not a rule.
res = sched.Place(r.orderedProfiles(m), nil)
}
return res
}
// What the model's class may take, in the order preemption would, each
// released on the copy only when free room has run out.
victims, nextVictim, preempting := r.victims(m), 0, false
for h.More+h.ByPreempting < limit {
res := place()
// Named only once they have made room: candidates tried on the way
// to finding there is none would otherwise read as whose replicas a
// placement would stop.
var tried []string
for res.Placement == nil && nextVictim < len(victims) {
v := victims[nextVictim]
nextVictim++
inv.ReleaseReplica(v.ID)
tried = append(tried, v.Model)
res = place()
}
if res.Placement != nil && len(tried) > 0 {
preempting = true
for _, m := range tried {
if !slices.Contains(h.Preempts, m) {
h.Preempts = append(h.Preempts, m)
}
}
}
if res.Placement == nil {
for _, rej := range res.Rejections {
h.Blocked = append(h.Blocked, rej.String())
@@ -103,8 +134,14 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
nodes = append(nodes, a.NodeID)
}
avoid = append(avoid, nodes...)
h.Nodes = append(h.Nodes, nodes)
h.More++
// Once anything was stopped, what fits after is counted as fitting
// by preempting: it may be sitting in the room that was freed.
if preempting {
h.ByPreempting++
} else {
h.Nodes = append(h.Nodes, nodes)
h.More++
}
}
h.Limited = true
return h, true
+104
View File
@@ -312,3 +312,107 @@ func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
}
}
}
func TestHeadroomCountsWhatAHigherClassCouldTake(t *testing.T) {
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
got, _ := h.r.Headroom("vital", 3)
if got.More != 0 || got.ByPreempting != 1 || len(got.Preempts) != 1 || got.Preempts[0] != "chat" {
t.Errorf("a critical model should see one more by stopping chat: %+v", got)
}
// Tried on a copy: nothing was stopped.
if replicasOf(h, "chat") != 1 {
t.Fatal("asking about room stopped a replica")
}
// A model of chat's own class may not take its minimum, so has no such room.
peer := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("peer", "", 1), 0)}, gpuNode("1", 8))
peer.settle(t, 3)
peer.advance(peer.r.cfg.MinLifetime + time.Minute)
if got, _ := peer.r.Headroom("peer", 3); got.ByPreempting != 0 {
t.Errorf("a standard model counted room in another's minimum: %+v", got)
}
}
func TestARestartBorrowsALowerClassesRoomAndGivesItBack(t *testing.T) {
// chat fills node 1; the only other node holds an opportunistic replica.
chat := gpuModel("chat", "", 1)
extra := gpuModel("extra", catalog.ClassOpportunistic, 1)
h := newFleetHarness(t, []catalog.Model{chat, extra}, gpuNode("1", 8), gpuNode("2", 8))
h.settle(t, 3)
h.advance(h.r.cfg.MinLifetime + time.Minute)
var old string
for _, rep := range h.r.Replicas() {
if rep.Model == "chat" {
old = rep.ID
}
}
if old == "" || replicasOf(h, "extra") != 1 {
t.Fatalf("setup: %+v", h.r.Replicas())
}
if err := h.r.Restart(old); err != nil {
t.Fatal(err)
}
// Replacement first, beside the old one, in the room extra gives up;
// the old one keeps serving until then.
servedThroughout := true
for range 8 {
h.r.Reconcile(t.Context())
h.markReady()
ready := 0
for _, rep := range h.r.Replicas() {
if rep.Model == "chat" && rep.State == Ready {
ready++
}
}
if ready == 0 {
servedThroughout = false
}
h.advance(h.r.cfg.PreemptGrace + time.Second)
}
if !servedThroughout {
t.Error("chat had no ready replica during its restart")
}
var newChat bool
for _, rep := range h.r.Replicas() {
if rep.Model == "chat" && rep.ID != old && rep.State == Ready {
newChat = true
}
if rep.ID == old {
t.Errorf("the old replica is still there: %+v", rep)
}
}
if !newChat {
t.Fatalf("no new chat replica: %+v", h.r.Replicas())
}
// And extra has its replica back, in the room the old chat left.
h.settle(t, 4)
if replicasOf(h, "extra") != 1 {
t.Errorf("the opportunistic model did not get its room back: %+v", h.r.Replicas())
}
}
func TestHeadroomNamesOnlyTheReplicasThatWouldMakeRoom(t *testing.T) {
// vital needs a whole node and chat is on the only one; a second
// critical can never fit. Asking for 3: one by stopping chat, then no
// candidates left — nothing else may be named.
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
got, _ := h.r.Headroom("vital", 3)
if got.ByPreempting != 1 || len(got.Preempts) != 1 {
t.Errorf("%+v", got)
}
// Free room for one, and candidates that cannot make room for a second:
// none of them may be listed.
small := catalog.Model{Name: "small", Source: "/m", Class: catalog.ClassOpportunistic,
Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{{Name: "tp2", Engine: "vllm", GPUClass: "h100-80g", TP: 2}}}
h2 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 8))
got2, _ := h2.r.Headroom("big", 3)
if got2.More != 1 || got2.ByPreempting != 1 || len(got2.Preempts) != 1 {
t.Errorf("one free, one by stopping small: %+v", got2)
}
h3 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 6))
got3, _ := h3.r.Headroom("big", 3)
if got3.ByPreempting != 0 || len(got3.Preempts) != 0 {
t.Errorf("stopping small frees a 6-GPU node, too small for big; nobody may be named: %+v", got3)
}
}
+28 -7
View File
@@ -255,6 +255,11 @@ type Reconciler struct {
// restarted and not yet begun. One is replaced at a time: replacing them
// all at once would need the memory of every replica twice over.
restartQueue map[string][]string
// restartInPlace are the replicas asked to restart by stopping first.
restartInPlace map[string]bool
// restartRefused says, by replica, why a restart was given up rather
// than leave the model without it.
restartRefused map[string]string
// parked holds models that failed ParkAfter times in a row.
parked map[string]parkRecord
// extra is how many replicas load has added above a model's base, and
@@ -304,6 +309,8 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
reservedPorts: map[int]struct{}{},
wanted: map[string]time.Time{},
restartQueue: map[string][]string{},
restartInPlace: map[string]bool{},
restartRefused: map[string]string{},
parked: map[string]parkRecord{},
extra: map[string]int{},
busySince: map[string]time.Time{},
@@ -729,6 +736,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
r.mu.Unlock()
desired := r.desiredFor(m)
outcome := notTried
switch {
case len(liveIDs) < desired && isParked:
// Not placed. Its failure is on every status the operator reads.
@@ -742,7 +750,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
// the real error in noise.
return
}
r.scaleUp(ctx, m, avoid)
outcome = r.scaleUp(ctx, m, avoid)
case len(liveIDs) > desired:
r.scaleDown(ctx, liveIDs[desired:])
}
@@ -781,7 +789,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
}
}
r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
r.retire(ctx, m, retiringIDs, readyLive, len(liveIDs), desired, outcome, isParked)
}
// scaleUp places and starts one replica per pass.
@@ -789,7 +797,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
// One at a time on purpose: placement decisions depend on what is already
// allocated, and starting several at once from the same inventory snapshot
// would hand them the same GPUs.
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) scaleOutcome {
profiles := r.orderedProfiles(m)
res := r.sched.Place(profiles, avoid)
if res.Placement == nil && len(avoid) > 0 {
@@ -813,13 +821,13 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
// the pass that reaps it, and the next pass places this model — which
// keeps the stop-then-start ordering in one place rather than two.
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
return
return makingRoom
}
if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
return
return makingRoom
}
r.notePlacementFailure(m.Name, res.Explain())
return
return noRoom
}
if why := r.overLimit(m, res.Placement); why != "" {
// Recorded as the placement's answer, so the model reads as not
@@ -829,13 +837,26 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
{Profile: res.Placement.Profile.Name, Reason: why}}}
r.mu.Unlock()
r.notePlacementFailure(m.Name, why)
return
return overLimit
}
if err := r.startReplica(ctx, m, res.Placement); err != nil {
r.log.Error("starting replica failed", "model", m.Name, "err", err)
return noRoom
}
return placed
}
// scaleOutcome is what one scaleUp did.
type scaleOutcome int
const (
notTried scaleOutcome = iota // nothing was asked of placement this pass
placed // a replica was started
makingRoom // room is being made: a preemption, an idle replica stopping
noRoom // placement found nowhere, and nothing is making room
overLimit // there is room, but the owner's quota does not allow it
)
func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *placement.Placement) error {
prof, ok := m.ProfileByName(pl.Profile.Name)
if !ok {
+64 -10
View File
@@ -114,6 +114,17 @@ type harness struct {
inv *inventory.Store
disp *fakeDispatcher
clock *clock
live *catalog.Live
}
// setModels replaces the catalog the reconciler works from, as an edit does.
func (h *harness) setModels(t *testing.T, models ...catalog.Model) {
t.Helper()
cat, err := catalog.New(models)
if err != nil {
t.Fatal(err)
}
h.live.Store(cat)
}
func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness {
@@ -135,7 +146,8 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
cfg.UnreachableAfter = 15 * time.Second
cfg.LostAfter = 60 * time.Second
r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
live := catalog.NewLive(cat)
r := New(cfg, live, inv, disp, slog.New(slog.DiscardHandler))
r.SetClock(c.now)
// Nodes were just registered, so their LastSeen is wall-clock now, which is
// far from the fake clock's epoch. Re-stamp them onto the fake timeline.
@@ -150,7 +162,7 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
for _, n := range nodes {
touchAt(inv, n.ID, c.now())
}
return &harness{r: r, inv: inv, disp: disp, clock: c}
return &harness{r: r, inv: inv, disp: disp, clock: c, live: live}
}
// touchAt rewrites a node's LastSeen onto the fake clock's timeline.
@@ -1787,7 +1799,9 @@ func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
// One node, room for one replica: the replacement can only start once the
// old one has gone. Waiting forever would mean never restarting at all.
// old one has gone. It is the only thing in the way, so it goes now —
// waiting out a grace period first changed nothing but the wait, which
// in the field read as a restart that never happened.
h := newHarness(t, testModel(1), gpuNode("1", 4))
h.r.Reconcile(t.Context())
h.markReady()
@@ -1796,14 +1810,8 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
t.Fatal(err)
}
h.r.Reconcile(t.Context())
if rep, _ := replicaByID(h, old); rep.State != Ready {
t.Fatalf("stopped before the grace period: %s", rep.State)
}
h.advance(h.r.cfg.DrainGrace + time.Second)
h.r.Reconcile(t.Context())
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
t.Fatal("still serving after the grace period with no replacement possible")
t.Fatal("still serving with no replacement possible beside it")
}
for i := 0; i < 3; i++ {
h.r.Reconcile(t.Context())
@@ -1815,6 +1823,52 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
}
}
func TestARestartWhoseReplacementFitsNowhereIsGivenUp(t *testing.T) {
// The profile was edited to need more cards than the node has: stopping
// the old replica would leave the model with nothing at all.
m := testModel(1)
h := newHarness(t, m, gpuNode("1", 4))
h.r.Reconcile(t.Context())
h.markReady()
old := h.r.ReadyReplicas()[0].ID
m.Profiles[0].TP, m.Profiles[0].GPUsPerNode = 8, 8
h.setModels(t, m)
if err := h.r.Restart(old); err != nil {
t.Fatal(err)
}
for range 3 {
h.r.Reconcile(t.Context())
h.advance(h.r.cfg.DrainGrace + time.Second)
}
if rep, ok := replicaByID(h, old); !ok || rep.State != Ready {
t.Fatalf("the only replica was stopped for a replacement that fits nowhere: %+v", rep)
}
if st := h.r.RestartState(old); !strings.HasPrefix(st, "refused: ") || !strings.Contains(st, "fits on no node") {
t.Errorf("restart state %q should say it was given up, and why", st)
}
// Asked again, it is tried again.
if err := h.r.Restart(old); err != nil || h.r.RestartState(old) != "queued" {
t.Errorf("a new restart did not clear the refusal: %q, %v", h.r.RestartState(old), err)
}
}
func TestARestartAskedInPlaceStopsFirst(t *testing.T) {
// Room beside it, but in place was asked for: an engine that must not
// run twice, or memory somebody knows is short.
h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
h.r.Reconcile(t.Context())
h.markReady()
old := h.r.ReadyReplicas()[0].ID
if err := h.r.RestartInPlace(old); err != nil {
t.Fatal(err)
}
h.r.Reconcile(t.Context())
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
t.Fatalf("asked in place, it kept serving beside its replacement: %+v", rep)
}
}
func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
for i := 0; i < 3; i++ {
+80 -13
View File
@@ -6,17 +6,34 @@ import (
"fmt"
"slices"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
// retire finishes restarting replicas.
//
// A retiring replica keeps serving until its replacement is ready, and is then
// drained like any other. When no replacement could be placed at all —
// nowhere else has the memory, which on a small CPU fleet is the usual case —
// it is stopped after DrainGrace so its replacement can take its place: a
// restart in place, with the model unserved while it loads if it had no other
// replica. Said in the log, since that is the one case with an outage.
func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
// drained like any other. When no replacement could be placed beside it, what
// happens depends on what is in the way:
//
// - the replica itself — one node, or every card that fits held: it is
// stopped now, and its replacement takes its place. Waiting would change
// nothing; it used to wait DrainGrace anyway, five minutes reading only
// "restart replacing", which looked like a restart that never would.
// - more than it: the replacement fits nowhere even with it stopped — its
// profile was edited to need more than any node has free. Stopping it
// would leave the model with nothing, so the restart is given up, and
// says why where the replica is shown.
// - something being moved out of the way (a preemption, an idle model
// stopping): waited for.
// - nothing tried this pass (a backoff): the old rule, stopped after
// DrainGrace with no replacement in sight.
//
// A restart asked for in place (ctl restart --in-place) stops first, always.
// Each stop is said in the log, since it is the one case with an outage.
func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string, readyLive, live, desired int,
outcome scaleOutcome, parked bool,
) {
for _, id := range ids {
r.mu.Lock()
rep, ok := r.replicas[id]
@@ -24,6 +41,7 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
if ok {
since = rep.RetiringAt
}
inPlace := r.restartInPlace[id]
r.mu.Unlock()
if !ok {
continue
@@ -31,15 +49,48 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
switch {
case readyLive >= desired:
r.drainForRestart(ctx, id)
case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
r.log.Warn("restarting a replica in place: no room for its replacement beside it",
case inPlace:
r.log.Warn("restarting a replica in place, as asked: stopped before its replacement starts", "replica", id)
r.stopForRestart(ctx, id)
case live >= desired:
// A replacement is placed and loading; it is waited for.
case parked:
r.refuseRestart(id, "its replacements fail to start; the model is parked (llmgw ctl retry "+m.Name+")")
case outcome == noRoom && r.fitsWithout(m, nil, []victim{{ID: id, Model: m.Name}}):
r.log.Warn("restarting a replica in place: it is the only thing in its replacement's way",
"replica", id, "model", m.Name)
r.stopForRestart(ctx, id)
case outcome == noRoom || outcome == overLimit:
r.refuseRestart(id, "its replacement fits on no node even with this replica stopped "+
"(llmgw ctl explain-placement "+m.Name+")")
case outcome == notTried && r.now().Sub(since) > r.cfg.DrainGrace:
r.log.Warn("restarting a replica in place: no replacement beside it",
"replica", id, "after", r.cfg.DrainGrace)
r.drainForRestart(ctx, id)
r.stopReplica(ctx, id)
r.stopForRestart(ctx, id)
}
}
}
// stopForRestart drains a replica and stops it at once, for its replacement
// to take its place.
func (r *Reconciler) stopForRestart(ctx context.Context, id string) {
r.drainForRestart(ctx, id)
r.stopReplica(ctx, id)
}
// refuseRestart gives up a replica's restart, leaving it serving, and keeps
// why for whoever looks at the replica.
func (r *Reconciler) refuseRestart(id, why string) {
r.mu.Lock()
if rep, ok := r.replicas[id]; ok {
rep.RetiringAt = time.Time{}
}
delete(r.restartInPlace, id)
r.restartRefused[id] = why
r.mu.Unlock()
r.log.Warn("not restarting a replica: it keeps serving", "replica", id, "because", why)
}
func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
r.mu.Lock()
rep, ok := r.replicas[id]
@@ -48,6 +99,7 @@ func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
shards = slices.Clone(rep.Shards)
}
delete(r.restartInPlace, id)
r.mu.Unlock()
for _, s := range shards {
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
@@ -78,7 +130,14 @@ var ErrNoSuchReplica = errors.New("no ready replica with that id")
// Restart queues replicas to be replaced, each by a new one started beside it
// before it is stopped — to pick up an edited profile, an engine key, a new
// engine version. Replicas of one model are replaced one at a time.
func (r *Reconciler) Restart(ids ...string) error {
func (r *Reconciler) Restart(ids ...string) error { return r.restart(false, ids) }
// RestartInPlace queues replicas to be replaced by stopping each first and
// then starting its replacement: for when there is knowingly no room beside
// it, and the outage while it loads is accepted.
func (r *Reconciler) RestartInPlace(ids ...string) error { return r.restart(true, ids) }
func (r *Reconciler) restart(inPlace bool, ids []string) error {
r.mu.Lock()
for _, id := range ids {
rep, ok := r.replicas[id]
@@ -86,6 +145,10 @@ func (r *Reconciler) Restart(ids ...string) error {
r.mu.Unlock()
return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
}
delete(r.restartRefused, id)
if inPlace {
r.restartInPlace[id] = true
}
if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
}
@@ -95,8 +158,8 @@ func (r *Reconciler) Restart(ids ...string) error {
return nil
}
// RestartState says where a replica's restart stands: "queued", "replacing",
// or "" when none was asked for.
// RestartState says where a replica's restart stands: "queued",
// "replacing", "in place", "refused: why", or "" when none was asked for.
func (r *Reconciler) RestartState(id string) string {
r.mu.Lock()
defer r.mu.Unlock()
@@ -104,10 +167,14 @@ func (r *Reconciler) RestartState(id string) string {
switch {
case !ok:
return ""
case !rep.RetiringAt.IsZero() && r.restartInPlace[id]:
return "in place"
case !rep.RetiringAt.IsZero():
return "replacing"
case slices.Contains(r.restartQueue[rep.Model], id):
return "queued"
case r.restartRefused[id] != "":
return "refused: " + r.restartRefused[id]
}
return ""
}
+56 -13
View File
@@ -10,6 +10,7 @@ import (
"context"
"errors"
"fmt"
"maps"
"net"
"os"
"os/exec"
@@ -86,10 +87,12 @@ type Facts struct {
// What installing vLLM here would meet: the version this llmgw installs,
// where, the space there, and whether it already has.
VLLMVersion string `json:"vllm_version,omitempty"`
EnginesDir string `json:"engines_dir,omitempty"`
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
VLLMInstalled bool `json:"vllm_installed"`
VLLMVersion string `json:"vllm_version,omitempty"`
// DisabledEngines are those agent.yaml says this node never runs.
DisabledEngines []string `json:"disabled_engines,omitempty"`
EnginesDir string `json:"engines_dir,omitempty"`
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
VLLMInstalled bool `json:"vllm_installed"`
}
// Modules checked, and why each matters.
@@ -103,6 +106,10 @@ type Options struct {
// VLLMVersion is the vLLM this llmgw installs; its prerequisites are
// checked when set.
VLLMVersion string
// Fabrics are fabrics about to be declared, by name, with their
// adapters: what `node add --fabric` will write into agent.yaml, which
// does not exist yet when doctor runs before it.
Fabrics map[string][]string
}
// Collect reads the node.
@@ -183,6 +190,14 @@ func Collect(ctx context.Context, o Options) Facts {
}
f.readConfig(o)
if len(o.Fabrics) > 0 {
declared := map[string][]string{}
for _, c := range f.Fabrics {
declared[c.Fabric] = c.Declared
}
maps.Copy(declared, o.Fabrics)
f.Fabrics = discovery.CheckFabrics(declared, f.HCAs)
}
f.Host.MinComputeCapability = discovery.MinComputeCapability(f.GPUs)
if f.VLLMVersion = o.VLLMVersion; f.VLLMVersion != "" {
if f.EnginesDir == "" {
@@ -238,6 +253,7 @@ func (f *Facts) readConfig(o Options) {
_, err = net.InterfaceByName(f.MgmtIface)
f.IfaceExists = err == nil
f.EnginesDir = cfg.EnginesDir
f.DisabledEngines = cfg.Engines.Disabled
f.LaunchMode = cfg.Engines.VLLM.LaunchMode
if f.LaunchMode == "" {
f.LaunchMode = "mp"
@@ -419,8 +435,15 @@ func Evaluate(f Facts) []Finding {
}
switch {
case p.Active() && p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID < 0:
add(Fail, "fabric", where+" is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang",
"give the interface an IPv4 address, or check the adapter's RoCE mode")
// Fatal only for a port a declared fabric uses. An adapter
// nothing is configured to use — a RoCE NIC on a node added
// for single-node inference — kept nodes out for a fabric
// nobody asked for.
msg := where + " is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang"
if declared[h.Name] == "" {
msg += "; no declared fabric uses it, so nothing will until one does"
}
add(level, "fabric", msg, "give the interface an IPv4 address, or check the adapter's RoCE mode")
case p.Active():
on := ""
switch {
@@ -430,17 +453,24 @@ func Evaluate(f Facts) []Finding {
on = ", network " + p.Network
}
add(OK, "fabric", fmt.Sprintf("%s %s at %d Gb/s (%s)%s", where, p.State, p.RateGbps, p.LinkLayer, on), "")
// Questions about which fabric a port is on matter to the
// fabrics that use it; for a port none does they are noted,
// not warned of.
quiet := Warn
if declared[h.Name] == "" {
quiet = Info
}
if p.LinkLayer == discovery.LinkInfiniBand && p.SM == nil {
why := f.SMErrors[where]
fix := ""
if strings.Contains(why, "infiniband-diags") {
fix = "install infiniband-diags"
}
add(Warn, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
add(quiet, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
"; without it, separate fabrics given one name cannot be caught", fix)
}
if p.LinkLayer == discovery.LinkInfiniBand && p.SMLID == 0 {
add(Warn, "fabric", where+" is active but reports no subnet manager LID", "")
add(quiet, "fabric", where+" is active but reports no subnet manager LID", "")
}
case strings.EqualFold(p.PhysState, "LinkUp"):
add(level, "fabric", where+" has a link but is "+p.State+": no subnet manager has configured it",
@@ -479,7 +509,14 @@ func Evaluate(f Facts) []Finding {
}
if len(f.HCAs) > 0 {
if !f.Modules["ib_uverbs"] {
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
// RDMA is for replicas spanning nodes; a node that declares no
// fabric runs none, and is a node without it.
if len(declared) > 0 {
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
} else {
add(Warn, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA, which a fabric declared "+
"later would need; this node runs single-node replicas without it", "modprobe ib_uverbs")
}
}
if len(f.GPUs) > 0 && !f.Modules["nvidia_peermem"] && !f.Modules["nv_peer_mem"] {
add(Warn, "fabric", "nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory",
@@ -525,7 +562,9 @@ func Evaluate(f Facts) []Finding {
}
}
// --- engines ---
if f.VLLMVersion != "" {
// An engine disabled on this node: whether it could be installed is not
// a question anybody is asking.
if f.VLLMVersion != "" && !slices.Contains(f.DisabledEngines, "vllm") {
vllmFindings(f, add)
}
@@ -549,15 +588,19 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "")
return
case err != nil:
add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(),
"upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one")
// A warning, not a failure: the node is still a node, and serves
// llama.cpp. Refusing it here kept older GPU generations out of a
// cluster for an engine nobody meant to run on them.
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
return
case f.VLLMInstalled:
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
return
}
if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) {
add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB",
add(Warn, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB; the node joins without it until there is",
v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB),
"free space there, or point engines_dir in agent.yaml at a larger disk")
return
+43 -2
View File
@@ -151,8 +151,14 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
says string
}{
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"},
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"},
// What vLLM cannot do keeps vLLM off the node, not the node out of
// the cluster: it still serves llama.cpp.
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Warn, "driver 580"},
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, VLLMVersion: "0.30.0"},
Warn, "before 0.26 is built for CUDA 12"},
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
}
@@ -200,3 +206,38 @@ func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) {
t.Errorf("ARM: %+v", f)
}
}
func TestAnEngineDisabledOnTheNodeIsNotChecked(t *testing.T) {
// The field: a V100 node meant for llama.cpp, refused for a vLLM it was
// never going to run.
f := Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0},
VLLMVersion: "0.30.0", DisabledEngines: []string{"vllm"}}
for _, fd := range Evaluate(f) {
if fd.Area == "engines" {
t.Errorf("a disabled vLLM was reported on: %s %q", fd.Level, fd.Message)
}
}
}
func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
// The field: a RoCE NIC with no v2 GID on a node added for single-node
// inference, no fabric given, and the node refused for it.
roce := discovery.HCA{Name: "mlx5_2", Ports: []discovery.Port{{Num: 1, State: "ACTIVE",
PhysState: "LinkUp", RateGbps: 100, LinkLayer: discovery.LinkEthernet, RoCEv2GID: -1}}}
f := healthy()
f.HCAs = append(f.HCAs, roce)
f.Modules["ib_uverbs"] = false
f.Fabrics = nil // nothing declared
fs := Evaluate(f)
if fails := levels(fs, Fail); len(fails) > 0 {
t.Errorf("an undeclared adapter failed the node: %v", fails)
}
has(t, fs, Warn, "no declared fabric uses it")
has(t, fs, Warn, "runs single-node replicas without it")
// Declared, the same adapter and module are what the fabric needs.
f.Fabrics = discovery.CheckFabrics(map[string][]string{"roce-a": {"mlx5_2"}}, f.HCAs)
fs = Evaluate(f)
has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
has(t, fs, Fail, "ib_uverbs is not loaded")
}
+41
View File
@@ -25,6 +25,10 @@ type Header struct {
Version uint32
Architecture string
Name string
// Type is general.type: "model", or "adapter" for a LoRA adapter, which
// is applied to a model rather than run on its own. Empty in files made
// before the key existed, which are models.
Type string
// The shape the KV cache is sized from, from the architecture's own keys
// (llama.block_count, qwen2.attention.head_count_kv, ...). Zero when the
@@ -235,6 +239,8 @@ func Read(r io.Reader) (Header, error) {
h.Architecture = v
case "general.name":
h.Name = v
case "general.type":
h.Type = v
}
if h.Architecture != "" {
switch strings.TrimPrefix(key, h.Architecture+".") {
@@ -290,6 +296,41 @@ func FromURL(ctx context.Context, client *http.Client, url string) (Header, erro
return Read(io.LimitReader(resp.Body, headerBytes))
}
// IsAdapter reports whether the file is a LoRA adapter rather than a model.
func (h Header) IsAdapter() bool { return h.Type == "adapter" }
// adapterProbe is how much of a file is read to tell an adapter from a model:
// general.type comes before the tokenizer, which is what makes a model's
// header megabytes long.
const adapterProbe = 256 << 10
// KindFromURL reads just enough of a GGUF file to say what it is — its type
// and architecture — over HTTP. A header cut short by the probe is fine: the
// keys asked for come first.
func KindFromURL(ctx context.Context, client *http.Client, url string) (Header, error) {
if client == nil {
client = &http.Client{Timeout: 30 * time.Second}
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return Header{}, err
}
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", adapterProbe-1))
resp, err := client.Do(req)
if err != nil {
return Header{}, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
return Header{}, fmt.Errorf("%s answered %s", url, resp.Status)
}
h, err := Read(io.LimitReader(resp.Body, adapterProbe))
if h.Architecture != "" {
return h, nil // what was asked for was read, whatever came after
}
return h, err
}
// IsGGUF reports whether a source names a GGUF file, or a directory of them.
func IsGGUF(source string) bool {
s := strings.ToLower(strings.TrimSuffix(source, "/"))
+31
View File
@@ -0,0 +1,31 @@
package gguf
import (
"os"
"testing"
)
// Run by hand against the real adapter this was written for:
//
// LLMGW_LIVE_GGUF=1 go test ./internal/gguf -run LiveAdapter -v
func TestLiveAdapterIsAnAdapter(t *testing.T) {
if os.Getenv("LLMGW_LIVE_GGUF") == "" {
t.Skip("set LLMGW_LIVE_GGUF to read a real adapter from Hugging Face")
}
h, err := KindFromURL(t.Context(), nil,
"https://huggingface.co/ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF/resolve/main/LoRA-Qwen2.5-Coder-7B-Instruct-f16.gguf")
if err != nil {
t.Fatal(err)
}
if !h.IsAdapter() || h.Architecture != "qwen2" {
t.Errorf("%+v", h)
}
m, err := KindFromURL(t.Context(), nil,
"https://huggingface.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf")
if err != nil {
t.Fatal(err)
}
if m.IsAdapter() || m.Architecture != "qwen2" {
t.Errorf("a model read as %+v", m)
}
}
+58
View File
@@ -0,0 +1,58 @@
package gguf
import (
"bytes"
"encoding/binary"
"net/http"
"net/http/httptest"
"testing"
)
// adapterHeader is the start of a LoRA adapter as llama.cpp's converter
// writes it, followed by padding that stands in for its tensors.
func adapterHeader(kind string, pad int) []byte {
var b bytes.Buffer
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
b.WriteString("GGUF")
w(uint32(3))
w(uint64(0))
w(uint64(3))
str("general.architecture")
w(uint32(tString))
str("qwen2")
str("general.type")
w(uint32(tString))
str(kind)
str("adapter.type")
w(uint32(tString))
str("lora")
b.Write(make([]byte, pad))
return b.Bytes()
}
func TestAnAdapterIsToldFromAModelByItsFirstBytes(t *testing.T) {
var asked string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
asked = r.Header.Get("Range")
w.WriteHeader(http.StatusPartialContent)
if r.URL.Path == "/adapter.gguf" {
_, _ = w.Write(adapterHeader("adapter", 1<<20))
return
}
_, _ = w.Write(header("qwen2"))
}))
defer srv.Close()
h, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/adapter.gguf")
if err != nil || !h.IsAdapter() || h.Architecture != "qwen2" {
t.Errorf("an adapter read as %+v, %v", h, err)
}
if asked != "bytes=0-262143" {
t.Errorf("asked for %q: the probe should read the start only", asked)
}
m, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/model.gguf")
if err != nil || m.IsAdapter() || m.Architecture != "qwen2" {
t.Errorf("a model read as %+v, %v", m, err)
}
}
+43 -1
View File
@@ -156,6 +156,10 @@ type Query struct {
// applies it, so a page of small models is a full page.
MaxParams int64
Limit int
// Filters are further registry tags every result must carry, such as
// "lora", or "base_model:adapter:Qwen/Qwen2.5-7B-Instruct" for the
// adapters made for one model.
Filters []string
}
// exploreFields is everything a listing shows. Asking for any field with
@@ -188,7 +192,10 @@ func (c *Client) Explore(ctx context.Context, q Query) ([]Model, error) {
v.Set("pipeline_tag", q.Task)
}
if q.GGUF {
v.Set("filter", "gguf")
v.Add("filter", "gguf")
}
for _, f := range q.Filters {
v.Add("filter", f)
}
if q.MaxParams > 0 {
v.Set("num_parameters", fmt.Sprintf("min:0,max:%d", q.MaxParams))
@@ -251,6 +258,41 @@ func (f File) IsWeights() bool {
return false
}
// companionNames are the files beside safetensors weights that an engine
// loading the directory reads: the model's shape, its tokenizer, how it was
// quantised, and which shard holds which tensor.
var companionNames = map[string]bool{
"config.json": true, "generation_config.json": true,
"tokenizer.json": true, "tokenizer_config.json": true, "tokenizer.model": true,
"special_tokens_map.json": true, "added_tokens.json": true,
"vocab.json": true, "vocab.txt": true, "merges.txt": true,
"chat_template.jinja": true, "chat_template.json": true,
"preprocessor_config.json": true, "processor_config.json": true,
"quantize_config.json": true, "quant_config.json": true,
}
// Companions are the files of a repository an engine reads beside a
// safetensors set: config.json and the tokenizer, the shard index. Only
// those at the top: the directory an engine is given is one level deep.
//
// Never code (a .py): a model that needs its own Python to load asks the
// engine to run what the repository ships, which is a decision for whoever
// runs the engine, made on purpose, not a file swept in with the tokenizer.
func Companions(files []File) []File {
var out []File
for _, f := range files {
name := f.Path
if strings.Contains(name, "/") {
continue
}
if companionNames[name] || strings.HasSuffix(name, ".safetensors.index.json") ||
strings.HasSuffix(name, ".tiktoken") {
out = append(out, f)
}
}
return out
}
// Repo is a repository's current revision and what is in it.
type Repo struct {
ID string
+10
View File
@@ -94,6 +94,13 @@ func addWith(ctx context.Context, client *http.Client, root, source string, e En
_ = os.Remove(part)
return Added{}, err
}
// The checksum beside the file, before the file is in place: a manifest
// listing the file then always has its checksum to hand on, and a node
// can tell a file damaged on the way from the one that was mirrored.
if err := writeChecksum(final, sum); err != nil {
_ = os.Remove(part)
return Added{}, err
}
if err := os.Rename(part, final); err != nil {
return Added{}, err
}
@@ -146,6 +153,9 @@ func Remove(root string, e Entry) error {
if err := os.Remove(path); err != nil {
return err
}
// Its checksum goes with it; left, it would keep the directory from
// being empty, and so from going.
_ = os.Remove(checksumPath(path))
versionDir := filepath.Dir(path)
if empty, _ := isEmpty(versionDir); empty {
_ = os.Remove(versionDir)
+102 -2
View File
@@ -16,12 +16,15 @@ package mirror
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
)
@@ -91,7 +94,7 @@ func List(root string) ([]Entry, error) {
continue
}
for _, f := range files {
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) {
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) || strings.HasPrefix(f.Name(), ".") {
continue
}
info, err := f.Info()
@@ -169,6 +172,31 @@ func normalizeURL(u string) string {
const partSuffix = ".part"
// checksumPath is where a file's sha256 is kept: beside it, hidden, so no
// listing takes it for a file of the model and no name a person gives a
// file (which may not start with a dot) can be the same.
func checksumPath(file string) string {
return filepath.Join(filepath.Dir(file), "."+filepath.Base(file)+".sha256")
}
func writeChecksum(file, sum string) error {
tmp := checksumPath(file) + partSuffix
if err := os.WriteFile(tmp, []byte(sum+"\n"), 0o644); err != nil { //nolint:gosec // a checksum, for anyone to read
return err
}
return os.Rename(tmp, checksumPath(file))
}
// readChecksum is a file's recorded sha256, or "" for a file put in place by
// hand, which has none.
func readChecksum(file string) string {
b, err := os.ReadFile(checksumPath(file)) //nolint:gosec // beside a file in the mirror
if err != nil {
return ""
}
return strings.TrimSpace(string(b))
}
// sha256File is the checksum of a file on disk.
//
// Reported after a download so an operator has something to compare against
@@ -205,8 +233,16 @@ type Manifest struct {
type ManifestFile struct {
Name string `json:"name"`
Bytes int64 `json:"bytes"`
// SHA256 is the file's checksum when the mirror fetched it, for a node
// to check what it downloaded; empty for a file put there by hand.
SHA256 string `json:"sha256,omitempty"`
}
// ErrIncomplete is a version not yet whole: a download still running, or a
// shard its own index names missing. Its manifest is not given out, so no
// node starts a model from part of it.
var ErrIncomplete = errors.New("this version is not complete")
// ManifestFor lists the files of one model version.
//
// Built from the directory rather than from a file written alongside it. A
@@ -219,15 +255,79 @@ func ManifestFor(root, model, version string) (Manifest, error) {
return Manifest{}, err
}
m := Manifest{Model: model, Version: version}
dir := filepath.Join(root, model, version)
for _, e := range entries {
if e.Model != model || e.Version != version {
continue
}
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes})
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes,
SHA256: readChecksum(filepath.Join(dir, e.File))})
m.Bytes += e.Bytes
}
if len(m.Files) == 0 {
return Manifest{}, fmt.Errorf("%s/%s: %w", model, version, os.ErrNotExist)
}
if err := complete(dir, m.Files); err != nil {
return Manifest{}, fmt.Errorf("%s/%s: %w: %v", model, version, ErrIncomplete, err)
}
return m, nil
}
// complete says whether a version's files are the whole of it: nothing still
// downloading, every shard a safetensors index maps a tensor to, and every
// part of a split ("-00002-of-00004").
//
// From what the files say of each other rather than from a list written when
// the fetch began, so a version filled by hand is held to the same.
func complete(dir string, files []ManifestFile) error {
if parts, _ := filepath.Glob(filepath.Join(dir, "*"+partSuffix)); len(parts) > 0 {
var names []string
for _, p := range parts {
if b := filepath.Base(p); !strings.HasPrefix(b, ".") {
names = append(names, strings.TrimSuffix(b, partSuffix))
}
}
if len(names) > 0 {
return fmt.Errorf("still downloading %s", strings.Join(names, ", "))
}
}
have := map[string]bool{}
for _, f := range files {
have[f.Name] = true
}
var missing []string
want := map[string]bool{}
for _, f := range files {
if strings.HasSuffix(f.Name, ".safetensors.index.json") {
var idx struct {
WeightMap map[string]string `json:"weight_map"`
}
b, err := os.ReadFile(filepath.Join(dir, f.Name)) //nolint:gosec // a file in the mirror
if err != nil || json.Unmarshal(b, &idx) != nil {
return fmt.Errorf("%s cannot be read", f.Name)
}
for _, shard := range idx.WeightMap {
want[shard] = true
}
}
if m := splitPart.FindStringSubmatch(f.Name); m != nil {
total, _ := strconv.Atoi(m[3])
for i := 1; i <= total; i++ {
want[fmt.Sprintf("%s-%0*d-of-%s%s", m[1], len(m[2]), i, m[3], m[4])] = true
}
}
}
for name := range want {
if !have[name] {
missing = append(missing, name)
}
}
if len(missing) > 0 {
sort.Strings(missing)
return fmt.Errorf("missing %s", strings.Join(missing, ", "))
}
return nil
}
// splitPart matches one part of a split: prefix, part, total, extension.
var splitPart = regexp.MustCompile(`^(.+)-(\d{5})-of-(\d{5})(\.[A-Za-z0-9]+)$`)
+60
View File
@@ -177,3 +177,63 @@ func TestANameThatLeavesTheMirrorIsRefused(t *testing.T) {
}
}
}
func TestAVersionIsGivenOutOnlyWhole(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "m", "v1")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
write := func(name, body string) {
t.Helper()
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
write("config.json", "{}")
write("model.safetensors.index.json",
`{"weight_map":{"a":"model-00001-of-00002.safetensors","b":"model-00002-of-00002.safetensors"}}`)
write("model-00001-of-00002.safetensors", "aa")
write("model-00002-of-00002.safetensors.part", "b")
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "still downloading") {
t.Fatalf("a version with a shard downloading was given out: %v", err)
}
// The download failed and its .part went: the index still names it.
_ = os.Remove(filepath.Join(dir, "model-00002-of-00002.safetensors.part"))
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) ||
!strings.Contains(err.Error(), "missing model-00002-of-00002.safetensors") {
t.Fatalf("a version missing a shard its index names was given out: %v", err)
}
write("model-00002-of-00002.safetensors", "bb")
m, err := ManifestFor(root, "m", "v1")
if err != nil || len(m.Files) != 4 {
t.Fatalf("the whole version: %+v, %v", m, err)
}
// A split GGUF with no index is held to its own numbering.
gg := filepath.Join(root, "g", "v1")
_ = os.MkdirAll(gg, 0o755)
_ = os.WriteFile(filepath.Join(gg, "q-00001-of-00003.gguf"), []byte("x"), 0o644)
_ = os.WriteFile(filepath.Join(gg, "q-00003-of-00003.gguf"), []byte("x"), 0o644)
if _, err := ManifestFor(root, "g", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "q-00002-of-00003.gguf") {
t.Fatalf("a split missing its second part was given out: %v", err)
}
}
func TestAMirroredFilesChecksumIsHandedOnAndHiddenFromListings(t *testing.T) {
root := t.TempDir()
e := Entry{Model: "m", Version: "v1", File: "f.gguf"}
added, err := Add(context.Background(), root, serving(t, "weights"), e, nil)
if err != nil {
t.Fatal(err)
}
entries, _ := List(root)
if len(entries) != 1 {
t.Errorf("the checksum was listed as a file of the model: %+v", entries)
}
m, err := ManifestFor(root, "m", "v1")
if err != nil || m.Files[0].SHA256 != added.SHA256 || added.SHA256 == "" {
t.Errorf("manifest %+v, checksum %q: %v", m, added.SHA256, err)
}
}
+27 -2
View File
@@ -18,7 +18,6 @@ import (
"encoding/pem"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"net"
"os"
"path/filepath"
@@ -27,6 +26,8 @@ import (
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
@@ -86,6 +87,15 @@ type Options struct {
Releases string
// IgnoreChecks goes on when doctor finds a problem.
IgnoreChecks bool
// DisableEngines are engines this node never installs or advertises,
// written into its agent.yaml, and not checked by doctor: vLLM on a node
// of cards it does not run on, meant for llama.cpp alone. Narrower than
// IgnoreChecks, which also passes over everything else.
DisableEngines []string
// VLLMVersion is the vLLM the control plane has nodes install, for
// doctor to check against; "none" when it has them install none, empty
// for doctor's own default.
VLLMVersion string
// SSHWait is how long an unreachable machine is waited for.
SSHWait time.Duration
@@ -390,6 +400,18 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
if r.Opts.IBRoot != "" {
args += " --ib-root " + shellQuote(r.Opts.IBRoot)
}
// The fabrics given, checked as the ones they will be; any adapter they
// do not name is checked as one nothing uses.
for _, fab := range r.Opts.Fabrics {
args += " --fabric " + shellQuote(fab.Name+":"+strings.Join(fab.HCAs, ","))
}
switch {
case slices.Contains(r.Opts.DisableEngines, "vllm"):
args += " --vllm none"
r.emit(StepCheck, Info, "vLLM is disabled on this node: it joins with llama.cpp alone", "")
case r.Opts.VLLMVersion != "":
args += " --vllm " + shellQuote(r.Opts.VLLMVersion)
}
out, err := r.Remote.Run(ctx, args, nil)
var rep struct {
Findings []doctor.Finding `json:"findings"`
@@ -596,8 +618,11 @@ func AgentYAML(o Options, res *Result) string {
fmt.Fprintf(&b, " ib_root: %q\n", o.IBRoot)
}
}
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" {
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" || len(o.DisableEngines) > 0 {
b.WriteString("engines:\n")
if len(o.DisableEngines) > 0 {
fmt.Fprintf(&b, " disabled: [%s]\n", strings.Join(o.DisableEngines, ", "))
}
if o.LlamaCppBin != "" {
fmt.Fprintf(&b, " llamacpp:\n bin: %q\n", o.LlamaCppBin)
}
+41
View File
@@ -280,3 +280,44 @@ func said(events []Event, level, s string) bool {
}
return false
}
func TestAnEngineDisabledOnANodeIsWrittenAndNotChecked(t *testing.T) {
m := newFake()
r, events := runner(t, m, Options{Version: "dev", DisableEngines: []string{"vllm"}, VLLMVersion: "0.30.0"})
if _, err := r.Run(t.Context()); err != nil {
t.Fatal(err)
}
if yaml := string(m.files["/etc/llmgw/agent.yaml"]); !strings.Contains(yaml, "engines:\n disabled: [vllm]") {
t.Errorf("agent.yaml:\n%s", yaml)
}
if !m.ran("sudo llmgw doctor --json --vllm none") {
t.Errorf("doctor was asked about a vLLM this node will never run: %v", m.scripts)
}
if !said(*events, Info, "joins with llama.cpp alone") {
t.Error("the reduced engine set went unsaid")
}
// Not disabled: doctor checks the vLLM the control plane installs, not
// its own default.
m = newFake()
r, _ = runner(t, m, Options{Version: "dev", VLLMVersion: "0.25.1"})
if _, err := r.Run(t.Context()); err != nil {
t.Fatal(err)
}
if !m.ran("sudo llmgw doctor --json --vllm '0.25.1'") && !m.ran("sudo llmgw doctor --json --vllm 0.25.1") {
t.Errorf("doctor was not told the control plane's vLLM: %v", m.scripts)
}
}
func TestTheFabricsGivenAreCheckedAsDeclared(t *testing.T) {
// agent.yaml is written after doctor runs, so the fabrics on the command
// line are handed to doctor: checked strictly, and nothing else is.
m := newFake()
r, _ := runner(t, m, Options{Version: "dev", Fabrics: []Fabric{{Name: "ib-a", HCAs: []string{"mlx5_0", "mlx5_1"}}}})
if _, err := r.Run(t.Context()); err != nil {
t.Fatal(err)
}
if !m.ran("--fabric 'ib-a:mlx5_0,mlx5_1'") {
t.Errorf("doctor was not told the fabric: %v", m.scripts)
}
}
+27 -4
View File
@@ -9,6 +9,7 @@ package release
import (
"bufio"
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
@@ -51,7 +52,7 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
asset := AssetName(version, goos, goarch)
root := strings.TrimSuffix(base, "/") + "/" + version + "/"
want, err := checksum(ctx, client, root+"checksums.txt", asset)
want, err := checksum(ctx, client, root, asset)
if err != nil {
return "", err
}
@@ -85,13 +86,35 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
return backup, nil
}
func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
body, err := get(ctx, client, url)
// checksum reads the release's checksum for asset — after checking the
// checksums' signature, when this build trusts a signing key. A release with
// no signature is then refused rather than taken on its checksums: a missing
// signature is exactly what replacing a release would look like.
func checksum(ctx context.Context, client *http.Client, root, asset string) (string, error) {
body, err := get(ctx, client, root+"checksums.txt")
if err != nil {
return "", err
}
defer body.Close()
sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
sums, err := io.ReadAll(io.LimitReader(body, 1<<20))
if err != nil {
return "", err
}
if key := TrustedKey(); key != "" {
sigBody, err := get(ctx, client, root+SignatureName)
if err != nil {
return "", fmt.Errorf("this build only installs signed releases, and %s could not be read: %w", SignatureName, err)
}
sig, err := io.ReadAll(io.LimitReader(sigBody, 4096))
sigBody.Close()
if err != nil {
return "", err
}
if err := Verify(key, sums, string(sig)); err != nil {
return "", err
}
}
sc := bufio.NewScanner(bytes.NewReader(sums))
for sc.Scan() {
f := strings.Fields(sc.Text())
if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
+72
View File
@@ -0,0 +1,72 @@
package release
import (
"crypto/ed25519"
"crypto/rand"
_ "embed"
"encoding/base64"
"errors"
"fmt"
"os"
"strings"
)
// trustedPub is the public key releases are signed with, base64, built into
// every binary from the repository. Empty — as it is until a key is made —
// trusts a release on its checksums alone, as every release before this did.
//
// Built in rather than fetched, because the point of a signature is that the
// server the release comes from cannot vouch for it: a machine that already
// runs llmgw checks the next release against the key the release it runs was
// built with, so replacing both the binaries and the checksums on the server
// is no longer enough.
//
//go:embed trusted.pub
var trustedPub string
// TrustedKey is the key releases are verified with: LLMGW_RELEASE_KEY when set
// (for rotating a key, or testing), the built-in one otherwise; empty is none.
func TrustedKey() string {
if k := strings.TrimSpace(os.Getenv("LLMGW_RELEASE_KEY")); k != "" {
return k
}
return strings.TrimSpace(trustedPub)
}
// SignatureName is the file a release's checksums are signed in.
const SignatureName = "checksums.txt.sig"
// GenerateKey makes a signing key pair, both base64: the private key is the
// 32-byte seed, which is what goes into the CI secret.
func GenerateKey() (public, private string, err error) {
pub, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
return "", "", err
}
return base64.StdEncoding.EncodeToString(pub), base64.StdEncoding.EncodeToString(priv.Seed()), nil
}
// Sign signs data with a base64 seed, returning the signature in base64.
func Sign(privateSeed string, data []byte) (string, error) {
seed, err := base64.StdEncoding.DecodeString(strings.TrimSpace(privateSeed))
if err != nil || len(seed) != ed25519.SeedSize {
return "", errors.New("the signing key is not a base64 ed25519 seed")
}
return base64.StdEncoding.EncodeToString(ed25519.Sign(ed25519.NewKeyFromSeed(seed), data)), nil
}
// Verify checks a base64 signature of data against a base64 public key.
func Verify(public string, data []byte, signature string) error {
pub, err := base64.StdEncoding.DecodeString(strings.TrimSpace(public))
if err != nil || len(pub) != ed25519.PublicKeySize {
return errors.New("the trusted release key is not a base64 ed25519 public key")
}
sig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(signature))
if err != nil {
return fmt.Errorf("the release signature is not base64: %w", err)
}
if !ed25519.Verify(pub, data, sig) {
return errors.New("the release's checksums are not signed by the trusted key")
}
return nil
}
+99
View File
@@ -0,0 +1,99 @@
package release
import (
"crypto/sha256"
"encoding/hex"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
// signedServer publishes a release whose checksums are signed with seed,
// or not signed at all when seed is empty; tamper rewrites the checksums
// after signing.
func signedServer(t *testing.T, seed string, body []byte, tamper bool) (*httptest.Server, string) {
t.Helper()
asset := AssetName("v2.0.0", "linux", "amd64")
sum := sha256.Sum256(body)
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
sig := ""
if seed != "" {
var err error
if sig, err = Sign(seed, []byte(sums)); err != nil {
t.Fatal(err)
}
}
if tamper {
sums += strings.Repeat("0", 64) + " extra-file\n"
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v2.0.0/checksums.txt":
_, _ = w.Write([]byte(sums))
case "/v2.0.0/" + SignatureName:
if sig == "" {
http.NotFound(w, r)
return
}
_, _ = w.Write([]byte(sig + "\n"))
case "/v2.0.0/" + asset:
_, _ = w.Write(body)
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
return srv, asset
}
func TestABuildThatTrustsAKeyInstallsOnlyReleasesSignedWithIt(t *testing.T) {
pub, priv, err := GenerateKey()
if err != nil {
t.Fatal(err)
}
_, otherPriv, _ := GenerateKey()
t.Setenv("LLMGW_RELEASE_KEY", pub)
install := func(srv *httptest.Server) error {
target := filepath.Join(t.TempDir(), "llmgw")
_ = os.WriteFile(target, []byte("old"), 0o755)
_, err := Install(t.Context(), srv.Client(), srv.URL, "v2.0.0", "linux", "amd64", target)
return err
}
body := []byte("new binary")
good, _ := signedServer(t, priv, body, false)
if err := install(good); err != nil {
t.Fatalf("a release signed with the trusted key was refused: %v", err)
}
for name, srv := range map[string]*httptest.Server{
"unsigned": func() *httptest.Server { s, _ := signedServer(t, "", body, false); return s }(),
"signed by another": func() *httptest.Server { s, _ := signedServer(t, otherPriv, body, false); return s }(),
"checksums rewritten": func() *httptest.Server { s, _ := signedServer(t, priv, body, true); return s }(),
} {
if err := install(srv); err == nil {
t.Errorf("%s: installed", name)
}
}
// A build that trusts no key installs on checksums, as before.
t.Setenv("LLMGW_RELEASE_KEY", "")
if TrustedKey() != "" {
t.Skip("this build embeds a trusted key; the unsigned path cannot be tested here")
}
unsigned, _ := signedServer(t, "", body, false)
if err := install(unsigned); err != nil {
t.Errorf("without a trusted key an unsigned release was refused: %v", err)
}
}
func TestAKeyThatIsNotOneIsSaid(t *testing.T) {
if _, err := Sign("not-a-key", []byte("x")); err == nil {
t.Error("signed with a malformed key")
}
if err := Verify("bm9wZQ==", []byte("x"), "c2ln"); err == nil || !strings.Contains(err.Error(), "public key") {
t.Errorf("a malformed trusted key: %v", err)
}
}
View File
Whitespace-only changes.
+98
View File
@@ -0,0 +1,98 @@
package router
import (
"encoding/json"
"net/http"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func adapterModel() catalog.Model {
m := catModel("coder", nil)
m.Profiles = []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4}}
m.Adapters = []catalog.Adapter{
{Name: "coder-sql", Source: "http://mirror/sql.gguf"},
{Name: "coder-docs", Source: "http://mirror/docs.gguf", Scale: 0.5},
}
return m
}
func TestAnAdapterIsAskedForByNameAndAppliedOnItsModel(t *testing.T) {
e := newFakeEngine(t, "coder")
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"coder"}})
cfg.Models = []catalog.Model{adapterModel()}
rt := newTestRouter(t, cfg)
sent := func(model string) map[string]any {
t.Helper()
rec := post(t, rt.Handler(), map[string]any{"model": model, "messages": []any{}}, nil)
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d: %s", model, rec.Code, rec.Body.String())
}
var body map[string]any
_ = json.Unmarshal(e.last.Load().([]byte), &body)
return body
}
body := sent("coder-docs")
if body["model"] != "coder" {
t.Errorf("the engine was asked for %v, not the model the adapter runs on", body["model"])
}
lora, _ := json.Marshal(body["lora"])
if string(lora) != `[{"id":1,"scale":0.5}]` {
t.Errorf("lora %s, expected the second adapter at its scale", lora)
}
if body := sent("coder-sql"); func() string { b, _ := json.Marshal(body["lora"]); return string(b) }() != `[{"id":0,"scale":1}]` {
t.Errorf("the first adapter, at the default scale: %v", body["lora"])
}
if body := sent("coder"); body["lora"] != nil {
t.Errorf("the base model was sent an adapter: %v", body["lora"])
}
}
func TestAKeyAllowedAnAdapterReachesThatAdapterOnly(t *testing.T) {
cfg := baseConfig()
cfg.Models = []catalog.Model{adapterModel()}
rt := newTestRouter(t, cfg)
key := func(allowed ...string) auth.Principal {
return auth.Principal{Limits: store.Effective{AllowedModels: allowed}}
}
for _, c := range []struct {
allowed []string
requested string
want bool
}{
{[]string{"coder-sql"}, "coder-sql", true},
{[]string{"coder-sql"}, "coder", false},
{[]string{"coder-sql"}, "coder-docs", false},
{[]string{"coder"}, "coder-sql", true}, // the model allows its adapters
{[]string{"coder"}, "coder", true},
} {
if got := rt.permits(key(c.allowed...), c.requested); got != c.want {
t.Errorf("allowed %v, asked for %s: %v", c.allowed, c.requested, got)
}
}
}
func TestAdaptersShareTheNamespaceOfModels(t *testing.T) {
a := adapterModel()
b := catModel("coder-sql", nil)
if _, err := catalog.New([]catalog.Model{a, b}); err == nil {
t.Error("an adapter named like another model was accepted")
}
bad := adapterModel()
bad.Profiles = []catalog.Profile{{Name: "gpu", Engine: "vllm", GPUClass: "h100-80g", TP: 1}}
if _, err := catalog.New([]catalog.Model{bad}); err == nil {
t.Error("adapters on a vLLM profile were accepted")
}
c, err := catalog.New([]catalog.Model{adapterModel()})
if err != nil {
t.Fatal(err)
}
if m, ad, i, ok := c.ResolveAdapter("coder-docs"); !ok || m.Name != "coder" || i != 1 || ad.EffectiveScale() != 0.5 {
t.Errorf("resolve: %v %+v %d %v", m.Name, ad, i, ok)
}
}
+54 -2
View File
@@ -34,6 +34,12 @@ type Config struct {
// QueueCapacityPerBand bounds each band's queue. Rejecting immediately
// beats accumulating silently.
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
// QueueCapacityTotal bounds everything queued, across bands. When it is
// reached, a request is not refused while a lower band has requests
// queued: the newest of the lowest band is turned away instead
// (displacement), since the oldest have waited longest and are nearest
// their TTL. Zero takes half of every band's capacity together.
QueueCapacityTotal int `koanf:"queue_capacity_total"`
// Interval is how often the dispatcher re-checks saturation. Releases also
// wake it, so this is only the backstop for saturation falling on its own.
Interval time.Duration `koanf:"interval"`
@@ -94,6 +100,9 @@ func (c *Config) normalize() error {
if c.QueueCapacityPerBand <= 0 {
c.QueueCapacityPerBand = 2000
}
if c.QueueCapacityTotal <= 0 {
c.QueueCapacityTotal = max(c.QueueCapacityPerBand*len(c.Bands)/2, 1)
}
if c.Interval <= 0 {
c.Interval = 10 * time.Millisecond
}
@@ -123,6 +132,10 @@ var (
ErrQueueTimeout = errors.New("admission queue timeout")
// ErrClosed means the controller is shutting down.
ErrClosed = errors.New("admission controller closed")
// ErrDisplaced means a queued request was turned away to make room for
// one of a higher band. Worth retrying: the queue is full now, not for
// good.
ErrDisplaced = errors.New("displaced from the admission queue by higher-priority requests")
)
// Stats is a snapshot of one band's queue, for metrics.
@@ -259,6 +272,12 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
c.mu.Unlock()
return nil, fmt.Errorf("%w: band %q", ErrQueueFull, band.Name)
}
if c.queuedLocked() >= c.cfg.QueueCapacityTotal && !c.displaceBelowLocked(idx) {
c.rejected[band.Name]++
c.mu.Unlock()
return nil, fmt.Errorf("%w: %d queued in all, and nothing of a lower band to make room",
ErrQueueFull, c.cfg.QueueCapacityTotal)
}
w := &waiter{
tenant: tenantKey(r),
@@ -279,6 +298,10 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
c.mu.Unlock()
return nil, ErrClosed
}
if w.err != nil {
c.mu.Unlock()
return nil, w.err
}
c.waitTotal[band.Name] += time.Since(w.enqueued)
c.mu.Unlock()
return &Lease{c: c, band: band.Name, model: r.Model}, nil
@@ -297,8 +320,9 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
func (c *Controller) remove(idx int, w *waiter, reason string) {
c.mu.Lock()
defer c.mu.Unlock()
if w.admitted {
// It won the race; the caller will see ready closed next time round.
if w.admitted || w.err != nil {
// It won the race, or was displaced; the caller will see ready
// closed next time round.
return
}
c.queues[idx].remove(w)
@@ -410,3 +434,31 @@ func tenantKey(r Request) string {
}
return "anonymous"
}
// queuedLocked is how many requests wait in every band. Called with mu held.
func (c *Controller) queuedLocked() int {
n := 0
for _, q := range c.queues {
n += q.len()
}
return n
}
// displaceBelowLocked turns away the newest request of the lowest band below
// idx that has any queued, and says whether there was one. Called with mu held.
func (c *Controller) displaceBelowLocked(idx int) bool {
// Bands are sorted highest priority first, so lower bands have larger
// indices, and the lowest is last.
for j := len(c.queues) - 1; j > idx; j-- {
w := c.queues[j].newest()
if w == nil {
continue
}
c.queues[j].remove(w)
w.err = fmt.Errorf("%w (band %q)", ErrDisplaced, c.cfg.Bands[j].Name)
close(w.ready)
c.rejected[c.cfg.Bands[j].Name+"/displaced"]++
return true
}
return false
}
@@ -0,0 +1,89 @@
package admission
import (
"context"
"errors"
"testing"
"time"
)
// queueUntil starts an Acquire in the background and waits until the band
// holds depth requests, returning where its result will arrive.
func queueUntil(t *testing.T, c *Controller, band string, bandIdx, depth int) <-chan error {
t.Helper()
out := make(chan error, 1)
go func() {
l, err := c.Acquire(context.Background(), Request{Model: "m", Band: band})
if err == nil {
l.Release()
}
out <- err
}()
deadline := time.Now().Add(2 * time.Second)
for c.Depths()[bandIdx].Depth < depth && time.Now().Before(deadline) {
time.Sleep(time.Millisecond)
}
if got := c.Depths()[bandIdx].Depth; got != depth {
t.Fatalf("band %s holds %d, expected %d", band, got, depth)
}
return out
}
func TestAHigherBandDisplacesTheNewestOfTheLowest(t *testing.T) {
s := &fixedSat{}
s.set(5) // past every ceiling: everything queues
cfg := testConfig()
cfg.QueueCapacityTotal = 3
c := newController(t, cfg, s.fn)
// Bands, highest first: platinum 0, gold 1, silver 2, bronze 3.
oldest := queueUntil(t, c, "bronze", 3, 1)
time.Sleep(2 * time.Millisecond)
silver := queueUntil(t, c, "silver", 2, 1)
time.Sleep(2 * time.Millisecond)
newest := queueUntil(t, c, "bronze", 3, 2)
// Full at three. Gold arrives: the newest bronze goes, not the oldest,
// and not silver, which outranks bronze.
gold := queueUntil(t, c, "gold", 1, 1)
select {
case err := <-newest:
if !errors.Is(err, ErrDisplaced) {
t.Fatalf("the newest bronze got %v", err)
}
case <-time.After(time.Second):
t.Fatal("nothing was displaced")
}
if d := c.Depths(); d[3].Depth != 1 || d[2].Depth != 1 || d[1].Depth != 1 {
t.Errorf("depths %+v: one bronze, the silver and the gold should remain", d)
}
// Bronze cannot displace anything: nothing is below it.
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
if !errors.Is(err, ErrQueueFull) {
t.Errorf("a bronze request at a full queue got %v", err)
}
// Another gold takes the remaining bronze, then silver.
queueUntil(t, c, "gold", 1, 2)
if err := <-oldest; !errors.Is(err, ErrDisplaced) {
t.Errorf("the last bronze got %v", err)
}
queueUntil(t, c, "gold", 1, 3)
if err := <-silver; !errors.Is(err, ErrDisplaced) {
t.Errorf("silver got %v", err)
}
// All gold now: a fourth has nobody lower to displace.
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueFull) {
t.Errorf("a gold request with only gold queued got %v", err)
}
// And they are served once there is room.
s.set(0)
select {
case err := <-gold:
if err != nil {
t.Errorf("the first gold was not served: %v", err)
}
case <-time.After(2 * time.Second):
t.Error("the queue did not drain")
}
}
+17
View File
@@ -12,6 +12,9 @@ type waiter struct {
// admitted guards the race between the dispatcher admitting a waiter and
// the waiter giving up on its own.
admitted bool
// err is why the waiter was turned away while queued — displaced by a
// higher band — set before ready is closed.
err error
}
// bandQueue holds one band's waiters, grouped by tenant.
@@ -166,3 +169,17 @@ func (q *bandQueue) dropExpired(now time.Time, onDrop func()) {
q.byTenant[tenant] = kept
}
}
// newest is the most recently queued waiter, across every tenant: the one
// displacement takes, since it has invested the least waiting.
func (q *bandQueue) newest() *waiter {
var out *waiter
for _, ws := range q.byTenant {
for _, w := range ws {
if out == nil || w.enqueued.After(out.enqueued) {
out = w
}
}
}
return out
}
+9 -1
View File
@@ -37,8 +37,16 @@ type Info struct {
Weight float64
// Priority expresses pool preference; higher pools are tried first.
Priority int
// UpstreamModel is what the endpoint calls the model, when that differs
// from the catalog's name: an external provider's. Requests are rewritten
// to it. Empty sends the name as the client did.
UpstreamModel string
}
// EngineExternal marks an endpoint that is a provider outside the cluster.
// It publishes no metrics, so it is never scraped and never reads as stale.
const EngineExternal = "external"
// Serves reports whether the replica serves the given model.
func (i Info) Serves(model string) bool {
return slices.Contains(i.Models, model)
@@ -242,7 +250,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
continue
}
var age time.Duration
stale := true
stale := l.info.Engine != EngineExternal
if !l.scrapedAt.IsZero() {
age = now.Sub(l.scrapedAt)
stale = age > r.cfg.StaleAfter
+89
View File
@@ -0,0 +1,89 @@
package router
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
func TestALocalModelWithNoCapacityFallsBackToAnExternalProvider(t *testing.T) {
local := newFakeEngine(t, "local")
local.status.Store(http.StatusServiceUnavailable)
var calls atomic.Int64
var gotModel, gotAuth atomic.Value
provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/metrics" {
t.Error("the provider was scraped for metrics")
}
calls.Add(1)
gotAuth.Store(r.Header.Get("Authorization"))
var body map[string]any
b, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(b, &body)
gotModel.Store(fmt.Sprint(body["model"]))
w.Header().Set("Content-Type", "application/json")
fmt.Fprint(w, `{"model":"gpt-4o-mini","choices":[{"message":{"content":"from outside"}}],"usage":{"prompt_tokens":3,"completion_tokens":2}}`)
}))
t.Cleanup(provider.Close)
cfg := baseConfig(config.Endpoint{ID: "l", URL: local.srv.URL, Models: []string{"local"}})
cloud := catalog.Model{Name: "cloud", Source: "external",
External: &catalog.External{URL: provider.URL, Model: "gpt-4o-mini"}}
cfg.Models = []catalog.Model{catModel("local", nil, "cloud"), cloud}
rt := newTestRouter(t, cfg)
rt.SetEndpoints(t.Context(), []endpoint.Info{
{ID: "l", URL: local.srv.URL, Models: []string{"local"}},
{ID: "external:cloud", URL: provider.URL, Models: []string{"cloud"}, APIKey: "provider-key",
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1, UpstreamModel: "gpt-4o-mini"},
})
rec := post(t, rt.Handler(), map[string]any{"model": "local", "messages": []any{}}, nil)
if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
if calls.Load() != 1 {
t.Fatalf("the provider was called %d times", calls.Load())
}
if gotModel.Load() != "gpt-4o-mini" {
t.Errorf("the provider was asked for %v, not its own name for the model", gotModel.Load())
}
if gotAuth.Load() != "Bearer provider-key" {
t.Errorf("the provider got %v, not its key", gotAuth.Load())
}
if got := rec.Header().Get("x-llmgw-served-model"); got != "cloud" {
t.Errorf("the client should be told it was served by cloud, got %q", got)
}
// Never scraped, and never stale for it.
for _, s := range rt.reg.Snapshots("cloud") {
if s.Stale {
t.Error("an external endpoint reads as stale")
}
}
}
func TestAnExternalModelIsNotAModelWithProfiles(t *testing.T) {
good := catalog.Model{Name: "cloud", Source: "external",
External: &catalog.External{URL: "https://api.example.com", Model: "gpt-4o-mini"}}
if _, err := catalog.New([]catalog.Model{good}); err != nil {
t.Fatalf("a valid external model was refused: %v", err)
}
for name, m := range map[string]catalog.Model{
"profiles": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b", Model: "m"},
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp"}}},
"no url": {Name: "x", Source: "external", External: &catalog.External{URL: "api.example.com", Model: "m"}},
"no model": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b"}},
} {
if _, err := catalog.New([]catalog.Model{m}); err == nil {
t.Errorf("%s: accepted", name)
}
}
}
+3
View File
@@ -88,6 +88,9 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
{"queued too long", func(w http.ResponseWriter) {
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
}, 504, ""},
{"displaced for a higher band", func(w http.ResponseWriter) {
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
}, 503, "5"},
{"shutting down", func(w http.ResponseWriter) {
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
}, 503, ""},
+58
View File
@@ -60,9 +60,15 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
return
case errors.Is(err, admission.ErrQueueTimeout):
status = http.StatusGatewayTimeout
case errors.Is(err, admission.ErrDisplaced):
// Turned away for higher-priority work, not for good: said so the
// client backs off and comes back rather than giving up.
w.Header().Set("Retry-After", "5")
}
reason := "unavailable"
switch {
case errors.Is(err, admission.ErrDisplaced):
reason = "displaced"
case errors.Is(err, admission.ErrQueueFull):
reason = "queue_full"
case errors.Is(err, admission.ErrQueueTimeout):
@@ -185,6 +191,11 @@ func (r *Router) Handler() http.Handler {
pages.RegisterAdminUsers(mux, r.store)
// Quotas are written on the same tree the accounts page builds, so
// they are mounted together.
// Capacity quotas are the control plane's, reached through the
// same admin API the CLI uses.
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
pages.SetCapacity(c)
}
pages.RegisterAdminQuotas(mux, r.store, r.acct)
// And the report of what that tree actually used.
pages.RegisterAdminUsage(mux, r.store, r.acct)
@@ -415,6 +426,12 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
// around the allowlist; matching only the canonical name would silently ignore
// an allowlist written in the names clients actually use.
func (r *Router) permits(p auth.Principal, requested string) bool {
// An adapter named in the allowlist allows that adapter: not its model,
// and not the model's other adapters, which the checks below would let
// through if the adapter's own name were not asked about first.
if _, ok := r.adapter(requested); ok && p.Allows(requested) {
return true
}
canonical, m := r.resolve(requested)
if p.Allows(canonical) {
return true
@@ -973,3 +990,44 @@ func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
w.Header().Set("x-ratelimit-reset-tokens", reset)
}
}
// withModel is body with its model set to name, for an endpoint that knows the
// model by another name — an external provider. The body unchanged when it is
// not a JSON object, which the endpoint then refuses as it would have anyway.
func withModel(body []byte, name string) []byte {
var m map[string]json.RawMessage
if err := json.Unmarshal(body, &m); err != nil {
return body
}
encoded, err := json.Marshal(name)
if err != nil {
return body
}
m["model"] = encoded
out, err := json.Marshal(m)
if err != nil {
return body
}
return out
}
// withAdapter is body asking for model with a LoRA adapter applied, in
// llama.cpp's form: "lora": [{"id": index, "scale": scale}]. Adapters not
// listed stay at zero, the scale the engine loads each one at.
func withAdapter(body []byte, model string, index int, scale float64) []byte {
var m map[string]json.RawMessage
if err := json.Unmarshal(body, &m); err != nil {
return body
}
name, err1 := json.Marshal(model)
lora, err2 := json.Marshal([]map[string]any{{"id": index, "scale": scale}})
if err1 != nil || err2 != nil {
return body
}
m["model"], m["lora"] = name, lora
out, err := json.Marshal(m)
if err != nil {
return body
}
return out
}
+46 -11
View File
@@ -61,6 +61,8 @@ type Router struct {
// router's half of not having to. A request takes the map once and works
// from it, so a swap mid-request is invisible rather than a race.
models atomic.Pointer[map[string]catalog.Model]
// adapters are the LoRA adapters by the name clients ask for them by.
adapters atomic.Pointer[map[string]adapterRef]
// extraMetrics are collectors other components publish on /metrics.
extraMetrics []prometheus.Collector
@@ -315,14 +317,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
// all. A control plane pushes over the top of this at startup and on every
// change after.
if len(cfg.Models) > 0 {
seed := map[string]catalog.Model{}
for _, m := range cfg.Models {
seed[m.Name] = m
for _, a := range m.Aliases {
seed[a] = m
}
}
r.models.Store(&seed)
r.storeModels(cfg.Models)
}
return r, nil
}
@@ -347,6 +342,7 @@ func admissionConfig(rc config.Router) admission.Config {
},
QueueTTL: rc.Admission.QueueTTL,
QueueCapacityPerBand: rc.Admission.QueueCapacityPerBand,
QueueCapacityTotal: rc.Admission.QueueCapacityTotal,
Interval: rc.Admission.Interval,
}
}
@@ -504,15 +500,44 @@ func (r *Router) modelMap() map[string]catalog.Model {
// The catalog's counterpart to SetEndpoints, and pushed for the same reason:
// the control plane owns what should exist, the router needs to know, and
// neither has to be in the other's process for that to work.
func (r *Router) SetCatalog(c *catalog.Catalog) {
func (r *Router) SetCatalog(c *catalog.Catalog) { r.storeModels(c.Models()) }
// storeModels makes models what names resolve to: each model by its name, its
// aliases and its adapters. One place for it, whether the models came from
// configuration at start or from the control plane since.
func (r *Router) storeModels(models []catalog.Model) {
m := map[string]catalog.Model{}
for _, entry := range c.Models() {
ads := map[string]adapterRef{}
for _, entry := range models {
m[entry.Name] = entry
for _, a := range entry.Aliases {
m[a] = entry
}
// An adapter resolves to the model it runs on, as an alias does, and
// is applied to the request on the way there.
for i, a := range entry.Adapters {
m[a.Name] = entry
ads[a.Name] = adapterRef{base: entry.Name, index: i, scale: a.EffectiveScale()}
}
}
r.models.Store(&m)
r.adapters.Store(&ads)
}
// adapterRef is where an adapter runs and how it is applied.
type adapterRef struct {
base string
index int
scale float64
}
// adapter finds one by the name a client asked for.
func (r *Router) adapter(name string) (adapterRef, bool) {
if p := r.adapters.Load(); p != nil {
a, ok := (*p)[name]
return a, ok
}
return adapterRef{}, false
}
// LastUsed reports when each endpoint was last sent a request, satisfying
@@ -714,10 +739,20 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
attribute.String("url.full", dec.Endpoint.URL+req.Path)))
body := req.UpstreamBody
if dec.Endpoint.UpstreamModel != "" {
body = withModel(body, dec.Endpoint.UpstreamModel)
}
// An adapter asked for by name, served by its model's replicas: the
// engine is asked for the model, with the adapter applied. Not on a
// fallback to another model, which does not have it.
if a, ok := r.adapter(req.Model); ok && a.base == model {
body = withAdapter(body, model, a.index, a.scale)
}
out, perr := r.px.Do(uctx, w, proxy.Request{
URL: dec.Endpoint.URL + req.Path,
APIKey: dec.Endpoint.APIKey,
Body: req.UpstreamBody,
Body: body,
Header: withTraceparent(uctx, req.Header),
Stream: req.Stream,
SuppressUsageChunk: req.SuppressUsageChunk,
+28 -18
View File
@@ -1,6 +1,9 @@
package scheduler
import (
"sort"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
@@ -51,34 +54,41 @@ func KVSaturated(ceiling float64) Filter {
}
}
// PoolPriority keeps only the endpoints of the highest-priority pool
// (plan §7.3, the two-stage decision: pool first, then replica).
// PoolPriority keeps the endpoints of one pool: the highest-priority one,
// unless it is saturated and a lower one is not (plan §7.3, the two-stage
// decision — pool first, then replica).
//
// Spillover arrives in phase 5, at which point this filter will fall through
// to the next pool once the preferred one is saturated.
func PoolPriority() Filter {
return poolPriorityFilter{}
// Saturated is what admission means by it, the same formula over the same
// thresholds, so the two layers agree on when a pool is full: the H100 pool is
// used until it is, and only then the A100 one — never the A100 because it
// happens to be emptier. When every pool is saturated the preferred one is
// kept: moving to a pool that is just as full buys nothing and loses whatever
// made the first preferred.
func PoolPriority(targetWaiting, kvCeiling float64) Filter {
return poolPriorityFilter{t: admission.Thresholds{TargetWaitingPerReplica: targetWaiting, KVCeiling: kvCeiling}}
}
type poolPriorityFilter struct{}
type poolPriorityFilter struct{ t admission.Thresholds }
func (poolPriorityFilter) Name() string { return "pool-priority" }
func (poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
func (f poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
if len(eps) == 0 {
return eps
}
best := eps[0].Priority
for _, e := range eps[1:] {
if e.Priority > best {
best = e.Priority
}
}
out := make([]endpoint.Snapshot, 0, len(eps))
byPriority := map[int][]endpoint.Snapshot{}
var order []int
for _, e := range eps {
if e.Priority == best {
out = append(out, e)
if _, ok := byPriority[e.Priority]; !ok {
order = append(order, e.Priority)
}
byPriority[e.Priority] = append(byPriority[e.Priority], e)
}
sort.Sort(sort.Reverse(sort.IntSlice(order)))
for _, p := range order {
if admission.Saturation(byPriority[p], 0, f.t) < 1 {
return byPriority[p]
}
}
return out
return byPriority[order[0]]
}
+3 -3
View File
@@ -50,7 +50,7 @@ func Chat(c ProfileConfig) *Profile {
c.normalize()
return &Profile{
Name: "chat",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
Scorers: c.withPrefix([]WeightedScorer{
{Session(c.SessionLoadFactor), 3.0},
{Queue(c.TargetWaiting), 1.0},
@@ -81,7 +81,7 @@ func Batch(c ProfileConfig) *Profile {
c.normalize()
return &Profile{
Name: "batch",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
Scorers: []WeightedScorer{
{Queue(c.TargetWaiting), 3.0},
{KV(c.KVCeiling), 1.0},
@@ -99,7 +99,7 @@ func LongContext(c ProfileConfig) *Profile {
c.normalize()
return &Profile{
Name: "long-context",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
// Prefix matching earns more here than anywhere else: a long prompt is
// mostly prefill, and prefill is what a cache hit skips.
Scorers: c.withPrefix([]WeightedScorer{
+36 -2
View File
@@ -177,7 +177,7 @@ func TestStaleEndpointIsPenalizedNotDropped(t *testing.T) {
func TestPoolPriority(t *testing.T) {
p := &Profile{
Name: "t",
Filters: []Filter{PoolPriority()},
Filters: []Filter{PoolPriority(4, 0.9)},
Scorers: []WeightedScorer{{Queue(4), 1}},
Picker: MaxScore(),
}
@@ -189,7 +189,8 @@ func TestPoolPriority(t *testing.T) {
if err != nil {
t.Fatal(err)
}
// The lower-priority pool must not win despite being emptier (spillover is phase 5).
// The lower-priority pool must not win for being emptier: the preferred
// one is not saturated (3 waiting of a healthy 4).
if d.Endpoint.ID != "h100" {
t.Errorf("pool priority was ignored: %s", d.Endpoint.ID)
}
@@ -383,3 +384,36 @@ func TestBuiltinProfilesSchedule(t *testing.T) {
}
}
}
func TestASaturatedPoolSpillsToTheNext(t *testing.T) {
p := &Profile{Name: "t", Filters: []Filter{PoolPriority(4, 0.9)},
Scorers: []WeightedScorer{{Queue(4), 1}}, Picker: MaxScore()}
h100 := func(w int) endpoint.Snapshot {
return ep("h100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 100; s.Pool = "ib-a/h100" })
}
a100 := func(w int) endpoint.Snapshot {
return ep("a100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 50; s.Pool = "ib-b/a100" })
}
pick := func(eps ...endpoint.Snapshot) string {
d, err := p.Schedule(newCtx(Request{Model: "m"}), eps)
if err != nil {
t.Fatal(err)
}
return d.Endpoint.ID
}
if got := pick(h100(8), a100(0)); got != "a100" {
t.Errorf("the preferred pool at twice its healthy queue did not spill: %s", got)
}
if got := pick(h100(8), a100(6)); got != "h100" {
t.Errorf("both pools saturated: the preferred one should be kept, got %s", got)
}
if got := pick(h100(4), a100(0)); got != "a100" {
t.Errorf("at exactly its healthy limit a pool is full: %s", got)
}
// KV pressure saturates a pool as queue depth does.
full := h100(0)
full.Stats.KVCacheUsage = 0.95
if got := pick(full, a100(0)); got != "a100" {
t.Errorf("a pool out of KV cache did not spill: %s", got)
}
}
+3
View File
@@ -96,6 +96,9 @@ func (p *Poller) Sync(ctx context.Context) {
want := make(map[string]endpoint.Info, len(infos))
for _, in := range infos {
if in.Engine == endpoint.EngineExternal {
continue // a provider elsewhere publishes no /metrics
}
want[in.ID] = in
}
+30
View File
@@ -0,0 +1,30 @@
package store
import (
"errors"
"fmt"
"testing"
"github.com/jackc/pgx/v5/pgconn"
)
func TestADuplicateIsRecognisedOnEitherDatabase(t *testing.T) {
for _, c := range []struct {
name string
err error
want bool
}{
{"sqlite", errors.New("constraint failed: UNIQUE constraint failed: users.email (2067)"), true},
// What pgx returns, wrapped as database/sql passes it on: "unique"
// in lower case, which a check for "UNIQUE" missed.
{"postgres", fmt.Errorf("exec: %w", &pgconn.PgError{Code: "23505",
Message: `duplicate key value violates unique constraint "users_email_key"`}), true},
{"postgres, as text", errors.New(`ERROR: duplicate key value violates unique constraint "users_email_key" (SQLSTATE 23505)`), true},
{"another postgres error", &pgconn.PgError{Code: "23502", Message: "null value in column"}, false},
{"nothing", nil, false},
} {
if got := isUniqueViolation(c.err); got != c.want {
t.Errorf("%s: %v, want %v", c.name, got, c.want)
}
}
}
+20 -1
View File
@@ -6,6 +6,8 @@ import (
"errors"
"strings"
"time"
"github.com/jackc/pgx/v5/pgconn"
)
// UserState is where an account is in its life.
@@ -66,12 +68,29 @@ func (s *DB) CreateUser(ctx context.Context, u User) error {
VALUES (?, ?, ?, ?, ?, ?, ?)`,
u.ID, u.Email, u.Name, u.PasswordHash, string(u.State),
nullString(u.ProjectID), millis(u.CreatedAt))
if err != nil && strings.Contains(err.Error(), "UNIQUE") {
if isUniqueViolation(err) {
return ErrEmailTaken
}
return err
}
// isUniqueViolation reports whether err is a row refused for a value another
// row already has. Each database says it its own way: SQLite in words
// ("UNIQUE constraint failed"), PostgreSQL by code, 23505, in a message with
// "unique" in lower case — which a check for the upper-case word missed, so a
// second sign-up with an address on PostgreSQL was "something went wrong".
func isUniqueViolation(err error) bool {
if err == nil {
return false
}
var pe *pgconn.PgError
if errors.As(err, &pe) {
return pe.Code == "23505"
}
m := err.Error()
return strings.Contains(m, "UNIQUE constraint failed") || strings.Contains(m, "SQLSTATE 23505")
}
// UserByEmail finds an account by its address.
func (s *DB) UserByEmail(ctx context.Context, email string) (User, error) {
return s.scanUser(s.db.QueryRowContext(ctx,
+21 -2
View File
@@ -4,11 +4,13 @@ import (
"context"
"crypto/rand"
"encoding/base32"
"encoding/hex"
"errors"
"log/slog"
"net/http"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/account"
@@ -61,6 +63,8 @@ type AccountPages struct {
hasMirror bool
// hasQuotas is set when /admin/quotas is registered.
hasQuotas bool
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
capacity CapacityAdmin
// personalOff refuses personal keys; see SetPersonalOff.
personalOff bool
// personalDefaults is what personal use falls under by default.
@@ -74,6 +78,9 @@ type AccountPages struct {
priced Priced
// registry browses a model registry, or is nil where there is no way out.
registry Registry
// kinds remembers what each held GGUF file is, by its address: a held
// version never changes, so it is read once.
kinds sync.Map
// catalog puts a mirrored model into service; fleet says whether anything
// could run it. Both nil where the mirror page is read-only.
catalog CatalogAdmin
@@ -163,8 +170,12 @@ func (p *AccountPages) signup(w http.ResponseWriter, r *http.Request) {
case errors.Is(err, account.ErrWeakPassword), errors.Is(err, account.ErrInvalidEmail):
d.Error = err.Error()
default:
p.log.Error("signup failed", "err", err)
d.Error = "something went wrong; try again"
// Nothing of the database's words on the page; a reference that finds
// them in the log for whoever runs the gateway.
ref := errorRef()
p.log.Error("signup failed", "ref", ref, "err", err)
d.Error = "something went wrong on our side, and your account was not created. " +
"Try again later; if it keeps happening, give an operator this reference: " + ref
}
render(w, "signup", d)
}
@@ -632,3 +643,11 @@ func (p *AccountPages) renderWithCSRF(w http.ResponseWriter, r *http.Request,
// SetLlamaCppVersion tells the pages which llama.cpp build the nodes run.
func (p *AccountPages) SetLlamaCppVersion(v string) { p.llamaVersion = v }
// errorRef is a short reference for an error a page does not explain, logged
// beside the error itself.
func errorRef() string {
var b [4]byte
_, _ = rand.Read(b[:])
return "err-" + hex.EncodeToString(b[:])
}
+138
View File
@@ -0,0 +1,138 @@
package web
import (
"context"
"fmt"
"net/http"
"strconv"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// CapacityAdmin sets what parts of the tenancy tree are guaranteed of the
// cluster and may hold. The admin API implements it; the page and the CLI go
// through the same checks.
type CapacityAdmin interface {
QuotaViews(ctx context.Context) ([]adminapi.QuotaView, error)
SetQuota(ctx context.Context, req adminapi.QuotaRequest, by string) (string, error)
ClearQuota(ctx context.Context, scope string) (string, error)
Scopes(ctx context.Context) ([]adminapi.ScopeName, error)
}
// SetCapacity gives the quotas page capacity quotas to show and set. Without
// it the section is left off.
func (p *AccountPages) SetCapacity(c CapacityAdmin) { p.capacity = c }
// capacityRow is one scope's capacity quota as the page shows it.
type capacityRow struct {
adminapi.QuotaView
GPUs, Cores, Memory string
}
// capacityForm is what the form starts with: a scope's quota to change, or
// nothing for a new one.
type capacityForm struct {
Scope string
GPUs, Cores, Memory string
LGPUs, LCores, LMem string
}
func (p *AccountPages) capacitySection(ctx context.Context, d *adminQuotasData, editing string) {
if p.capacity == nil {
return
}
d.HasCapacity = true
views, err := p.capacity.QuotaViews(ctx)
if err != nil {
d.Error = err.Error()
return
}
for _, v := range views {
d.Capacity = append(d.Capacity, capacityRow{QuotaView: v,
GPUs: capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
Cores: capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)})
if v.Scope == editing {
d.CapForm = capacityForm{Scope: v.Scope,
GPUs: intStr(v.Guarantee.GPUs), Cores: intStr(v.Guarantee.CPUCores), Memory: intStr(v.Guarantee.MemoryGB),
LGPUs: intStr(v.Limit.GPUs), LCores: intStr(v.Limit.CPUCores), LMem: intStr(v.Limit.MemoryGB)}
}
}
if d.Scopes, err = p.capacity.Scopes(ctx); err != nil {
d.Error = err.Error()
}
}
// capCell is "6 of 8 (at most 16)": held, guaranteed, limit.
func capCell(holds int, g, l *int) string {
if g == nil && l == nil {
return ""
}
out := strconv.Itoa(holds)
if g != nil {
out += " of " + strconv.Itoa(*g)
}
if l != nil {
out += fmt.Sprintf(" (at most %d)", *l)
}
return out
}
func intStr(v *int) string {
if v == nil {
return ""
}
return strconv.Itoa(*v)
}
// applyCapacity sets or clears one scope's capacity quota from the form.
func (p *AccountPages) applyCapacity(r *http.Request, admin store.User) (string, error) {
scope := strings.TrimSpace(r.FormValue("scope"))
if scope == "" {
return "", fmt.Errorf("choose where the quota applies")
}
if r.FormValue("clear") != "" {
key, err := p.capacity.ClearQuota(r.Context(), scope)
if err != nil {
return "", err
}
return "capacity quota removed from " + key, nil
}
field := func(name, what string) (*int, error) {
v := strings.TrimSpace(r.FormValue(name))
if v == "" {
return nil, nil
}
n, err := strconv.Atoi(v)
if err != nil || n < 0 {
return nil, fmt.Errorf("%s must be a whole number, or blank for none", what)
}
return &n, nil
}
var req adminapi.QuotaRequest
req.Scope = scope
var err error
for _, f := range []struct {
dst **int
name string
what string
}{
{&req.Guarantee.GPUs, "gpus", "guaranteed GPUs"},
{&req.Guarantee.CPUCores, "cores", "guaranteed cores"},
{&req.Guarantee.MemoryGB, "memory", "guaranteed memory"},
{&req.Limit.GPUs, "limit_gpus", "the GPU limit"},
{&req.Limit.CPUCores, "limit_cores", "the core limit"},
{&req.Limit.MemoryGB, "limit_memory", "the memory limit"},
} {
if *f.dst, err = field(f.name, f.what); err != nil {
return "", err
}
}
key, err := p.capacity.SetQuota(r.Context(), req, admin.Email)
if err != nil {
return "", err
}
return "capacity quota set on " + key + "; it takes effect within half a minute", nil
}
+88
View File
@@ -0,0 +1,88 @@
package web
import (
"context"
"errors"
"log/slog"
"net/http"
"net/url"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type fakeCapacity struct {
views []adminapi.QuotaView
set []adminapi.QuotaRequest
cleared []string
}
func (f *fakeCapacity) QuotaViews(context.Context) ([]adminapi.QuotaView, error) { return f.views, nil }
func (f *fakeCapacity) SetQuota(_ context.Context, r adminapi.QuotaRequest, _ string) (string, error) {
if r.Scope == "team:nope" {
return "", errors.New(`there is no team "nope"`)
}
f.set = append(f.set, r)
return r.Scope, nil
}
func (f *fakeCapacity) ClearQuota(_ context.Context, s string) (string, error) {
f.cleared = append(f.cleared, s)
return s, nil
}
func (f *fakeCapacity) Scopes(context.Context) ([]adminapi.ScopeName, error) {
return []adminapi.ScopeName{{Key: "org:o1", Name: "Acme"}, {Key: "team:t1", Name: "Acme / ml"}}, nil
}
func TestCapacityQuotasAreShownAndSetFromTheQuotasPage(t *testing.T) {
eight, sixteen := 8, 16
fc := &fakeCapacity{views: []adminapi.QuotaView{{Scope: "team:t1", Name: "Acme / ml",
Guarantee: store.Resources{GPUs: &eight}, Limit: store.Resources{GPUs: &sixteen},
Holds: quota.Amount{GPUs: 6}, UpdatedBy: "ops@acme"}}}
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
p.SetCapacity(fc)
mux := http.NewServeMux()
p.RegisterAdminQuotas(mux, &fakeQuotas{}, pricedYes{})
body := mirrorGet(t, mux, "/admin/quotas").Body.String()
for _, want := range []string{`id="capacity"`, "Acme / ml", "6 of 8 (at most 16)", "ops@acme",
`<option value="team:t1">Acme / ml</option>`} {
if !strings.Contains(body, want) {
t.Errorf("the page lacks %q", want)
}
}
// "change" fills the form with the scope's figures.
edit := mirrorGet(t, mux, "/admin/quotas?cap=team:t1").Body.String()
if !strings.Contains(edit, `<option value="team:t1" selected>`) || !strings.Contains(edit, `name="limit_gpus" value="16"`) {
t.Error("change did not open the form on the scope's quota")
}
res := mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"},
"gpus": {"8"}, "cores": {""}, "limit_memory": {"64"}}).Body.String()
if len(fc.set) != 1 || *fc.set[0].Guarantee.GPUs != 8 || fc.set[0].Guarantee.CPUCores != nil || *fc.set[0].Limit.MemoryGB != 64 {
t.Errorf("sent %+v", fc.set)
}
if !strings.Contains(res, "capacity quota set on team:t1") {
t.Errorf("no word of it: %s", res)
}
for _, bad := range []url.Values{{"scope": {"team:t1"}, "gpus": {"eight"}}, {"scope": {"team:nope"}, "gpus": {"1"}}, {"gpus": {"1"}}} {
bad.Set("csrf", "tok")
n := len(fc.set)
out := mirrorPost(t, mux, "/admin/quotas/capacity", bad).Body.String()
if len(fc.set) != n || !strings.Contains(out, `class="error"`) {
t.Errorf("%v was not refused with a reason", bad)
}
}
mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"}, "clear": {"1"}})
if len(fc.cleared) != 1 || fc.cleared[0] != "team:t1" {
t.Errorf("cleared %v", fc.cleared)
}
// Without a CSRF token nothing happens.
mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"scope": {"team:t1"}, "clear": {"1"}})
if len(fc.cleared) != 1 {
t.Error("a change was made without the CSRF token")
}
}
+92 -4
View File
@@ -11,6 +11,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
@@ -46,6 +47,10 @@ type Registry interface {
// repoSet is one thing worth fetching: a file, or every shard of one.
type repoSet struct {
// Kind is what the file says it is, read from its first bytes: "adapter"
// for a LoRA adapter, "model", or empty when it was not read. Arch is the
// architecture it names, which an adapter must share with its model.
Kind, Arch string
// Verdict is whether the nodes' llama.cpp can load it, from the
// architecture the registry reports; a set that cannot be served is
// shown with the reason and no fetch button.
@@ -55,6 +60,9 @@ type repoSet struct {
// A shard set goes as one action — a fifteen-shard model is not fifteen
// decisions, it is one.
Sources []repoSource
// Companions are the files fetched with a safetensors set that are not
// weights: config.json, the tokenizer, the shard index.
Companions []string
// Model and Version are what the fetch form starts with. Editable, because
// the name a catalog entry uses is the operator's to choose and the
// registry's idea of it is only a good guess.
@@ -71,6 +79,13 @@ type modelUse struct {
Replicas int
// Min and Max are its scaling as the catalog holds it.
Min, Max int
// Class is what it set, empty for its default; EffClass what that comes
// to. Owner is org:ID or empty for shared. MemoryGB and CPUCores are what
// a replica of its first profile reserves.
Class, EffClass, Owner string
MemoryGB, CPUCores int
// Adapters are the LoRA adapters it serves.
Adapters []catalog.Adapter
// Ready, Wanted and Nodes are how it runs now, for the row itself:
// replicas ready of those it should have, and the nodes they are on.
// Fleet is false where nothing could be asked, so the row says nothing
@@ -100,6 +115,9 @@ type mirrorRow struct {
// Detail is the card opened beneath the row: what runs it, on which
// nodes, and how many replicas it runs.
Detail *heldDetail
// Kind is what a single GGUF file is — "adapter" or "model" — and Arch
// its architecture; empty when not read.
Kind, Arch string
// Open is the row's ?open= value.
Open string
@@ -293,7 +311,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
return name + " now runs all the time", nil
}))
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setReplicas))
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setModel))
mux.HandleFunc("POST /admin/mirror/adapters", p.mirrorAction(m, p.setAdapters))
mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
@@ -301,6 +320,16 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
if err != nil {
return "", err
}
// The CPU while GPUs sit free is a choice to make on purpose:
// made by default, it was a model running at a tenth of the
// speed with nothing saying why.
if model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" {
if n := freeGPUs(p.fleet); n > 0 {
return "", fmt.Errorf("this fleet has %s free and %s was set to run on the CPU; "+
"choose a GPU placement, or tick \"run on the CPU anyway\"",
plural(n, "GPU", "GPUs"), model.Name)
}
}
// The same check `llmgw ctl model add` makes. This page builds
// the source out of a row the mirror reported, so it is right
// far more often than something typed — but a file removed
@@ -316,8 +345,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
// Placed on the next pass, not now: saying it is serving would
// be wrong for however long the weights take to reach a node.
return fmt.Sprintf(
"%s added to the catalog, %s; it is placed on the next pass",
model.Name, plural(model.Scaling.Min, "replica", "replicas")), nil
"%s added to the catalog, %s: %s; it is placed on the next pass",
model.Name, plural(model.Scaling.Min, "replica", "replicas"), describeServe(model)), nil
}))
}
@@ -463,8 +492,13 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
u := modelUse{
Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
Min: m.Scaling.Min, Max: m.Scaling.Max,
Class: m.Class, EffClass: m.EffectiveClass(), Owner: m.Owner,
Adapters: m.Adapters,
Declared: p.catalog.Declared(m.Name),
}
if len(m.Profiles) > 0 {
u.MemoryGB, u.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
}
if v, ok := views[m.Name]; ok {
u.Fleet, u.Ready, u.Wanted = true, v.Ready, v.Desired
u.Parked, u.Unplaceable = v.Parked, v.Unplaceable
@@ -478,13 +512,16 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
}
}
p.heldKinds(r.Context(), d.Rows)
// The one row whose card is open. From the form too, so a change made in
// the card comes back with the card still open to show it.
open := r.FormValue("open")
for i := range d.Rows {
d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
d.Rows[i].Detail = p.heldDetailFor(d.Rows[i])
d.Rows[i].Detail = p.heldDetailFor(r.Context(), d.Rows[i])
d.Rows[i].Detail.addAdapterChoices(d.Rows[i], d.Rows)
}
}
@@ -744,8 +781,59 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
rs.Sources = append(rs.Sources,
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
}
// Safetensors are not a model on their own: vLLM loads a directory,
// and reads its config and tokenizer before any weight. Fetched as
// part of the set, or the set arrives as shards nothing can load.
if strings.HasSuffix(strings.ToLower(set.Name), ".safetensors") {
for _, f := range hf.Companions(got.Files) {
rs.Sources = append(rs.Sources,
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
rs.Companions = append(rs.Companions, f.Path)
rs.Bytes += f.Bytes
}
}
d.RepoSets = append(d.RepoSets, rs)
}
p.probeKinds(r.Context(), d.RepoSets)
}
// kindOf reads what a GGUF file is; a variable so tests need no registry.
var kindOf = func(ctx context.Context, url string) (gguf.Header, error) {
return gguf.KindFromURL(ctx, nil, url)
}
// probeKinds reads the start of each single-file GGUF set, to tell LoRA
// adapters from models: a repository tagged "lora" holds a model with the
// adapter merged in as often as an adapter, and only the file says which.
// A few at once and a few in all, since each is a request to the registry.
func (p *AccountPages) probeKinds(ctx context.Context, sets []repoSet) {
const most, together = 12, 6
sem := make(chan struct{}, together)
var wg sync.WaitGroup
n := 0
for i := range sets {
s := &sets[i]
if len(s.Sources) != 1 || !gguf.IsGGUF(s.Sources[0].Path) || n == most {
continue
}
n++
wg.Add(1)
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
cctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if h, err := kindOf(cctx, s.Sources[0].URL); err == nil {
s.Arch = h.Architecture
s.Kind = "model"
if h.IsAdapter() {
s.Kind = "adapter"
}
}
}()
}
wg.Wait()
}
// defaultModelName is the catalog name a repository suggests.
+34
View File
@@ -710,3 +710,37 @@ func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
t.Errorf("a started download should be shown where it is followed: %s", body)
}
}
func TestASafetensorsSetIsFetchedWithWhatTheEngineReadsBesideIt(t *testing.T) {
// The field: a four-shard AWQ model mirrored as its shards alone, and
// vLLM failing at start for want of config.json.
reg := &fakeRegistry{repo: hf.Repo{
ID: "Qwen/Qwen2.5-32B-Instruct-AWQ", Revision: "abc123def456",
Files: []hf.File{
{Path: "model-00001-of-00002.safetensors", Bytes: 4_000_000_000},
{Path: "model-00002-of-00002.safetensors", Bytes: 3_000_000_000},
{Path: "model.safetensors.index.json", Bytes: 90_000},
{Path: "config.json", Bytes: 900},
{Path: "tokenizer.json", Bytes: 7_000_000},
{Path: "tokenizer_config.json", Bytes: 7_000},
{Path: "README.md", Bytes: 9_000},
{Path: "modeling_custom.py", Bytes: 40_000},
{Path: "onnx/config.json", Bytes: 900},
},
}}
body := mirrorGet(t, mirrorPageWith(t, &fakeMirror{}, reg, true),
"/admin/mirror?repo=Qwen/Qwen2.5-32B-Instruct-AWQ").Body.String()
for _, want := range []string{"config.json", "tokenizer.json", "tokenizer_config.json", "model.safetensors.index.json"} {
if !strings.Contains(body, `name="file" value="`+want+`"`) {
t.Errorf("%s is not fetched with the set", want)
}
}
for _, not := range []string{"README.md", "modeling_custom.py", "onnx/config.json"} {
if strings.Contains(body, `name="file" value="`+not+`"`) {
t.Errorf("%s was swept in with the set", not)
}
}
if !strings.Contains(body, "with 4 files the engine reads beside them") {
t.Errorf("the companions are not said; got %s", body)
}
}
+31 -6
View File
@@ -9,6 +9,7 @@ import (
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -129,12 +130,17 @@ type adminQuotasData struct {
// Editing is the "scope:id" whose form is open, so the template can keep
// the link on every other row.
Editing string
Since string
Message string
Error string
CSRF string
Version string
Now string
// The capacity section: quotas on GPUs, cores and memory.
HasCapacity bool
Capacity []capacityRow
Scopes []adminapi.ScopeName
CapForm capacityForm
Since string
Message string
Error string
CSRF string
Version string
Now string
}
// RegisterAdminQuotas mounts quota administration under /admin/quotas.
@@ -176,6 +182,24 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
}
p.renderQuotas(w, r, s, msg, "")
}))
mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
func(w http.ResponseWriter, r *http.Request, admin store.User) {
sess, _, _ := p.session(r)
if p.capacity == nil || !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
http.Error(w, "bad request", http.StatusBadRequest)
return
}
msg, err := p.applyCapacity(r, admin)
if err != nil {
p.log.Warn("a capacity quota change was refused",
"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
p.renderQuotas(w, r, s, "", err.Error())
return
}
p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
p.renderQuotas(w, r, s, msg, "")
}))
}
// applyQuota writes one scope's limits, and says what it did.
@@ -259,6 +283,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
d.Editing = r.URL.Query().Get("edit")
p.capacitySection(r.Context(), &d, r.FormValue("cap"))
limits, err := s.AllLimits(r.Context())
if err != nil {
+188 -6
View File
@@ -22,6 +22,12 @@ type CatalogAdmin interface {
// Scale changes how a model added at runtime runs: always, on demand, or
// not at all. One declared in configuration is refused.
Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error)
// SetClass, SetOwner and Resources change what a model's capacity is
// worth, whose quota it counts against, and what a replica reserves.
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
}
// serveForm is the offer to put one mirrored version into service.
@@ -37,8 +43,20 @@ type serveForm struct {
Engine string
// Profile is the profile name, which doubles as how a node_selector would
// be written later. Named for the hardware rather than the model, because
// a profile is about where a model runs.
// a profile is about where a model runs. Derived from the placement, not
// typed: a name is only a name, and a field for it read as though calling
// a profile "gpu" would put it on one.
Profile string
// Placements are where it can run: each GPU class the fleet has, and for
// llama.cpp the CPU. One is selected.
Placements []placeChoice
// GPUs is how many cards a replica takes, when placed on a GPU class.
GPUs int
// Parallel is how many requests one replica serves at once.
Parallel int
// FreeGPUs is how many cards no replica holds, across the fleet: what
// choosing the CPU leaves unused.
FreeGPUs int
// Files is how many files the version holds, above one for a set.
Files int
// MinMemoryGB is the host RAM to ask for, estimated from the weights.
@@ -120,6 +138,13 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
default:
f.Profile = "gpu"
}
f.GPUs, f.Parallel = 1, defaultParallel
f.Placements, f.FreeGPUs = placementsFor(fleet, f.Engine)
for _, pc := range f.Placements {
if pc.Selected {
f.Profile = pc.profile()
}
}
// Whether any node can actually run it. Agents report their installed
// engines now, so this is a fact rather than the guess it started as —
// which was cards, and would have passed a GPU node with no vLLM on it.
@@ -268,12 +293,38 @@ func buildModel(get func(string) string) (catalog.Model, error) {
if engine == "" {
return m, errors.New("which engine should run it?")
}
p := catalog.Profile{
Name: strings.TrimSpace(get("profile")), Engine: engine,
Weight: 1, Priority: 100,
p := catalog.Profile{Engine: engine, Weight: 1, Priority: 100}
gpus := 1
if v := strings.TrimSpace(get("gpus")); v != "" {
if gpus, err = positive(v, "the GPUs per replica"); err != nil {
return m, err
}
gpus = max(gpus, 1)
}
if p.Name == "" {
p.Name = engine
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
case onGPU && class != "":
// Placement is decided by gpu_class, which is what the selector
// sets; the count is gpus_per_node for llama.cpp, which spreads
// layers over a node's cards, and tp for the others.
p.GPUClass = class
if engine == "llamacpp" {
p.GPUsPerNode = gpus
} else {
p.TP = gpus
}
place.GPUs = gpus
case place.Value == "" || place.Value == "cpu":
if engine != "llamacpp" {
return m, fmt.Errorf("%s runs on GPUs; choose a GPU class to place it on", engine)
}
place.Value = "cpu"
default:
return m, fmt.Errorf("%q is not a placement", place.Value)
}
p.Name = place.profile()
if v := strings.TrimSpace(get("profile")); v != "" {
p.Name = v
}
// The pool, not the per-request limit: llama.cpp divides it across its
// parallel slots, so the number a client may use is smaller and is what
@@ -284,6 +335,15 @@ func buildModel(get func(string) string) (catalog.Model, error) {
}
p.Args = []string{"--ctx-size", v}
}
if v := strings.TrimSpace(get("parallel")); v != "" && engine == "llamacpp" {
n, err := positive(v, "the parallel requests")
if err != nil {
return m, err
}
if n > 0 {
p.Args = append(p.Args, "--parallel", strconv.Itoa(n))
}
}
if v := strings.TrimSpace(get("min_memory_gb")); v != "" {
if p.MinMemoryGB, err = positive(v, "the memory requirement"); err != nil {
return m, err
@@ -402,3 +462,125 @@ func modeOf(sc catalog.Scaling) string {
// defaultCtxPool is the llama.cpp context pool the serve form starts with.
// --parallel divides it among slots; it does not multiply it.
const defaultCtxPool = 16384
// defaultParallel is the requests a llama.cpp replica serves at once when
// the form is not changed: the agent's own default.
const defaultParallel = 4
// placeChoice is one place a model can run: a GPU class, or the CPU.
type placeChoice struct {
// Value is "gpu:<class>" or "cpu", as the form submits it.
Value string
Label string
// GPUs is how many cards a replica takes, for naming the profile.
GPUs int
Selected bool
}
// profile names the profile for where it runs: the class, with the count when
// it is more than one card ("t4", "t4-x4"), or "cpu".
func (c placeChoice) profile() string {
class, ok := strings.CutPrefix(c.Value, "gpu:")
if !ok || class == "" {
return "cpu"
}
if c.GPUs > 1 {
return fmt.Sprintf("%s-x%d", class, c.GPUs)
}
return class
}
// placementsFor lists where engine can run on this fleet, with one selected,
// and how many GPUs are free.
//
// A GPU when the fleet has one: a GGUF model was put on the CPU by this form
// on a cluster of GPUs, ran at a tenth of their speed, and nothing on the way
// said why — the form had no choice to offer and named the profile "cpu". The
// class selected is the one with the most cards free.
func placementsFor(fleet Source, engine string) ([]placeChoice, int) {
var out []placeChoice
free, best := 0, -1
if fleet != nil {
for _, c := range fleet.Summary().GPUs {
n := c.Free()
free += n
out = append(out, placeChoice{
Value: "gpu:" + c.Class,
Label: fmt.Sprintf("GPU: %s, %d of %d free", c.Class, n, c.Total),
})
if best < 0 || n > fleet.Summary().GPUs[best].Free() {
best = len(out) - 1
}
}
}
if best >= 0 {
out[best].Selected = true
}
if engine == "llamacpp" {
label := "CPU only: no GPU, many times slower"
if len(out) == 0 {
label = "CPU: this fleet has no GPU"
}
out = append(out, placeChoice{Value: "cpu", Label: label, Selected: best < 0})
}
return out, free
}
// freeGPUs is how many cards no replica holds across the fleet.
func freeGPUs(fleet Source) int {
if fleet == nil {
return 0
}
n := 0
for _, c := range fleet.Summary().GPUs {
n += c.Free()
}
return n
}
// describeServe says, in one line, what a model will run as: the summary
// shown before it is added and in the message after.
func describeServe(m catalog.Model) string {
if len(m.Profiles) == 0 {
return ""
}
p := m.Profiles[0]
engine := map[string]string{"llamacpp": "llama.cpp", "vllm": "vLLM", "sglang": "SGLang"}[p.Engine]
if engine == "" {
engine = p.Engine
}
where := "on the CPU"
if p.GPUClass != "" {
n := max(p.GPUsPerNode, p.TP, 1)
where = fmt.Sprintf("on %s %s", plural(n, "GPU", "GPUs"), p.GPUClass)
if n > 1 {
where = fmt.Sprintf("on %d %s GPUs of one node", n, p.GPUClass)
}
}
parts := []string{engine + " " + where}
if p.MinMemoryGB > 0 {
parts = append(parts, fmt.Sprintf("%d GB of host memory", p.MinMemoryGB))
}
if p.GPUClass == "" && p.CPUCores > 0 {
parts = append(parts, plural(p.CPUCores, "core", "cores"))
}
pool, parallel := argInt(p.Args, "--ctx-size"), argInt(p.Args, "--parallel")
switch {
case pool > 0 && parallel > 0:
parts = append(parts, fmt.Sprintf("context pool %d over %d parallel requests (%d each)", pool, parallel, pool/parallel))
case pool > 0:
parts = append(parts, fmt.Sprintf("context pool %d", pool))
}
return strings.Join(parts, ", ")
}
// argInt is the number after flag in args, or 0.
func argInt(args []string, flag string) int {
for i := 0; i+1 < len(args); i++ {
if args[i] == flag {
n, _ := strconv.Atoi(args[i+1])
return n
}
}
return 0
}
+126 -7
View File
@@ -3,6 +3,7 @@ package web
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
@@ -15,12 +16,16 @@ import (
)
type fakeCatalog struct {
added []catalog.Model
declared string
sources map[string][]catalog.Model
scaled []string
lastScale catalog.Scaling
err error
added []catalog.Model
declared string
sources map[string][]catalog.Model
scaled []string
lastScale catalog.Scaling
err error
classes []string
owners []string
resources []string
adapterSets [][]catalog.Adapter
}
func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
@@ -35,6 +40,26 @@ func (f *fakeCatalog) Sources(context.Context) (map[string][]catalog.Model, erro
return f.sources, nil
}
func (f *fakeCatalog) SetAdapters(_ context.Context, name string, as []catalog.Adapter, _ string) (catalog.Model, error) {
f.adapterSets = append(f.adapterSets, as)
return catalog.Model{Name: name, Adapters: as}, f.err
}
func (f *fakeCatalog) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
f.classes = append(f.classes, name+"="+class)
return catalog.Model{Name: name, Class: class}, f.err
}
func (f *fakeCatalog) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
f.owners = append(f.owners, name+"="+owner)
return catalog.Model{Name: name, Owner: owner}, f.err
}
func (f *fakeCatalog) Resources(_ context.Context, name, _ string, mem, cores int, _ string) (catalog.Model, error) {
f.resources = append(f.resources, fmt.Sprintf("%s=%d/%d", name, mem, cores))
return catalog.Model{Name: name}, f.err
}
func (f *fakeCatalog) Scale(_ context.Context, name string, sc catalog.Scaling, _ string) (catalog.Model, error) {
f.scaled = append(f.scaled, name)
f.lastScale = sc
@@ -46,6 +71,8 @@ type fakeFleet struct {
gpus int
engines []string
memoryGB int
// used is how many of the cards replicas hold.
used int
}
func (f fakeFleet) Nodes() []adminapi.NodeView {
@@ -56,7 +83,12 @@ func (f fakeFleet) Nodes() []adminapi.NodeView {
func (f fakeFleet) Models() []adminapi.ModelView { return nil }
func (f fakeFleet) Replicas(string) []adminapi.ReplicaView { return nil }
func (f fakeFleet) Explain(string) adminapi.ExplainView { return adminapi.ExplainView{} }
func (f fakeFleet) Summary() adminapi.Summary { return adminapi.Summary{} }
func (f fakeFleet) Summary() adminapi.Summary {
if f.gpus == 0 {
return adminapi.Summary{}
}
return adminapi.Summary{GPUs: []adminapi.ClassUse{{Class: "t4", Total: f.gpus, Used: f.used}}}
}
func held(model, version, file string) mirror.Status {
return mirror.Status{Model: model, Version: version, File: file,
@@ -427,3 +459,90 @@ func TestServingOnDemandLoadsWhenAskedFor(t *testing.T) {
t.Errorf("on demand was added as %+v", sc)
}
}
func TestAGGUFOnAFleetWithGPUsIsOfferedTheGPUFirst(t *testing.T) {
gguf := []mirror.Status{held("coder", "v1", "coder-q4_k_m.gguf")}
body := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 4, "llamacpp"),
"/admin/mirror?serve=coder:v1").Body.String()
// Found in the field: no choice offered, a profile called "cpu", and a
// model on the CPU of a GPU cluster at a tenth of the speed.
if !strings.Contains(body, `<option value="gpu:t4" selected>GPU: t4, 4 of 4 free</option>`) {
t.Errorf("the free GPU class should be offered and chosen; got %s", body)
}
if !strings.Contains(body, `<option value="cpu">CPU only`) {
t.Error("the CPU should still be on offer, unselected")
}
if !strings.Contains(body, `name="cpu_ok"`) {
t.Error("choosing the CPU with GPUs free needs saying so")
}
cpuOnly := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 0, "llamacpp"),
"/admin/mirror?serve=coder:v1").Body.String()
if !strings.Contains(cpuOnly, `<option value="cpu" selected>CPU: this fleet has no GPU</option>`) ||
strings.Contains(cpuOnly, `name="cpu_ok"`) {
t.Errorf("a fleet without GPUs has only the CPU, and nothing to confirm; got %s", cpuOnly)
}
}
func TestTheChosenPlacementDecidesWhereItRuns(t *testing.T) {
form := func(engine, place, gpus string) url.Values {
return url.Values{
"name": {"m"}, "source": {"http://x/m"}, "engine": {engine}, "replicas": {"1"},
"context_length": {"4096"}, "ctx_size": {"16384"}, "parallel": {"4"},
"placement": {place}, "gpus": {gpus}, "min_memory_gb": {"8"},
}
}
m, err := buildModel(form("llamacpp", "gpu:t4", "4").Get)
if err != nil {
t.Fatal(err)
}
p := m.Profiles[0]
if p.GPUClass != "t4" || p.GPUsPerNode != 4 || p.TP > 1 || p.Name != "t4-x4" {
t.Errorf("llama.cpp on four t4s: %+v", p)
}
if got := describeServe(m); got != "llama.cpp on 4 t4 GPUs of one node, 8 GB of host memory, "+
"context pool 16384 over 4 parallel requests (4096 each)" {
t.Errorf("summary %q", got)
}
if m, _ := buildModel(form("vllm", "gpu:t4", "2").Get); m.Profiles[0].TP != 2 || m.Profiles[0].GPUsPerNode != 0 {
t.Errorf("vLLM's GPU count is tp: %+v", m.Profiles[0])
}
if m, _ := buildModel(form("llamacpp", "cpu", "1").Get); m.Profiles[0].GPUClass != "" || m.Profiles[0].Name != "cpu" {
t.Errorf("the CPU: %+v", m.Profiles[0])
}
if _, err := buildModel(form("vllm", "cpu", "1").Get); err == nil {
t.Error("vLLM on the CPU was accepted")
}
}
func TestTheCPUWithGPUsFreeIsAChoiceMadeOnPurpose(t *testing.T) {
values := func(extra ...string) url.Values {
v := url.Values{
"csrf": {"tok"}, "name": {"coder"}, "source": {reachableSource(t)}, "engine": {"llamacpp"},
"replicas": {"1"}, "context_length": {"4096"}, "placement": {"cpu"}, "min_memory_gb": {"8"},
}
for i := 0; i+1 < len(extra); i += 2 {
v.Set(extra[i], extra[i+1])
}
return v
}
cat := &fakeCatalog{}
page := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
Fleet: fakeFleet{gpus: 4, used: 1, engines: []string{"llamacpp"}}}, true)
body := mirrorPost(t, page, "/admin/mirror/serve", values()).Body.String()
if len(cat.added) != 0 || !strings.Contains(body, "3 GPUs free") {
t.Fatalf("the CPU with three GPUs free went through unasked: %s", body)
}
mirrorPost(t, page, "/admin/mirror/serve", values("cpu_ok", "yes"))
if len(cat.added) != 1 {
t.Fatal("ticking the box should add it")
}
// Every card held: nothing to ask.
cat = &fakeCatalog{}
full := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
Fleet: fakeFleet{gpus: 4, used: 4, engines: []string{"llamacpp"}}}, true)
mirrorPost(t, full, "/admin/mirror/serve", values())
if len(cat.added) != 1 {
t.Error("with no GPU free the CPU is the only choice and should not be questioned")
}
}
+235 -9
View File
@@ -4,13 +4,19 @@ import (
"context"
"errors"
"fmt"
"html/template"
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -29,14 +35,23 @@ const maxReplicas = 64
// and room to change it.
type heldDetail struct {
Models []heldModel
// Scopes are the parts of the tenancy tree a model can belong to; empty
// when capacity quotas are not set up, and the owner choice is left off.
Scopes []adminapi.ScopeName
}
// heldModel is one catalog model running a held version.
type heldModel struct {
modelUse
View adminapi.ModelView
Replicas []heldReplica
Headroom *adminapi.HeadroomView
// Adapters are the model's own; AdapterChoices the held adapter files
// of its architecture not attached yet; FindAdapters the explorer's
// list of adapters made for it.
Adapters []catalog.Adapter
AdapterChoices []adapterChoice
FindAdapters template.URL
View adminapi.ModelView
Replicas []heldReplica
Headroom *adminapi.HeadroomView
// MostNow is how many replicas could always run, Current plus the room
// found; zero when the room was not measured or had no end in sight.
MostNow int
@@ -64,11 +79,14 @@ func openKey(model, version string) string { return model + ":" + version }
// heldDetailFor reads, for each model using a row, how it runs and how much
// more room it has.
func (p *AccountPages) heldDetailFor(row mirrorRow) *heldDetail {
func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDetail {
d := &heldDetail{}
if p.capacity != nil {
d.Scopes, _ = p.capacity.Scopes(ctx)
}
if p.fleet == nil {
for _, u := range row.UsedBy {
d.Models = append(d.Models, heldModel{modelUse: u})
d.Models = append(d.Models, heldModel{modelUse: u, Adapters: u.Adapters})
}
return d
}
@@ -82,7 +100,7 @@ func (p *AccountPages) heldDetailFor(row mirrorRow) *heldDetail {
}
hs, _ := p.fleet.(HeadroomSource)
for _, u := range row.UsedBy {
hm := heldModel{modelUse: u, View: views[u.Name]}
hm := heldModel{modelUse: u, View: views[u.Name], Adapters: u.Adapters}
for _, rep := range p.fleet.Replicas(u.Name) {
// Built field by field: a ReplicaView carries the engine's key,
// which is for the admin listener and never for a page.
@@ -98,7 +116,7 @@ func (p *AccountPages) heldDetailFor(row mirrorRow) *heldDetail {
if h, ok := hs.Headroom(u.Name, 16); ok {
hm.Headroom = &h
if !h.Limited {
hm.MostNow = h.Current + h.More
hm.MostNow = h.Current + h.More + h.ByPreempting
}
}
}
@@ -144,7 +162,7 @@ func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *htt
if !ok {
return "", fmt.Errorf("no model %q", name)
}
fits := h.Current + h.More
fits := h.Current + h.More + h.ByPreempting
if lo > fits && !h.Limited {
why := "nothing more fits"
if len(h.Blocked) > 0 {
@@ -156,6 +174,11 @@ func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *htt
if hi > fits && !h.Limited {
note = fmt.Sprintf("; there is room for %d now, so it grows past that only if room is made", fits)
}
// Room its class would make by stopping others: said, since it is
// somebody else's replica that goes.
if lo > h.Current+h.More && len(h.Preempts) > 0 {
note += "; to make room, replicas of " + strings.Join(h.Preempts, ", ") + " will be stopped"
}
}
sc := catalog.Scaling{Min: lo, Max: hi, IdleTimeout: current.Scaling.IdleTimeout}
@@ -171,7 +194,7 @@ func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *htt
case lo == 0:
return fmt.Sprintf("%s loads when asked for, up to %d under load%s", name, hi, note), nil
case lo == hi:
return fmt.Sprintf("%s runs %s", name, plural(lo, "replica", "replicas")), nil
return fmt.Sprintf("%s runs %s%s", name, plural(lo, "replica", "replicas"), note), nil
}
return fmt.Sprintf("%s runs %d, and up to %d under load%s", name, lo, hi, note), nil
}
@@ -205,3 +228,206 @@ func age(sec int64) string {
}
return fmt.Sprintf("%dd", int(d.Hours()/24))
}
// setModel applies the card's settings form: what a replica reserves, the
// model's class and owner, then how many replicas it runs — in that order, so
// the room checked for the replicas is the room its new class and reservation
// would get. Only what changed is applied, and each change is said.
func (p *AccountPages) setModel(ctx context.Context, admin store.User, r *http.Request) (string, error) {
name := r.FormValue("model")
current, err := p.currentModel(ctx, name)
if err != nil {
return "", err
}
if p.catalog.Declared(name) {
return "", fmt.Errorf("%s is set in server.yaml; change it there", name)
}
var done []string
if r.Form.Has("memory") && len(current.Profiles) > 0 {
mem, err1 := strconv.Atoi(strings.TrimSpace(r.FormValue("memory")))
cores, err2 := strconv.Atoi(strings.TrimSpace(r.FormValue("cores")))
if err1 != nil || err2 != nil || mem < 0 || cores < 0 {
return "", errors.New("memory and cores are whole numbers")
}
if cur := current.Profiles[0]; mem != cur.MinMemoryGB || cores != cur.CPUCores {
if _, err := p.catalog.Resources(ctx, name, "", mem, cores, admin.Email); err != nil {
return "", err
}
done = append(done, fmt.Sprintf("a replica reserves %d GB and %d cores (running ones keep theirs until restarted)", mem, cores))
}
}
if r.Form.Has("class") {
if c := r.FormValue("class"); c != current.Class {
m, err := p.catalog.SetClass(ctx, name, c, admin.Email)
if err != nil {
return "", err
}
done = append(done, "is "+m.EffectiveClass())
}
}
if r.Form.Has("owner") && p.capacity != nil {
if o := r.FormValue("owner"); o != current.Owner {
if _, err := p.catalog.SetOwner(ctx, name, o, admin.Email); err != nil {
return "", err
}
if o == "" {
done = append(done, "is shared")
} else {
done = append(done, "counts against "+o)
}
}
}
lo, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("min")))
hi, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("max")))
if r.Form.Has("min") && (lo != current.Scaling.Min || hi != current.Scaling.Max) {
msg, err := p.setReplicas(ctx, admin, r)
if err != nil {
if len(done) > 0 {
return "", fmt.Errorf("%s %s, but not the replicas: %w", name, strings.Join(done, ", "), err)
}
return "", err
}
done = append(done, strings.TrimPrefix(msg, name+" "))
}
if len(done) == 0 {
return name + ": nothing changed", nil
}
return name + " " + strings.Join(done, "; "), nil
}
// heldKinds reads what each held single-file GGUF is — an adapter or a model —
// once per file, remembered after.
func (p *AccountPages) heldKinds(ctx context.Context, rows []mirrorRow) {
var wg sync.WaitGroup
sem := make(chan struct{}, 6)
for i := range rows {
row := &rows[i]
if row.State != mirror.Present || row.Files != 1 || !gguf.IsGGUF(row.File) {
continue
}
if h, ok := p.kinds.Load(row.URL); ok {
row.Kind, row.Arch = kindName(h.(gguf.Header)), h.(gguf.Header).Architecture
continue
}
wg.Add(1)
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if h, err := kindOf(cctx, row.URL); err == nil {
p.kinds.Store(row.URL, h)
row.Kind, row.Arch = kindName(h), h.Architecture
}
}()
}
wg.Wait()
}
func kindName(h gguf.Header) string {
if h.IsAdapter() {
return "adapter"
}
return "model"
}
// adapterChoice is a held adapter file that could be attached to a model.
type adapterChoice struct {
URL, Label, Suggest string
}
// addAdapterChoices offers, on each model of a held model's card, the held
// adapter files of the same architecture, and a way to find more.
func (d *heldDetail) addAdapterChoices(row mirrorRow, rows []mirrorRow) {
for i := range d.Models {
hm := &d.Models[i]
hm.FindAdapters = findAdaptersHref(row.Source)
if row.Kind == "adapter" {
continue
}
for _, r := range rows {
if r.Kind != "adapter" || (row.Arch != "" && r.Arch != row.Arch) {
continue
}
attached := false
for _, a := range hm.Adapters {
attached = attached || a.Source == r.URL
}
if !attached {
hm.AdapterChoices = append(hm.AdapterChoices, adapterChoice{URL: r.URL,
Label: r.Model + " " + r.Version + " · " + r.File, Suggest: hm.Name + "-" + r.Model})
}
}
}
}
// findAdaptersHref is the explorer's list of adapters for a held model: those
// made for the model its repository was made from, when it came from the
// registry, and every adapter otherwise.
func findAdaptersHref(source string) template.URL {
const base = "/admin/mirror?tab=explore&view=adapters"
u, err := url.Parse(source)
if err != nil || u.Host != "huggingface.co" {
return base
}
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
if len(parts) < 2 {
return base
}
return template.URL(base + "&from=" + url.QueryEscape(parts[0]+"/"+parts[1]))
}
// setAdapters attaches or detaches one adapter on a model, from its card.
// An adapter is checked before it is attached: a file that is not an adapter,
// or one made for another architecture, is a replica that fails to start.
func (p *AccountPages) setAdapters(ctx context.Context, admin store.User, r *http.Request) (string, error) {
name := r.FormValue("model")
current, err := p.currentModel(ctx, name)
if err != nil {
return "", err
}
adapters := slices.Clone(current.Adapters)
switch r.FormValue("op") {
case "remove":
gone := r.FormValue("name")
adapters = slices.DeleteFunc(adapters, func(a catalog.Adapter) bool { return a.Name == gone })
if len(adapters) == len(current.Adapters) {
return "", fmt.Errorf("%s has no adapter %q", name, gone)
}
if _, err := p.catalog.SetAdapters(ctx, name, adapters, admin.Email); err != nil {
return "", err
}
return fmt.Sprintf("%s: adapter %s removed; replicas drop it when restarted (llmgw ctl restart --model %s)",
name, gone, name), nil
case "add":
an := strings.TrimSpace(r.FormValue("name"))
src := strings.TrimSpace(r.FormValue("source"))
scale := 1.0
if v := strings.TrimSpace(r.FormValue("scale")); v != "" {
if scale, err = strconv.ParseFloat(v, 64); err != nil || scale <= 0 {
return "", errors.New("the scale is a positive number")
}
}
if an == "" || src == "" {
return "", errors.New("an adapter needs a name and a file")
}
h, err := kindOf(ctx, src)
if err != nil {
return "", fmt.Errorf("the adapter file could not be read: %w", err)
}
if !h.IsAdapter() {
return "", errors.New("that file is a model, not a LoRA adapter")
}
if base, err := kindOf(ctx, current.Source); err == nil && base.Architecture != h.Architecture {
return "", fmt.Errorf("that adapter is for %s models and %s is %s", h.Architecture, name, base.Architecture)
}
adapters = append(adapters, catalog.Adapter{Name: an, Source: src, Scale: scale})
if _, err := p.catalog.SetAdapters(ctx, name, adapters, admin.Email); err != nil {
return "", err
}
return fmt.Sprintf("%s: adapter %s added; ask for it by that name once the replicas are restarted "+
"(llmgw ctl restart --model %s)", name, an, name), nil
}
return "", errors.New("add or remove an adapter")
}
+155 -3
View File
@@ -1,21 +1,28 @@
package web
import (
"context"
"log/slog"
"net/http"
"net/url"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// roomyFleet runs one replica of "coder" on n2 and has room for `more`.
type roomyFleet struct {
fakeFleet
more int
blocked []string
asked int
more int
blocked []string
asked int
preempt int
preempts []string
}
func (f *roomyFleet) Nodes() []adminapi.NodeView {
@@ -41,6 +48,7 @@ func (f *roomyFleet) Headroom(model string, limit int) (adminapi.HeadroomView, b
} else {
h.Limited = true
}
h.ByPreempting, h.Preempts = f.preempt, f.preempts
return h, true
}
@@ -181,3 +189,147 @@ func TestNodeNamesLoseOnlyWhatEveryNodeShares(t *testing.T) {
}
}
}
func settingsPage(t *testing.T, cat *fakeCatalog, withQuotas bool) *http.ServeMux {
t.Helper()
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
if withQuotas {
p.SetCapacity(&fakeCapacity{})
}
mux := http.NewServeMux()
p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
}}, Catalog: cat, Fleet: &roomyFleet{more: 3}})
return mux
}
func TestACardSetsClassOwnerAndReservationAndOnlyWhatChanged(t *testing.T) {
cat := coderCatalog()
cat.sources["http://mirror:8081/coder/v1/c.gguf"][0].Profiles = []catalog.Profile{{Name: "cpu", MinMemoryGB: 12, CPUCores: 4}}
mux := settingsPage(t, cat, true)
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
for _, want := range []string{`name="class"`, `<option value="" selected>default (standard)</option>`,
`name="owner"`, `<option value="team:t1">Acme / ml</option>`, `name="memory" value="12"`, `name="cores" value="4"`} {
if !strings.Contains(card, want) {
t.Errorf("the card lacks %q", want)
}
}
post := func(v url.Values) string {
v.Set("csrf", "tok")
v.Set("model", "coder")
v.Set("open", "coder:v1")
return mirrorPost(t, mux, "/admin/mirror/replicas", v).Body.String()
}
same := url.Values{"min": {"1"}, "max": {"1"}, "class": {""}, "owner": {""}, "memory": {"12"}, "cores": {"4"}}
if body := post(same); !strings.Contains(body, "coder: nothing changed") ||
len(cat.classes)+len(cat.owners)+len(cat.resources)+len(cat.scaled) != 0 {
t.Errorf("an unchanged form changed something: %v %v %v %v", cat.classes, cat.owners, cat.resources, cat.scaled)
}
changed := url.Values{"min": {"1"}, "max": {"1"}, "class": {"critical"}, "owner": {"team:t1"}, "memory": {"14"}, "cores": {"4"}}
body := post(changed)
if strings.Join(cat.resources, ",") != "coder=14/4" || strings.Join(cat.classes, ",") != "coder=critical" ||
strings.Join(cat.owners, ",") != "coder=team:t1" || len(cat.scaled) != 0 {
t.Errorf("applied %v %v %v %v", cat.resources, cat.classes, cat.owners, cat.scaled)
}
if !strings.Contains(body, "a replica reserves 14 GB and 4 cores") || !strings.Contains(body, "counts against team:t1") {
t.Errorf("the change was not said: %s", body)
}
// Without capacity quotas there is no owner to choose, and none is sent.
plain := mirrorGet(t, settingsPage(t, coderCatalog(), false), "/admin/mirror?open=coder:v1").Body.String()
if strings.Contains(plain, `name="owner"`) {
t.Error("an owner choice with no quotas to count against")
}
}
func TestRoomAHigherClassWouldMakeIsOfferedAndSaid(t *testing.T) {
fleet := &roomyFleet{more: 0, preempt: 1, preempts: []string{"chat"}, blocked: []string{"n1: full"}}
cat := coderCatalog()
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
}}, Catalog: cat, Fleet: fleet}, true)
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
if !strings.Contains(card, "No free room for another replica now.") ||
!strings.Contains(card, "1 by stopping replicas its class outranks: chat.") ||
!strings.Contains(card, `name="min" value="1" min="0" max="2"`) {
t.Errorf("the card should offer the room preemption makes, and say whose")
}
body := mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
"model": {"coder"}, "min": {"2"}, "max": {"2"}, "open": {"coder:v1"}}).Body.String()
if len(cat.scaled) != 1 || cat.lastScale.Min != 2 {
t.Fatalf("two were refused though preemption makes room: %s", body)
}
if !strings.Contains(body, "replicas of chat will be stopped") {
t.Errorf("stopping somebody else's replica should be said: %s", body)
}
body = mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
"model": {"coder"}, "min": {"3"}, "max": {"3"}, "open": {"coder:v1"}}).Body.String()
if len(cat.scaled) != 1 || !strings.Contains(body, "at most 2") {
t.Errorf("three is past even what preemption makes: %s", body)
}
}
func TestAnAdapterHeldInTheMirrorIsAttachedFromItsModelsCard(t *testing.T) {
prev := kindOf
t.Cleanup(func() { kindOf = prev })
kindOf = func(_ context.Context, u string) (gguf.Header, error) {
switch {
case strings.Contains(u, "/sql/"):
return gguf.Header{Architecture: "qwen2", Type: "adapter"}, nil
case strings.Contains(u, "/llama-lora/"):
return gguf.Header{Architecture: "llama", Type: "adapter"}, nil
}
return gguf.Header{Architecture: "qwen2"}, nil
}
cat := coderCatalog()
coder := &cat.sources["http://mirror:8081/coder/v1/c.gguf"][0]
coder.Source = "http://mirror:8081/coder/v1/c.gguf"
coder.Profiles = []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12}}
m := &fakeMirror{models: []mirror.Status{
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1,
Source: "https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/abc/c.gguf"},
{Model: "sql", Version: "v1", File: "sql.gguf", State: mirror.Present, BytesDone: 1},
{Model: "llama-lora", Version: "v1", File: "l.gguf", State: mirror.Present, BytesDone: 1},
}}
mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Catalog: cat, Fleet: &roomyFleet{more: 1}}, true)
list := mirrorGet(t, mux, "/admin/mirror").Body.String()
if strings.Count(list, ">LoRA adapter</span>") != 2 {
t.Error("the two adapter files are not marked as adapters in the list")
}
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
if !strings.Contains(card, `<option value="http://mirror:8081/sql/v1/sql.gguf" data-suggest="coder-sql">`) {
t.Error("the qwen2 adapter is not offered for the qwen2 model")
}
if strings.Contains(card, "llama-lora") && strings.Contains(card, `value="http://mirror:8081/llama-lora/v1/l.gguf"`) {
t.Error("an adapter for another architecture is offered")
}
if !strings.Contains(card, "view=adapters&amp;from=bartowski%2FQwen2.5-Coder-7B-Instruct-GGUF") {
t.Error("finding adapters should start from the model's own repository")
}
post := func(v url.Values) string {
v.Set("csrf", "tok")
v.Set("model", "coder")
v.Set("open", "coder:v1")
return mirrorPost(t, mux, "/admin/mirror/adapters", v).Body.String()
}
body := post(url.Values{"op": {"add"}, "name": {"coder-sql"}, "source": {"http://mirror:8081/sql/v1/sql.gguf"}, "scale": {"0.8"}})
if len(cat.adapterSets) != 1 || len(cat.adapterSets[0]) != 1 || cat.adapterSets[0][0].Scale != 0.8 {
t.Fatalf("attached %+v: %s", cat.adapterSets, body)
}
if !strings.Contains(body, "adapter coder-sql added") {
t.Errorf("the attach was not said: %s", body)
}
for name, src := range map[string]string{"another architecture": "http://mirror:8081/llama-lora/v1/l.gguf",
"a model": "http://mirror:8081/coder/v1/c.gguf"} {
n := len(cat.adapterSets)
out := post(url.Values{"op": {"add"}, "name": {"x"}, "source": {src}})
if len(cat.adapterSets) != n || !strings.Contains(out, `class="error"`) {
t.Errorf("%s was attached", name)
}
}
}
+41 -2
View File
@@ -18,6 +18,9 @@ const (
viewTrending = "trending"
viewNew = "new"
viewDownloads = "downloads"
// viewAdapters lists LoRA adapters, all of them or those made for one
// model.
viewAdapters = "adapters"
)
// newWithin is how recent a repository has to be to count as new.
@@ -31,6 +34,7 @@ const exploreShown = 25
var exploreViews = []struct{ key, label string }{
{viewTrending, "Trending"}, {viewNew, "New this month"}, {viewDownloads, "Most downloaded"},
{viewAdapters, "LoRA adapters"},
}
var exploreTasks = []struct{ key, label, tag string }{
@@ -48,6 +52,9 @@ var exploreSizes = []struct {
// open: a listing, its filters, and links that keep them.
type explorer struct {
View, Task, Size string
// Of is the model whose adapters are listed, in the adapters view; empty
// lists adapters for any model.
Of string
// Here keeps to what the nodes' llama.cpp can load. On by default: a
// trending list is mostly things this cluster cannot run, and finding
// that out one repository at a time is what the explorer is for avoiding.
@@ -123,11 +130,12 @@ func trimZero(s string) string { return strings.TrimSuffix(s, ".0") }
// — the page opened from the menu — it is trending chat models this cluster
// can run.
func exploreState(q url.Values) explorer {
e := explorer{View: q.Get("view"), Task: q.Get("task"), Size: q.Get("size")}
e := explorer{View: q.Get("view"), Task: q.Get("task"), Size: q.Get("size"),
Of: strings.Trim(strings.TrimSpace(q.Get("of")), "/")}
// "x" marks a submitted form, so an unticked box reads as off rather than
// as not yet chosen.
e.Here = q.Get("x") != "1" || q.Get("here") == "1"
if !oneOf(e.View, viewTrending, viewNew, viewDownloads) {
if !oneOf(e.View, viewTrending, viewNew, viewDownloads, viewAdapters) {
e.View = viewTrending
// A search with no view asks for what a name usually means, which is
// the one everybody downloads.
@@ -165,6 +173,10 @@ func (e explorer) href(query, key, value string) template.URL {
if key != "" {
v.Set(key, value)
}
// Whose adapters, kept while it is adapters being looked at.
if e.Of != "" && v.Get("view") == viewAdapters {
v.Set("of", e.Of)
}
return template.URL("/admin/mirror?" + v.Encode())
}
@@ -181,6 +193,16 @@ func (e explorer) hfQuery(search string) hf.Query {
// Trending, then kept to the recent: sorted by creation date alone,
// the newest repositories are mostly empty uploads nobody has used.
q.Limit = 100
case viewAdapters:
// Adapters rarely carry a task or a parameter count, so neither is
// asked for; what makes one useful is the model it was made for.
q.Sort = hf.SortDownloads
if e.Of != "" {
q.Filters = []string{"base_model:adapter:" + e.Of}
} else {
q.Filters = []string{"lora"}
}
return q
}
for _, t := range exploreTasks {
if t.key == e.Task {
@@ -198,6 +220,15 @@ func (e explorer) hfQuery(search string) hf.Query {
// exploreList fills in the listing.
func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
e := exploreState(r.URL.Query())
// From a held model's card: its repository, whose adapters are those of
// the model it was made from — a GGUF repository is a quantisation, and
// adapters are made for the original.
if from := strings.TrimSpace(r.URL.Query().Get("from")); e.View == viewAdapters && e.Of == "" && from != "" {
e.Of = from
if repo, err := p.registry.Repo(r.Context(), from); err == nil && len(repo.BaseModel) > 0 {
e.Of = repo.BaseModel[0]
}
}
for _, v := range exploreViews {
e.Views = append(e.Views, exploreLink{v.label, e.href(d.Query, "view", v.key), v.key == e.View})
}
@@ -225,6 +256,14 @@ func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
if a := m.Architecture(); a != "" && p.llamaVersion != "" {
row.Verdict = gguf.LlamaCpp(p.llamaVersion, a)
}
// Adapters are small and say little about their architecture in a
// listing; whether one fits is read from its file when it is opened.
if e.View == viewAdapters {
if len(e.Rows) < exploreShown {
e.Rows = append(e.Rows, row)
}
continue
}
// An adapter or a fragment, not a model: tens of megabytes of LoRA
// weights listed among the models they need a model to run on.
if e.Here && m.Params() > 0 && m.Params() < minParams {
+52
View File
@@ -1,15 +1,18 @@
package web
import (
"context"
"errors"
"html"
"net/http"
"net/http/httptest"
"net/url"
"os"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
)
@@ -195,3 +198,52 @@ func TestAFetchFromTheListAnswersWithItsOutcomeAlone(t *testing.T) {
t.Errorf("a refusal: %d %s", w.Code, w.Body.String())
}
}
// noProbe answers every file as an unreadable one, for tests that are not
// about what the files are.
func TestMain(m *testing.M) {
kindOf = func(context.Context, string) (gguf.Header, error) {
return gguf.Header{}, errors.New("not probed in tests")
}
os.Exit(m.Run())
}
func TestAdaptersAreListedForAModelAndToldFromModelsByTheirFiles(t *testing.T) {
prev := kindOf
t.Cleanup(func() { kindOf = prev })
kindOf = func(_ context.Context, url string) (gguf.Header, error) {
if strings.Contains(url, "adapter") {
return gguf.Header{Architecture: "qwen2", Type: "adapter"}, nil
}
return gguf.Header{Architecture: "qwen2"}, nil
}
reg := &fakeRegistry{
results: []hf.Model{{ID: "ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF", Downloads: 900}},
repo: hf.Repo{ID: "ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF", Revision: "776530df0000",
BaseModel: []string{"Qwen/Qwen2.5-Coder-7B-Instruct"},
Files: []hf.File{{Path: "lora-adapter-f16.gguf", Bytes: 726 << 20}, {Path: "merged-Q4_K_M.gguf", Bytes: 4 << 30}}},
}
mux := mirrorPageWith(t, &fakeMirror{}, reg, true)
// From a held model's card: its repository is looked up, and the
// adapters listed are those of the model it was made from.
body := mirrorGet(t, mux, "/admin/mirror?tab=explore&view=adapters&from=bartowski/Qwen2.5-Coder-7B-Instruct-GGUF").Body.String()
want := "base_model:adapter:Qwen/Qwen2.5-Coder-7B-Instruct"
if len(reg.query.Filters) != 1 || reg.query.Filters[0] != want {
t.Errorf("asked the registry for %v, expected %s", reg.query.Filters, want)
}
if !strings.Contains(body, "LoRA adapters made for <code>Qwen/Qwen2.5-Coder-7B-Instruct</code>") {
t.Error("the page does not say whose adapters these are")
}
if strings.Contains(body, `aria-label="Task"`) {
t.Error("task and size filters shown for adapters, which rarely carry either")
}
repo := mirrorGet(t, mux, "/admin/mirror?repo=ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF").Body.String()
if !strings.Contains(repo, "LoRA adapter · qwen2") {
t.Error("the adapter file is not marked as one")
}
if !strings.Contains(repo, `<span class="opt">model · qwen2</span>`) {
t.Error("the merged file is not marked as a model")
}
}
+54
View File
@@ -0,0 +1,54 @@
package web
import (
"context"
"errors"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
)
// signupAccounts is an account service whose sign-up answers with err.
type signupAccounts struct {
fakeAccounts
err error
}
func (s signupAccounts) SignUp(context.Context, string, string, string) error { return s.err }
func (signupAccounts) SignupOpen() bool { return true }
func signupBody(t *testing.T, err error) string {
t.Helper()
mux := http.NewServeMux()
NewAccountPages(signupAccounts{err: err}, nil, nil, "v0", slog.New(slog.DiscardHandler)).Register(mux)
form := url.Values{"email": {"a@b.c"}, "name": {"A"}, "password": {"long enough password"}}
req := httptest.NewRequest(http.MethodPost, "/signup", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
return rec.Body.String()
}
func TestASignupThatFailsSaysSoWithAReferenceAndNoDatabaseWords(t *testing.T) {
body := signupBody(t, errors.New(`duplicate key value violates unique constraint "users_email_key"`))
if strings.Contains(body, "users_email_key") || strings.Contains(body, "constraint") {
t.Errorf("the database's own words reached the page: %s", body)
}
if !strings.Contains(body, "was not created") || !strings.Contains(body, "reference: err-") {
t.Errorf("no plain failure with a reference to give an operator: %s", body)
}
}
func TestASignupSaysWhatToDoIfTheAddressHadAnAccount(t *testing.T) {
// The service answers a taken address as it does a new one, so nobody
// can find out from this page who has an account; what the page can do
// is say what to do in either case.
body := signupBody(t, nil)
if !strings.Contains(body, "already had an account") || !strings.Contains(body, `href="/login"`) ||
!strings.Contains(body, "reset it") {
t.Errorf("the done page leaves someone who already had an account stuck: %s", body)
}
}
+7
View File
@@ -855,3 +855,10 @@ table.explore .from { margin-left: .5rem; font-size: .78rem; color: var(--muted)
.clip-inline { display: inline-block; max-width: 24rem; overflow: hidden; text-overflow: ellipsis;
vertical-align: bottom; font-size: .8rem; color: var(--muted); margin-left: .4rem; }
table.rows td.warn { color: var(--warn); cursor: help; }
.held-card .replicas-form select, .capacity-form select { display: block; margin-top: .25rem; }
.held-card .replicas-form .field-row { flex-wrap: wrap; gap: .75rem 1rem; }
.quota-tree td.q-act form.inline { margin-left: .75rem; }
.held-card .adapters { margin: .75rem 0; }
.held-card .adapters h4 { font-size: .9rem; margin: 0 0 .35rem; display: flex; gap: .75rem; align-items: baseline; }
.held-card .adapter-form .field-row { align-items: flex-end; flex-wrap: wrap; gap: .5rem 1rem; }
.held-card .adapter-form select { display: block; margin-top: .25rem; max-width: 28rem; }
+5
View File
@@ -11,6 +11,11 @@
<p class="note">You will not get an email from this gateway — it does not send any.
Tell whoever you asked for access that you have signed up.</p>
<p><a href="/login">Sign in</a> once you have been approved.</p>
{{/* The same page whether or not the address already had an account, so
it says what to do in either case without saying which this was. */}}
<p class="note">If this address already had an account, nothing new was made:
<a href="/login">sign in</a> with it, or, if you have forgotten its password, ask
an operator to reset it.</p>
{{else}}
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
<form method="post" autocomplete="on">
Loaded 100 of 105 files, more files were not shown because too many files have changed in this diff. Show more