Compare commits
18
Commits
No files matched your search
+238
-2
@@ -22,8 +22,193 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
Changes since v0.10.5, published as release candidates (v0.10.6-rc.N) until
|
||||
there are enough for v0.10.6.
|
||||
From the field report in issue #2, after a deployment on GPU nodes.
|
||||
|
||||
### A GPU placement no longer runs on the CPU unnoticed
|
||||
|
||||
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
|
||||
where the loader looks), came up on the CPU, answered at CPU speed, and was
|
||||
reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||||
|
||||
- **Installing** the CUDA build asks it which devices it sees
|
||||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||||
own `ldd` could not say it: the backend is opened at run time.
|
||||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||||
included). One that does not is stopped and reported failed: "placed on GPU
|
||||
0 but computing on the CPU", with the devices the engine lists. This covers
|
||||
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
|
||||
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
|
||||
when the engine sees no GPU, and otherwise prints the devices it computes on.
|
||||
|
||||
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
|
||||
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
|
||||
node with two kinds of GPU, a shard could get a different card from the one it
|
||||
was given.
|
||||
|
||||
### A model of many files, mirrored whole for vLLM
|
||||
|
||||
vLLM loads a directory — shards, config.json, the tokenizer — and does not
|
||||
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
|
||||
shards alone, a node could be handed a version still downloading or missing a
|
||||
shard, and a repository's page given as a model's source was fetched as one
|
||||
file of HTML.
|
||||
|
||||
- **The explorer fetches a safetensors set with what the engine reads beside
|
||||
it**: config.json, generation_config.json, the tokenizer files, the shard
|
||||
index, the quantisation config. Never code: a `.py` would be the engine
|
||||
running what the repository ships, which is to be chosen on purpose.
|
||||
- **A version is offered to nodes only whole.** The mirror refuses its
|
||||
manifest (409, with what is missing) while a file is still downloading, or
|
||||
while a shard its `model.safetensors.index.json` names, or a part of a
|
||||
split (`-00002-of-00004`), is absent. A node reports "not complete on the
|
||||
mirror yet" and is tried again.
|
||||
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
|
||||
it and the manifest carries it; a node checks every file of a set against
|
||||
it before using the set, and fetches one that differs again.
|
||||
- **`llmgw mirror add` takes several URLs** into one version.
|
||||
- **A Hugging Face repository page is refused as a source**, saying to mirror
|
||||
it or to give the repository id for vLLM to fetch itself.
|
||||
|
||||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||||
such a model, as before.
|
||||
|
||||
### Signing up with an address that has an account
|
||||
|
||||
On PostgreSQL, signing up with an address already registered showed
|
||||
"something went wrong; try again": the store recognised a duplicate by the
|
||||
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
|
||||
code 23505. It now knows both, and the address is answered as on SQLite —
|
||||
with the same page as a new sign-up, so the page cannot be used to find out
|
||||
who has an account. That page now also says what to do if the address had
|
||||
one: sign in, or ask an operator to reset the password.
|
||||
|
||||
An error the page does not explain says the account was not created and
|
||||
gives a reference (`err-…`), logged beside the error, for an operator to find
|
||||
it; the database's own words stay in the log.
|
||||
|
||||
### An adapter no fabric uses does not keep a node out
|
||||
|
||||
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
|
||||
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
|
||||
added for single-node inference, with no fabric given, that refused it for
|
||||
RDMA nobody had asked for.
|
||||
|
||||
- Those are failures only for ports a declared fabric uses, and only when a
|
||||
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
|
||||
- Which fabric an undeclared InfiniBand port is on, and whether it has a
|
||||
subnet manager LID, are notes rather than warnings; `node add` still warns
|
||||
once that adapters are up with no fabric given.
|
||||
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
|
||||
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
|
||||
written afterwards.
|
||||
|
||||
### A restart with no room beside it
|
||||
|
||||
`ctl restart` places each replacement beside the replica it replaces. With
|
||||
no room for it — one GPU, every card that fits held by the replica itself —
|
||||
the old replica was stopped only after the drain grace period, five minutes
|
||||
of "restart replacing" with nothing saying why, which read as a restart that
|
||||
never happened; scaling to zero and back was the way out. And a replacement
|
||||
that fit nowhere even with the old replica gone was stopped for all the same,
|
||||
leaving the model with nothing. Now:
|
||||
|
||||
- **The replica is the only thing in the way**, found by placing the
|
||||
replacement on a copy of the inventory without it: stopped at once, the
|
||||
replacement taking its place.
|
||||
- **The replacement fits nowhere even then**: the restart is given up and the
|
||||
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
|
||||
- **Room being made** — a lower class preempted, an idle on-demand replica
|
||||
stopping — is waited for, as before.
|
||||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||||
|
||||
### Every GPU node gets a vLLM it can run, chosen on the cluster page
|
||||
|
||||
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
|
||||
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
|
||||
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
|
||||
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
|
||||
across the cluster, so the flags llmgw builds — checked against that version
|
||||
— hold on every node.
|
||||
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation in the new environment. A build without kernels for the card
|
||||
imports and prints its version, and failed only on its first model; now it
|
||||
fails to install, saying so.
|
||||
- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
|
||||
CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
|
||||
to the node at once.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
A cluster with older GPUs could not add them cleanly: `node add` refused a
|
||||
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
|
||||
and `--ignore-checks` passed over every other problem with it.
|
||||
|
||||
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
|
||||
node joins without it and serves what it does run. `node add` checks the
|
||||
vLLM the control plane installs, not the binary's own default.
|
||||
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: a CUDA 12 vLLM on
|
||||
Volta beside a CUDA 13 one on newer cards.
|
||||
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
|
||||
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
|
||||
the threshold had been CUDA 13's for every version, so the doctor's advice
|
||||
to set an older version could not work. A refusal now says which versions
|
||||
would.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
|
||||
### Serving from the mirror says where a model will run
|
||||
|
||||
The serve form put every GGUF model on the CPU: it offered no choice, and its
|
||||
"Profile name" field, pre-filled "cpu", read as though renaming it would move
|
||||
the model to a GPU. Placement is decided by `gpu_class`, which the form never
|
||||
set. Now:
|
||||
|
||||
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
|
||||
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
|
||||
cards is chosen.
|
||||
- **GPUs per replica** sets how many cards a replica takes on one node
|
||||
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
|
||||
how many a llama.cpp replica serves at once.
|
||||
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
|
||||
typed.
|
||||
- A line under the form says what the model will run as — engine, CPU or how
|
||||
many of which GPU, memory, context pool and how it is divided — and turns
|
||||
red when the CPU is chosen with GPUs free. The message after adding repeats
|
||||
it.
|
||||
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
|
||||
anyway** is ticked.
|
||||
|
||||
### llama.cpp on several GPUs of one node
|
||||
|
||||
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
|
||||
placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||||
placed — the replica stayed unplaced or ran on one card — and is now refused
|
||||
when the model is added, saying to write `gpus_per_node` instead; so is
|
||||
`nodes` above one.
|
||||
|
||||
## v0.10.6 — 2026-10-01
|
||||
|
||||
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
|
||||
format check and has no binaries.
|
||||
|
||||
Tried on the PoC cluster before release: LoRA adapters asked for by name
|
||||
through the gateway, two on one model, each answering as trained and the model
|
||||
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
|
||||
requests on the preferred replica, 4 on the other); and a bronze request
|
||||
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
|
||||
live: a model served from an external provider, which needs a provider key.
|
||||
|
||||
### Capacity from the admin pages
|
||||
|
||||
@@ -42,6 +227,57 @@ What v0.10.5 made settable from the command line is now on the pages too:
|
||||
critical model's card no longer refuses a replica that preemption would make
|
||||
room for.
|
||||
|
||||
### LoRA adapters
|
||||
|
||||
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
|
||||
nodes fetch them with the model's weights and llama.cpp loads them all at start,
|
||||
applied to nothing (`--lora-scaled FILE:0`); a request that names an
|
||||
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
|
||||
so one replica serves the model and all its adapters. Adapter names share the
|
||||
namespace of models, are listed in `/v1/models`, and follow key allowlists on
|
||||
their own — a key allowed an adapter does not reach the model under it. Adapter
|
||||
files are kept from weight cleanup like the model's own. Not yet for vLLM or
|
||||
SGLang profiles, whose adapters are loaded differently.
|
||||
|
||||
### Finding and attaching adapters from the mirror page
|
||||
|
||||
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
|
||||
held model's card, *find adapters on Hugging Face* — those made for the model
|
||||
its repository was made from, since a GGUF repository is a quantisation and
|
||||
adapters are made for the original.
|
||||
- **Opening a repository** reads the first bytes of each GGUF file and marks it
|
||||
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
|
||||
the adapter merged in about as often as an adapter, and only the file says.
|
||||
- **Held adapter files** are marked in the list, and a model's card lists its
|
||||
adapters, offers the held adapter files of its architecture to attach under a
|
||||
name and scale, and removes them. A file that is not an adapter, or is for
|
||||
another architecture, is refused before it reaches the catalog.
|
||||
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
|
||||
lists and changes them from the command line, through
|
||||
`POST /api/v1/models/{name}/adapters`, with the same checks.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
|
||||
model, api_key_file}` and no profiles. Nothing is placed for it; the router
|
||||
sends its requests to the provider, rewritten to the provider's name for the
|
||||
model and with its key, read from the file on every publish so a rotated key is
|
||||
picked up. Named in another model's `fallbacks`, it is used only when the
|
||||
cluster has no capacity left. It is never scraped for metrics, and never reads
|
||||
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
|
||||
model — and since its prompts leave the cluster, the allowlist is how to decide
|
||||
who may send them (docs/install.md). `ctl models` and the cluster page mark it
|
||||
external.
|
||||
|
||||
### Pool spillover
|
||||
|
||||
A model served from pools of different priority — H100 before A100, say — was
|
||||
routed to the preferred pool however full it was. Now a pool whose replicas are
|
||||
saturated, by the same measure admission uses (queue past its healthy depth, or
|
||||
KV cache past its ceiling), passes requests to the next pool that is not. When
|
||||
every pool is saturated the preferred one is kept: a pool that is just as full
|
||||
gains nothing.
|
||||
|
||||
### Displacement: a higher band is not refused while a lower one waits
|
||||
|
||||
Each band's queue was bounded, and nothing bounded them together, so a full
|
||||
|
||||
+112
-31
@@ -2370,9 +2370,12 @@ func (x *Provision) GetEngines() []*EngineWant {
|
||||
|
||||
// EngineWant names one engine at one version.
|
||||
type EngineWant struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
Variant string `protobuf:"bytes,3,opt,name=variant,proto3" json:"variant,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -2421,6 +2424,13 @@ func (x *EngineWant) GetVersion() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *EngineWant) GetVariant() string {
|
||||
if x != nil {
|
||||
return x.Variant
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
type ProvisionStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -2873,8 +2883,11 @@ type EngineProfile struct {
|
||||
Env map[string]string `protobuf:"bytes,9,rep,name=env,proto3" json:"env,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
ModelPath string `protobuf:"bytes,10,opt,name=model_path,json=modelPath,proto3" json:"model_path,omitempty"`
|
||||
ServedModelName string `protobuf:"bytes,11,opt,name=served_model_name,json=servedModelName,proto3" json:"served_model_name,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
// adapters are LoRA adapters loaded beside the model, in order: the index
|
||||
// is the id a request applies one by.
|
||||
Adapters []*LoraAdapter `protobuf:"bytes,12,rep,name=adapters,proto3" json:"adapters,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *EngineProfile) Reset() {
|
||||
@@ -2984,6 +2997,13 @@ func (x *EngineProfile) GetServedModelName() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *EngineProfile) GetAdapters() []*LoraAdapter {
|
||||
if x != nil {
|
||||
return x.Adapters
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemoveWeights deletes cached model files a node no longer needs.
|
||||
//
|
||||
// The paths are computed by the control plane from the catalog, but the agent
|
||||
@@ -3777,6 +3797,60 @@ func (x *UpgradeReply) GetError() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// LoraAdapter is one adapter: the name clients ask for it by, and its source,
|
||||
// which the agent fetches as it does the model's weights.
|
||||
type LoraAdapter struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
|
||||
Source string `protobuf:"bytes,2,opt,name=source,proto3" json:"source,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *LoraAdapter) Reset() {
|
||||
*x = LoraAdapter{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[36]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *LoraAdapter) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*LoraAdapter) ProtoMessage() {}
|
||||
|
||||
func (x *LoraAdapter) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[36]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use LoraAdapter.ProtoReflect.Descriptor instead.
|
||||
func (*LoraAdapter) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{36}
|
||||
}
|
||||
|
||||
func (x *LoraAdapter) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *LoraAdapter) GetSource() string {
|
||||
if x != nil {
|
||||
return x.Source
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
@@ -4004,11 +4078,12 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||
"\tProvision\x12.\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\">\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\"X\n" +
|
||||
"\n" +
|
||||
"EngineWant\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\"\x8b\x01\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
"\avariant\x18\x03 \x01(\tR\avariant\"\x8b\x01\n" +
|
||||
"\x0fProvisionStatus\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
@@ -4066,7 +4141,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
|
||||
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" +
|
||||
"\vcpu_threads\x18\x11 \x01(\x05R\n" +
|
||||
"cpuThreads\"\xfd\x02\n" +
|
||||
"cpuThreads\"\xb0\x03\n" +
|
||||
"\rEngineProfile\x12\x12\n" +
|
||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||
"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
|
||||
@@ -4080,7 +4155,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"model_path\x18\n" +
|
||||
" \x01(\tR\tmodelPath\x12*\n" +
|
||||
"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x1a6\n" +
|
||||
"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x121\n" +
|
||||
"\badapters\x18\f \x03(\v2\x15.llmgw.v1.LoraAdapterR\badapters\x1a6\n" +
|
||||
"\bEnvEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"%\n" +
|
||||
@@ -4143,7 +4219,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x14\n" +
|
||||
"\x05error\x18\x03 \x01(\tR\x05error2O\n" +
|
||||
"\x05error\x18\x03 \x01(\tR\x05error\"9\n" +
|
||||
"\vLoraAdapter\x12\x12\n" +
|
||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||
"\x06source\x18\x02 \x01(\tR\x06source2O\n" +
|
||||
"\fAgentService\x12?\n" +
|
||||
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
|
||||
"\fcom.llmgw.v1B\n" +
|
||||
@@ -4162,7 +4241,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 42)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 43)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -4201,12 +4280,13 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*FabricTestReply)(nil), // 34: llmgw.v1.FabricTestReply
|
||||
(*UpgradeAgent)(nil), // 35: llmgw.v1.UpgradeAgent
|
||||
(*UpgradeReply)(nil), // 36: llmgw.v1.UpgradeReply
|
||||
nil, // 37: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 38: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 39: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 40: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 41: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 42: llmgw.v1.EngineProfile.EnvEntry
|
||||
(*LoraAdapter)(nil), // 37: llmgw.v1.LoraAdapter
|
||||
nil, // 38: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 39: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 40: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 41: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 42: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 43: llmgw.v1.EngineProfile.EnvEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
3, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
@@ -4226,24 +4306,24 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
31, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
33, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
35, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
37, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
38, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
7, // 18: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
8, // 19: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
38, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
39, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
6, // 21: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
9, // 22: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
5, // 23: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
4, // 24: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
15, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 26: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
39, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
40, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
12, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
11, // 29: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
40, // 30: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
41, // 30: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
14, // 31: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
5, // 32: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 33: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
41, // 34: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
42, // 34: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
20, // 35: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
10, // 36: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
13, // 37: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
@@ -4254,14 +4334,15 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
0, // 42: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
22, // 43: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
24, // 44: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
42, // 45: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
1, // 46: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
2, // 47: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
47, // [47:48] is the sub-list for method output_type
|
||||
46, // [46:47] is the sub-list for method input_type
|
||||
46, // [46:46] is the sub-list for extension type_name
|
||||
46, // [46:46] is the sub-list for extension extendee
|
||||
0, // [0:46] is the sub-list for field type_name
|
||||
43, // 45: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
37, // 46: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 47: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
2, // 48: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
48, // [48:49] is the sub-list for method output_type
|
||||
47, // [47:48] is the sub-list for method input_type
|
||||
47, // [47:47] is the sub-list for extension type_name
|
||||
47, // [47:47] is the sub-list for extension extendee
|
||||
0, // [0:47] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -4296,7 +4377,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 42,
|
||||
NumMessages: 43,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -465,6 +465,9 @@ message Provision {
|
||||
message EngineWant {
|
||||
string engine = 1; // "llamacpp"
|
||||
string version = 2; // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
string variant = 3;
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
@@ -572,6 +575,9 @@ message EngineProfile {
|
||||
map<string, string> env = 9;
|
||||
string model_path = 10;
|
||||
string served_model_name = 11;
|
||||
// adapters are LoRA adapters loaded beside the model, in order: the index
|
||||
// is the id a request applies one by.
|
||||
repeated LoraAdapter adapters = 12;
|
||||
}
|
||||
|
||||
// RemoveWeights deletes cached model files a node no longer needs.
|
||||
@@ -704,3 +710,10 @@ message UpgradeReply {
|
||||
string version = 2;
|
||||
string error = 3;
|
||||
}
|
||||
|
||||
// LoraAdapter is one adapter: the name clients ask for it by, and its source,
|
||||
// which the agent fetches as it does the model's weights.
|
||||
message LoraAdapter {
|
||||
string name = 1;
|
||||
string source = 2;
|
||||
}
|
||||
+129
-8
@@ -376,8 +376,13 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
|
||||
vLLM is built for CUDA 13. A V100 cannot run it;
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
|
||||
run it — an older driver, a V100 — gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
|
||||
(7.0) or newer. Either way the install ends by computing something on the
|
||||
node's GPU, and a build that cannot fails to install, saying why;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -394,8 +399,51 @@ engines:
|
||||
```
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself instead, set its version to `""` and name its path in
|
||||
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
|
||||
*CUDA 12.9*, or *none* — or from the command line:
|
||||
|
||||
```
|
||||
llmgw ctl engine-build gpu07 vllm cu129
|
||||
```
|
||||
|
||||
The choice is kept by the control plane and the node is told at once; it
|
||||
installs the build in the background, and the column shows how that stands.
|
||||
|
||||
#### Different GPUs, different engines
|
||||
|
||||
A cluster of two GPU generations may need two builds of an engine, or one on
|
||||
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
|
||||
id, or by labels, every one given having to match — and lays engines over the
|
||||
cluster's for them, in order:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
|
||||
A node told nothing it can run of an engine joins without it: `llmgw doctor`
|
||||
and `node add` warn, and the node serves what it does run. To keep an engine
|
||||
off a node for good, whatever the control plane says, add the node with
|
||||
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
|
||||
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
|
||||
than `--ignore-checks`, which passes over every other problem as well.
|
||||
|
||||
A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
|
||||
asks it which devices it sees, and a replica placed on GPUs is not ready until
|
||||
its process holds memory on each of them. A build missing a CUDA library fails
|
||||
there, naming the library, rather than serving on the CPU.
|
||||
|
||||
After changing a version, check it still takes the flags llmgw builds for it,
|
||||
which drift between releases:
|
||||
@@ -671,6 +719,74 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
|
||||
which names every candidate and why each was rejected, rather than leaving you
|
||||
to infer it.
|
||||
|
||||
### LoRA adapters
|
||||
|
||||
A llama.cpp model can carry LoRA adapters, each asked for by its own name as
|
||||
though it were a model:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-coder-7b-instruct
|
||||
source: http://mirror:8081/qwen2.5-coder-7b-instruct/5d567d26/model.gguf
|
||||
adapters:
|
||||
- name: coder-sql
|
||||
source: http://mirror:8081/coder-sql/v1/adapter.gguf
|
||||
scale: 1.0 # optional; 1 when left out
|
||||
profiles: [...]
|
||||
```
|
||||
|
||||
Every replica of the model fetches every adapter and loads it at start, applied
|
||||
to nothing. A request for `coder-sql` goes to the model's replicas with the
|
||||
adapter applied to it alone; a request for the model gets the model. Adapter
|
||||
names share the namespace of models and aliases, appear in `/v1/models`, and
|
||||
follow key allowlists on their own: a key allowed `coder-sql` reaches that
|
||||
adapter and not the model under it, while a key allowed the model reaches its
|
||||
adapters too. Adding or removing an adapter is a change to the model; running
|
||||
replicas load the new set when restarted (`llmgw ctl restart --model M`).
|
||||
Besides the mirror page's model card, adapters are attached from the command
|
||||
line, which reads the file first and refuses one that is not an adapter for
|
||||
this model's architecture:
|
||||
|
||||
llmgw ctl model adapters coder --add coder-sql=http://mirror:8081/sql-lora/v1/sql.gguf@0.8
|
||||
llmgw ctl model adapters coder --remove coder-sql
|
||||
|
||||
The adapter file must be a GGUF LoRA made for this model — llama.cpp's
|
||||
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
|
||||
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
another gateway — rather than replicas here. Nothing is placed for it; the
|
||||
router sends its requests on, with the provider's key and the provider's name
|
||||
for the model:
|
||||
|
||||
```yaml
|
||||
- name: cloud-mini
|
||||
source: external
|
||||
external:
|
||||
url: https://api.openai.com # without /v1
|
||||
model: gpt-4o-mini # what the provider calls it
|
||||
api_key_file: /etc/llmgw/secrets/openai.key
|
||||
```
|
||||
|
||||
Named in another model's `fallbacks`, it is used only when the cluster has no
|
||||
capacity left for that model:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-coder-7b-instruct
|
||||
fallbacks: [cloud-mini]
|
||||
...
|
||||
```
|
||||
|
||||
The key is read from its file on the control plane, never stored in the
|
||||
catalog, and a key rotated in the file is used within seconds. A model whose key
|
||||
file cannot be read is not offered. Everything else applies as to any model:
|
||||
prices and budgets, rate limits, usage records, and key allowlists.
|
||||
|
||||
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
@@ -964,7 +1080,12 @@ A replica may come back marked **`ready (stale profile)`**. That means it is
|
||||
running a profile that has since been edited — the reconciler leaves a serving
|
||||
replica alone on purpose, because churning one costs more than the difference.
|
||||
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
|
||||
the new one before stopping the old.
|
||||
the new one before stopping the old. Where the old one is all that stands in
|
||||
the new one's way — a single GPU — it is stopped first, at once, and the model
|
||||
is unserved while the new one loads; where the new profile fits nowhere even
|
||||
then, the restart is refused and the old one keeps serving, which `ctl
|
||||
replicas` shows as `restart refused:` with the reason. `--in-place` asks for
|
||||
stop-then-start whatever room there is.
|
||||
|
||||
### Signed releases
|
||||
|
||||
@@ -1148,9 +1269,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
|
||||
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
|
||||
install a vLLM built for your CUDA yourself.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -15,6 +15,9 @@ rather than repeated.
|
||||
|
||||
| Key | Type | In |
|
||||
|---|---|---|
|
||||
| `name` | `string` | [Adapter](#adapter) |
|
||||
| `source` | `string` | [Adapter](#adapter) |
|
||||
| `scale` | `float64` | [Adapter](#adapter) |
|
||||
| `enabled` | `bool` | [Admission](#admission) |
|
||||
| `bands` | `[]Band` | [Admission](#admission) |
|
||||
| `default_band` | `string` | [Admission](#admission) |
|
||||
@@ -78,6 +81,13 @@ rather than repeated.
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
@@ -95,6 +105,8 @@ rather than repeated.
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -175,6 +187,7 @@ rather than repeated.
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
@@ -196,6 +209,27 @@ rather than repeated.
|
||||
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
|
||||
## Adapter
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name is what clients ask for, like a model name.
|
||||
|
||||
### `source`
|
||||
|
||||
`string`
|
||||
|
||||
Source is the adapter file (GGUF for llama.cpp), fetched by each node
|
||||
as the model's weights are.
|
||||
|
||||
### `scale`
|
||||
|
||||
`float64`
|
||||
|
||||
Scale is how strongly it is applied; zero takes 1.
|
||||
|
||||
## Admission
|
||||
|
||||
### `enabled`
|
||||
@@ -616,6 +650,54 @@ called. Empty means vLLM's names.
|
||||
|
||||
`int`
|
||||
|
||||
## EngineOverride
|
||||
|
||||
### `gpu_class`
|
||||
|
||||
`string`
|
||||
|
||||
GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes picks nodes by id.
|
||||
|
||||
### `labels`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Labels picks nodes carrying all of these.
|
||||
|
||||
### `engines`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
URL is the provider's base address, without /v1: https://api.openai.com.
|
||||
|
||||
### `model`
|
||||
|
||||
`string`
|
||||
|
||||
Model is what the provider calls it; requests are rewritten to it.
|
||||
|
||||
### `api_key_file`
|
||||
|
||||
`string`
|
||||
|
||||
APIKeyFile holds the provider's key, on the control plane. A file
|
||||
rather than the key itself, so the key is in neither the catalog, the
|
||||
store nor the audit log; empty for a provider that asks for none.
|
||||
|
||||
## Landing
|
||||
|
||||
### `contact`
|
||||
@@ -765,6 +847,28 @@ Owner is the part of the tenancy tree whose capacity quota this model
|
||||
counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
held by the cluster, counted against nobody.
|
||||
|
||||
### `external`
|
||||
|
||||
`*External`
|
||||
|
||||
External is a model served by someone else — an OpenAI-compatible API
|
||||
outside the cluster — rather than by replicas placed here. Such a model
|
||||
has no profiles and nothing is placed for it; the router sends its
|
||||
requests to the provider. Most useful named in another model's
|
||||
fallbacks: used only when the cluster has no capacity left. Its
|
||||
prompts leave the cluster, so which keys may name it is worth an
|
||||
allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
|
||||
Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
for by its own name as though it were a model. Every replica loads
|
||||
every adapter, applying one only to a request that names it.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
@@ -1339,6 +1443,16 @@ never trusted to already have what somebody once put there.
|
||||
Unset keeps DefaultEngines. A version replaces the default; an empty
|
||||
one leaves that engine to be installed by hand.
|
||||
|
||||
### `engine_overrides`
|
||||
|
||||
`[]EngineOverride`
|
||||
|
||||
EngineOverrides give some nodes different engines from the rest: a
|
||||
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
engines over the cluster's, in order, later ones last; an empty version
|
||||
removes that engine from the nodes it picks.
|
||||
|
||||
### `backup_dir`
|
||||
|
||||
`string`
|
||||
|
||||
+20
-1
@@ -450,7 +450,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
// vLLM accepted a vLLM shard and failed it minutes later.
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = provision.Installable()
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
@@ -891,6 +891,17 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
stop()
|
||||
spec.Profile.ModelPath = path
|
||||
}
|
||||
// Adapters are fetched like the model, and before the engine starts: it
|
||||
// loads them at start, and one missing is a replica that answers for the
|
||||
// base model under the adapter's name.
|
||||
for i, ad := range spec.Profile.Adapters {
|
||||
path, err := a.weights().Ensure(ctx, ad.Path)
|
||||
if err != nil {
|
||||
report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil)
|
||||
return
|
||||
}
|
||||
spec.Profile.Adapters[i].Path = path
|
||||
}
|
||||
|
||||
cmds, err := eng.Commands(spec)
|
||||
if err != nil {
|
||||
@@ -920,6 +931,14 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
} else {
|
||||
err = a.sup.WaitAlive(ctx, sh)
|
||||
}
|
||||
if err == nil {
|
||||
err = a.onItsGPUs(ctx, eng, spec, sh.Status().PID)
|
||||
if err != nil {
|
||||
// Serving, but not where it was placed: stopped, so the GPU it
|
||||
// holds in the inventory is not taken by a CPU process.
|
||||
_ = a.sup.Stop(ctx, shardID)
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
report(supervisor.Failed, err.Error(), sh.Logs(20))
|
||||
return
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNeitherInstalledNorAdvertised(t *testing.T) {
|
||||
a := &Agent{}
|
||||
a.cfg.Engines.Disabled = []string{"vllm"}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{
|
||||
{Engine: "llamacpp", Version: "b1"}, {Engine: "vllm", Version: "0.30.0"},
|
||||
}})
|
||||
if len(a.provWants) != 1 || a.provWants[0].Engine != "llamacpp" {
|
||||
t.Errorf("wants %+v: vLLM is disabled here and must not be installed", a.provWants)
|
||||
}
|
||||
a.configuredVers = map[string]string{"vllm": "0.25.0"}
|
||||
a.installedVers = map[string]string{"llamacpp": "b1"}
|
||||
if got := a.engineReportLocked(); got["vllm"] != "" || got["llamacpp"] != "b1" {
|
||||
t.Errorf("advertised %v: a disabled engine found on disk is still not offered", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineNamedByHandIsNotInstalledOverIt(t *testing.T) {
|
||||
a := &Agent{reg: engines.NewRegistry(engines.Config{LlamaCpp: engines.LlamaCppConfig{Bin: "/opt/own/llama-server"}})}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "llamacpp", Version: "b1"}}})
|
||||
if len(a.provWants) != 0 {
|
||||
t.Errorf("wants %+v: an installed build would be used over the operator's own", a.provWants)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
|
||||
// nvidia-smi index: what a process actually computes on, as opposed to what it
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
|
||||
// backend did not load runs on the CPU without a word, and this is where that
|
||||
// shows — the process is on no GPU's list.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
if binary == "" {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUProcesses(string(gpus), string(apps)), nil
|
||||
}
|
||||
|
||||
// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
|
||||
// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
|
||||
func parseGPUProcesses(gpus, apps string) map[int][]int {
|
||||
index := map[string]int{}
|
||||
for _, line := range strings.Split(gpus, "\n") {
|
||||
i, uuid, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.Atoi(strings.TrimSpace(i))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
index[strings.TrimSpace(uuid)] = n
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for _, line := range strings.Split(apps, "\n") {
|
||||
uuid, p, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(p))
|
||||
i, known := index[strings.TrimSpace(uuid)]
|
||||
if err != nil || !known {
|
||||
continue
|
||||
}
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
|
||||
gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
|
||||
// Two processes on GPU 1, one on GPU 2, and a GPU this list has never
|
||||
// heard of, which is dropped rather than guessed at.
|
||||
apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
|
||||
got := parseGPUProcesses(gpus, apps)
|
||||
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
|
||||
t.Errorf("no compute apps should be no GPU in use: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"slices"
|
||||
@@ -40,6 +41,9 @@ type Profile struct {
|
||||
ModelPath string
|
||||
// ServedModelName is the name clients use, which need not match the path.
|
||||
ServedModelName string
|
||||
// Adapters are LoRA adapters to load beside the model, in order: an
|
||||
// adapter's index is the id requests apply it by.
|
||||
Adapters []Adapter
|
||||
}
|
||||
|
||||
// ShardSpec is one shard of one replica on this node.
|
||||
@@ -143,6 +147,15 @@ type Engine interface {
|
||||
MetricsPath() string
|
||||
}
|
||||
|
||||
// CPUFallback is an engine that can come up on the CPU when it was placed on
|
||||
// GPUs, and keep serving there without an error. The agent checks such a shard
|
||||
// is on its GPUs before calling it ready.
|
||||
type CPUFallback interface {
|
||||
FallsBackToCPU() bool
|
||||
// Devices lists the GPUs the engine's build can see, for saying why.
|
||||
Devices(ctx context.Context) ([]string, error)
|
||||
}
|
||||
|
||||
// GPUInfo is the subset of GPU facts validation needs.
|
||||
type GPUInfo struct {
|
||||
Index int
|
||||
@@ -200,6 +213,10 @@ func (r *Registry) Names() []string {
|
||||
|
||||
// Config configures the engine adapters.
|
||||
type Config struct {
|
||||
// Disabled are engines this node never installs or advertises, whatever
|
||||
// the control plane asks: vLLM on a node of cards no build of it runs on,
|
||||
// that is meant for llama.cpp alone.
|
||||
Disabled []string `koanf:"disabled"`
|
||||
VLLM VLLMConfig `koanf:"vllm"`
|
||||
SGLang SGLangConfig `koanf:"sglang"`
|
||||
LlamaCpp LlamaCppConfig `koanf:"llamacpp"`
|
||||
@@ -229,6 +246,11 @@ func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
|
||||
func baseEnv(s ShardSpec) map[string]string {
|
||||
env := map[string]string{
|
||||
"CUDA_VISIBLE_DEVICES": joinInts(s.GPUIndices),
|
||||
// The indices above are nvidia-smi's, which counts by PCI bus. CUDA
|
||||
// counts fastest first unless told otherwise, and on a node with two
|
||||
// kinds of GPU the shard would get a different card than placement
|
||||
// gave it — one another shard may hold.
|
||||
"CUDA_DEVICE_ORDER": "PCI_BUS_ID",
|
||||
}
|
||||
|
||||
if len(s.HCAs) > 0 {
|
||||
@@ -373,3 +395,10 @@ func quantizationOf(args []string) string {
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Adapter is a LoRA adapter an engine loads beside its model. Path is a
|
||||
// source until the agent has fetched it, a local file after.
|
||||
type Adapter struct {
|
||||
Name string `json:"name"`
|
||||
Path string `json:"path"`
|
||||
}
|
||||
@@ -1,8 +1,10 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
@@ -65,25 +67,25 @@ func (*LlamaCpp) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// Validate checks the profile against the GPUs it would run on.
|
||||
//
|
||||
// The two refusals here are about not pretending: llama.cpp has no equivalent
|
||||
// of a tensor-parallel group spanning machines, and its multi-GPU support is a
|
||||
// different mechanism (--split-mode, --tensor-split) that this adapter does not
|
||||
// drive. Accepting either would produce a replica that starts and then serves
|
||||
// from one machine while the catalog believes it has several.
|
||||
// llama.cpp spreads a model over a node's GPUs by layer (--split-mode layer),
|
||||
// so gpus_per_node is how many it gets. It has no tensor parallelism and no
|
||||
// group spanning machines, and accepting tp or nodes would produce a replica
|
||||
// that serves from fewer cards, or one machine, while the catalog believes
|
||||
// otherwise.
|
||||
func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if err := validateCommon(p, gpus); err != nil {
|
||||
return err
|
||||
}
|
||||
if p.Nodes > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
p.Name, p.Nodes)
|
||||
}
|
||||
if p.TP > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp does not do tensor parallelism; "+
|
||||
"it splits a model across local GPUs with --tensor-split, which this adapter does not drive",
|
||||
p.Name)
|
||||
if p.TP > 1 || p.PP > 1 {
|
||||
return fmt.Errorf("profile %q: llamacpp splits a model across a node's GPUs by layer, not by tp or pp; "+
|
||||
"write gpus_per_node: %d instead", p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
}
|
||||
return nil
|
||||
// The common checks count a replica's GPUs as tp; for llama.cpp the
|
||||
// count is gpus_per_node.
|
||||
q := p
|
||||
q.TP = max(p.GPUsPerNode, 1)
|
||||
return validateCommon(q, gpus)
|
||||
}
|
||||
|
||||
// Commands returns the processes for one shard, which is always one process.
|
||||
@@ -132,6 +134,13 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
|
||||
args = append(args, "--n-gpu-layers", "999")
|
||||
}
|
||||
// Several GPUs: the layers spread across all of them, which is the only
|
||||
// split llama.cpp does without a matched interconnect. In proportion to
|
||||
// each card's free memory, which for the whole cards placement gives is
|
||||
// an even split.
|
||||
if len(s.GPUIndices) > 1 && !hasFlag(p.Args, "--split-mode", "-sm") {
|
||||
args = append(args, "--split-mode", "layer")
|
||||
}
|
||||
if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
|
||||
// What placement reserved first: it is what the node's other
|
||||
// replicas were placed around.
|
||||
@@ -144,6 +153,15 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
}
|
||||
args = append(args, "--threads", strconv.Itoa(n))
|
||||
}
|
||||
// Loaded at start, applied per request: a request that names an adapter
|
||||
// carries "lora": [{"id": i, "scale": s}], and one that names none is the
|
||||
// base model. Each is loaded at scale zero, not with --lora and
|
||||
// --lora-init-without-apply: that leaves an adapter's scale at 1, and
|
||||
// llama-server applies it at that scale to every request with no "lora"
|
||||
// of its own — every request for the model itself.
|
||||
for _, a := range p.Adapters {
|
||||
args = append(args, "--lora-scaled", a.Path+":0")
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
|
||||
env := baseEnv(s)
|
||||
@@ -225,3 +243,38 @@ func (l *LlamaCpp) bin() string {
|
||||
}
|
||||
return "llama-server"
|
||||
}
|
||||
|
||||
// FallsBackToCPU says that llama.cpp keeps running when its GPU backend does
|
||||
// not load: --n-gpu-layers asks for layers on a GPU it then has none of, and
|
||||
// the server comes up and answers on the CPU, at a tenth of the speed, with no
|
||||
// error anywhere. Its readiness on a GPU placement needs checking separately.
|
||||
func (*LlamaCpp) FallsBackToCPU() bool { return true }
|
||||
|
||||
// Devices lists the GPUs this build can compute on, as llama-server's
|
||||
// --list-devices names them ("CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)").
|
||||
// Empty, on a machine with GPUs, is a build whose GPU backend did not load:
|
||||
// the CPU build, or a CUDA build missing a library its backend links.
|
||||
func (l *LlamaCpp) Devices(ctx context.Context) ([]string, error) {
|
||||
out, err := exec.CommandContext(ctx, l.bin(), "--list-devices").CombinedOutput() //nolint:gosec // the engine's own binary
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s --list-devices: %v: %s", l.bin(), err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return parseDevices(string(out)), nil
|
||||
}
|
||||
|
||||
// parseDevices reads --list-devices: the lines after "Available devices:",
|
||||
// which says "(none)" when there are none. Log lines before it are skipped.
|
||||
func parseDevices(out string) []string {
|
||||
var devs []string
|
||||
listing := false
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
switch {
|
||||
case strings.HasPrefix(line, "Available devices"):
|
||||
listing = true
|
||||
case listing && line != "" && line != "(none)":
|
||||
devs = append(devs, line)
|
||||
}
|
||||
}
|
||||
return devs
|
||||
}
|
||||
@@ -306,3 +306,58 @@ func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
|
||||
t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
|
||||
s := llamaSpec(nil)
|
||||
s.Profile.Adapters = []Adapter{{Name: "sql", Path: "/w/sql.gguf"}, {Name: "docs", Path: "/w/docs.gguf"}}
|
||||
cmds, err := (&LlamaCpp{}).Commands(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
if !strings.Contains(args, "--lora-scaled /w/sql.gguf:0 --lora-scaled /w/docs.gguf:0") {
|
||||
t.Errorf("args %s: both adapters, in order, loaded at scale zero", args)
|
||||
}
|
||||
plain, _ := (&LlamaCpp{}).Commands(llamaSpec(nil))
|
||||
if strings.Contains(strings.Join(plain[0].Args, " "), "--lora") {
|
||||
t.Error("a model with no adapters was given lora flags")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevicesAreReadFromListDevices(t *testing.T) {
|
||||
cuda := "ggml_cuda_init: found 2 CUDA devices:\nAvailable devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n CUDA1: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
|
||||
if got := parseDevices(cuda); len(got) != 2 || !strings.HasPrefix(got[0], "CUDA0: NVIDIA L4") {
|
||||
t.Errorf("CUDA build: %q", got)
|
||||
}
|
||||
// A CUDA build whose backend did not load says what the CPU build says.
|
||||
if got := parseDevices("Available devices:\n (none)\n"); len(got) != 0 {
|
||||
t.Errorf("no devices: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLlamaCppSpreadsAReplicaOverItsNodesGPUsByLayer(t *testing.T) {
|
||||
four := Profile{Name: "t4x4", Engine: "llamacpp", GPUClass: "t4", TP: 1, GPUsPerNode: 4, ModelPath: "/m"}
|
||||
if err := (&LlamaCpp{}).Validate(four, h100(4)); err != nil {
|
||||
t.Fatalf("four GPUs on a four-GPU node: %v", err)
|
||||
}
|
||||
if err := (&LlamaCpp{}).Validate(four, h100(2)); err == nil {
|
||||
t.Error("four GPUs were accepted on a node with two")
|
||||
}
|
||||
|
||||
s := llamaSpec([]int{0, 1, 2, 3})
|
||||
s.Profile.GPUsPerNode = 4
|
||||
cmds, err := (&LlamaCpp{}).Commands(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, _ := flagValue(cmds[0].Args, "--split-mode"); v != "layer" {
|
||||
t.Errorf("--split-mode %q over four GPUs", v)
|
||||
}
|
||||
if !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0,1,2,3") {
|
||||
t.Errorf("env %v: the replica must see all four of its GPUs", cmds[0].Env)
|
||||
}
|
||||
one, _ := (&LlamaCpp{}).Commands(llamaSpec([]int{2}))
|
||||
if hasFlag(one[0].Args, "--split-mode") {
|
||||
t.Error("a one-GPU replica was given a split mode")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
// gpuProcesses and processGroup are variables so tests can stand in for
|
||||
// nvidia-smi and /proc.
|
||||
var (
|
||||
gpuProcesses = discovery.GPUProcesses
|
||||
processGroup = pgidOf
|
||||
// gpuCheckWait is how long a shard has, once it answers, to show up on
|
||||
// its GPUs: the weights are on them by then, but nvidia-smi can lag.
|
||||
gpuCheckWait = 15 * time.Second
|
||||
)
|
||||
|
||||
// onItsGPUs checks that a shard placed on GPUs computes on them.
|
||||
//
|
||||
// For an engine that falls back to the CPU when its GPU backend does not load,
|
||||
// "it answers" is not "it is ready": llama.cpp built for CUDA but missing
|
||||
// cuBLAS comes up on the CPU, answers at a tenth of the speed, and the GPU it
|
||||
// was given sits idle and reserved. What it computes on is what nvidia-smi
|
||||
// lists it on, so a shard none of whose processes hold memory on one of its
|
||||
// GPUs is refused, with what the engine itself can see.
|
||||
func (a *Agent) onItsGPUs(ctx context.Context, eng engines.Engine, spec engines.ShardSpec, pid int) error {
|
||||
fb, ok := eng.(engines.CPUFallback)
|
||||
if !ok || !fb.FallsBackToCPU() || len(spec.GPUIndices) == 0 || pid <= 0 {
|
||||
return nil
|
||||
}
|
||||
group := processGroup(pid)
|
||||
deadline := time.Now().Add(gpuCheckWait)
|
||||
var missing []int
|
||||
for {
|
||||
procs, err := gpuProcesses(ctx, a.cfg.Discovery.NvidiaSMI)
|
||||
if err != nil || procs == nil {
|
||||
// No nvidia-smi to ask is no evidence either way, and a node that
|
||||
// was given GPUs has one; say so and let the shard serve.
|
||||
a.log.Warn("could not check that a GPU shard runs on its GPUs", "shard", spec.ShardID, "err", err)
|
||||
return nil
|
||||
}
|
||||
missing = missing[:0]
|
||||
for _, i := range spec.GPUIndices {
|
||||
if !holds(procs[i], pid, group) {
|
||||
missing = append(missing, i)
|
||||
}
|
||||
}
|
||||
if len(missing) == 0 {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
seen := "it lists no GPU at all"
|
||||
if devs, err := fb.Devices(ctx); err != nil {
|
||||
seen = "listing its devices failed: " + err.Error()
|
||||
} else if len(devs) > 0 {
|
||||
seen = "it lists " + strings.Join(devs, "; ")
|
||||
}
|
||||
return fmt.Errorf("placed on GPU %s but computing on the CPU: the engine holds no memory there. "+
|
||||
"Its GPU backend did not load (%s). Check the build with `ldd` on its libggml-cuda.so, "+
|
||||
"or run `llmgw doctor` on this node", joinIndices(missing), seen)
|
||||
}
|
||||
|
||||
// holds reports whether any of pids is the shard's process or in its group:
|
||||
// an engine that computes in a child process is still the shard.
|
||||
func holds(pids []int, pid, group int) bool {
|
||||
for _, p := range pids {
|
||||
if p == pid || (group > 0 && processGroup(p) == group) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// pgidOf reads a process's group from /proc; zero when it cannot.
|
||||
func pgidOf(pid int) int {
|
||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
// The command name, in parentheses, may hold spaces; the fields after its
|
||||
// closing parenthesis are state, ppid, pgrp.
|
||||
s := string(b)
|
||||
i := strings.LastIndexByte(s, ')')
|
||||
if i < 0 {
|
||||
return 0
|
||||
}
|
||||
f := strings.Fields(s[i+1:])
|
||||
if len(f) < 3 {
|
||||
return 0
|
||||
}
|
||||
n, _ := strconv.Atoi(f[2])
|
||||
return n
|
||||
}
|
||||
|
||||
func joinIndices(is []int) string {
|
||||
s := make([]string, len(is))
|
||||
for k, i := range is {
|
||||
s[k] = strconv.Itoa(i)
|
||||
}
|
||||
return strings.Join(s, ", ")
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
// fallbackEngine is an engine that comes up on the CPU when its GPU backend
|
||||
// does not load, and lists the devices it can see.
|
||||
type fallbackEngine struct {
|
||||
engines.Engine
|
||||
devices []string
|
||||
}
|
||||
|
||||
func (fallbackEngine) FallsBackToCPU() bool { return true }
|
||||
func (f fallbackEngine) Devices(context.Context) ([]string, error) {
|
||||
return f.devices, nil
|
||||
}
|
||||
|
||||
func TestAGPUShardComputingOnTheCPUIsNotReady(t *testing.T) {
|
||||
procs := map[int][]int{}
|
||||
oldProcs, oldGroup, oldWait := gpuProcesses, processGroup, gpuCheckWait
|
||||
gpuProcesses = func(context.Context, string) (map[int][]int, error) { return procs, nil }
|
||||
processGroup = func(pid int) int {
|
||||
if pid >= 500 && pid < 600 {
|
||||
return 500 // the shard's group: 500 and its children
|
||||
}
|
||||
return pid
|
||||
}
|
||||
gpuCheckWait = 0
|
||||
t.Cleanup(func() { gpuProcesses, processGroup, gpuCheckWait = oldProcs, oldGroup, oldWait })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
spec := engines.ShardSpec{ShardID: "s", GPUIndices: []int{0, 1}}
|
||||
eng := fallbackEngine{}
|
||||
|
||||
// On no GPU at all: refused, saying what the engine sees.
|
||||
err := a.onItsGPUs(context.Background(), eng, spec, 500)
|
||||
if err == nil || !strings.Contains(err.Error(), "computing on the CPU") || !strings.Contains(err.Error(), "lists no GPU") {
|
||||
t.Fatalf("a shard on no GPU was let through: %v", err)
|
||||
}
|
||||
// On one of its two: refused, naming the one it is missing.
|
||||
procs[0] = []int{500}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil || !strings.Contains(err.Error(), "GPU 1 ") {
|
||||
t.Fatalf("a shard on half its GPUs: %v", err)
|
||||
}
|
||||
// Another process on GPU 1 is not the shard.
|
||||
procs[1] = []int{900}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err == nil {
|
||||
t.Fatal("another process's memory was taken for the shard's")
|
||||
}
|
||||
// A child of the shard holding GPU 1 is the shard.
|
||||
procs[1] = []int{900, 512}
|
||||
if err := a.onItsGPUs(context.Background(), eng, spec, 500); err != nil {
|
||||
t.Fatalf("a shard on both its GPUs: %v", err)
|
||||
}
|
||||
|
||||
// An engine that fails rather than falling back is not checked, nor is a
|
||||
// shard with no GPUs.
|
||||
procs = map[int][]int{}
|
||||
if err := a.onItsGPUs(context.Background(), struct{ engines.Engine }{}, spec, 500); err != nil {
|
||||
t.Errorf("checked an engine that does not fall back: %v", err)
|
||||
}
|
||||
if err := a.onItsGPUs(context.Background(), eng, engines.ShardSpec{}, 500); err != nil {
|
||||
t.Errorf("checked a CPU shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCheckWaitsForTheShardToShowUpOnItsGPU(t *testing.T) {
|
||||
calls := 0
|
||||
oldProcs, oldWait := gpuProcesses, gpuCheckWait
|
||||
gpuProcesses = func(context.Context, string) (map[int][]int, error) {
|
||||
calls++
|
||||
if calls < 2 {
|
||||
return map[int][]int{}, nil
|
||||
}
|
||||
return map[int][]int{0: {77}}, nil
|
||||
}
|
||||
gpuCheckWait = 5 * time.Second
|
||||
t.Cleanup(func() { gpuProcesses, gpuCheckWait = oldProcs, oldWait })
|
||||
|
||||
a := &Agent{log: slog.Default()}
|
||||
if err := a.onItsGPUs(context.Background(), fallbackEngine{}, engines.ShardSpec{GPUIndices: []int{0}}, 77); err != nil {
|
||||
t.Fatalf("nvidia-smi lagging by a second failed the shard: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -46,9 +46,30 @@ func (a *Agent) engineReportLocked() map[string]string {
|
||||
out = map[string]string{}
|
||||
}
|
||||
maps.Copy(out, a.installedVers)
|
||||
maps.DeleteFunc(out, func(e, _ string) bool { return a.disabled(e) })
|
||||
return out
|
||||
}
|
||||
|
||||
// byHand reports whether agent.yaml names this engine's binary, which opts
|
||||
// it out of being installed.
|
||||
func (a *Agent) byHand(engine string) bool {
|
||||
if a.reg == nil {
|
||||
return false
|
||||
}
|
||||
e, err := a.reg.Get(engine)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
c, ok := e.(engines.Configured)
|
||||
return ok && c.Configured()
|
||||
}
|
||||
|
||||
// disabled reports whether agent.yaml says this node never runs an engine:
|
||||
// not installed when asked, and not advertised when found.
|
||||
func (a *Agent) disabled(engine string) bool {
|
||||
return slices.Contains(a.cfg.Engines.Disabled, engine)
|
||||
}
|
||||
|
||||
// provisionEvery is how often installed engines are checked again. A library
|
||||
// removed by an unrelated upgrade is found here rather than by the next
|
||||
// replica to start.
|
||||
@@ -58,7 +79,16 @@ const provisionEvery = 10 * time.Minute
|
||||
func (a *Agent) setProvision(p *llmgwv1.Provision) {
|
||||
wants := []provision.Want{}
|
||||
for _, e := range p.GetEngines() {
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
|
||||
if a.disabled(e.GetEngine()) {
|
||||
continue // a control plane older than the node's say in it
|
||||
}
|
||||
if a.byHand(e.GetEngine()) {
|
||||
// The operator named a binary for it: a local build, a CUDA
|
||||
// runtime of their own. Installing one beside it would win —
|
||||
// the installed build is used first — and quietly replace theirs.
|
||||
continue
|
||||
}
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion(), Variant: e.GetVariant()})
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.provWants = wants
|
||||
|
||||
@@ -27,6 +27,9 @@ import (
|
||||
type Want struct {
|
||||
Engine string
|
||||
Version string
|
||||
// Variant is a build chosen for this node ("cu129"); empty leaves it to
|
||||
// the node's hardware.
|
||||
Variant string
|
||||
}
|
||||
|
||||
// The states an engine's installation can be in.
|
||||
@@ -135,7 +138,12 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
"; install it by hand and name it in the agent's configuration"
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, in.Host)
|
||||
if w.Variant != "" && !validVersion.MatchString(w.Variant) {
|
||||
st.State = Unsupported
|
||||
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, w.Variant, in.Host)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Unsupported, err.Error()
|
||||
var nh NotHere
|
||||
@@ -299,9 +307,55 @@ func (in *Installer) verify(ctx context.Context, dir string, b Build) (bin, vers
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("%s --version failed: %v: %s", b.Binary, err, lastLine(out))
|
||||
}
|
||||
if b.GPUBackend != "" {
|
||||
if err := in.verifyGPUBackend(tctx, dir, bin, b); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
}
|
||||
return bin, firstVersionLine(out), nil
|
||||
}
|
||||
|
||||
// verifyGPUBackend checks a GPU build can compute on this node's GPUs.
|
||||
//
|
||||
// The binary's own ldd says nothing about it: the GPU backend is a library
|
||||
// the server opens at run time, and when it cannot — cuBLAS missing, a driver
|
||||
// too old — the server carries on with its CPU backend. So the build is asked
|
||||
// which devices it sees, and when it sees none, ldd on the backend says why.
|
||||
func (in *Installer) verifyGPUBackend(ctx context.Context, dir, bin string, b Build) error {
|
||||
out, err := in.run(ctx, nil, bin, "--list-devices")
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s --list-devices failed: %v: %s", b.Binary, err, lastLine(out))
|
||||
}
|
||||
if listsDevice(string(out)) {
|
||||
return nil
|
||||
}
|
||||
why := "its GPU backend did not load"
|
||||
if lib, err := findFile(dir, b.GPUBackend); err != nil {
|
||||
why = "the build has no " + b.GPUBackend
|
||||
} else if out, err := in.run(ctx, nil, "ldd", lib); err == nil {
|
||||
if missing := lddMissing(string(out)); len(missing) > 0 {
|
||||
why = b.GPUBackend + " cannot load: " + strings.Join(missing, "; ")
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("the %s build of %s sees no GPU, and would run on the CPU: %s", b.Variant, b.Binary, why)
|
||||
}
|
||||
|
||||
// listsDevice reads llama-server --list-devices: a device line after
|
||||
// "Available devices:", which holds "(none)" when there is none.
|
||||
func listsDevice(out string) bool {
|
||||
listing := false
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
switch {
|
||||
case strings.HasPrefix(line, "Available devices"):
|
||||
listing = true
|
||||
case listing && line != "" && line != "(none)":
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// packages installs the operating system packages a build needs.
|
||||
//
|
||||
// Only those missing, and from the node's own package sources: a package the
|
||||
|
||||
@@ -89,7 +89,11 @@ type fakeSystem struct {
|
||||
installed map[string]bool
|
||||
tools map[string]bool
|
||||
lddOut string
|
||||
calls []string
|
||||
// devices is what llama-server --list-devices prints, and backendLdd
|
||||
// what ldd says of its GPU backend.
|
||||
devices string
|
||||
backendLdd string
|
||||
calls []string
|
||||
// locked is how many more package commands find the lock held.
|
||||
locked int
|
||||
}
|
||||
@@ -127,8 +131,12 @@ func (f *fakeSystem) run(_ context.Context, _ []string, name string, args ...str
|
||||
}
|
||||
}
|
||||
return nil, nil
|
||||
case name == "ldd" && strings.HasSuffix(args[0], "libggml-cuda.so"):
|
||||
return []byte(f.backendLdd), nil
|
||||
case name == "ldd":
|
||||
return []byte(f.lddOut), nil
|
||||
case strings.HasSuffix(name, "llama-server") && args[0] == "--list-devices":
|
||||
return []byte(f.devices), nil
|
||||
case strings.HasSuffix(name, "llama-server"):
|
||||
return []byte("version: 10991 (abc)\n"), nil
|
||||
}
|
||||
@@ -247,11 +255,11 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
|
||||
func TestTheBuildFollowsTheHardware(t *testing.T) {
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
b, err := chooseLlamaCpp("b1", gpu)
|
||||
b, err := chooseLlamaCpp("b1", "", gpu)
|
||||
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
|
||||
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
|
||||
}
|
||||
if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
|
||||
}
|
||||
|
||||
@@ -388,3 +396,32 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
|
||||
t.Errorf("link %q %v", target, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
|
||||
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
|
||||
// loader looks, and llama-server serving a GPU placement on the CPU.
|
||||
cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
|
||||
rel := newRelease(map[string][]byte{
|
||||
"llama-b1-" + cuda: tarGz(t, map[string]string{
|
||||
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
|
||||
"cudart-llama-b1-" + cuda: tarGz(t, map[string]string{"libcudart.so.12": "x"}, nil),
|
||||
})
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
sys := &fakeSystem{installed: map[string]bool{"libgomp1": true},
|
||||
devices: "Available devices:\n (none)\n",
|
||||
backendLdd: "\tlibcublas.so.12 => not found\n\tlibcudart.so.12 => /x/libcudart.so.12\n"}
|
||||
in, _ := installer(t, rel.serve(t), sys, gpu)
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "sees no GPU") ||
|
||||
!strings.Contains(got[0].Detail, "libcublas.so.12 is not installed") {
|
||||
t.Fatalf("expected a failure naming the missing library: %+v", got[0])
|
||||
}
|
||||
|
||||
// The same build seeing its GPU is installed.
|
||||
sys.devices = "Available devices:\n CUDA0: NVIDIA L4 (22994 MiB, 22713 MiB free)\n"
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "llamacpp", Version: "b1"}}); got[0].State != Ready {
|
||||
t.Fatalf("a CUDA build that sees its GPU: %+v", got[0])
|
||||
}
|
||||
}
|
||||
@@ -86,28 +86,62 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
|
||||
reqs := b.Requirements
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s's digest", b.Wheel.Asset)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum, ok := sums[b.Wheel.Asset]
|
||||
if !ok {
|
||||
return fmt.Errorf("%s %s publishes no %s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Asset)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + b.Wheel.Asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
// is taken from whichever of the two has the version resolved — both are
|
||||
// the publishers' own — and each file is still pinned by its hash.
|
||||
var index []string
|
||||
if b.ExtraIndex != "" {
|
||||
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
|
||||
}
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
|
||||
lock := filepath.Join(tmp, lockName)
|
||||
args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock, "-"}
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
|
||||
args = append(args, "-")
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
|
||||
strings.Join(b.Requirements, " "))
|
||||
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
strings.Join(reqs, " "))
|
||||
install := append([]string{"pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock}, index...)
|
||||
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "checking that %s runs", w.Engine)
|
||||
if _, _, err := in.verify(ctx, tmp, b); err != nil {
|
||||
return err
|
||||
}
|
||||
if b.GPUCheck && in.Host.NVIDIADriver != "" {
|
||||
in.progress(st, "checking that %s (%s) computes on this node's GPU", w.Engine, b.Variant)
|
||||
if err := in.gpuCheck(ctx, python, w, b); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
m, _ := json.MarshalIndent(markerFile{
|
||||
Engine: w.Engine, Version: w.Version, Variant: b.Variant,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: reqs,
|
||||
ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
|
||||
}, "", " ")
|
||||
if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
|
||||
@@ -203,3 +237,26 @@ func tail(b []byte) string {
|
||||
}
|
||||
return strings.Join(lines, " / ")
|
||||
}
|
||||
|
||||
// gpuProbe is what the environment is asked to run on the GPU: one small
|
||||
// operation, which needs the torch to have kernels for the card.
|
||||
const gpuProbe = `import torch
|
||||
assert torch.cuda.is_available(), "torch sees no CUDA device"
|
||||
x = torch.ones(64, device="cuda")
|
||||
print(float((x * 2).sum()), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))`
|
||||
|
||||
// gpuCheck runs gpuProbe in a freshly installed environment.
|
||||
func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Build) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
|
||||
defer cancel()
|
||||
out, err := in.run(ctx, nil, python, "-c", gpuProbe)
|
||||
if err != nil {
|
||||
hint := ""
|
||||
if strings.Contains(string(out), "no kernel image") {
|
||||
hint = "; this build has no kernels for this GPU, so try the other build for this node"
|
||||
}
|
||||
return fmt.Errorf("%s %s (%s) installed but cannot compute on this node's GPU: %s%s",
|
||||
w.Engine, w.Version, b.Variant, tail(out), hint)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@@ -23,6 +24,9 @@ type fakePython struct {
|
||||
pmu sync.Mutex
|
||||
commands []string
|
||||
failPip bool
|
||||
// noKernel makes the GPU check fail as a torch without kernels for the
|
||||
// card does.
|
||||
noKernel bool
|
||||
}
|
||||
|
||||
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
@@ -43,6 +47,11 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
|
||||
if f.noKernel {
|
||||
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
|
||||
}
|
||||
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
|
||||
}
|
||||
return f.fakeSystem.run(ctx, env, name, args...)
|
||||
}
|
||||
@@ -59,6 +68,8 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
uv := newRelease(map[string][]byte{
|
||||
"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
|
||||
map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
|
||||
// The fake GitHub serves every repository's release from this one.
|
||||
"vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl": []byte("the cu129 wheel"),
|
||||
})
|
||||
gh := uv.serve(t)
|
||||
mux := http.NewServeMux()
|
||||
@@ -130,8 +141,9 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.0 or newer"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -182,3 +194,77 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
t.Errorf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderGPUGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A V100: CUDA 13 dropped it. The same vLLM's CUDA 12.9 build, from the
|
||||
// project's GitHub release, with the torch PyTorch publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a V100 node: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
wheel := "vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
|
||||
for _, want := range []string{
|
||||
// Pinned to the digest GitHub publishes for the wheel.
|
||||
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
|
||||
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
|
||||
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
|
||||
"--require-hashes --no-deps -r",
|
||||
// Computing on the card, not only importing.
|
||||
"/bin/python -c import torch",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
|
||||
t.Errorf("installed without the index its torch comes from: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
|
||||
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
|
||||
}
|
||||
volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
|
||||
if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
|
||||
t.Error("a build that does not exist was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernel: true}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "no kernels for this GPU") {
|
||||
t.Fatalf("a build without kernels for the card: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu13", marker)); err == nil {
|
||||
t.Error("marked installed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
// A machine vLLM is not published for is refused, not given the other
|
||||
// build to fail with.
|
||||
if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no Linux build") {
|
||||
t.Errorf("an unknown architecture: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -43,12 +44,33 @@ type Build struct {
|
||||
// VersionArgs is what Binary is run with to prove the build works, when
|
||||
// --version is not it.
|
||||
VersionArgs []string
|
||||
// GPUBackend, when set, is the shared library the build computes on GPUs
|
||||
// with, loaded at run time rather than linked: "libggml-cuda.so". The
|
||||
// build must list a device with it, or it is not installed — a llama.cpp
|
||||
// whose CUDA backend cannot load runs on the CPU without complaint.
|
||||
GPUBackend string
|
||||
// Wheel, when set, is the engine itself as a wheel from a GitHub
|
||||
// release rather than from PyPI, checked against the digest GitHub
|
||||
// publishes for it. ExtraIndex is a package index besides PyPI that the
|
||||
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
|
||||
Wheel *Wheel
|
||||
ExtraIndex string
|
||||
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
|
||||
// compute something on one before the build counts as installed. A build
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
// imports and prints its version, and fails on its first model.
|
||||
GPUCheck bool
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Asset string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
type Recipe struct {
|
||||
// Repo is the GitHub repository whose releases carry the builds; empty
|
||||
@@ -56,8 +78,10 @@ type Recipe struct {
|
||||
Repo string
|
||||
// PyPI is the project an engine installed from PyPI is published as.
|
||||
PyPI string
|
||||
// Choose picks the build for a machine, or says why none fits.
|
||||
Choose func(version string, h discovery.Host) (Build, error)
|
||||
// Choose picks the build for a machine, or says why none fits. variant
|
||||
// is a build chosen for the node ("cu129"); empty leaves it to the
|
||||
// machine.
|
||||
Choose func(version, variant string, h discovery.Host) (Build, error)
|
||||
}
|
||||
|
||||
// Recipes are the engines this agent can install.
|
||||
@@ -99,7 +123,7 @@ type NotHere struct{ Reason string }
|
||||
|
||||
func (e NotHere) Error() string { return e.Reason }
|
||||
|
||||
func chooseUV(version string, h discovery.Host) (Build, error) {
|
||||
func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -124,25 +148,72 @@ const minDriverForCUDA13 = 580
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// cudaGate says whether a node can run an engine built on CUDA 13.
|
||||
func cudaGate(engine string, h discovery.Host) error {
|
||||
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
|
||||
// version compatibility, and on Volta.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine built for CUDA 13.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
|
||||
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older build (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
return "", fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
}
|
||||
if h.NVIDIADriver == "" {
|
||||
return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, minDriverForCUDA13, h.NVIDIADriver)
|
||||
variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
|
||||
engine, minComputeForCUDA13, cc)
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
return nil
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
hint := ""
|
||||
if cuda == "13" {
|
||||
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
|
||||
}
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, hint)
|
||||
}
|
||||
return variant, nil
|
||||
}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
func versionBefore(a, b string) bool {
|
||||
if b == "" {
|
||||
return false
|
||||
}
|
||||
pa, pb := strings.Split(strings.TrimPrefix(a, "v"), "."), strings.Split(b, ".")
|
||||
for i := 0; i < max(len(pa), len(pb)); i++ {
|
||||
x, y := 0, 0
|
||||
if i < len(pa) {
|
||||
x = leadingInt(pa[i])
|
||||
}
|
||||
if i < len(pb) {
|
||||
y = leadingInt(pb[i])
|
||||
}
|
||||
if x != y {
|
||||
return x < y
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
@@ -150,28 +221,129 @@ func cudaGate(engine string, h discovery.Host) error {
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
func chooseVLLM(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("vLLM", h); err != nil {
|
||||
return Build{}, err
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for the drivers and GPUs CUDA 13 left behind.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// chooseVLLM picks the vLLM build for a machine.
|
||||
//
|
||||
// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
|
||||
// same version's CUDA 12.9 build where only that does: one version across
|
||||
// the cluster, which matters because the flags llmgw builds are checked
|
||||
// against one version, and an older vLLM on the older cards would be a
|
||||
// second set of flags nobody checked. A variant chosen for the node is that
|
||||
// build, if the node can run it.
|
||||
func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
// What no build of it gets past: a machine it is not published for, or
|
||||
// one with no NVIDIA GPU. Only a CUDA requirement falls back to the
|
||||
// CUDA 12.9 build.
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
if h.NVIDIADriver == "" {
|
||||
return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
pypi := func(v string) Build {
|
||||
return Build{
|
||||
Variant: v,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := func() (Build, error) {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeForCUDA12, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
|
||||
b := pypi("cu129")
|
||||
b.Requirements = nil
|
||||
b.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
|
||||
Asset: fmt.Sprintf("vllm-%s+cu129-cp38-abi3-manylinux_2_28_%s.whl", version, arch),
|
||||
}
|
||||
b.ExtraIndex = pytorchCU129
|
||||
return b, nil
|
||||
}
|
||||
modern := !versionBefore(version, cuda13Since["vLLM"])
|
||||
switch variant {
|
||||
case "":
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err == nil {
|
||||
return pypi(v), nil
|
||||
}
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) || !modern {
|
||||
return Build{}, err
|
||||
}
|
||||
b, err129 := cu129()
|
||||
if err129 != nil {
|
||||
return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
|
||||
}
|
||||
return b, nil
|
||||
case "cu13":
|
||||
if !modern {
|
||||
return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi("cu13"), nil
|
||||
case "cu129":
|
||||
if !modern {
|
||||
// Before 0.26 PyPI's own build is CUDA 12, and is the one given.
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi(v), nil
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) {
|
||||
return Build{}, err
|
||||
}
|
||||
}
|
||||
return cu129()
|
||||
}
|
||||
return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
|
||||
}
|
||||
|
||||
func chooseSGLang(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("SGLang", h); err != nil {
|
||||
// cudaFits checks a node against a CUDA's driver and GPU requirements.
|
||||
func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f",
|
||||
engine, version, cuda, minCC, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
|
||||
if chosen != "" {
|
||||
return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
|
||||
}
|
||||
variant, err := cudaGate("SGLang", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
Binary: "python",
|
||||
@@ -198,7 +370,7 @@ func leadingInt(s string) int {
|
||||
// else — no toolkit.
|
||||
const cudaForLlamaCpp = "12.8"
|
||||
|
||||
func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
|
||||
func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them
|
||||
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
|
||||
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
|
||||
@@ -234,6 +406,7 @@ func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
|
||||
if h.NVIDIADriver != "" && arch == "x64" {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
|
||||
b.Variant = "cuda-" + cudaForLlamaCpp
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
b.Assets = []string{
|
||||
fmt.Sprintf("llama-%s-%s", version, name),
|
||||
fmt.Sprintf("cudart-llama-%s-%s", version, name),
|
||||
|
||||
@@ -2,6 +2,8 @@ package weights
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -26,8 +28,9 @@ type manifest struct {
|
||||
Model string `json:"model"`
|
||||
Version string `json:"version"`
|
||||
Files []struct {
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
} `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
@@ -87,6 +90,25 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error)
|
||||
}
|
||||
}
|
||||
|
||||
// Each file against the checksum the mirror took when it fetched it,
|
||||
// before the set is called complete: a shard damaged on the way loads as
|
||||
// a model that answers nonsense, not as an error.
|
||||
for _, file := range m.Files {
|
||||
if file.SHA256 == "" {
|
||||
continue
|
||||
}
|
||||
target := filepath.Join(dir, file.Name)
|
||||
got, err := fileSHA256(target)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !strings.EqualFold(got, file.SHA256) {
|
||||
_ = os.Remove(target)
|
||||
return "", fmt.Errorf("%s%s: sha256 %s, the mirror has %s; removed, to be fetched again",
|
||||
source, file.Name, got, file.SHA256)
|
||||
}
|
||||
}
|
||||
|
||||
if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -108,6 +130,12 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
|
||||
return manifest{}, fmt.Errorf("asking %s what it holds: %w", source, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusConflict {
|
||||
// Still downloading on the mirror, or a shard missing: said in the
|
||||
// mirror's words, which name what is not there yet.
|
||||
why, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10))
|
||||
return manifest{}, fmt.Errorf("%s is not complete on the mirror yet: %s", source, strings.TrimSpace(string(why)))
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return manifest{}, fmt.Errorf("%s answered %d", source, resp.StatusCode)
|
||||
}
|
||||
@@ -127,3 +155,16 @@ func (f *Fetcher) manifestFor(ctx context.Context, source string) (manifest, err
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func fileSHA256(path string) (string, error) {
|
||||
f, err := os.Open(path) //nolint:gosec // a file this fetcher wrote
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(h, f); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
package weights
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -165,3 +167,40 @@ func TestOnlyATrailingSlashMeansASet(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetFileThatDoesNotMatchTheMirrorsChecksumIsRefused(t *testing.T) {
|
||||
good := "the shard as mirrored"
|
||||
sum := sha256.Sum256([]byte(good))
|
||||
served := "the shard damaged on the way"
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasSuffix(r.URL.Path, "/") {
|
||||
fmt.Fprintf(w, `{"model":"m","version":"v","files":[{"name":"s.safetensors","bytes":%d,"sha256":%q}]}`,
|
||||
len(served), hex.EncodeToString(sum[:]))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(served))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err == nil || !strings.Contains(err.Error(), "sha256") {
|
||||
t.Fatalf("a damaged shard was accepted: %v", err)
|
||||
}
|
||||
served = good
|
||||
// The size differs, so it is fetched again; this time it matches.
|
||||
if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil {
|
||||
t.Fatalf("the shard as mirrored: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "m/v: this version is not complete: still downloading model-00002-of-00002.safetensors", http.StatusConflict)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
_, err := f.Ensure(t.Context(), srv.URL+"/m/v/")
|
||||
if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") ||
|
||||
!strings.Contains(err.Error(), "model-00002-of-00002") {
|
||||
t.Fatalf("got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -143,6 +143,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
newEngineCheckCmd(),
|
||||
newEngineBuildCmd(client),
|
||||
newToolCheckCmd(),
|
||||
newWeightsCmd(client, &asJSON),
|
||||
newCapacityCmd(client, &asJSON),
|
||||
@@ -254,6 +255,8 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
switch {
|
||||
case m.Parked:
|
||||
return "parked"
|
||||
case m.External != "":
|
||||
return "external: " + m.External
|
||||
case m.HeldFor != "":
|
||||
return "gave room to " + m.HeldFor
|
||||
case m.Unplaceable:
|
||||
|
||||
@@ -12,8 +12,9 @@ import (
|
||||
|
||||
func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
var (
|
||||
models []string
|
||||
all bool
|
||||
models []string
|
||||
all bool
|
||||
inPlace bool
|
||||
)
|
||||
c := &cobra.Command{
|
||||
Use: "restart [REPLICA...]",
|
||||
@@ -22,10 +23,14 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
"an edited profile, a new engine version.\n\n" +
|
||||
"Each replacement is placed beside the replica it replaces, and the old one\n" +
|
||||
"keeps serving until the new one is ready. Replicas of one model are\n" +
|
||||
"replaced one at a time. Where nothing else has room for a second copy —\n" +
|
||||
"the usual case on a small CPU fleet — the old one is stopped after the\n" +
|
||||
"drain grace period (5 minutes by default) and the new one starts in its\n" +
|
||||
"place; a model with only that replica is unserved while it loads.\n\n" +
|
||||
"replaced one at a time.\n\n" +
|
||||
"Where there is no room for a second copy and the old replica is all that\n" +
|
||||
"is in the way — one GPU, every card that fits held — it is stopped at once\n" +
|
||||
"and the new one starts in its place; a model with only that replica is\n" +
|
||||
"unserved while it loads. Where the new one would not fit even then — a\n" +
|
||||
"profile edited to need more than any node has — the restart is given up\n" +
|
||||
"and the old replica keeps serving: llmgw ctl replicas says so, and why.\n\n" +
|
||||
"--in-place stops each replica first, whatever room there is.\n\n" +
|
||||
" llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
|
||||
" llmgw ctl restart --model qwen2.5-7b\n" +
|
||||
" llmgw ctl restart --all",
|
||||
@@ -33,7 +38,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
if len(args) == 0 && len(models) == 0 && !all {
|
||||
return errors.New("name replicas, --model, or --all")
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all, InPlace: inPlace})
|
||||
var res adminapi.RestartResult
|
||||
if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
|
||||
return err
|
||||
@@ -44,6 +49,7 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
||||
}
|
||||
c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
|
||||
c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
|
||||
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before starting its replacement")
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
+17
-2
@@ -23,6 +23,7 @@ func newDoctorCmd() *cobra.Command {
|
||||
opts doctor.Options
|
||||
asJSON bool
|
||||
verbose bool
|
||||
fabrics []string
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
Use: "doctor",
|
||||
@@ -38,8 +39,21 @@ that file when the report is not enough to go on.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
doctor.SetConfigLoader(config.LoadAgent)
|
||||
if opts.VLLMVersion == "" {
|
||||
for _, spec := range fabrics {
|
||||
name, hcas, _ := strings.Cut(spec, ":")
|
||||
if name == "" || hcas == "" {
|
||||
return fmt.Errorf("--fabric %q: give it as name:mlx5_0,mlx5_1", spec)
|
||||
}
|
||||
if opts.Fabrics == nil {
|
||||
opts.Fabrics = map[string][]string{}
|
||||
}
|
||||
opts.Fabrics[name] = strings.Split(hcas, ",")
|
||||
}
|
||||
switch opts.VLLMVersion {
|
||||
case "":
|
||||
opts.VLLMVersion = config.DefaultEngines["vllm"]
|
||||
case "none":
|
||||
opts.VLLMVersion = ""
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
@@ -66,7 +80,8 @@ that file when the report is not enough to go on.`,
|
||||
f.StringVar(&opts.ConfigPath, "config", "", "agent configuration (default /etc/llmgw/agent.yaml when present)")
|
||||
f.StringVar(&opts.IBRoot, "ib-root", discovery.SysfsRoot, "sysfs InfiniBand directory")
|
||||
f.StringVar(&opts.NvidiaSMI, "nvidia-smi", "", "path to nvidia-smi")
|
||||
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs)")
|
||||
f.StringVar(&opts.VLLMVersion, "vllm", "", "the vLLM version to check for (default: the one this llmgw installs; none to skip)")
|
||||
f.StringArrayVar(&fabrics, "fabric", nil, "a fabric about to be declared, name:mlx5_0,mlx5_1, checked as if it were; repeatable")
|
||||
f.BoolVar(&asJSON, "json", false, "write every fact and finding as JSON, to send back")
|
||||
f.BoolVar(&verbose, "verbose", false, "also print the raw nvidia-smi topology")
|
||||
return cmd
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "engine-build <node> <engine> <build>",
|
||||
Short: "Choose which build of an engine a GPU node installs",
|
||||
Long: "A node installs the build of vLLM its hardware runs: CUDA 13 where the\n" +
|
||||
"driver and GPUs allow, else the same version built for CUDA 12.9. This\n" +
|
||||
"chooses instead, as the cluster page's vLLM build column does:\n\n" +
|
||||
" auto what the hardware runs (the default)\n" +
|
||||
" cu13 the CUDA 13 build, from PyPI\n" +
|
||||
" cu129 the CUDA 12.9 build, from vLLM's GitHub release\n" +
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
body, _ := json.Marshal(adminapi.NodeEngineRequest{Engine: args[1], Build: args[2]})
|
||||
if err := client().post(cmd.Context(), "/api/v1/nodes/"+args[0]+"/engines", body, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s build %s; the node installs it in the background (llmgw ctl nodes shows it)\n",
|
||||
args[0], args[1], args[2])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
@@ -126,6 +127,14 @@ func newEngineCheckCmd() *cobra.Command {
|
||||
}
|
||||
sort.Strings(unknown)
|
||||
|
||||
// Flags accepted is not the same as running where it is placed: a
|
||||
// GPU build whose backend does not load accepts every flag and
|
||||
// computes on the CPU. Asked of the engine here, not of a help.
|
||||
var deviceErr error
|
||||
if fb, ok := eng.(engines.CPUFallback); ok && help == "" {
|
||||
deviceErr = checkDevices(cmd.Context(), name, fb, out)
|
||||
}
|
||||
|
||||
fmt.Fprintf(out, "%s: %d flags checked", name, checked)
|
||||
if skipped > 0 {
|
||||
fmt.Fprintf(out, ", %d commands skipped", skipped)
|
||||
@@ -137,7 +146,7 @@ func newEngineCheckCmd() *cobra.Command {
|
||||
return fmt.Errorf("nothing was checked; install the engine here or pass --help-output")
|
||||
}
|
||||
fmt.Fprintln(out, "every flag llmgw builds is accepted")
|
||||
return nil
|
||||
return deviceErr
|
||||
}
|
||||
fmt.Fprintf(out, "\n%d flags not mentioned by the program they are passed to:\n", len(unknown))
|
||||
for _, f := range unknown {
|
||||
@@ -249,3 +258,22 @@ func isFlagChar(b byte) bool {
|
||||
return b == '-' || b == '_' || (b >= 'a' && b <= 'z') ||
|
||||
(b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9')
|
||||
}
|
||||
|
||||
// checkDevices says which GPUs an engine that can fall back to the CPU sees on
|
||||
// this machine, and fails when the machine has NVIDIA GPUs and it sees none.
|
||||
func checkDevices(ctx context.Context, name string, fb engines.CPUFallback, out io.Writer) error {
|
||||
if _, err := exec.LookPath("nvidia-smi"); err != nil {
|
||||
fmt.Fprintln(out, "no NVIDIA GPU here: which devices it computes on was not checked")
|
||||
return nil
|
||||
}
|
||||
devs, err := fb.Devices(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(devs) == 0 {
|
||||
return fmt.Errorf("this machine has NVIDIA GPUs and %s sees none: it would serve a GPU placement on the CPU. "+
|
||||
"Its GPU backend did not load; ldd on its libggml-cuda.so names the missing library", name)
|
||||
}
|
||||
fmt.Fprintf(out, "computes on: %s\n", strings.Join(devs, "; "))
|
||||
return nil
|
||||
}
|
||||
+54
-29
@@ -2,6 +2,7 @@ package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
@@ -56,52 +57,69 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
var model, version, as string
|
||||
|
||||
c := &cobra.Command{
|
||||
Use: "add <url>",
|
||||
Use: "add <url>...",
|
||||
Short: "Download a model into the mirror",
|
||||
Long: "Downloads to a temporary name and renames when complete, so an\n" +
|
||||
"interrupted download never leaves something a catalog could be pointed\n" +
|
||||
"at. Reports the checksum: a mirror is the one place in this system where\n" +
|
||||
"a corrupted file is copied to every node rather than caught once.\n\n" +
|
||||
"Several URLs go into one version, as the files of one model: the shards\n" +
|
||||
"of a safetensors set with its config.json and tokenizer, for vLLM. The\n" +
|
||||
"version is offered to nodes only once it is whole — nothing still\n" +
|
||||
"downloading, every shard its index names present — and its source is\n" +
|
||||
"the version, ending in a slash.\n\n" +
|
||||
"Refuses to replace a version that is already here. Add a new one beside\n" +
|
||||
"it instead — that is what makes the old one still available when the new\n" +
|
||||
"one turns out to be worse.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
source := args[0]
|
||||
file := as
|
||||
if file == "" {
|
||||
file = mirror.LastPathElement(source)
|
||||
}
|
||||
if model == "" || version == "" {
|
||||
return fmt.Errorf("--model and --version are required: they are the layout")
|
||||
}
|
||||
|
||||
e := mirror.Entry{Model: model, Version: version, File: file}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "fetching %s\n", source)
|
||||
|
||||
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
|
||||
if total > 0 {
|
||||
fmt.Fprintf(out, " %s of %s (%d%%)\n",
|
||||
fileSize(done), fileSize(total), done*100/total)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, " %s so far\n", fileSize(done))
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
if as != "" && len(args) > 1 {
|
||||
return fmt.Errorf("--as names one file; with %d URLs each keeps its own name", len(args))
|
||||
}
|
||||
fmt.Fprintf(out, "\n%s/%s/%s %s\n sha256 %s\n",
|
||||
added.Entry.Model, added.Entry.Version, added.Entry.File,
|
||||
fileSize(added.Entry.Bytes), added.SHA256)
|
||||
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n",
|
||||
added.Entry.URL("http://<this-host>:8081"))
|
||||
out := cmd.OutOrStdout()
|
||||
var last mirror.Added
|
||||
for _, source := range args {
|
||||
file := as
|
||||
if file == "" {
|
||||
file = mirror.LastPathElement(source)
|
||||
}
|
||||
e := mirror.Entry{Model: model, Version: version, File: file}
|
||||
fmt.Fprintf(out, "fetching %s\n", source)
|
||||
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
|
||||
if total > 0 {
|
||||
fmt.Fprintf(out, " %s of %s (%d%%)\n",
|
||||
fileSize(done), fileSize(total), done*100/total)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, " %s so far\n", fileSize(done))
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s/%s %s\n sha256 %s\n",
|
||||
added.Entry.Model, added.Entry.Version, added.Entry.File,
|
||||
fileSize(added.Entry.Bytes), added.SHA256)
|
||||
last = added
|
||||
}
|
||||
src := last.Entry.URL("http://<this-host>:8081")
|
||||
if len(args) > 1 {
|
||||
src = strings.TrimSuffix(mirror.Entry{Model: model, Version: version}.URL("http://<this-host>:8081"), "/") + "/"
|
||||
if m, err := mirror.ManifestFor(*dir, model, version); err != nil {
|
||||
fmt.Fprintf(out, "\nNot yet offered to nodes: %v\n", err)
|
||||
} else {
|
||||
fmt.Fprintf(out, "\n%d files, %s, whole.\n", len(m.Files), fileSize(m.Bytes))
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, "\nPoint a model at it with:\n source: %s\n", src)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
|
||||
c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
|
||||
c.Flags().StringVar(&as, "as", "", "the filename to save as; taken from the URL when omitted")
|
||||
c.Flags().StringVar(&as, "as", "", "the filename to save as, for one URL; taken from the URL when omitted")
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -405,7 +423,14 @@ func withManifest(root string, next http.Handler) http.Handler {
|
||||
return
|
||||
}
|
||||
m, err := mirror.ManifestFor(root, model, version)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, mirror.ErrIncomplete):
|
||||
// Said, not hidden behind a 404: a node asking now is told why
|
||||
// it cannot have the version yet, and tries again later.
|
||||
w.Header().Set("Retry-After", "60")
|
||||
http.Error(w, err.Error(), http.StatusConflict)
|
||||
return
|
||||
case err != nil:
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
+70
-1
@@ -5,6 +5,9 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/knadh/koanf/parsers/yaml"
|
||||
@@ -31,7 +34,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newTargetCmd(client, asJSON),
|
||||
newTargetCmd(client, asJSON), newAdaptersCmd(client),
|
||||
newModelServeCmd(client))
|
||||
return c
|
||||
}
|
||||
@@ -264,3 +267,69 @@ func newClassCmd(client func() *ctlClient) *cobra.Command {
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newAdaptersCmd(client func() *ctlClient) *cobra.Command {
|
||||
var add, remove []string
|
||||
c := &cobra.Command{
|
||||
Use: "adapters <model> [--add NAME=URL[@SCALE]] [--remove NAME]",
|
||||
Short: "List, add or remove a model's LoRA adapters",
|
||||
Long: "An adapter is asked for by its NAME as though it were a model, and is served\n" +
|
||||
"by this model's replicas with it applied. URL is the adapter file, usually in\n" +
|
||||
"the mirror; it must be a GGUF LoRA adapter for this model's architecture,\n" +
|
||||
"which is checked before it is attached. Replicas load a changed set when\n" +
|
||||
"restarted: llmgw ctl restart --model MODEL.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
name := args[0]
|
||||
var models []adminapi.ModelView
|
||||
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
|
||||
return err
|
||||
}
|
||||
var current []catalog.Adapter
|
||||
found := false
|
||||
for _, m := range models {
|
||||
if m.Name == name {
|
||||
current, found = m.Adapters, true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("no model %q", name)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
if len(add) == 0 && len(remove) == 0 {
|
||||
if len(current) == 0 {
|
||||
fmt.Fprintf(out, "%s has no adapters\n", name)
|
||||
}
|
||||
for _, a := range current {
|
||||
fmt.Fprintf(out, "%s\tscale %g\t%s\n", a.Name, a.EffectiveScale(), a.Source)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
next := slices.DeleteFunc(slices.Clone(current), func(a catalog.Adapter) bool { return slices.Contains(remove, a.Name) })
|
||||
for _, spec := range add {
|
||||
n, rest, ok := strings.Cut(spec, "=")
|
||||
if !ok || n == "" || rest == "" {
|
||||
return fmt.Errorf("--add %q is not NAME=URL[@SCALE]", spec)
|
||||
}
|
||||
ad := catalog.Adapter{Name: n, Source: rest}
|
||||
if i := strings.LastIndex(rest, "@"); i > 0 {
|
||||
if s, err := strconv.ParseFloat(rest[i+1:], 64); err == nil {
|
||||
ad.Source, ad.Scale = rest[:i], s
|
||||
}
|
||||
}
|
||||
next = append(next, ad)
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.AdaptersRequest{Adapters: next})
|
||||
var res adminapi.AdaptersRequest
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+name+"/adapters", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%s serves %d adapter(s); restart its replicas to load them: llmgw ctl restart --model %s\n",
|
||||
name, len(res.Adapters), name)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringArrayVar(&add, "add", nil, "NAME=URL[@SCALE]; repeatable")
|
||||
c.Flags().StringArrayVar(&remove, "remove", nil, "an adapter's name; repeatable")
|
||||
return c
|
||||
}
|
||||
+20
-1
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
@@ -113,13 +114,14 @@ engines are kept.`,
|
||||
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
|
||||
f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
|
||||
f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
|
||||
f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
|
||||
f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
|
||||
f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
|
||||
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
|
||||
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
|
||||
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
|
||||
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
|
||||
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
|
||||
f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
|
||||
@@ -148,6 +150,23 @@ func (na *nodeAdd) prepare() error {
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, e := range o.DisableEngines {
|
||||
if !slices.Contains(provision.Installable(), e) {
|
||||
return fmt.Errorf("--disable-engine %s: the engines are %s", e, strings.Join(provision.Installable(), ", "))
|
||||
}
|
||||
}
|
||||
// What the control plane will ask the node to install, so doctor checks
|
||||
// that vLLM rather than its own default. Its overrides for a GPU class
|
||||
// are not known until the node says what it has, and are not applied
|
||||
// here; doctor's verdict on vLLM is a warning either way.
|
||||
if o.VLLMVersion == "" {
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil {
|
||||
o.VLLMVersion = cfg.WantedEngines()["vllm"]
|
||||
if o.VLLMVersion == "" {
|
||||
o.VLLMVersion = "none"
|
||||
}
|
||||
}
|
||||
}
|
||||
o.Version = version.Version
|
||||
if o.Binary == "" {
|
||||
exe, err := os.Executable()
|
||||
|
||||
+18
-6
@@ -5,19 +5,23 @@ import (
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"net/http"
|
||||
"os/signal"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"golang.org/x/net/http2"
|
||||
"golang.org/x/net/http2/h2c"
|
||||
@@ -491,13 +495,21 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
// And what each model was measured to serve, for scaling and for
|
||||
// choosing between profiles.
|
||||
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
|
||||
// The engine builds chosen per node on the cluster page.
|
||||
ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
}
|
||||
svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
|
||||
svc.SetEngines(cfg.WantedEngines())
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines())
|
||||
svc.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
|
||||
})
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
|
||||
if ch := ctl.EngineChoices(); ch != nil {
|
||||
svc.SetEngineChoices(ch.For)
|
||||
ch.OnChange(svc.Reprovision)
|
||||
}
|
||||
// A standby must refuse agents as well as refusing to place. Commands
|
||||
// travel down the stream the agent opened, so a standby holding one is a
|
||||
// node the leader cannot command — and the leader's placements would fail
|
||||
|
||||
@@ -8,6 +8,7 @@ package config
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -85,6 +86,13 @@ type Server struct {
|
||||
// one leaves that engine to be installed by hand.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
|
||||
// EngineOverrides give some nodes different engines from the rest: a
|
||||
// vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
// all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
// engines over the cluster's, in order, later ones last; an empty version
|
||||
// removes that engine from the nodes it picks.
|
||||
EngineOverrides []EngineOverride `koanf:"engine_overrides"`
|
||||
|
||||
// BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
// Empty means /var/backups/llmgw-server.
|
||||
@@ -548,6 +556,14 @@ func Load(path string) (Server, error) {
|
||||
// Validate rejects configurations that would otherwise fail confusingly at
|
||||
// request time.
|
||||
func (s Server) Validate() error {
|
||||
for i, o := range s.EngineOverrides {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
|
||||
}
|
||||
if len(o.Engines) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] changes no engine: give it engines, with \"\" for none", i)
|
||||
}
|
||||
}
|
||||
if s.Listen == "" {
|
||||
return fmt.Errorf("listen is empty")
|
||||
}
|
||||
@@ -626,6 +642,58 @@ var DefaultEngines = map[string]string{
|
||||
"vllm": "0.30.0",
|
||||
}
|
||||
|
||||
// EngineOverride gives the nodes it picks different engines. Every selector
|
||||
// set must match; at least one must be set.
|
||||
type EngineOverride struct {
|
||||
// GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
GPUClass string `koanf:"gpu_class"`
|
||||
// Nodes picks nodes by id.
|
||||
Nodes []string `koanf:"nodes"`
|
||||
// Labels picks nodes carrying all of these.
|
||||
Labels map[string]string `koanf:"labels"`
|
||||
// Engines are laid over the cluster's: a version, or "" for none.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
}
|
||||
|
||||
// Picks reports whether the override applies to a node with this id, these
|
||||
// GPU classes and these labels.
|
||||
func (o EngineOverride) Picks(node string, classes []string, labels map[string]string) bool {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return false
|
||||
}
|
||||
if o.GPUClass != "" && !slices.Contains(classes, o.GPUClass) {
|
||||
return false
|
||||
}
|
||||
if len(o.Nodes) > 0 && !slices.Contains(o.Nodes, node) {
|
||||
return false
|
||||
}
|
||||
for k, v := range o.Labels {
|
||||
if labels[k] != v {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// EnginesFor is what one node installs: WantedEngines with every override
|
||||
// that picks it laid over, in order.
|
||||
func (s Server) EnginesFor(node string, classes []string, labels map[string]string) map[string]string {
|
||||
out := s.WantedEngines()
|
||||
for _, o := range s.EngineOverrides {
|
||||
if !o.Picks(node, classes, labels) {
|
||||
continue
|
||||
}
|
||||
for k, v := range o.Engines {
|
||||
if v == "" {
|
||||
delete(out, k)
|
||||
continue
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// WantedEngines is DefaultEngines with the configuration laid over it.
|
||||
func (s Server) WantedEngines() map[string]string {
|
||||
out := map[string]string{}
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) {
|
||||
s := Server{
|
||||
Engines: map[string]string{"vllm": "0.30.0"},
|
||||
EngineOverrides: []EngineOverride{
|
||||
// Volta: CUDA 13 dropped it, CUDA 12 vLLM still runs.
|
||||
{GPUClass: "v100-32g", Engines: map[string]string{"vllm": "0.25.1"}},
|
||||
// Nodes meant for llama.cpp alone.
|
||||
{Labels: map[string]string{"role": "gguf"}, Engines: map[string]string{"vllm": ""}},
|
||||
},
|
||||
}
|
||||
if got := s.EnginesFor("h1", []string{"h100-80g"}, nil); got["vllm"] != "0.30.0" || got["llamacpp"] == "" {
|
||||
t.Errorf("an H100 node: %v", got)
|
||||
}
|
||||
if got := s.EnginesFor("v1", []string{"v100-32g"}, nil); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("a V100 node: %v", got)
|
||||
}
|
||||
got := s.EnginesFor("v2", []string{"v100-32g"}, map[string]string{"role": "gguf"})
|
||||
if _, has := got["vllm"]; has || got["llamacpp"] == "" {
|
||||
t.Errorf("a V100 node for llama.cpp alone, the later override last: %v", got)
|
||||
}
|
||||
|
||||
for _, bad := range []EngineOverride{
|
||||
{Engines: map[string]string{"vllm": ""}},
|
||||
{GPUClass: "v100-32g"},
|
||||
} {
|
||||
s := Server{Listen: ":8080", EngineOverrides: []EngineOverride{bad}}
|
||||
if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "engine_overrides[0]") {
|
||||
t.Errorf("%+v: %v", bad, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// ggufStart is the start of a GGUF file with these string keys.
|
||||
func ggufStart(kv ...string) []byte {
|
||||
var b bytes.Buffer
|
||||
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
|
||||
str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
|
||||
b.WriteString("GGUF")
|
||||
w(uint32(3))
|
||||
w(uint64(0))
|
||||
w(uint64(len(kv) / 2))
|
||||
for i := 0; i < len(kv); i += 2 {
|
||||
str(kv[i])
|
||||
w(uint32(8)) // string
|
||||
str(kv[i+1])
|
||||
}
|
||||
b.Write(make([]byte, 4096))
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func TestOnlyAnAdapterForTheModelsArchitectureIsAttached(t *testing.T) {
|
||||
files := map[string][]byte{
|
||||
"/base.gguf": ggufStart("general.architecture", "qwen2"),
|
||||
"/model.gguf": ggufStart("general.architecture", "qwen2", "general.type", "model"),
|
||||
"/llama.gguf": ggufStart("general.architecture", "llama", "general.type", "adapter"),
|
||||
"/lora.gguf": ggufStart("general.architecture", "qwen2", "general.type", "adapter"),
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusPartialContent)
|
||||
_, _ = w.Write(files[r.URL.Path])
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
models := []catalog.Model{{
|
||||
Name: "coder", Source: srv.URL + "/base.gguf",
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 8}},
|
||||
}}
|
||||
for _, tc := range []struct {
|
||||
file, want string
|
||||
code int
|
||||
}{
|
||||
{"/model.gguf", "not a LoRA adapter", http.StatusBadRequest},
|
||||
{"/llama.gguf", "for llama models", http.StatusBadRequest},
|
||||
{"/lora.gguf", "", http.StatusOK},
|
||||
} {
|
||||
a, _ := setup(t, models)
|
||||
fm := &fakeModels{}
|
||||
a.SetModels(fm)
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/v1/models/coder/adapters",
|
||||
strings.NewReader(`{"adapters":[{"name":"fix","source":"`+srv.URL+tc.file+`"}]}`))
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
if rec.Code != tc.code || !strings.Contains(rec.Body.String(), tc.want) {
|
||||
t.Errorf("%s: %d %s; want %d mentioning %q", tc.file, rec.Code, rec.Body.String(), tc.code, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,17 +9,21 @@ package adminapi
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
@@ -60,6 +64,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/adapters", a.setAdapters)
|
||||
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
@@ -107,6 +112,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/engines", a.setNodeEngine)
|
||||
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
|
||||
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
|
||||
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
|
||||
@@ -152,6 +158,10 @@ type NodeView struct {
|
||||
// Provision is where installing each wanted engine stands. An engine is in
|
||||
// Engines only once its installation here is ready.
|
||||
Provision []inventory.EngineInstall `json:"provision,omitempty"`
|
||||
// EngineBuilds are the builds chosen for this node, by engine: "auto"
|
||||
// (its hardware decides), a build ("cu129"), or "none". Only on a node
|
||||
// with GPUs, the only kind with a choice to make.
|
||||
EngineBuilds map[string]string `json:"engine_builds,omitempty"`
|
||||
// OS is the operating system and architecture, as the agent read them.
|
||||
OS string `json:"os,omitempty"`
|
||||
// Warnings are facts about the machine that make it slow or unusable
|
||||
@@ -204,10 +214,13 @@ type NodeView struct {
|
||||
|
||||
// GPUView is one card as a node's page shows it.
|
||||
type GPUView struct {
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
// ComputeCapability is what decides which engine builds run on the card:
|
||||
// CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones.
|
||||
ComputeCapability string `json:"compute_capability,omitempty"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
// NearestNIC is the adapter closest to the card, and how close.
|
||||
NearestNIC string `json:"nearest_nic,omitempty"`
|
||||
NICPath string `json:"nic_path,omitempty"`
|
||||
@@ -312,10 +325,15 @@ type ModelView struct {
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
// Class is its capacity class; Preempted the replicas it lost to a higher
|
||||
// class; HeldFor the model it is giving room to now, if any.
|
||||
Class string `json:"class"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Preempted int `json:"preempted,omitempty"`
|
||||
HeldFor string `json:"held_for,omitempty"`
|
||||
Class string `json:"class"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
// External is where a model served outside the cluster goes, provider
|
||||
// host and its name for the model; empty for one served here.
|
||||
External string `json:"external,omitempty"`
|
||||
// Adapters are the LoRA adapters it serves.
|
||||
Adapters []catalog.Adapter `json:"adapters,omitempty"`
|
||||
Preempted int `json:"preempted,omitempty"`
|
||||
HeldFor string `json:"held_for,omitempty"`
|
||||
}
|
||||
|
||||
// ReplicaView is a replica as `ctl replicas` shows it.
|
||||
@@ -461,6 +479,13 @@ func (a *API) Nodes() []NodeView {
|
||||
}
|
||||
slices.Sort(v.Engines)
|
||||
v.Provision = n.Provision
|
||||
if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
|
||||
chosen := ch.For(n.ID)
|
||||
v.EngineBuilds = map[string]string{"vllm": "auto"}
|
||||
for e, b := range chosen {
|
||||
v.EngineBuilds[e] = b
|
||||
}
|
||||
}
|
||||
if h := n.Host; h.OSID != "" {
|
||||
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
|
||||
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
|
||||
@@ -488,7 +513,8 @@ func (a *API) Nodes() []NodeView {
|
||||
v.Load = loadView(n.Load, time.Now())
|
||||
v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
|
||||
for _, g := range n.GPUs {
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability,
|
||||
NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
for hca, rel := range g.NICPaths {
|
||||
if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
|
||||
(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
|
||||
@@ -569,6 +595,7 @@ func (a *API) Models() []ModelView {
|
||||
Desired: h.Desired, Failures: h.Failures,
|
||||
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
|
||||
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
|
||||
External: externalLabel(m), Adapters: m.Adapters,
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
@@ -651,6 +678,7 @@ type Models interface {
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// ResourcesRequest sets what a model's profiles reserve of a node.
|
||||
@@ -1227,6 +1255,9 @@ type RestartRequest struct {
|
||||
Replicas []string `json:"replicas,omitempty"`
|
||||
Models []string `json:"models,omitempty"`
|
||||
All bool `json:"all,omitempty"`
|
||||
// InPlace stops each replica before starting its replacement, rather
|
||||
// than starting the replacement beside it first.
|
||||
InPlace bool `json:"in_place,omitempty"`
|
||||
}
|
||||
|
||||
// RestartResult lists the replicas queued.
|
||||
@@ -1250,7 +1281,11 @@ func (a *API) restart(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "nothing ready matches; name replicas, models, or all")
|
||||
return
|
||||
}
|
||||
if err := a.ctl.Reconciler().Restart(ids...); err != nil {
|
||||
restart := a.ctl.Reconciler().Restart
|
||||
if req.InPlace {
|
||||
restart = a.ctl.Reconciler().RestartInPlace
|
||||
}
|
||||
if err := restart(ids...); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
@@ -1443,3 +1478,114 @@ func reservationWarnings(m catalog.Model, measuredGB float64) []string {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// externalLabel is "api.openai.com · gpt-4o-mini" for a model served elsewhere.
|
||||
func externalLabel(m catalog.Model) string {
|
||||
if !m.IsExternal() {
|
||||
return ""
|
||||
}
|
||||
host := m.External.URL
|
||||
if u, err := url.Parse(host); err == nil && u.Host != "" {
|
||||
host = u.Host
|
||||
}
|
||||
return host + " · " + m.External.Model
|
||||
}
|
||||
|
||||
// AdaptersRequest replaces a model's LoRA adapters with these.
|
||||
type AdaptersRequest struct {
|
||||
Adapters []catalog.Adapter `json:"adapters"`
|
||||
}
|
||||
|
||||
// setAdapters replaces a model's adapters, after reading each new file to
|
||||
// check it is an adapter for the model's architecture: one that is not is a
|
||||
// replica that fails to start, found minutes later on a node.
|
||||
func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req AdaptersRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
name := r.PathValue("name")
|
||||
m, ok := a.ctl.Catalog().Resolve(name)
|
||||
if !ok {
|
||||
writeError(w, http.StatusBadRequest, "no model "+name)
|
||||
return
|
||||
}
|
||||
base, baseErr := gguf.KindFromURL(r.Context(), nil, m.Source)
|
||||
have := map[string]bool{}
|
||||
for _, ad := range m.Adapters {
|
||||
have[ad.Source] = true
|
||||
}
|
||||
for _, ad := range req.Adapters {
|
||||
if have[ad.Source] {
|
||||
continue // already checked when it was attached
|
||||
}
|
||||
h, err := gguf.KindFromURL(r.Context(), nil, ad.Source)
|
||||
switch {
|
||||
case err != nil:
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
|
||||
return
|
||||
case !h.IsAdapter():
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
|
||||
return
|
||||
case baseErr == nil && h.Architecture != base.Architecture:
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
|
||||
ad.Name, h.Architecture, m.Name, base.Architecture))
|
||||
return
|
||||
}
|
||||
}
|
||||
out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
|
||||
}
|
||||
|
||||
// NodeEngineRequest chooses one engine's build for a node.
|
||||
type NodeEngineRequest struct {
|
||||
Engine string `json:"engine"`
|
||||
// Build is "auto", a build such as "cu129", or "none".
|
||||
Build string `json:"build"`
|
||||
}
|
||||
|
||||
// setNodeEngine chooses an engine's build for a node, and has the node
|
||||
// install it.
|
||||
func (a *API) setNodeEngine(w http.ResponseWriter, r *http.Request) {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the choice in")
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if _, ok := a.ctl.Inventory().Get(id); !ok {
|
||||
writeError(w, http.StatusNotFound, "no node "+id)
|
||||
return
|
||||
}
|
||||
var req NodeEngineRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := ch.Set(r.Context(), id, req.Engine, req.Build, actorOf(r)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, req)
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node, for the cluster page.
|
||||
func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string) error {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
return errors.New("this control plane has no store to keep the choice in")
|
||||
}
|
||||
if _, ok := a.ctl.Inventory().Get(node); !ok {
|
||||
return errors.New("no node " + node)
|
||||
}
|
||||
return ch.Set(ctx, node, engine, build, by)
|
||||
}
|
||||
@@ -258,6 +258,10 @@ type fakeModels struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetAdapters(_ context.Context, name string, as []catalog.Adapter, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Adapters: as}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Owner: owner}, f.err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// choiceStore keeps engine choices in memory.
|
||||
type choiceStore struct{ m map[string]map[string]string }
|
||||
|
||||
func (c *choiceStore) NodeEngines(context.Context) (map[string]map[string]string, error) {
|
||||
return c.m, nil
|
||||
}
|
||||
func (c *choiceStore) SetNodeEngine(_ context.Context, node, engine, variant, _ string) error {
|
||||
if c.m[node] == nil {
|
||||
c.m[node] = map[string]string{}
|
||||
}
|
||||
c.m[node][engine] = variant
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestAnEngineBuildIsChosenForANodeAndTheNodeIsTold(t *testing.T) {
|
||||
a, ctl := setup(t, testModels(), gpuNode("gpu07", 8), inventory.Node{ID: "cpu01", State: inventory.Ready})
|
||||
ch := control.NewEngineChoices(t.Context(), &choiceStore{m: map[string]map[string]string{}}, slog.New(slog.DiscardHandler))
|
||||
var told []string
|
||||
ch.OnChange(func(node string) error { told = append(told, node); return nil })
|
||||
ctl.SetEngineChoices(ch)
|
||||
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"cu129"}`, nil); code != http.StatusOK {
|
||||
t.Fatalf("status %d", code)
|
||||
}
|
||||
if len(told) != 1 || told[0] != "gpu07" {
|
||||
t.Errorf("the node was not told: %v", told)
|
||||
}
|
||||
var nodes []NodeView
|
||||
call(t, a, http.MethodGet, "/api/v1/nodes", "", &nodes)
|
||||
for _, n := range nodes {
|
||||
switch n.ID {
|
||||
case "gpu07":
|
||||
if n.EngineBuilds["vllm"] != "cu129" {
|
||||
t.Errorf("gpu07 shows %v", n.EngineBuilds)
|
||||
}
|
||||
case "cpu01":
|
||||
if n.EngineBuilds != nil {
|
||||
t.Errorf("a node with no GPU has no build to choose: %v", n.EngineBuilds)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{`{"engine":"vllm","build":"cu200"}`, `{"engine":"llamacpp","build":"cu129"}`} {
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", bad, nil); code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status %d", bad, code)
|
||||
}
|
||||
}
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"auto"}`, nil); code != http.StatusOK ||
|
||||
ch.For("gpu07")["vllm"] != "" {
|
||||
t.Errorf("auto should leave it to the node again: %d %v", code, ch.For("gpu07"))
|
||||
}
|
||||
}
|
||||
@@ -31,6 +31,11 @@ func (a *API) Orphans() []OrphanView {
|
||||
var wanted []string
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
wanted = append(wanted, m.Source)
|
||||
// A model's adapters are its weights too; collecting them would
|
||||
// leave replicas that cannot restart.
|
||||
for _, ad := range m.Adapters {
|
||||
wanted = append(wanted, ad.Source)
|
||||
}
|
||||
}
|
||||
|
||||
out := make([]OrphanView, 0)
|
||||
|
||||
@@ -189,6 +189,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
|
||||
GpusPerNode: int32(p.GPUsPerNode), Nodes: int32(p.Nodes),
|
||||
Args: p.Args, Env: p.Env,
|
||||
ModelPath: p.ModelPath, ServedModelName: p.ServedModelName,
|
||||
Adapters: adaptersToWire(p.Adapters),
|
||||
},
|
||||
StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(),
|
||||
Pool: cmd.Pool,
|
||||
@@ -224,6 +225,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
|
||||
GPUsPerNode: int(p.GetGpusPerNode()), Nodes: int(p.GetNodes()),
|
||||
Args: p.GetArgs(), Env: p.GetEnv(),
|
||||
ModelPath: p.GetModelPath(), ServedModelName: p.GetServedModelName(),
|
||||
Adapters: adaptersFromWire(p.GetAdapters()),
|
||||
},
|
||||
}
|
||||
return spec, time.Duration(m.GetStartupTimeoutMs()) * time.Millisecond
|
||||
@@ -292,3 +294,19 @@ func fabricFrom(state, status, cluster, clique, bandwidth string) *discovery.GPU
|
||||
}
|
||||
return &discovery.GPUFabric{State: state, Status: status, ClusterUUID: cluster, CliqueID: clique, Bandwidth: bandwidth}
|
||||
}
|
||||
|
||||
func adaptersToWire(as []engines.Adapter) []*llmgwv1.LoraAdapter {
|
||||
var out []*llmgwv1.LoraAdapter
|
||||
for _, a := range as {
|
||||
out = append(out, &llmgwv1.LoraAdapter{Name: a.Name, Source: a.Path})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func adaptersFromWire(as []*llmgwv1.LoraAdapter) []engines.Adapter {
|
||||
var out []engines.Adapter
|
||||
for _, a := range as {
|
||||
out = append(out, engines.Adapter{Name: a.GetName(), Path: a.GetSource()})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestAdaptersTravelToTheAgentInOrder(t *testing.T) {
|
||||
cmd := reconciler.ShardCommand{ReplicaID: "r", ShardID: "s", Spec: engines.ShardSpec{
|
||||
Profile: engines.Profile{Name: "cpu", Engine: "llamacpp", ModelPath: "/m.gguf",
|
||||
Adapters: []engines.Adapter{{Name: "sql", Path: "http://mirror/sql.gguf"}, {Name: "docs", Path: "http://mirror/docs.gguf"}}},
|
||||
}}
|
||||
spec, _ := ShardSpecFromStart(StartShardFromCommand(cmd))
|
||||
got := spec.Profile.Adapters
|
||||
if len(got) != 2 || got[0].Name != "sql" || got[1].Path != "http://mirror/docs.gguf" {
|
||||
t.Errorf("adapters %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
@@ -25,3 +26,57 @@ func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
t.Error("uv, a tool for the others, is offered as an engine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
if n.ID == "volta" {
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.25.1"}
|
||||
}
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.30.0"}
|
||||
})
|
||||
reg := &llmgwv1.Register{InstallableEngines: []string{"llamacpp", "vllm"}}
|
||||
want := func(p *llmgwv1.Provision) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, e := range p.GetEngines() {
|
||||
out[e.GetEngine()] = e.GetVersion()
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, installable(reg))); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("the Volta node: %v", got)
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, installable(reg))); got["vllm"] != "0.30.0" {
|
||||
t.Errorf("the Hopper node: %v", got)
|
||||
}
|
||||
// A node that disabled vLLM does not offer it, and is not asked for it.
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, []string{"llamacpp"})); got["vllm"] != "" {
|
||||
t.Errorf("asked a node for an engine it does not install: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsWhatItIsTold(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEngines(map[string]string{"llamacpp": "b1", "vllm": "0.30.0"})
|
||||
s.SetEngineChoices(func(node string) map[string]string {
|
||||
return map[string]map[string]string{"gpu07": {"vllm": "cu129"}, "gpu08": {"vllm": "none"}}[node]
|
||||
})
|
||||
can := []string{"llamacpp", "vllm"}
|
||||
variant := func(node string) (string, bool) {
|
||||
for _, e := range s.provisionFor(inventory.Node{ID: node}, can).GetEngines() {
|
||||
if e.GetEngine() == "vllm" {
|
||||
return e.GetVariant(), true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
if v, ok := variant("gpu07"); !ok || v != "cu129" {
|
||||
t.Errorf("gpu07: %q, %v", v, ok)
|
||||
}
|
||||
if _, ok := variant("gpu08"); ok {
|
||||
t.Error("gpu08 was asked for a vLLM it was chosen to run none of")
|
||||
}
|
||||
if v, ok := variant("gpu09"); !ok || v != "" {
|
||||
t.Errorf("gpu09, left to its hardware: %q, %v", v, ok)
|
||||
}
|
||||
}
|
||||
@@ -53,8 +53,13 @@ type Service struct {
|
||||
mu sync.RWMutex
|
||||
conns map[string]*conn
|
||||
|
||||
// engines is what every node is told to install, set once at start.
|
||||
engines map[string]string
|
||||
// enginesFor is what a node is told to install, set once at start: the
|
||||
// cluster's engines, with any override that picks the node laid over.
|
||||
enginesFor func(node inventory.Node) map[string]string
|
||||
// choices are the engine builds chosen per node, and installableBy what
|
||||
// each connected node said it can install, for telling it again.
|
||||
choices func(nodeID string) map[string]string
|
||||
installableBy map[string][]string
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -68,7 +73,37 @@ type Service struct {
|
||||
}
|
||||
|
||||
// SetEngines sets the engines every node installs, by name and version.
|
||||
func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
|
||||
func (s *Service) SetEngines(e map[string]string) {
|
||||
e = maps.Clone(e)
|
||||
s.enginesFor = func(inventory.Node) map[string]string { return e }
|
||||
}
|
||||
|
||||
// SetEnginesFor sets what each node installs, decided per node: a cluster
|
||||
// of two GPU generations needs two vLLM builds, or vLLM on only one.
|
||||
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
|
||||
|
||||
// SetEngineChoices sets where the builds chosen for each node come from: by
|
||||
// engine, a build ("cu129") or "none". An engine not named is left to the
|
||||
// node's hardware.
|
||||
func (s *Service) SetEngineChoices(f func(nodeID string) map[string]string) { s.choices = f }
|
||||
|
||||
// Reprovision tells a connected node again what to install, after a choice
|
||||
// for it changed; one not connected is told when it next connects.
|
||||
func (s *Service) Reprovision(nodeID string) error {
|
||||
if s.enginesFor == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.RLock()
|
||||
can, ok := s.installableBy[nodeID]
|
||||
s.mu.RUnlock()
|
||||
node, known := s.inv.Get(nodeID)
|
||||
if !ok || !known {
|
||||
return nil
|
||||
}
|
||||
return s.send(nodeID, &llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: s.provisionFor(node, can)},
|
||||
})
|
||||
}
|
||||
|
||||
// conn is one agent's open stream.
|
||||
type conn struct {
|
||||
@@ -179,15 +214,15 @@ func (s *Service) Connect(
|
||||
// What this node should have installed. Sent on every registration, so a
|
||||
// node rebuilt from nothing is told the same as one that has been here a
|
||||
// month, and a changed version reaches every node on its next connect.
|
||||
if s.engines != nil {
|
||||
p := &llmgwv1.Provision{}
|
||||
if s.enginesFor != nil {
|
||||
can := installable(reg)
|
||||
for _, e := range slices.Sorted(maps.Keys(s.engines)) {
|
||||
if !slices.Contains(can, e) {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
|
||||
s.mu.Lock()
|
||||
if s.installableBy == nil {
|
||||
s.installableBy = map[string][]string{}
|
||||
}
|
||||
s.installableBy[nodeID] = can
|
||||
s.mu.Unlock()
|
||||
p := s.provisionFor(node, can)
|
||||
if err := stream.Send(&llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
|
||||
}); err != nil {
|
||||
@@ -853,6 +888,24 @@ func rate(before, now uint64, secs float64) float64 {
|
||||
return float64(now-before) / secs
|
||||
}
|
||||
|
||||
// provisionFor is what a node is told to install: the engines decided for
|
||||
// it, those it can install.
|
||||
func (s *Service) provisionFor(node inventory.Node, can []string) *llmgwv1.Provision {
|
||||
p := &llmgwv1.Provision{}
|
||||
wants := s.enginesFor(node)
|
||||
var chosen map[string]string
|
||||
if s.choices != nil {
|
||||
chosen = s.choices(node.ID)
|
||||
}
|
||||
for _, e := range slices.Sorted(maps.Keys(wants)) {
|
||||
if !slices.Contains(can, e) || chosen[e] == "none" {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e], Variant: chosen[e]})
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// installable are the engines a node can be asked to install. An agent older
|
||||
// than the list said nothing, and could install llama.cpp alone: asked for
|
||||
// vLLM too, it reported it unsupported, and a cluster half way through an
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -160,8 +161,20 @@ type Model struct {
|
||||
// Owner is the part of the tenancy tree whose capacity quota this model
|
||||
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
// held by the cluster, counted against nobody.
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
// External is a model served by someone else — an OpenAI-compatible API
|
||||
// outside the cluster — rather than by replicas placed here. Such a model
|
||||
// has no profiles and nothing is placed for it; the router sends its
|
||||
// requests to the provider. Most useful named in another model's
|
||||
// fallbacks: used only when the cluster has no capacity left. Its
|
||||
// prompts leave the cluster, so which keys may name it is worth an
|
||||
// allowlist.
|
||||
External *External `json:"external,omitempty" koanf:"external"`
|
||||
// Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
// for by its own name as though it were a model. Every replica loads
|
||||
// every adapter, applying one only to a request that names it.
|
||||
Adapters []Adapter `json:"adapters,omitempty" koanf:"adapters"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
// RoutingProfile selects how the router scores endpoints for this model.
|
||||
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
|
||||
SLO SLO `json:"slo,omitempty" koanf:"slo"`
|
||||
@@ -204,14 +217,17 @@ type Capabilities struct {
|
||||
type Catalog struct {
|
||||
models map[string]Model
|
||||
aliases map[string]string
|
||||
order []string
|
||||
// adapters maps an adapter's name to the model it runs on.
|
||||
adapters map[string]string
|
||||
order []string
|
||||
}
|
||||
|
||||
// New builds a catalog, rejecting definitions that would fail confusingly later.
|
||||
func New(models []Model) (*Catalog, error) {
|
||||
c := &Catalog{
|
||||
models: make(map[string]Model, len(models)),
|
||||
aliases: map[string]string{},
|
||||
models: make(map[string]Model, len(models)),
|
||||
aliases: map[string]string{},
|
||||
adapters: map[string]string{},
|
||||
}
|
||||
|
||||
for i := range models {
|
||||
@@ -240,6 +256,18 @@ func New(models []Model) (*Catalog, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// Adapters are named like models, so they share the namespace too.
|
||||
for _, name := range c.order {
|
||||
for _, a := range c.models[name].Adapters {
|
||||
_, isModel := c.models[a.Name]
|
||||
_, isAlias := c.aliases[a.Name]
|
||||
if _, isAdapter := c.adapters[a.Name]; isModel || isAlias || isAdapter {
|
||||
return nil, fmt.Errorf("model %q: adapter %q collides with a model, an alias or another adapter", name, a.Name)
|
||||
}
|
||||
c.adapters[a.Name] = name
|
||||
}
|
||||
}
|
||||
|
||||
// Fallbacks are resolved now rather than at request time: a typo in a
|
||||
// fallback should stop a deployment, not surface during an outage when the
|
||||
// fallback is finally needed.
|
||||
@@ -305,7 +333,21 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if len(m.Profiles) == 0 {
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
|
||||
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.External != nil {
|
||||
if err := m.External.validate(m.Name, len(m.Profiles)); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
@@ -346,6 +388,21 @@ func (m Model) validate() error {
|
||||
}
|
||||
continue
|
||||
}
|
||||
// llama.cpp on GPUs splits a model across one node's cards by layer,
|
||||
// and how many is gpus_per_node. Not tp: llama.cpp has no tensor
|
||||
// parallelism, and a tp it was given was a width nothing placed — the
|
||||
// replica stayed unplaced, or ran on one card while four were meant.
|
||||
if p.Engine == "llamacpp" {
|
||||
switch {
|
||||
case p.Nodes > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
m.Name, p.Name, p.Nodes)
|
||||
case p.TP > 1 || p.PP > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
|
||||
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
}
|
||||
continue
|
||||
}
|
||||
// The arithmetic has to add up, or the engine is handed a shard spec it
|
||||
// cannot satisfy and fails minutes later during model loading.
|
||||
if want := p.GPUsPerNode * p.Nodes; want != p.TP*max(p.PP, 1) {
|
||||
@@ -357,6 +414,23 @@ func (m Model) validate() error {
|
||||
}
|
||||
|
||||
// Resolve maps a client-supplied name or alias onto a model.
|
||||
// ResolveAdapter finds an adapter by its name: the model it runs on, the
|
||||
// adapter, and its index among the model's adapters — which is the id the
|
||||
// engine knows it by.
|
||||
func (c *Catalog) ResolveAdapter(name string) (Model, Adapter, int, bool) {
|
||||
base, ok := c.adapters[name]
|
||||
if !ok {
|
||||
return Model{}, Adapter{}, 0, false
|
||||
}
|
||||
m := c.models[base]
|
||||
for i, a := range m.Adapters {
|
||||
if a.Name == name {
|
||||
return m, a, i, true
|
||||
}
|
||||
}
|
||||
return Model{}, Adapter{}, 0, false
|
||||
}
|
||||
|
||||
func (c *Catalog) Resolve(name string) (Model, bool) {
|
||||
if m, ok := c.models[name]; ok {
|
||||
return m, true
|
||||
@@ -433,6 +507,7 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
Env: p.Env,
|
||||
ModelPath: m.Source,
|
||||
ServedModelName: m.Name,
|
||||
Adapters: m.engineAdapters(),
|
||||
}, true
|
||||
}
|
||||
return engines.Profile{}, false
|
||||
@@ -564,3 +639,108 @@ func validOwner(o string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// External is a provider outside the cluster that serves a model.
|
||||
type External struct {
|
||||
// URL is the provider's base address, without /v1: https://api.openai.com.
|
||||
URL string `json:"url" koanf:"url"`
|
||||
// Model is what the provider calls it; requests are rewritten to it.
|
||||
Model string `json:"model" koanf:"model"`
|
||||
// APIKeyFile holds the provider's key, on the control plane. A file
|
||||
// rather than the key itself, so the key is in neither the catalog, the
|
||||
// store nor the audit log; empty for a provider that asks for none.
|
||||
APIKeyFile string `json:"api_key_file,omitempty" koanf:"api_key_file"`
|
||||
}
|
||||
|
||||
func (e *External) validate(model string, profiles int) error {
|
||||
if profiles > 0 {
|
||||
return fmt.Errorf("model %q: an external model is served elsewhere and has no profiles", model)
|
||||
}
|
||||
u, err := url.Parse(e.URL)
|
||||
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
|
||||
return fmt.Errorf("model %q: external url %q is not an http(s) address", model, e.URL)
|
||||
}
|
||||
if strings.TrimSpace(e.Model) == "" {
|
||||
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsExternal reports whether a model is served outside the cluster.
|
||||
func (m Model) IsExternal() bool { return m.External != nil }
|
||||
|
||||
// Adapter is a LoRA adapter served on a model's replicas.
|
||||
type Adapter struct {
|
||||
// Name is what clients ask for, like a model name.
|
||||
Name string `json:"name" koanf:"name"`
|
||||
// Source is the adapter file (GGUF for llama.cpp), fetched by each node
|
||||
// as the model's weights are.
|
||||
Source string `json:"source" koanf:"source"`
|
||||
// Scale is how strongly it is applied; zero takes 1.
|
||||
Scale float64 `json:"scale,omitempty" koanf:"scale"`
|
||||
}
|
||||
|
||||
// EffectiveScale is the scale an adapter is applied at.
|
||||
func (a Adapter) EffectiveScale() float64 {
|
||||
if a.Scale <= 0 {
|
||||
return 1
|
||||
}
|
||||
return a.Scale
|
||||
}
|
||||
|
||||
func validateAdapters(m Model) error {
|
||||
if len(m.Adapters) == 0 {
|
||||
return nil
|
||||
}
|
||||
if m.External != nil {
|
||||
return fmt.Errorf("model %q: an external model cannot carry adapters", m.Name)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
// llama.cpp loads adapters at start and applies them per request;
|
||||
// the other engines are not driven that way yet.
|
||||
if p.Engine != "llamacpp" && p.Engine != "sim" {
|
||||
return fmt.Errorf("model %q profile %q: adapters are served by llamacpp profiles only", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, a := range m.Adapters {
|
||||
switch {
|
||||
case a.Name == "" || a.Source == "":
|
||||
return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)
|
||||
case a.Name == m.Name:
|
||||
return fmt.Errorf("model %q: an adapter cannot have the model's own name", m.Name)
|
||||
case seen[a.Name]:
|
||||
return fmt.Errorf("model %q: adapter %q is declared twice", m.Name, a.Name)
|
||||
case a.Scale < 0:
|
||||
return fmt.Errorf("model %q: adapter %q has a negative scale", m.Name, a.Name)
|
||||
}
|
||||
seen[a.Name] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m Model) engineAdapters() []engines.Adapter {
|
||||
var out []engines.Adapter
|
||||
for _, a := range m.Adapters {
|
||||
out = append(out, engines.Adapter{Name: a.Name, Path: a.Source})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// repositoryPage reports whether source is a Hugging Face repository's page
|
||||
// (https://huggingface.co/org/repo, or one of its tree views) rather than a
|
||||
// file under /resolve/, and the repository's id.
|
||||
func repositoryPage(source string) (string, bool) {
|
||||
for _, host := range []string{"https://huggingface.co/", "http://huggingface.co/", "https://hf.co/"} {
|
||||
rest, ok := strings.CutPrefix(source, host)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
parts := strings.Split(strings.Trim(rest, "/"), "/")
|
||||
if len(parts) < 2 || (len(parts) > 2 && parts[2] == "resolve") {
|
||||
return "", false
|
||||
}
|
||||
return parts[0] + "/" + parts[1], true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
@@ -321,3 +321,48 @@ func TestAProfileCannotSetWhatRunsCodeInTheEngine(t *testing.T) {
|
||||
t.Errorf("ordinary tuning refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
|
||||
four := model("big", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", GPUsPerNode: 4})
|
||||
c, err := New([]Model{four})
|
||||
if err != nil {
|
||||
t.Fatalf("gpus_per_node 4 on llama.cpp: %v", err)
|
||||
}
|
||||
m, _ := c.Resolve("big")
|
||||
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
|
||||
t.Errorf("placement was not asked for four GPUs: %+v", pl)
|
||||
}
|
||||
// tp is a width llama.cpp has no way to use; refused, saying what to
|
||||
// write instead.
|
||||
for _, p := range []Profile{
|
||||
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
|
||||
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
|
||||
} {
|
||||
_, err := New([]Model{model("x", p)})
|
||||
if err == nil {
|
||||
t.Errorf("%s: accepted", p.Name)
|
||||
}
|
||||
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
|
||||
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryPageIsNotASource(t *testing.T) {
|
||||
for _, src := range []string{
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
|
||||
} {
|
||||
m := model("awq")
|
||||
m.Source = src
|
||||
_, err := New([]Model{m})
|
||||
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
|
||||
t.Errorf("%s: %v", src, err)
|
||||
}
|
||||
}
|
||||
m := model("file")
|
||||
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{m}); err != nil {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
+49
-10
@@ -9,6 +9,7 @@ import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -74,15 +75,18 @@ func DefaultConfig() Config {
|
||||
|
||||
// Controller owns the control plane's state.
|
||||
type Controller struct {
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
// engineChoices are the engine builds chosen per node; nil without a
|
||||
// store to keep them in.
|
||||
engineChoices *EngineChoices
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
|
||||
mu sync.Mutex
|
||||
last string // fingerprint of the last published list
|
||||
@@ -338,6 +342,34 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
Priority: prof.Priority,
|
||||
})
|
||||
}
|
||||
return append(out, c.externalEndpoints()...)
|
||||
}
|
||||
|
||||
// externalEndpoints are the models served outside the cluster, as endpoints:
|
||||
// one each, with the provider's key read from its file every time, so a key
|
||||
// rotated on disk is used from the next publish on.
|
||||
func (c *Controller) externalEndpoints() []endpoint.Info {
|
||||
var out []endpoint.Info
|
||||
for _, m := range c.cat.Load().Models() {
|
||||
if !m.IsExternal() {
|
||||
continue
|
||||
}
|
||||
key := ""
|
||||
if f := m.External.APIKeyFile; f != "" {
|
||||
b, err := os.ReadFile(f) //nolint:gosec // the operator's own path
|
||||
if err != nil {
|
||||
c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
key = strings.TrimSpace(string(b))
|
||||
}
|
||||
out = append(out, endpoint.Info{
|
||||
ID: "external:" + m.Name, URL: strings.TrimRight(m.External.URL, "/"),
|
||||
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
|
||||
UpstreamModel: m.External.Model,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -345,7 +377,8 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
func fingerprint(infos []endpoint.Info) string {
|
||||
var b strings.Builder
|
||||
for _, i := range infos {
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority)
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority,
|
||||
i.UpstreamModel, sha256.Sum256([]byte(i.APIKey)))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -378,3 +411,9 @@ func EngineKeys(secret []byte) func(replicaID string) string {
|
||||
|
||||
// QuotaUse is every capacity quota beside what its scope holds.
|
||||
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
|
||||
|
||||
// SetEngineChoices gives the controller the engine builds chosen per node.
|
||||
func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
|
||||
|
||||
// EngineChoices are the engine builds chosen per node, or nil.
|
||||
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
|
||||
@@ -0,0 +1,108 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
)
|
||||
|
||||
// EngineChoiceStore is where the engine builds chosen per node are kept.
|
||||
type EngineChoiceStore interface {
|
||||
NodeEngines(ctx context.Context) (map[string]map[string]string, error)
|
||||
SetNodeEngine(ctx context.Context, node, engine, variant, by string) error
|
||||
}
|
||||
|
||||
// EngineChoices are the engine builds chosen for nodes, rather than left to
|
||||
// their hardware: vLLM's CUDA 12.9 build on a node its CUDA 13 build would
|
||||
// also install on, or no vLLM at all. Kept in the store, held here for the
|
||||
// agent service to ask on every registration, and pushed to a node the
|
||||
// moment its choice changes.
|
||||
type EngineChoices struct {
|
||||
st EngineChoiceStore
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.RWMutex
|
||||
byNode map[string]map[string]string
|
||||
notify func(node string) error
|
||||
}
|
||||
|
||||
// NewEngineChoices reads what was chosen before.
|
||||
func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logger) *EngineChoices {
|
||||
c := &EngineChoices{st: st, log: log, byNode: map[string]map[string]string{}}
|
||||
if got, err := st.NodeEngines(ctx); err != nil {
|
||||
log.Error("could not read the engine builds chosen per node; every node installs what its hardware runs", "err", err)
|
||||
} else {
|
||||
c.byNode = got
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// OnChange is called with a node whose choice changed, to tell it.
|
||||
func (c *EngineChoices) OnChange(f func(node string) error) {
|
||||
c.mu.Lock()
|
||||
c.notify = f
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// For is what was chosen for one node, by engine: a build, or "none".
|
||||
func (c *EngineChoices) For(node string) map[string]string {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return maps.Clone(c.byNode[node])
|
||||
}
|
||||
|
||||
// EngineBuilds are what can be chosen for each engine that has a choice:
|
||||
// "auto" is the node's hardware deciding, "none" no engine at all.
|
||||
func EngineBuilds(engine string) []string {
|
||||
switch engine {
|
||||
case "vllm":
|
||||
return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
|
||||
case "sglang":
|
||||
return []string{"auto", "none"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Set chooses an engine's build for a node, keeps it, and tells the node.
|
||||
func (c *EngineChoices) Set(ctx context.Context, node, engine, build, by string) error {
|
||||
builds := EngineBuilds(engine)
|
||||
if builds == nil {
|
||||
return fmt.Errorf("%s has no builds to choose between", engine)
|
||||
}
|
||||
if !slices.Contains(builds, build) {
|
||||
return fmt.Errorf("%s has no build %q; choose %s", engine, build, strings.Join(builds, ", "))
|
||||
}
|
||||
variant := build
|
||||
if build == "auto" {
|
||||
variant = ""
|
||||
}
|
||||
if err := c.st.SetNodeEngine(ctx, node, engine, variant, by); err != nil {
|
||||
return err
|
||||
}
|
||||
c.mu.Lock()
|
||||
if variant == "" {
|
||||
delete(c.byNode[node], engine)
|
||||
} else {
|
||||
if c.byNode[node] == nil {
|
||||
c.byNode[node] = map[string]string{}
|
||||
}
|
||||
c.byNode[node][engine] = variant
|
||||
}
|
||||
notify := c.notify
|
||||
c.mu.Unlock()
|
||||
c.log.Info("engine build chosen for a node", "node", node, "engine", engine, "build", build, "by", by)
|
||||
if notify != nil {
|
||||
if err := notify(node); err != nil {
|
||||
// Kept and applied on its next connection; said, since a node that
|
||||
// is not connected is the usual reason.
|
||||
c.log.Warn("the node was not told yet; it is when it next connects", "node", node, "err", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
|
||||
type recordSink struct {
|
||||
mu sync.Mutex
|
||||
infos []endpoint.Info
|
||||
times int
|
||||
}
|
||||
|
||||
func (s *recordSink) SetEndpoints(_ context.Context, infos []endpoint.Info) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.infos, s.times = infos, s.times+1
|
||||
}
|
||||
|
||||
type noDispatch struct{}
|
||||
|
||||
func (noDispatch) StartShard(context.Context, reconciler.ShardCommand) error { return nil }
|
||||
func (noDispatch) StopShard(context.Context, string, string) error { return nil }
|
||||
func (noDispatch) DrainShard(context.Context, string, string) error { return nil }
|
||||
|
||||
func TestAnExternalModelIsPublishedWithItsKeyAndNothingIsPlacedForIt(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
keyFile := filepath.Join(dir, "provider.key")
|
||||
if err := os.WriteFile(keyFile, []byte("sk-provider-1\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
models := []catalog.Model{
|
||||
{Name: "cloud", Source: "external", Aliases: []string{"gpt"}, Scaling: catalog.Scaling{Min: 3, Max: 3},
|
||||
External: &catalog.External{URL: "https://api.example.com/", Model: "gpt-4o-mini", APIKeyFile: keyFile}},
|
||||
{Name: "gone", Source: "external",
|
||||
External: &catalog.External{URL: "https://api.example.com", Model: "m", APIKeyFile: filepath.Join(dir, "missing")}},
|
||||
}
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sink := &recordSink{}
|
||||
c := New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler))
|
||||
c.Publish(t.Context())
|
||||
|
||||
if len(sink.infos) != 1 {
|
||||
t.Fatalf("published %+v; the model whose key is missing must not be offered", sink.infos)
|
||||
}
|
||||
e := sink.infos[0]
|
||||
if e.ID != "external:cloud" || e.URL != "https://api.example.com" || e.APIKey != "sk-provider-1" ||
|
||||
e.UpstreamModel != "gpt-4o-mini" || e.Engine != endpoint.EngineExternal || len(e.Models) != 2 {
|
||||
t.Errorf("endpoint %+v", e)
|
||||
}
|
||||
// A key rotated on disk is published on the next pass.
|
||||
_ = os.WriteFile(keyFile, []byte("sk-provider-2"), 0o600)
|
||||
c.Publish(t.Context())
|
||||
if sink.times != 2 || sink.infos[0].APIKey != "sk-provider-2" {
|
||||
t.Errorf("the rotated key was not published: %d pushes, key %q", sink.times, sink.infos[0].APIKey)
|
||||
}
|
||||
// Scaling of 3 or not, nothing is wanted of the cluster for a model
|
||||
// served elsewhere, and so it never reads as short or unplaceable.
|
||||
for _, h := range c.Reconciler().ModelHealth() {
|
||||
if h.Model == "cloud" && (h.Desired != 0 || h.Unplaceable) {
|
||||
t.Errorf("an external model wants replicas: %+v", h)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -87,6 +87,9 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
|
||||
if m.Source != "" {
|
||||
out[m.Source] = append(out[m.Source], m)
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
out[ad.Source] = append(out[ad.Source], m)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -268,6 +271,23 @@ func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog
|
||||
return final, err
|
||||
}
|
||||
|
||||
// SetAdapters replaces the LoRA adapters a model serves. Replicas load the
|
||||
// new set when they next start.
|
||||
func (s *Service) SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its adapters", func(m *catalog.Model) error {
|
||||
m.Adapters = adapters
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
names := make([]string, 0, len(adapters))
|
||||
for _, a := range adapters {
|
||||
names = append(names, a.Name)
|
||||
}
|
||||
s.log.Info("model adapters set", "model", name, "adapters", names, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// update changes one model added at runtime and keeps the change.
|
||||
//
|
||||
// Built before anything is written, like Add: catalog.New is what applies the
|
||||
|
||||
@@ -94,6 +94,9 @@ func (r *Reconciler) desiredFor(m catalog.Model) int {
|
||||
// baseFor is what a model runs without load: its minimum, or for an on-demand
|
||||
// model one while somebody is asking and none once they stop.
|
||||
func (r *Reconciler) baseFor(m catalog.Model) int {
|
||||
if m.IsExternal() {
|
||||
return 0 // served elsewhere: nothing to place, whatever its scaling says
|
||||
}
|
||||
if !m.Scaling.OnDemand() {
|
||||
return m.Scaling.Min
|
||||
}
|
||||
|
||||
@@ -255,6 +255,11 @@ type Reconciler struct {
|
||||
// restarted and not yet begun. One is replaced at a time: replacing them
|
||||
// all at once would need the memory of every replica twice over.
|
||||
restartQueue map[string][]string
|
||||
// restartInPlace are the replicas asked to restart by stopping first.
|
||||
restartInPlace map[string]bool
|
||||
// restartRefused says, by replica, why a restart was given up rather
|
||||
// than leave the model without it.
|
||||
restartRefused map[string]string
|
||||
// parked holds models that failed ParkAfter times in a row.
|
||||
parked map[string]parkRecord
|
||||
// extra is how many replicas load has added above a model's base, and
|
||||
@@ -304,6 +309,8 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
reservedPorts: map[int]struct{}{},
|
||||
wanted: map[string]time.Time{},
|
||||
restartQueue: map[string][]string{},
|
||||
restartInPlace: map[string]bool{},
|
||||
restartRefused: map[string]string{},
|
||||
parked: map[string]parkRecord{},
|
||||
extra: map[string]int{},
|
||||
busySince: map[string]time.Time{},
|
||||
@@ -729,6 +736,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.mu.Unlock()
|
||||
|
||||
desired := r.desiredFor(m)
|
||||
outcome := notTried
|
||||
switch {
|
||||
case len(liveIDs) < desired && isParked:
|
||||
// Not placed. Its failure is on every status the operator reads.
|
||||
@@ -742,7 +750,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// the real error in noise.
|
||||
return
|
||||
}
|
||||
r.scaleUp(ctx, m, avoid)
|
||||
outcome = r.scaleUp(ctx, m, avoid)
|
||||
case len(liveIDs) > desired:
|
||||
r.scaleDown(ctx, liveIDs[desired:])
|
||||
}
|
||||
@@ -781,7 +789,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
}
|
||||
}
|
||||
|
||||
r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
|
||||
r.retire(ctx, m, retiringIDs, readyLive, len(liveIDs), desired, outcome, isParked)
|
||||
}
|
||||
|
||||
// scaleUp places and starts one replica per pass.
|
||||
@@ -789,7 +797,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// One at a time on purpose: placement decisions depend on what is already
|
||||
// allocated, and starting several at once from the same inventory snapshot
|
||||
// would hand them the same GPUs.
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) scaleOutcome {
|
||||
profiles := r.orderedProfiles(m)
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
@@ -813,13 +821,13 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
// the pass that reaps it, and the next pass places this model — which
|
||||
// keeps the stop-then-start ordering in one place rather than two.
|
||||
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
|
||||
return
|
||||
return makingRoom
|
||||
}
|
||||
if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
|
||||
return
|
||||
return makingRoom
|
||||
}
|
||||
r.notePlacementFailure(m.Name, res.Explain())
|
||||
return
|
||||
return noRoom
|
||||
}
|
||||
if why := r.overLimit(m, res.Placement); why != "" {
|
||||
// Recorded as the placement's answer, so the model reads as not
|
||||
@@ -829,13 +837,26 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
{Profile: res.Placement.Profile.Name, Reason: why}}}
|
||||
r.mu.Unlock()
|
||||
r.notePlacementFailure(m.Name, why)
|
||||
return
|
||||
return overLimit
|
||||
}
|
||||
if err := r.startReplica(ctx, m, res.Placement); err != nil {
|
||||
r.log.Error("starting replica failed", "model", m.Name, "err", err)
|
||||
return noRoom
|
||||
}
|
||||
return placed
|
||||
}
|
||||
|
||||
// scaleOutcome is what one scaleUp did.
|
||||
type scaleOutcome int
|
||||
|
||||
const (
|
||||
notTried scaleOutcome = iota // nothing was asked of placement this pass
|
||||
placed // a replica was started
|
||||
makingRoom // room is being made: a preemption, an idle replica stopping
|
||||
noRoom // placement found nowhere, and nothing is making room
|
||||
overLimit // there is room, but the owner's quota does not allow it
|
||||
)
|
||||
|
||||
func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *placement.Placement) error {
|
||||
prof, ok := m.ProfileByName(pl.Profile.Name)
|
||||
if !ok {
|
||||
|
||||
@@ -114,6 +114,17 @@ type harness struct {
|
||||
inv *inventory.Store
|
||||
disp *fakeDispatcher
|
||||
clock *clock
|
||||
live *catalog.Live
|
||||
}
|
||||
|
||||
// setModels replaces the catalog the reconciler works from, as an edit does.
|
||||
func (h *harness) setModels(t *testing.T, models ...catalog.Model) {
|
||||
t.Helper()
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.live.Store(cat)
|
||||
}
|
||||
|
||||
func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness {
|
||||
@@ -135,7 +146,8 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
|
||||
cfg.UnreachableAfter = 15 * time.Second
|
||||
cfg.LostAfter = 60 * time.Second
|
||||
|
||||
r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
|
||||
live := catalog.NewLive(cat)
|
||||
r := New(cfg, live, inv, disp, slog.New(slog.DiscardHandler))
|
||||
r.SetClock(c.now)
|
||||
// Nodes were just registered, so their LastSeen is wall-clock now, which is
|
||||
// far from the fake clock's epoch. Re-stamp them onto the fake timeline.
|
||||
@@ -150,7 +162,7 @@ func newHarness(t *testing.T, m catalog.Model, nodes ...inventory.Node) *harness
|
||||
for _, n := range nodes {
|
||||
touchAt(inv, n.ID, c.now())
|
||||
}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c, live: live}
|
||||
}
|
||||
|
||||
// touchAt rewrites a node's LastSeen onto the fake clock's timeline.
|
||||
@@ -1787,7 +1799,9 @@ func TestARestartedReplicaServesUntilItsReplacementIsReady(t *testing.T) {
|
||||
|
||||
func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
// One node, room for one replica: the replacement can only start once the
|
||||
// old one has gone. Waiting forever would mean never restarting at all.
|
||||
// old one has gone. It is the only thing in the way, so it goes now —
|
||||
// waiting out a grace period first changed nothing but the wait, which
|
||||
// in the field read as a restart that never happened.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
@@ -1796,14 +1810,8 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, _ := replicaByID(h, old); rep.State != Ready {
|
||||
t.Fatalf("stopped before the grace period: %s", rep.State)
|
||||
}
|
||||
|
||||
h.advance(h.r.cfg.DrainGrace + time.Second)
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
|
||||
t.Fatal("still serving after the grace period with no replacement possible")
|
||||
t.Fatal("still serving with no replacement possible beside it")
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
h.r.Reconcile(t.Context())
|
||||
@@ -1815,6 +1823,52 @@ func TestARestartWithNoRoomBesideItHappensInPlace(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartWhoseReplacementFitsNowhereIsGivenUp(t *testing.T) {
|
||||
// The profile was edited to need more cards than the node has: stopping
|
||||
// the old replica would leave the model with nothing at all.
|
||||
m := testModel(1)
|
||||
h := newHarness(t, m, gpuNode("1", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
old := h.r.ReadyReplicas()[0].ID
|
||||
|
||||
m.Profiles[0].TP, m.Profiles[0].GPUsPerNode = 8, 8
|
||||
h.setModels(t, m)
|
||||
if err := h.r.Restart(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for range 3 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.advance(h.r.cfg.DrainGrace + time.Second)
|
||||
}
|
||||
if rep, ok := replicaByID(h, old); !ok || rep.State != Ready {
|
||||
t.Fatalf("the only replica was stopped for a replacement that fits nowhere: %+v", rep)
|
||||
}
|
||||
if st := h.r.RestartState(old); !strings.HasPrefix(st, "refused: ") || !strings.Contains(st, "fits on no node") {
|
||||
t.Errorf("restart state %q should say it was given up, and why", st)
|
||||
}
|
||||
// Asked again, it is tried again.
|
||||
if err := h.r.Restart(old); err != nil || h.r.RestartState(old) != "queued" {
|
||||
t.Errorf("a new restart did not clear the refusal: %q, %v", h.r.RestartState(old), err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartAskedInPlaceStopsFirst(t *testing.T) {
|
||||
// Room beside it, but in place was asked for: an engine that must not
|
||||
// run twice, or memory somebody knows is short.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 4), gpuNode("2", 4))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
old := h.r.ReadyReplicas()[0].ID
|
||||
if err := h.r.RestartInPlace(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.Reconcile(t.Context())
|
||||
if rep, ok := replicaByID(h, old); ok && rep.State == Ready {
|
||||
t.Fatalf("asked in place, it kept serving beside its replacement: %+v", rep)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
|
||||
h := newHarness(t, testModel(2), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
|
||||
for i := 0; i < 3; i++ {
|
||||
|
||||
@@ -6,17 +6,34 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// retire finishes restarting replicas.
|
||||
//
|
||||
// A retiring replica keeps serving until its replacement is ready, and is then
|
||||
// drained like any other. When no replacement could be placed at all —
|
||||
// nowhere else has the memory, which on a small CPU fleet is the usual case —
|
||||
// it is stopped after DrainGrace so its replacement can take its place: a
|
||||
// restart in place, with the model unserved while it loads if it had no other
|
||||
// replica. Said in the log, since that is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
|
||||
// drained like any other. When no replacement could be placed beside it, what
|
||||
// happens depends on what is in the way:
|
||||
//
|
||||
// - the replica itself — one node, or every card that fits held: it is
|
||||
// stopped now, and its replacement takes its place. Waiting would change
|
||||
// nothing; it used to wait DrainGrace anyway, five minutes reading only
|
||||
// "restart replacing", which looked like a restart that never would.
|
||||
// - more than it: the replacement fits nowhere even with it stopped — its
|
||||
// profile was edited to need more than any node has free. Stopping it
|
||||
// would leave the model with nothing, so the restart is given up, and
|
||||
// says why where the replica is shown.
|
||||
// - something being moved out of the way (a preemption, an idle model
|
||||
// stopping): waited for.
|
||||
// - nothing tried this pass (a backoff): the old rule, stopped after
|
||||
// DrainGrace with no replacement in sight.
|
||||
//
|
||||
// A restart asked for in place (ctl restart --in-place) stops first, always.
|
||||
// Each stop is said in the log, since it is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, m catalog.Model, ids []string, readyLive, live, desired int,
|
||||
outcome scaleOutcome, parked bool,
|
||||
) {
|
||||
for _, id := range ids {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -24,6 +41,7 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
|
||||
if ok {
|
||||
since = rep.RetiringAt
|
||||
}
|
||||
inPlace := r.restartInPlace[id]
|
||||
r.mu.Unlock()
|
||||
if !ok {
|
||||
continue
|
||||
@@ -31,15 +49,48 @@ func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live,
|
||||
switch {
|
||||
case readyLive >= desired:
|
||||
r.drainForRestart(ctx, id)
|
||||
case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no room for its replacement beside it",
|
||||
case inPlace:
|
||||
r.log.Warn("restarting a replica in place, as asked: stopped before its replacement starts", "replica", id)
|
||||
r.stopForRestart(ctx, id)
|
||||
case live >= desired:
|
||||
// A replacement is placed and loading; it is waited for.
|
||||
case parked:
|
||||
r.refuseRestart(id, "its replacements fail to start; the model is parked (llmgw ctl retry "+m.Name+")")
|
||||
case outcome == noRoom && r.fitsWithout(m, nil, []victim{{ID: id, Model: m.Name}}):
|
||||
r.log.Warn("restarting a replica in place: it is the only thing in its replacement's way",
|
||||
"replica", id, "model", m.Name)
|
||||
r.stopForRestart(ctx, id)
|
||||
case outcome == noRoom || outcome == overLimit:
|
||||
r.refuseRestart(id, "its replacement fits on no node even with this replica stopped "+
|
||||
"(llmgw ctl explain-placement "+m.Name+")")
|
||||
case outcome == notTried && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no replacement beside it",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
r.stopForRestart(ctx, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// stopForRestart drains a replica and stops it at once, for its replacement
|
||||
// to take its place.
|
||||
func (r *Reconciler) stopForRestart(ctx context.Context, id string) {
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
|
||||
// refuseRestart gives up a replica's restart, leaving it serving, and keeps
|
||||
// why for whoever looks at the replica.
|
||||
func (r *Reconciler) refuseRestart(id, why string) {
|
||||
r.mu.Lock()
|
||||
if rep, ok := r.replicas[id]; ok {
|
||||
rep.RetiringAt = time.Time{}
|
||||
}
|
||||
delete(r.restartInPlace, id)
|
||||
r.restartRefused[id] = why
|
||||
r.mu.Unlock()
|
||||
r.log.Warn("not restarting a replica: it keeps serving", "replica", id, "because", why)
|
||||
}
|
||||
|
||||
func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -48,6 +99,7 @@ func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
|
||||
shards = slices.Clone(rep.Shards)
|
||||
}
|
||||
delete(r.restartInPlace, id)
|
||||
r.mu.Unlock()
|
||||
for _, s := range shards {
|
||||
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
|
||||
@@ -78,7 +130,14 @@ var ErrNoSuchReplica = errors.New("no ready replica with that id")
|
||||
// Restart queues replicas to be replaced, each by a new one started beside it
|
||||
// before it is stopped — to pick up an edited profile, an engine key, a new
|
||||
// engine version. Replicas of one model are replaced one at a time.
|
||||
func (r *Reconciler) Restart(ids ...string) error {
|
||||
func (r *Reconciler) Restart(ids ...string) error { return r.restart(false, ids) }
|
||||
|
||||
// RestartInPlace queues replicas to be replaced by stopping each first and
|
||||
// then starting its replacement: for when there is knowingly no room beside
|
||||
// it, and the outage while it loads is accepted.
|
||||
func (r *Reconciler) RestartInPlace(ids ...string) error { return r.restart(true, ids) }
|
||||
|
||||
func (r *Reconciler) restart(inPlace bool, ids []string) error {
|
||||
r.mu.Lock()
|
||||
for _, id := range ids {
|
||||
rep, ok := r.replicas[id]
|
||||
@@ -86,6 +145,10 @@ func (r *Reconciler) Restart(ids ...string) error {
|
||||
r.mu.Unlock()
|
||||
return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
|
||||
}
|
||||
delete(r.restartRefused, id)
|
||||
if inPlace {
|
||||
r.restartInPlace[id] = true
|
||||
}
|
||||
if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
|
||||
r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
|
||||
}
|
||||
@@ -95,8 +158,8 @@ func (r *Reconciler) Restart(ids ...string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// RestartState says where a replica's restart stands: "queued", "replacing",
|
||||
// or "" when none was asked for.
|
||||
// RestartState says where a replica's restart stands: "queued",
|
||||
// "replacing", "in place", "refused: why", or "" when none was asked for.
|
||||
func (r *Reconciler) RestartState(id string) string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
@@ -104,10 +167,14 @@ func (r *Reconciler) RestartState(id string) string {
|
||||
switch {
|
||||
case !ok:
|
||||
return ""
|
||||
case !rep.RetiringAt.IsZero() && r.restartInPlace[id]:
|
||||
return "in place"
|
||||
case !rep.RetiringAt.IsZero():
|
||||
return "replacing"
|
||||
case slices.Contains(r.restartQueue[rep.Model], id):
|
||||
return "queued"
|
||||
case r.restartRefused[id] != "":
|
||||
return "refused: " + r.restartRefused[id]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
+57
-14
@@ -10,6 +10,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
@@ -86,10 +87,12 @@ type Facts struct {
|
||||
|
||||
// What installing vLLM here would meet: the version this llmgw installs,
|
||||
// where, the space there, and whether it already has.
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
// DisabledEngines are those agent.yaml says this node never runs.
|
||||
DisabledEngines []string `json:"disabled_engines,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
}
|
||||
|
||||
// Modules checked, and why each matters.
|
||||
@@ -103,6 +106,10 @@ type Options struct {
|
||||
// VLLMVersion is the vLLM this llmgw installs; its prerequisites are
|
||||
// checked when set.
|
||||
VLLMVersion string
|
||||
// Fabrics are fabrics about to be declared, by name, with their
|
||||
// adapters: what `node add --fabric` will write into agent.yaml, which
|
||||
// does not exist yet when doctor runs before it.
|
||||
Fabrics map[string][]string
|
||||
}
|
||||
|
||||
// Collect reads the node.
|
||||
@@ -183,6 +190,14 @@ func Collect(ctx context.Context, o Options) Facts {
|
||||
}
|
||||
|
||||
f.readConfig(o)
|
||||
if len(o.Fabrics) > 0 {
|
||||
declared := map[string][]string{}
|
||||
for _, c := range f.Fabrics {
|
||||
declared[c.Fabric] = c.Declared
|
||||
}
|
||||
maps.Copy(declared, o.Fabrics)
|
||||
f.Fabrics = discovery.CheckFabrics(declared, f.HCAs)
|
||||
}
|
||||
f.Host.MinComputeCapability = discovery.MinComputeCapability(f.GPUs)
|
||||
if f.VLLMVersion = o.VLLMVersion; f.VLLMVersion != "" {
|
||||
if f.EnginesDir == "" {
|
||||
@@ -238,6 +253,7 @@ func (f *Facts) readConfig(o Options) {
|
||||
_, err = net.InterfaceByName(f.MgmtIface)
|
||||
f.IfaceExists = err == nil
|
||||
f.EnginesDir = cfg.EnginesDir
|
||||
f.DisabledEngines = cfg.Engines.Disabled
|
||||
f.LaunchMode = cfg.Engines.VLLM.LaunchMode
|
||||
if f.LaunchMode == "" {
|
||||
f.LaunchMode = "mp"
|
||||
@@ -419,8 +435,15 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
switch {
|
||||
case p.Active() && p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID < 0:
|
||||
add(Fail, "fabric", where+" is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang",
|
||||
"give the interface an IPv4 address, or check the adapter's RoCE mode")
|
||||
// Fatal only for a port a declared fabric uses. An adapter
|
||||
// nothing is configured to use — a RoCE NIC on a node added
|
||||
// for single-node inference — kept nodes out for a fabric
|
||||
// nobody asked for.
|
||||
msg := where + " is RoCE with no RoCE v2 GID entry: NCCL would use a v1 entry and hang"
|
||||
if declared[h.Name] == "" {
|
||||
msg += "; no declared fabric uses it, so nothing will until one does"
|
||||
}
|
||||
add(level, "fabric", msg, "give the interface an IPv4 address, or check the adapter's RoCE mode")
|
||||
case p.Active():
|
||||
on := ""
|
||||
switch {
|
||||
@@ -430,17 +453,24 @@ func Evaluate(f Facts) []Finding {
|
||||
on = ", network " + p.Network
|
||||
}
|
||||
add(OK, "fabric", fmt.Sprintf("%s %s at %d Gb/s (%s)%s", where, p.State, p.RateGbps, p.LinkLayer, on), "")
|
||||
// Questions about which fabric a port is on matter to the
|
||||
// fabrics that use it; for a port none does they are noted,
|
||||
// not warned of.
|
||||
quiet := Warn
|
||||
if declared[h.Name] == "" {
|
||||
quiet = Info
|
||||
}
|
||||
if p.LinkLayer == discovery.LinkInfiniBand && p.SM == nil {
|
||||
why := f.SMErrors[where]
|
||||
fix := ""
|
||||
if strings.Contains(why, "infiniband-diags") {
|
||||
fix = "install infiniband-diags"
|
||||
}
|
||||
add(Warn, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
|
||||
add(quiet, "fabric", where+": which fabric it is on cannot be told — "+orUnknown(why)+
|
||||
"; without it, separate fabrics given one name cannot be caught", fix)
|
||||
}
|
||||
if p.LinkLayer == discovery.LinkInfiniBand && p.SMLID == 0 {
|
||||
add(Warn, "fabric", where+" is active but reports no subnet manager LID", "")
|
||||
add(quiet, "fabric", where+" is active but reports no subnet manager LID", "")
|
||||
}
|
||||
case strings.EqualFold(p.PhysState, "LinkUp"):
|
||||
add(level, "fabric", where+" has a link but is "+p.State+": no subnet manager has configured it",
|
||||
@@ -479,7 +509,14 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
if len(f.HCAs) > 0 {
|
||||
if !f.Modules["ib_uverbs"] {
|
||||
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
|
||||
// RDMA is for replicas spanning nodes; a node that declares no
|
||||
// fabric runs none, and is a node without it.
|
||||
if len(declared) > 0 {
|
||||
add(Fail, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA", "modprobe ib_uverbs")
|
||||
} else {
|
||||
add(Warn, "fabric", "ib_uverbs is not loaded: nothing in user space can use RDMA, which a fabric declared "+
|
||||
"later would need; this node runs single-node replicas without it", "modprobe ib_uverbs")
|
||||
}
|
||||
}
|
||||
if len(f.GPUs) > 0 && !f.Modules["nvidia_peermem"] && !f.Modules["nv_peer_mem"] {
|
||||
add(Warn, "fabric", "nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory",
|
||||
@@ -525,7 +562,9 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
}
|
||||
// --- engines ---
|
||||
if f.VLLMVersion != "" {
|
||||
// An engine disabled on this node: whether it could be installed is not
|
||||
// a question anybody is asking.
|
||||
if f.VLLMVersion != "" && !slices.Contains(f.DisabledEngines, "vllm") {
|
||||
vllmFindings(f, add)
|
||||
}
|
||||
|
||||
@@ -542,22 +581,26 @@ func Evaluate(f Facts) []Finding {
|
||||
// the same decision the agent makes, so the report and the node agree.
|
||||
func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
v := f.VLLMVersion
|
||||
b, err := provision.Recipes["vllm"].Choose(v, f.Host)
|
||||
b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
|
||||
var nh provision.NotHere
|
||||
switch {
|
||||
case errors.As(err, &nh):
|
||||
add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "")
|
||||
return
|
||||
case err != nil:
|
||||
add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(),
|
||||
"upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one")
|
||||
// A warning, not a failure: the node is still a node, and serves
|
||||
// llama.cpp. Refusing it here kept older GPU generations out of a
|
||||
// cluster for an engine nobody meant to run on them.
|
||||
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
|
||||
"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
|
||||
"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
return
|
||||
case f.VLLMInstalled:
|
||||
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
|
||||
return
|
||||
}
|
||||
if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) {
|
||||
add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB",
|
||||
add(Warn, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB; the node joins without it until there is",
|
||||
v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB),
|
||||
"free space there, or point engines_dir in agent.yaml at a larger disk")
|
||||
return
|
||||
|
||||
@@ -151,8 +151,15 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
|
||||
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"},
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// What CUDA 13 cannot run, its CUDA 12.9 build of the same version can.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
}
|
||||
@@ -200,3 +207,38 @@ func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) {
|
||||
t.Errorf("ARM: %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNotChecked(t *testing.T) {
|
||||
// The field: a V100 node meant for llama.cpp, refused for a vLLM it was
|
||||
// never going to run.
|
||||
f := Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", DisabledEngines: []string{"vllm"}}
|
||||
for _, fd := range Evaluate(f) {
|
||||
if fd.Area == "engines" {
|
||||
t.Errorf("a disabled vLLM was reported on: %s %q", fd.Level, fd.Message)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
|
||||
// The field: a RoCE NIC with no v2 GID on a node added for single-node
|
||||
// inference, no fabric given, and the node refused for it.
|
||||
roce := discovery.HCA{Name: "mlx5_2", Ports: []discovery.Port{{Num: 1, State: "ACTIVE",
|
||||
PhysState: "LinkUp", RateGbps: 100, LinkLayer: discovery.LinkEthernet, RoCEv2GID: -1}}}
|
||||
f := healthy()
|
||||
f.HCAs = append(f.HCAs, roce)
|
||||
f.Modules["ib_uverbs"] = false
|
||||
f.Fabrics = nil // nothing declared
|
||||
fs := Evaluate(f)
|
||||
if fails := levels(fs, Fail); len(fails) > 0 {
|
||||
t.Errorf("an undeclared adapter failed the node: %v", fails)
|
||||
}
|
||||
has(t, fs, Warn, "no declared fabric uses it")
|
||||
has(t, fs, Warn, "runs single-node replicas without it")
|
||||
|
||||
// Declared, the same adapter and module are what the fabric needs.
|
||||
f.Fabrics = discovery.CheckFabrics(map[string][]string{"roce-a": {"mlx5_2"}}, f.HCAs)
|
||||
fs = Evaluate(f)
|
||||
has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
|
||||
has(t, fs, Fail, "ib_uverbs is not loaded")
|
||||
}
|
||||
@@ -25,6 +25,10 @@ type Header struct {
|
||||
Version uint32
|
||||
Architecture string
|
||||
Name string
|
||||
// Type is general.type: "model", or "adapter" for a LoRA adapter, which
|
||||
// is applied to a model rather than run on its own. Empty in files made
|
||||
// before the key existed, which are models.
|
||||
Type string
|
||||
|
||||
// The shape the KV cache is sized from, from the architecture's own keys
|
||||
// (llama.block_count, qwen2.attention.head_count_kv, ...). Zero when the
|
||||
@@ -235,6 +239,8 @@ func Read(r io.Reader) (Header, error) {
|
||||
h.Architecture = v
|
||||
case "general.name":
|
||||
h.Name = v
|
||||
case "general.type":
|
||||
h.Type = v
|
||||
}
|
||||
if h.Architecture != "" {
|
||||
switch strings.TrimPrefix(key, h.Architecture+".") {
|
||||
@@ -290,6 +296,41 @@ func FromURL(ctx context.Context, client *http.Client, url string) (Header, erro
|
||||
return Read(io.LimitReader(resp.Body, headerBytes))
|
||||
}
|
||||
|
||||
// IsAdapter reports whether the file is a LoRA adapter rather than a model.
|
||||
func (h Header) IsAdapter() bool { return h.Type == "adapter" }
|
||||
|
||||
// adapterProbe is how much of a file is read to tell an adapter from a model:
|
||||
// general.type comes before the tokenizer, which is what makes a model's
|
||||
// header megabytes long.
|
||||
const adapterProbe = 256 << 10
|
||||
|
||||
// KindFromURL reads just enough of a GGUF file to say what it is — its type
|
||||
// and architecture — over HTTP. A header cut short by the probe is fine: the
|
||||
// keys asked for come first.
|
||||
func KindFromURL(ctx context.Context, client *http.Client, url string) (Header, error) {
|
||||
if client == nil {
|
||||
client = &http.Client{Timeout: 30 * time.Second}
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return Header{}, err
|
||||
}
|
||||
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", adapterProbe-1))
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return Header{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
|
||||
return Header{}, fmt.Errorf("%s answered %s", url, resp.Status)
|
||||
}
|
||||
h, err := Read(io.LimitReader(resp.Body, adapterProbe))
|
||||
if h.Architecture != "" {
|
||||
return h, nil // what was asked for was read, whatever came after
|
||||
}
|
||||
return h, err
|
||||
}
|
||||
|
||||
// IsGGUF reports whether a source names a GGUF file, or a directory of them.
|
||||
func IsGGUF(source string) bool {
|
||||
s := strings.ToLower(strings.TrimSuffix(source, "/"))
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
package gguf
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Run by hand against the real adapter this was written for:
|
||||
//
|
||||
// LLMGW_LIVE_GGUF=1 go test ./internal/gguf -run LiveAdapter -v
|
||||
func TestLiveAdapterIsAnAdapter(t *testing.T) {
|
||||
if os.Getenv("LLMGW_LIVE_GGUF") == "" {
|
||||
t.Skip("set LLMGW_LIVE_GGUF to read a real adapter from Hugging Face")
|
||||
}
|
||||
h, err := KindFromURL(t.Context(), nil,
|
||||
"https://huggingface.co/ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF/resolve/main/LoRA-Qwen2.5-Coder-7B-Instruct-f16.gguf")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !h.IsAdapter() || h.Architecture != "qwen2" {
|
||||
t.Errorf("%+v", h)
|
||||
}
|
||||
m, err := KindFromURL(t.Context(), nil,
|
||||
"https://huggingface.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m.IsAdapter() || m.Architecture != "qwen2" {
|
||||
t.Errorf("a model read as %+v", m)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package gguf
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// adapterHeader is the start of a LoRA adapter as llama.cpp's converter
|
||||
// writes it, followed by padding that stands in for its tensors.
|
||||
func adapterHeader(kind string, pad int) []byte {
|
||||
var b bytes.Buffer
|
||||
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
|
||||
str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
|
||||
b.WriteString("GGUF")
|
||||
w(uint32(3))
|
||||
w(uint64(0))
|
||||
w(uint64(3))
|
||||
str("general.architecture")
|
||||
w(uint32(tString))
|
||||
str("qwen2")
|
||||
str("general.type")
|
||||
w(uint32(tString))
|
||||
str(kind)
|
||||
str("adapter.type")
|
||||
w(uint32(tString))
|
||||
str("lora")
|
||||
b.Write(make([]byte, pad))
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func TestAnAdapterIsToldFromAModelByItsFirstBytes(t *testing.T) {
|
||||
var asked string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
asked = r.Header.Get("Range")
|
||||
w.WriteHeader(http.StatusPartialContent)
|
||||
if r.URL.Path == "/adapter.gguf" {
|
||||
_, _ = w.Write(adapterHeader("adapter", 1<<20))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write(header("qwen2"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
h, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/adapter.gguf")
|
||||
if err != nil || !h.IsAdapter() || h.Architecture != "qwen2" {
|
||||
t.Errorf("an adapter read as %+v, %v", h, err)
|
||||
}
|
||||
if asked != "bytes=0-262143" {
|
||||
t.Errorf("asked for %q: the probe should read the start only", asked)
|
||||
}
|
||||
m, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/model.gguf")
|
||||
if err != nil || m.IsAdapter() || m.Architecture != "qwen2" {
|
||||
t.Errorf("a model read as %+v, %v", m, err)
|
||||
}
|
||||
}
|
||||
+43
-1
@@ -156,6 +156,10 @@ type Query struct {
|
||||
// applies it, so a page of small models is a full page.
|
||||
MaxParams int64
|
||||
Limit int
|
||||
// Filters are further registry tags every result must carry, such as
|
||||
// "lora", or "base_model:adapter:Qwen/Qwen2.5-7B-Instruct" for the
|
||||
// adapters made for one model.
|
||||
Filters []string
|
||||
}
|
||||
|
||||
// exploreFields is everything a listing shows. Asking for any field with
|
||||
@@ -188,7 +192,10 @@ func (c *Client) Explore(ctx context.Context, q Query) ([]Model, error) {
|
||||
v.Set("pipeline_tag", q.Task)
|
||||
}
|
||||
if q.GGUF {
|
||||
v.Set("filter", "gguf")
|
||||
v.Add("filter", "gguf")
|
||||
}
|
||||
for _, f := range q.Filters {
|
||||
v.Add("filter", f)
|
||||
}
|
||||
if q.MaxParams > 0 {
|
||||
v.Set("num_parameters", fmt.Sprintf("min:0,max:%d", q.MaxParams))
|
||||
@@ -251,6 +258,41 @@ func (f File) IsWeights() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// companionNames are the files beside safetensors weights that an engine
|
||||
// loading the directory reads: the model's shape, its tokenizer, how it was
|
||||
// quantised, and which shard holds which tensor.
|
||||
var companionNames = map[string]bool{
|
||||
"config.json": true, "generation_config.json": true,
|
||||
"tokenizer.json": true, "tokenizer_config.json": true, "tokenizer.model": true,
|
||||
"special_tokens_map.json": true, "added_tokens.json": true,
|
||||
"vocab.json": true, "vocab.txt": true, "merges.txt": true,
|
||||
"chat_template.jinja": true, "chat_template.json": true,
|
||||
"preprocessor_config.json": true, "processor_config.json": true,
|
||||
"quantize_config.json": true, "quant_config.json": true,
|
||||
}
|
||||
|
||||
// Companions are the files of a repository an engine reads beside a
|
||||
// safetensors set: config.json and the tokenizer, the shard index. Only
|
||||
// those at the top: the directory an engine is given is one level deep.
|
||||
//
|
||||
// Never code (a .py): a model that needs its own Python to load asks the
|
||||
// engine to run what the repository ships, which is a decision for whoever
|
||||
// runs the engine, made on purpose, not a file swept in with the tokenizer.
|
||||
func Companions(files []File) []File {
|
||||
var out []File
|
||||
for _, f := range files {
|
||||
name := f.Path
|
||||
if strings.Contains(name, "/") {
|
||||
continue
|
||||
}
|
||||
if companionNames[name] || strings.HasSuffix(name, ".safetensors.index.json") ||
|
||||
strings.HasSuffix(name, ".tiktoken") {
|
||||
out = append(out, f)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Repo is a repository's current revision and what is in it.
|
||||
type Repo struct {
|
||||
ID string
|
||||
|
||||
@@ -94,6 +94,13 @@ func addWith(ctx context.Context, client *http.Client, root, source string, e En
|
||||
_ = os.Remove(part)
|
||||
return Added{}, err
|
||||
}
|
||||
// The checksum beside the file, before the file is in place: a manifest
|
||||
// listing the file then always has its checksum to hand on, and a node
|
||||
// can tell a file damaged on the way from the one that was mirrored.
|
||||
if err := writeChecksum(final, sum); err != nil {
|
||||
_ = os.Remove(part)
|
||||
return Added{}, err
|
||||
}
|
||||
if err := os.Rename(part, final); err != nil {
|
||||
return Added{}, err
|
||||
}
|
||||
@@ -146,6 +153,9 @@ func Remove(root string, e Entry) error {
|
||||
if err := os.Remove(path); err != nil {
|
||||
return err
|
||||
}
|
||||
// Its checksum goes with it; left, it would keep the directory from
|
||||
// being empty, and so from going.
|
||||
_ = os.Remove(checksumPath(path))
|
||||
versionDir := filepath.Dir(path)
|
||||
if empty, _ := isEmpty(versionDir); empty {
|
||||
_ = os.Remove(versionDir)
|
||||
|
||||
+102
-2
@@ -16,12 +16,15 @@ package mirror
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
@@ -91,7 +94,7 @@ func List(root string) ([]Entry, error) {
|
||||
continue
|
||||
}
|
||||
for _, f := range files {
|
||||
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) {
|
||||
if f.IsDir() || strings.HasSuffix(f.Name(), partSuffix) || strings.HasPrefix(f.Name(), ".") {
|
||||
continue
|
||||
}
|
||||
info, err := f.Info()
|
||||
@@ -169,6 +172,31 @@ func normalizeURL(u string) string {
|
||||
|
||||
const partSuffix = ".part"
|
||||
|
||||
// checksumPath is where a file's sha256 is kept: beside it, hidden, so no
|
||||
// listing takes it for a file of the model and no name a person gives a
|
||||
// file (which may not start with a dot) can be the same.
|
||||
func checksumPath(file string) string {
|
||||
return filepath.Join(filepath.Dir(file), "."+filepath.Base(file)+".sha256")
|
||||
}
|
||||
|
||||
func writeChecksum(file, sum string) error {
|
||||
tmp := checksumPath(file) + partSuffix
|
||||
if err := os.WriteFile(tmp, []byte(sum+"\n"), 0o644); err != nil { //nolint:gosec // a checksum, for anyone to read
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, checksumPath(file))
|
||||
}
|
||||
|
||||
// readChecksum is a file's recorded sha256, or "" for a file put in place by
|
||||
// hand, which has none.
|
||||
func readChecksum(file string) string {
|
||||
b, err := os.ReadFile(checksumPath(file)) //nolint:gosec // beside a file in the mirror
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
// sha256File is the checksum of a file on disk.
|
||||
//
|
||||
// Reported after a download so an operator has something to compare against
|
||||
@@ -205,8 +233,16 @@ type Manifest struct {
|
||||
type ManifestFile struct {
|
||||
Name string `json:"name"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
// SHA256 is the file's checksum when the mirror fetched it, for a node
|
||||
// to check what it downloaded; empty for a file put there by hand.
|
||||
SHA256 string `json:"sha256,omitempty"`
|
||||
}
|
||||
|
||||
// ErrIncomplete is a version not yet whole: a download still running, or a
|
||||
// shard its own index names missing. Its manifest is not given out, so no
|
||||
// node starts a model from part of it.
|
||||
var ErrIncomplete = errors.New("this version is not complete")
|
||||
|
||||
// ManifestFor lists the files of one model version.
|
||||
//
|
||||
// Built from the directory rather than from a file written alongside it. A
|
||||
@@ -219,15 +255,79 @@ func ManifestFor(root, model, version string) (Manifest, error) {
|
||||
return Manifest{}, err
|
||||
}
|
||||
m := Manifest{Model: model, Version: version}
|
||||
dir := filepath.Join(root, model, version)
|
||||
for _, e := range entries {
|
||||
if e.Model != model || e.Version != version {
|
||||
continue
|
||||
}
|
||||
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes})
|
||||
m.Files = append(m.Files, ManifestFile{Name: e.File, Bytes: e.Bytes,
|
||||
SHA256: readChecksum(filepath.Join(dir, e.File))})
|
||||
m.Bytes += e.Bytes
|
||||
}
|
||||
if len(m.Files) == 0 {
|
||||
return Manifest{}, fmt.Errorf("%s/%s: %w", model, version, os.ErrNotExist)
|
||||
}
|
||||
if err := complete(dir, m.Files); err != nil {
|
||||
return Manifest{}, fmt.Errorf("%s/%s: %w: %v", model, version, ErrIncomplete, err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// complete says whether a version's files are the whole of it: nothing still
|
||||
// downloading, every shard a safetensors index maps a tensor to, and every
|
||||
// part of a split ("-00002-of-00004").
|
||||
//
|
||||
// From what the files say of each other rather than from a list written when
|
||||
// the fetch began, so a version filled by hand is held to the same.
|
||||
func complete(dir string, files []ManifestFile) error {
|
||||
if parts, _ := filepath.Glob(filepath.Join(dir, "*"+partSuffix)); len(parts) > 0 {
|
||||
var names []string
|
||||
for _, p := range parts {
|
||||
if b := filepath.Base(p); !strings.HasPrefix(b, ".") {
|
||||
names = append(names, strings.TrimSuffix(b, partSuffix))
|
||||
}
|
||||
}
|
||||
if len(names) > 0 {
|
||||
return fmt.Errorf("still downloading %s", strings.Join(names, ", "))
|
||||
}
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, f := range files {
|
||||
have[f.Name] = true
|
||||
}
|
||||
var missing []string
|
||||
want := map[string]bool{}
|
||||
for _, f := range files {
|
||||
if strings.HasSuffix(f.Name, ".safetensors.index.json") {
|
||||
var idx struct {
|
||||
WeightMap map[string]string `json:"weight_map"`
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, f.Name)) //nolint:gosec // a file in the mirror
|
||||
if err != nil || json.Unmarshal(b, &idx) != nil {
|
||||
return fmt.Errorf("%s cannot be read", f.Name)
|
||||
}
|
||||
for _, shard := range idx.WeightMap {
|
||||
want[shard] = true
|
||||
}
|
||||
}
|
||||
if m := splitPart.FindStringSubmatch(f.Name); m != nil {
|
||||
total, _ := strconv.Atoi(m[3])
|
||||
for i := 1; i <= total; i++ {
|
||||
want[fmt.Sprintf("%s-%0*d-of-%s%s", m[1], len(m[2]), i, m[3], m[4])] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
for name := range want {
|
||||
if !have[name] {
|
||||
missing = append(missing, name)
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
sort.Strings(missing)
|
||||
return fmt.Errorf("missing %s", strings.Join(missing, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// splitPart matches one part of a split: prefix, part, total, extension.
|
||||
var splitPart = regexp.MustCompile(`^(.+)-(\d{5})-of-(\d{5})(\.[A-Za-z0-9]+)$`)
|
||||
@@ -177,3 +177,63 @@ func TestANameThatLeavesTheMirrorIsRefused(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAVersionIsGivenOutOnlyWhole(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "m", "v1")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write := func(name, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
write("config.json", "{}")
|
||||
write("model.safetensors.index.json",
|
||||
`{"weight_map":{"a":"model-00001-of-00002.safetensors","b":"model-00002-of-00002.safetensors"}}`)
|
||||
write("model-00001-of-00002.safetensors", "aa")
|
||||
write("model-00002-of-00002.safetensors.part", "b")
|
||||
|
||||
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "still downloading") {
|
||||
t.Fatalf("a version with a shard downloading was given out: %v", err)
|
||||
}
|
||||
// The download failed and its .part went: the index still names it.
|
||||
_ = os.Remove(filepath.Join(dir, "model-00002-of-00002.safetensors.part"))
|
||||
if _, err := ManifestFor(root, "m", "v1"); !errors.Is(err, ErrIncomplete) ||
|
||||
!strings.Contains(err.Error(), "missing model-00002-of-00002.safetensors") {
|
||||
t.Fatalf("a version missing a shard its index names was given out: %v", err)
|
||||
}
|
||||
write("model-00002-of-00002.safetensors", "bb")
|
||||
m, err := ManifestFor(root, "m", "v1")
|
||||
if err != nil || len(m.Files) != 4 {
|
||||
t.Fatalf("the whole version: %+v, %v", m, err)
|
||||
}
|
||||
|
||||
// A split GGUF with no index is held to its own numbering.
|
||||
gg := filepath.Join(root, "g", "v1")
|
||||
_ = os.MkdirAll(gg, 0o755)
|
||||
_ = os.WriteFile(filepath.Join(gg, "q-00001-of-00003.gguf"), []byte("x"), 0o644)
|
||||
_ = os.WriteFile(filepath.Join(gg, "q-00003-of-00003.gguf"), []byte("x"), 0o644)
|
||||
if _, err := ManifestFor(root, "g", "v1"); !errors.Is(err, ErrIncomplete) || !strings.Contains(err.Error(), "q-00002-of-00003.gguf") {
|
||||
t.Fatalf("a split missing its second part was given out: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMirroredFilesChecksumIsHandedOnAndHiddenFromListings(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
e := Entry{Model: "m", Version: "v1", File: "f.gguf"}
|
||||
added, err := Add(context.Background(), root, serving(t, "weights"), e, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := List(root)
|
||||
if len(entries) != 1 {
|
||||
t.Errorf("the checksum was listed as a file of the model: %+v", entries)
|
||||
}
|
||||
m, err := ManifestFor(root, "m", "v1")
|
||||
if err != nil || m.Files[0].SHA256 != added.SHA256 || added.SHA256 == "" {
|
||||
t.Errorf("manifest %+v, checksum %q: %v", m, added.SHA256, err)
|
||||
}
|
||||
}
|
||||
@@ -18,7 +18,6 @@ import (
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -27,6 +26,8 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
@@ -86,6 +87,15 @@ type Options struct {
|
||||
Releases string
|
||||
// IgnoreChecks goes on when doctor finds a problem.
|
||||
IgnoreChecks bool
|
||||
// DisableEngines are engines this node never installs or advertises,
|
||||
// written into its agent.yaml, and not checked by doctor: vLLM on a node
|
||||
// of cards it does not run on, meant for llama.cpp alone. Narrower than
|
||||
// IgnoreChecks, which also passes over everything else.
|
||||
DisableEngines []string
|
||||
// VLLMVersion is the vLLM the control plane has nodes install, for
|
||||
// doctor to check against; "none" when it has them install none, empty
|
||||
// for doctor's own default.
|
||||
VLLMVersion string
|
||||
// SSHWait is how long an unreachable machine is waited for.
|
||||
SSHWait time.Duration
|
||||
|
||||
@@ -390,6 +400,18 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
if r.Opts.IBRoot != "" {
|
||||
args += " --ib-root " + shellQuote(r.Opts.IBRoot)
|
||||
}
|
||||
// The fabrics given, checked as the ones they will be; any adapter they
|
||||
// do not name is checked as one nothing uses.
|
||||
for _, fab := range r.Opts.Fabrics {
|
||||
args += " --fabric " + shellQuote(fab.Name+":"+strings.Join(fab.HCAs, ","))
|
||||
}
|
||||
switch {
|
||||
case slices.Contains(r.Opts.DisableEngines, "vllm"):
|
||||
args += " --vllm none"
|
||||
r.emit(StepCheck, Info, "vLLM is disabled on this node: it joins with llama.cpp alone", "")
|
||||
case r.Opts.VLLMVersion != "":
|
||||
args += " --vllm " + shellQuote(r.Opts.VLLMVersion)
|
||||
}
|
||||
out, err := r.Remote.Run(ctx, args, nil)
|
||||
var rep struct {
|
||||
Findings []doctor.Finding `json:"findings"`
|
||||
@@ -596,8 +618,11 @@ func AgentYAML(o Options, res *Result) string {
|
||||
fmt.Fprintf(&b, " ib_root: %q\n", o.IBRoot)
|
||||
}
|
||||
}
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" {
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" || len(o.DisableEngines) > 0 {
|
||||
b.WriteString("engines:\n")
|
||||
if len(o.DisableEngines) > 0 {
|
||||
fmt.Fprintf(&b, " disabled: [%s]\n", strings.Join(o.DisableEngines, ", "))
|
||||
}
|
||||
if o.LlamaCppBin != "" {
|
||||
fmt.Fprintf(&b, " llamacpp:\n bin: %q\n", o.LlamaCppBin)
|
||||
}
|
||||
|
||||
@@ -280,3 +280,44 @@ func said(events []Event, level, s string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnANodeIsWrittenAndNotChecked(t *testing.T) {
|
||||
m := newFake()
|
||||
r, events := runner(t, m, Options{Version: "dev", DisableEngines: []string{"vllm"}, VLLMVersion: "0.30.0"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if yaml := string(m.files["/etc/llmgw/agent.yaml"]); !strings.Contains(yaml, "engines:\n disabled: [vllm]") {
|
||||
t.Errorf("agent.yaml:\n%s", yaml)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm none") {
|
||||
t.Errorf("doctor was asked about a vLLM this node will never run: %v", m.scripts)
|
||||
}
|
||||
if !said(*events, Info, "joins with llama.cpp alone") {
|
||||
t.Error("the reduced engine set went unsaid")
|
||||
}
|
||||
|
||||
// Not disabled: doctor checks the vLLM the control plane installs, not
|
||||
// its own default.
|
||||
m = newFake()
|
||||
r, _ = runner(t, m, Options{Version: "dev", VLLMVersion: "0.25.1"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm '0.25.1'") && !m.ran("sudo llmgw doctor --json --vllm 0.25.1") {
|
||||
t.Errorf("doctor was not told the control plane's vLLM: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFabricsGivenAreCheckedAsDeclared(t *testing.T) {
|
||||
// agent.yaml is written after doctor runs, so the fabrics on the command
|
||||
// line are handed to doctor: checked strictly, and nothing else is.
|
||||
m := newFake()
|
||||
r, _ := runner(t, m, Options{Version: "dev", Fabrics: []Fabric{{Name: "ib-a", HCAs: []string{"mlx5_0", "mlx5_1"}}}})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("--fabric 'ib-a:mlx5_0,mlx5_1'") {
|
||||
t.Errorf("doctor was not told the fabric: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func adapterModel() catalog.Model {
|
||||
m := catModel("coder", nil)
|
||||
m.Profiles = []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4}}
|
||||
m.Adapters = []catalog.Adapter{
|
||||
{Name: "coder-sql", Source: "http://mirror/sql.gguf"},
|
||||
{Name: "coder-docs", Source: "http://mirror/docs.gguf", Scale: 0.5},
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func TestAnAdapterIsAskedForByNameAndAppliedOnItsModel(t *testing.T) {
|
||||
e := newFakeEngine(t, "coder")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"coder"}})
|
||||
cfg.Models = []catalog.Model{adapterModel()}
|
||||
rt := newTestRouter(t, cfg)
|
||||
|
||||
sent := func(model string) map[string]any {
|
||||
t.Helper()
|
||||
rec := post(t, rt.Handler(), map[string]any{"model": model, "messages": []any{}}, nil)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("%s: status %d: %s", model, rec.Code, rec.Body.String())
|
||||
}
|
||||
var body map[string]any
|
||||
_ = json.Unmarshal(e.last.Load().([]byte), &body)
|
||||
return body
|
||||
}
|
||||
body := sent("coder-docs")
|
||||
if body["model"] != "coder" {
|
||||
t.Errorf("the engine was asked for %v, not the model the adapter runs on", body["model"])
|
||||
}
|
||||
lora, _ := json.Marshal(body["lora"])
|
||||
if string(lora) != `[{"id":1,"scale":0.5}]` {
|
||||
t.Errorf("lora %s, expected the second adapter at its scale", lora)
|
||||
}
|
||||
if body := sent("coder-sql"); func() string { b, _ := json.Marshal(body["lora"]); return string(b) }() != `[{"id":0,"scale":1}]` {
|
||||
t.Errorf("the first adapter, at the default scale: %v", body["lora"])
|
||||
}
|
||||
if body := sent("coder"); body["lora"] != nil {
|
||||
t.Errorf("the base model was sent an adapter: %v", body["lora"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAKeyAllowedAnAdapterReachesThatAdapterOnly(t *testing.T) {
|
||||
cfg := baseConfig()
|
||||
cfg.Models = []catalog.Model{adapterModel()}
|
||||
rt := newTestRouter(t, cfg)
|
||||
key := func(allowed ...string) auth.Principal {
|
||||
return auth.Principal{Limits: store.Effective{AllowedModels: allowed}}
|
||||
}
|
||||
for _, c := range []struct {
|
||||
allowed []string
|
||||
requested string
|
||||
want bool
|
||||
}{
|
||||
{[]string{"coder-sql"}, "coder-sql", true},
|
||||
{[]string{"coder-sql"}, "coder", false},
|
||||
{[]string{"coder-sql"}, "coder-docs", false},
|
||||
{[]string{"coder"}, "coder-sql", true}, // the model allows its adapters
|
||||
{[]string{"coder"}, "coder", true},
|
||||
} {
|
||||
if got := rt.permits(key(c.allowed...), c.requested); got != c.want {
|
||||
t.Errorf("allowed %v, asked for %s: %v", c.allowed, c.requested, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdaptersShareTheNamespaceOfModels(t *testing.T) {
|
||||
a := adapterModel()
|
||||
b := catModel("coder-sql", nil)
|
||||
if _, err := catalog.New([]catalog.Model{a, b}); err == nil {
|
||||
t.Error("an adapter named like another model was accepted")
|
||||
}
|
||||
bad := adapterModel()
|
||||
bad.Profiles = []catalog.Profile{{Name: "gpu", Engine: "vllm", GPUClass: "h100-80g", TP: 1}}
|
||||
if _, err := catalog.New([]catalog.Model{bad}); err == nil {
|
||||
t.Error("adapters on a vLLM profile were accepted")
|
||||
}
|
||||
c, err := catalog.New([]catalog.Model{adapterModel()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m, ad, i, ok := c.ResolveAdapter("coder-docs"); !ok || m.Name != "coder" || i != 1 || ad.EffectiveScale() != 0.5 {
|
||||
t.Errorf("resolve: %v %+v %d %v", m.Name, ad, i, ok)
|
||||
}
|
||||
}
|
||||
@@ -37,8 +37,16 @@ type Info struct {
|
||||
Weight float64
|
||||
// Priority expresses pool preference; higher pools are tried first.
|
||||
Priority int
|
||||
// UpstreamModel is what the endpoint calls the model, when that differs
|
||||
// from the catalog's name: an external provider's. Requests are rewritten
|
||||
// to it. Empty sends the name as the client did.
|
||||
UpstreamModel string
|
||||
}
|
||||
|
||||
// EngineExternal marks an endpoint that is a provider outside the cluster.
|
||||
// It publishes no metrics, so it is never scraped and never reads as stale.
|
||||
const EngineExternal = "external"
|
||||
|
||||
// Serves reports whether the replica serves the given model.
|
||||
func (i Info) Serves(model string) bool {
|
||||
return slices.Contains(i.Models, model)
|
||||
@@ -242,7 +250,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
|
||||
continue
|
||||
}
|
||||
var age time.Duration
|
||||
stale := true
|
||||
stale := l.info.Engine != EngineExternal
|
||||
if !l.scrapedAt.IsZero() {
|
||||
age = now.Sub(l.scrapedAt)
|
||||
stale = age > r.cfg.StaleAfter
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
|
||||
func TestALocalModelWithNoCapacityFallsBackToAnExternalProvider(t *testing.T) {
|
||||
local := newFakeEngine(t, "local")
|
||||
local.status.Store(http.StatusServiceUnavailable)
|
||||
|
||||
var calls atomic.Int64
|
||||
var gotModel, gotAuth atomic.Value
|
||||
provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/metrics" {
|
||||
t.Error("the provider was scraped for metrics")
|
||||
}
|
||||
calls.Add(1)
|
||||
gotAuth.Store(r.Header.Get("Authorization"))
|
||||
var body map[string]any
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
_ = json.Unmarshal(b, &body)
|
||||
gotModel.Store(fmt.Sprint(body["model"]))
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprint(w, `{"model":"gpt-4o-mini","choices":[{"message":{"content":"from outside"}}],"usage":{"prompt_tokens":3,"completion_tokens":2}}`)
|
||||
}))
|
||||
t.Cleanup(provider.Close)
|
||||
|
||||
cfg := baseConfig(config.Endpoint{ID: "l", URL: local.srv.URL, Models: []string{"local"}})
|
||||
cloud := catalog.Model{Name: "cloud", Source: "external",
|
||||
External: &catalog.External{URL: provider.URL, Model: "gpt-4o-mini"}}
|
||||
cfg.Models = []catalog.Model{catModel("local", nil, "cloud"), cloud}
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "l", URL: local.srv.URL, Models: []string{"local"}},
|
||||
{ID: "external:cloud", URL: provider.URL, Models: []string{"cloud"}, APIKey: "provider-key",
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1, UpstreamModel: "gpt-4o-mini"},
|
||||
})
|
||||
|
||||
rec := post(t, rt.Handler(), map[string]any{"model": "local", "messages": []any{}}, nil)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Fatalf("the provider was called %d times", calls.Load())
|
||||
}
|
||||
if gotModel.Load() != "gpt-4o-mini" {
|
||||
t.Errorf("the provider was asked for %v, not its own name for the model", gotModel.Load())
|
||||
}
|
||||
if gotAuth.Load() != "Bearer provider-key" {
|
||||
t.Errorf("the provider got %v, not its key", gotAuth.Load())
|
||||
}
|
||||
if got := rec.Header().Get("x-llmgw-served-model"); got != "cloud" {
|
||||
t.Errorf("the client should be told it was served by cloud, got %q", got)
|
||||
}
|
||||
// Never scraped, and never stale for it.
|
||||
for _, s := range rt.reg.Snapshots("cloud") {
|
||||
if s.Stale {
|
||||
t.Error("an external endpoint reads as stale")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnExternalModelIsNotAModelWithProfiles(t *testing.T) {
|
||||
good := catalog.Model{Name: "cloud", Source: "external",
|
||||
External: &catalog.External{URL: "https://api.example.com", Model: "gpt-4o-mini"}}
|
||||
if _, err := catalog.New([]catalog.Model{good}); err != nil {
|
||||
t.Fatalf("a valid external model was refused: %v", err)
|
||||
}
|
||||
for name, m := range map[string]catalog.Model{
|
||||
"profiles": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b", Model: "m"},
|
||||
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp"}}},
|
||||
"no url": {Name: "x", Source: "external", External: &catalog.External{URL: "api.example.com", Model: "m"}},
|
||||
"no model": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b"}},
|
||||
} {
|
||||
if _, err := catalog.New([]catalog.Model{m}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -196,6 +196,11 @@ func (r *Router) Handler() http.Handler {
|
||||
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
|
||||
pages.SetCapacity(c)
|
||||
}
|
||||
// The engine build each GPU node installs, chosen on the cluster
|
||||
// page.
|
||||
if e, ok := r.clusterView.(web.NodeEngineAdmin); ok {
|
||||
pages.RegisterNodeEngines(mux, e)
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
@@ -426,6 +431,12 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
// around the allowlist; matching only the canonical name would silently ignore
|
||||
// an allowlist written in the names clients actually use.
|
||||
func (r *Router) permits(p auth.Principal, requested string) bool {
|
||||
// An adapter named in the allowlist allows that adapter: not its model,
|
||||
// and not the model's other adapters, which the checks below would let
|
||||
// through if the adapter's own name were not asked about first.
|
||||
if _, ok := r.adapter(requested); ok && p.Allows(requested) {
|
||||
return true
|
||||
}
|
||||
canonical, m := r.resolve(requested)
|
||||
if p.Allows(canonical) {
|
||||
return true
|
||||
@@ -984,3 +995,44 @@ func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
|
||||
w.Header().Set("x-ratelimit-reset-tokens", reset)
|
||||
}
|
||||
}
|
||||
|
||||
// withModel is body with its model set to name, for an endpoint that knows the
|
||||
// model by another name — an external provider. The body unchanged when it is
|
||||
// not a JSON object, which the endpoint then refuses as it would have anyway.
|
||||
func withModel(body []byte, name string) []byte {
|
||||
var m map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &m); err != nil {
|
||||
return body
|
||||
}
|
||||
encoded, err := json.Marshal(name)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
m["model"] = encoded
|
||||
out, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// withAdapter is body asking for model with a LoRA adapter applied, in
|
||||
// llama.cpp's form: "lora": [{"id": index, "scale": scale}]. Adapters not
|
||||
// listed stay at zero, the scale the engine loads each one at.
|
||||
func withAdapter(body []byte, model string, index int, scale float64) []byte {
|
||||
var m map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &m); err != nil {
|
||||
return body
|
||||
}
|
||||
name, err1 := json.Marshal(model)
|
||||
lora, err2 := json.Marshal([]map[string]any{{"id": index, "scale": scale}})
|
||||
if err1 != nil || err2 != nil {
|
||||
return body
|
||||
}
|
||||
m["model"], m["lora"] = name, lora
|
||||
out, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
return out
|
||||
}
|
||||
+45
-11
@@ -61,6 +61,8 @@ type Router struct {
|
||||
// router's half of not having to. A request takes the map once and works
|
||||
// from it, so a swap mid-request is invisible rather than a race.
|
||||
models atomic.Pointer[map[string]catalog.Model]
|
||||
// adapters are the LoRA adapters by the name clients ask for them by.
|
||||
adapters atomic.Pointer[map[string]adapterRef]
|
||||
|
||||
// extraMetrics are collectors other components publish on /metrics.
|
||||
extraMetrics []prometheus.Collector
|
||||
@@ -315,14 +317,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
// all. A control plane pushes over the top of this at startup and on every
|
||||
// change after.
|
||||
if len(cfg.Models) > 0 {
|
||||
seed := map[string]catalog.Model{}
|
||||
for _, m := range cfg.Models {
|
||||
seed[m.Name] = m
|
||||
for _, a := range m.Aliases {
|
||||
seed[a] = m
|
||||
}
|
||||
}
|
||||
r.models.Store(&seed)
|
||||
r.storeModels(cfg.Models)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
@@ -505,15 +500,44 @@ func (r *Router) modelMap() map[string]catalog.Model {
|
||||
// The catalog's counterpart to SetEndpoints, and pushed for the same reason:
|
||||
// the control plane owns what should exist, the router needs to know, and
|
||||
// neither has to be in the other's process for that to work.
|
||||
func (r *Router) SetCatalog(c *catalog.Catalog) {
|
||||
func (r *Router) SetCatalog(c *catalog.Catalog) { r.storeModels(c.Models()) }
|
||||
|
||||
// storeModels makes models what names resolve to: each model by its name, its
|
||||
// aliases and its adapters. One place for it, whether the models came from
|
||||
// configuration at start or from the control plane since.
|
||||
func (r *Router) storeModels(models []catalog.Model) {
|
||||
m := map[string]catalog.Model{}
|
||||
for _, entry := range c.Models() {
|
||||
ads := map[string]adapterRef{}
|
||||
for _, entry := range models {
|
||||
m[entry.Name] = entry
|
||||
for _, a := range entry.Aliases {
|
||||
m[a] = entry
|
||||
}
|
||||
// An adapter resolves to the model it runs on, as an alias does, and
|
||||
// is applied to the request on the way there.
|
||||
for i, a := range entry.Adapters {
|
||||
m[a.Name] = entry
|
||||
ads[a.Name] = adapterRef{base: entry.Name, index: i, scale: a.EffectiveScale()}
|
||||
}
|
||||
}
|
||||
r.models.Store(&m)
|
||||
r.adapters.Store(&ads)
|
||||
}
|
||||
|
||||
// adapterRef is where an adapter runs and how it is applied.
|
||||
type adapterRef struct {
|
||||
base string
|
||||
index int
|
||||
scale float64
|
||||
}
|
||||
|
||||
// adapter finds one by the name a client asked for.
|
||||
func (r *Router) adapter(name string) (adapterRef, bool) {
|
||||
if p := r.adapters.Load(); p != nil {
|
||||
a, ok := (*p)[name]
|
||||
return a, ok
|
||||
}
|
||||
return adapterRef{}, false
|
||||
}
|
||||
|
||||
// LastUsed reports when each endpoint was last sent a request, satisfying
|
||||
@@ -715,10 +739,20 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
|
||||
trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
|
||||
attribute.String("url.full", dec.Endpoint.URL+req.Path)))
|
||||
body := req.UpstreamBody
|
||||
if dec.Endpoint.UpstreamModel != "" {
|
||||
body = withModel(body, dec.Endpoint.UpstreamModel)
|
||||
}
|
||||
// An adapter asked for by name, served by its model's replicas: the
|
||||
// engine is asked for the model, with the adapter applied. Not on a
|
||||
// fallback to another model, which does not have it.
|
||||
if a, ok := r.adapter(req.Model); ok && a.base == model {
|
||||
body = withAdapter(body, model, a.index, a.scale)
|
||||
}
|
||||
out, perr := r.px.Do(uctx, w, proxy.Request{
|
||||
URL: dec.Endpoint.URL + req.Path,
|
||||
APIKey: dec.Endpoint.APIKey,
|
||||
Body: req.UpstreamBody,
|
||||
Body: body,
|
||||
Header: withTraceparent(uctx, req.Header),
|
||||
Stream: req.Stream,
|
||||
SuppressUsageChunk: req.SuppressUsageChunk,
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
@@ -51,34 +54,41 @@ func KVSaturated(ceiling float64) Filter {
|
||||
}
|
||||
}
|
||||
|
||||
// PoolPriority keeps only the endpoints of the highest-priority pool
|
||||
// (plan §7.3, the two-stage decision: pool first, then replica).
|
||||
// PoolPriority keeps the endpoints of one pool: the highest-priority one,
|
||||
// unless it is saturated and a lower one is not (plan §7.3, the two-stage
|
||||
// decision — pool first, then replica).
|
||||
//
|
||||
// Spillover arrives in phase 5, at which point this filter will fall through
|
||||
// to the next pool once the preferred one is saturated.
|
||||
func PoolPriority() Filter {
|
||||
return poolPriorityFilter{}
|
||||
// Saturated is what admission means by it, the same formula over the same
|
||||
// thresholds, so the two layers agree on when a pool is full: the H100 pool is
|
||||
// used until it is, and only then the A100 one — never the A100 because it
|
||||
// happens to be emptier. When every pool is saturated the preferred one is
|
||||
// kept: moving to a pool that is just as full buys nothing and loses whatever
|
||||
// made the first preferred.
|
||||
func PoolPriority(targetWaiting, kvCeiling float64) Filter {
|
||||
return poolPriorityFilter{t: admission.Thresholds{TargetWaitingPerReplica: targetWaiting, KVCeiling: kvCeiling}}
|
||||
}
|
||||
|
||||
type poolPriorityFilter struct{}
|
||||
type poolPriorityFilter struct{ t admission.Thresholds }
|
||||
|
||||
func (poolPriorityFilter) Name() string { return "pool-priority" }
|
||||
|
||||
func (poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
|
||||
func (f poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
|
||||
if len(eps) == 0 {
|
||||
return eps
|
||||
}
|
||||
best := eps[0].Priority
|
||||
for _, e := range eps[1:] {
|
||||
if e.Priority > best {
|
||||
best = e.Priority
|
||||
}
|
||||
}
|
||||
out := make([]endpoint.Snapshot, 0, len(eps))
|
||||
byPriority := map[int][]endpoint.Snapshot{}
|
||||
var order []int
|
||||
for _, e := range eps {
|
||||
if e.Priority == best {
|
||||
out = append(out, e)
|
||||
if _, ok := byPriority[e.Priority]; !ok {
|
||||
order = append(order, e.Priority)
|
||||
}
|
||||
byPriority[e.Priority] = append(byPriority[e.Priority], e)
|
||||
}
|
||||
sort.Sort(sort.Reverse(sort.IntSlice(order)))
|
||||
for _, p := range order {
|
||||
if admission.Saturation(byPriority[p], 0, f.t) < 1 {
|
||||
return byPriority[p]
|
||||
}
|
||||
}
|
||||
return out
|
||||
return byPriority[order[0]]
|
||||
}
|
||||
@@ -50,7 +50,7 @@ func Chat(c ProfileConfig) *Profile {
|
||||
c.normalize()
|
||||
return &Profile{
|
||||
Name: "chat",
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority()},
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
|
||||
Scorers: c.withPrefix([]WeightedScorer{
|
||||
{Session(c.SessionLoadFactor), 3.0},
|
||||
{Queue(c.TargetWaiting), 1.0},
|
||||
@@ -81,7 +81,7 @@ func Batch(c ProfileConfig) *Profile {
|
||||
c.normalize()
|
||||
return &Profile{
|
||||
Name: "batch",
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority()},
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
|
||||
Scorers: []WeightedScorer{
|
||||
{Queue(c.TargetWaiting), 3.0},
|
||||
{KV(c.KVCeiling), 1.0},
|
||||
@@ -99,7 +99,7 @@ func LongContext(c ProfileConfig) *Profile {
|
||||
c.normalize()
|
||||
return &Profile{
|
||||
Name: "long-context",
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority()},
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
|
||||
// Prefix matching earns more here than anywhere else: a long prompt is
|
||||
// mostly prefill, and prefill is what a cache hit skips.
|
||||
Scorers: c.withPrefix([]WeightedScorer{
|
||||
|
||||
@@ -177,7 +177,7 @@ func TestStaleEndpointIsPenalizedNotDropped(t *testing.T) {
|
||||
func TestPoolPriority(t *testing.T) {
|
||||
p := &Profile{
|
||||
Name: "t",
|
||||
Filters: []Filter{PoolPriority()},
|
||||
Filters: []Filter{PoolPriority(4, 0.9)},
|
||||
Scorers: []WeightedScorer{{Queue(4), 1}},
|
||||
Picker: MaxScore(),
|
||||
}
|
||||
@@ -189,7 +189,8 @@ func TestPoolPriority(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The lower-priority pool must not win despite being emptier (spillover is phase 5).
|
||||
// The lower-priority pool must not win for being emptier: the preferred
|
||||
// one is not saturated (3 waiting of a healthy 4).
|
||||
if d.Endpoint.ID != "h100" {
|
||||
t.Errorf("pool priority was ignored: %s", d.Endpoint.ID)
|
||||
}
|
||||
@@ -383,3 +384,36 @@ func TestBuiltinProfilesSchedule(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestASaturatedPoolSpillsToTheNext(t *testing.T) {
|
||||
p := &Profile{Name: "t", Filters: []Filter{PoolPriority(4, 0.9)},
|
||||
Scorers: []WeightedScorer{{Queue(4), 1}}, Picker: MaxScore()}
|
||||
h100 := func(w int) endpoint.Snapshot {
|
||||
return ep("h100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 100; s.Pool = "ib-a/h100" })
|
||||
}
|
||||
a100 := func(w int) endpoint.Snapshot {
|
||||
return ep("a100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 50; s.Pool = "ib-b/a100" })
|
||||
}
|
||||
pick := func(eps ...endpoint.Snapshot) string {
|
||||
d, err := p.Schedule(newCtx(Request{Model: "m"}), eps)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return d.Endpoint.ID
|
||||
}
|
||||
if got := pick(h100(8), a100(0)); got != "a100" {
|
||||
t.Errorf("the preferred pool at twice its healthy queue did not spill: %s", got)
|
||||
}
|
||||
if got := pick(h100(8), a100(6)); got != "h100" {
|
||||
t.Errorf("both pools saturated: the preferred one should be kept, got %s", got)
|
||||
}
|
||||
if got := pick(h100(4), a100(0)); got != "a100" {
|
||||
t.Errorf("at exactly its healthy limit a pool is full: %s", got)
|
||||
}
|
||||
// KV pressure saturates a pool as queue depth does.
|
||||
full := h100(0)
|
||||
full.Stats.KVCacheUsage = 0.95
|
||||
if got := pick(full, a100(0)); got != "a100" {
|
||||
t.Errorf("a pool out of KV cache did not spill: %s", got)
|
||||
}
|
||||
}
|
||||
@@ -96,6 +96,9 @@ func (p *Poller) Sync(ctx context.Context) {
|
||||
|
||||
want := make(map[string]endpoint.Info, len(infos))
|
||||
for _, in := range infos {
|
||||
if in.Engine == endpoint.EngineExternal {
|
||||
continue // a provider elsewhere publishes no /metrics
|
||||
}
|
||||
want[in.ID] = in
|
||||
}
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ var copyOrder = []string{
|
||||
// derived from this, and a new secret would lock the router out of all
|
||||
// of them at the moment of the move.
|
||||
"cluster_secrets",
|
||||
"node_engines",
|
||||
}
|
||||
|
||||
// notCopied are tables deliberately left behind.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,44 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NodeEngines are the engine builds chosen per node: node id, then engine,
|
||||
// then the build ("cu129") or "none". A node or engine not here is left to
|
||||
// the node's hardware.
|
||||
func (s *DB) NodeEngines(ctx context.Context) (map[string]map[string]string, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `SELECT node_id, engine, variant FROM node_engines`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := map[string]map[string]string{}
|
||||
for rows.Next() {
|
||||
var node, engine, variant string
|
||||
if err := rows.Scan(&node, &engine, &variant); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if out[node] == nil {
|
||||
out[node] = map[string]string{}
|
||||
}
|
||||
out[node][engine] = variant
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node; an empty variant
|
||||
// leaves it to the node's hardware again.
|
||||
func (s *DB) SetNodeEngine(ctx context.Context, node, engine, variant, by string) error {
|
||||
if variant == "" {
|
||||
_, err := s.db.ExecContext(ctx, `DELETE FROM node_engines WHERE node_id = ? AND engine = ?`, node, engine)
|
||||
return err
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO node_engines (node_id, engine, variant, updated_at, updated_by) VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT (node_id, engine) DO UPDATE SET variant = excluded.variant,
|
||||
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
|
||||
node, engine, variant, time.Now().UnixMilli(), by)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestAnEngineBuildIsChosenPerNodeAndCanBeLeftToItAgain(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
ctx := t.Context()
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "cu129", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "none", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := db.NodeEngines(ctx)
|
||||
if err != nil || got["gpu07"]["vllm"] != "none" {
|
||||
t.Fatalf("got %v, %v; the second choice replaces the first", got, err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, _ := db.NodeEngines(ctx); len(got["gpu07"]) != 0 {
|
||||
t.Errorf("left to the node again, still %v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
func TestADuplicateIsRecognisedOnEitherDatabase(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{"sqlite", errors.New("constraint failed: UNIQUE constraint failed: users.email (2067)"), true},
|
||||
// What pgx returns, wrapped as database/sql passes it on: "unique"
|
||||
// in lower case, which a check for "UNIQUE" missed.
|
||||
{"postgres", fmt.Errorf("exec: %w", &pgconn.PgError{Code: "23505",
|
||||
Message: `duplicate key value violates unique constraint "users_email_key"`}), true},
|
||||
{"postgres, as text", errors.New(`ERROR: duplicate key value violates unique constraint "users_email_key" (SQLSTATE 23505)`), true},
|
||||
{"another postgres error", &pgconn.PgError{Code: "23502", Message: "null value in column"}, false},
|
||||
{"nothing", nil, false},
|
||||
} {
|
||||
if got := isUniqueViolation(c.err); got != c.want {
|
||||
t.Errorf("%s: %v, want %v", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
+20
-1
@@ -6,6 +6,8 @@ import (
|
||||
"errors"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
// UserState is where an account is in its life.
|
||||
@@ -66,12 +68,29 @@ func (s *DB) CreateUser(ctx context.Context, u User) error {
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
u.ID, u.Email, u.Name, u.PasswordHash, string(u.State),
|
||||
nullString(u.ProjectID), millis(u.CreatedAt))
|
||||
if err != nil && strings.Contains(err.Error(), "UNIQUE") {
|
||||
if isUniqueViolation(err) {
|
||||
return ErrEmailTaken
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// isUniqueViolation reports whether err is a row refused for a value another
|
||||
// row already has. Each database says it its own way: SQLite in words
|
||||
// ("UNIQUE constraint failed"), PostgreSQL by code, 23505, in a message with
|
||||
// "unique" in lower case — which a check for the upper-case word missed, so a
|
||||
// second sign-up with an address on PostgreSQL was "something went wrong".
|
||||
func isUniqueViolation(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
var pe *pgconn.PgError
|
||||
if errors.As(err, &pe) {
|
||||
return pe.Code == "23505"
|
||||
}
|
||||
m := err.Error()
|
||||
return strings.Contains(m, "UNIQUE constraint failed") || strings.Contains(m, "SQLSTATE 23505")
|
||||
}
|
||||
|
||||
// UserByEmail finds an account by its address.
|
||||
func (s *DB) UserByEmail(ctx context.Context, email string) (User, error) {
|
||||
return s.scanUser(s.db.QueryRowContext(ctx,
|
||||
|
||||
+19
-2
@@ -4,11 +4,13 @@ import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base32"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/account"
|
||||
@@ -76,6 +78,9 @@ type AccountPages struct {
|
||||
priced Priced
|
||||
// registry browses a model registry, or is nil where there is no way out.
|
||||
registry Registry
|
||||
// kinds remembers what each held GGUF file is, by its address: a held
|
||||
// version never changes, so it is read once.
|
||||
kinds sync.Map
|
||||
// catalog puts a mirrored model into service; fleet says whether anything
|
||||
// could run it. Both nil where the mirror page is read-only.
|
||||
catalog CatalogAdmin
|
||||
@@ -165,8 +170,12 @@ func (p *AccountPages) signup(w http.ResponseWriter, r *http.Request) {
|
||||
case errors.Is(err, account.ErrWeakPassword), errors.Is(err, account.ErrInvalidEmail):
|
||||
d.Error = err.Error()
|
||||
default:
|
||||
p.log.Error("signup failed", "err", err)
|
||||
d.Error = "something went wrong; try again"
|
||||
// Nothing of the database's words on the page; a reference that finds
|
||||
// them in the log for whoever runs the gateway.
|
||||
ref := errorRef()
|
||||
p.log.Error("signup failed", "ref", ref, "err", err)
|
||||
d.Error = "something went wrong on our side, and your account was not created. " +
|
||||
"Try again later; if it keeps happening, give an operator this reference: " + ref
|
||||
}
|
||||
render(w, "signup", d)
|
||||
}
|
||||
@@ -634,3 +643,11 @@ func (p *AccountPages) renderWithCSRF(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
// SetLlamaCppVersion tells the pages which llama.cpp build the nodes run.
|
||||
func (p *AccountPages) SetLlamaCppVersion(v string) { p.llamaVersion = v }
|
||||
|
||||
// errorRef is a short reference for an error a page does not explain, logged
|
||||
// beside the error itself.
|
||||
func errorRef() string {
|
||||
var b [4]byte
|
||||
_, _ = rand.Read(b[:])
|
||||
return "err-" + hex.EncodeToString(b[:])
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
@@ -46,6 +47,10 @@ type Registry interface {
|
||||
|
||||
// repoSet is one thing worth fetching: a file, or every shard of one.
|
||||
type repoSet struct {
|
||||
// Kind is what the file says it is, read from its first bytes: "adapter"
|
||||
// for a LoRA adapter, "model", or empty when it was not read. Arch is the
|
||||
// architecture it names, which an adapter must share with its model.
|
||||
Kind, Arch string
|
||||
// Verdict is whether the nodes' llama.cpp can load it, from the
|
||||
// architecture the registry reports; a set that cannot be served is
|
||||
// shown with the reason and no fetch button.
|
||||
@@ -55,6 +60,9 @@ type repoSet struct {
|
||||
// A shard set goes as one action — a fifteen-shard model is not fifteen
|
||||
// decisions, it is one.
|
||||
Sources []repoSource
|
||||
// Companions are the files fetched with a safetensors set that are not
|
||||
// weights: config.json, the tokenizer, the shard index.
|
||||
Companions []string
|
||||
// Model and Version are what the fetch form starts with. Editable, because
|
||||
// the name a catalog entry uses is the operator's to choose and the
|
||||
// registry's idea of it is only a good guess.
|
||||
@@ -76,6 +84,8 @@ type modelUse struct {
|
||||
// a replica of its first profile reserves.
|
||||
Class, EffClass, Owner string
|
||||
MemoryGB, CPUCores int
|
||||
// Adapters are the LoRA adapters it serves.
|
||||
Adapters []catalog.Adapter
|
||||
// Ready, Wanted and Nodes are how it runs now, for the row itself:
|
||||
// replicas ready of those it should have, and the nodes they are on.
|
||||
// Fleet is false where nothing could be asked, so the row says nothing
|
||||
@@ -105,6 +115,9 @@ type mirrorRow struct {
|
||||
// Detail is the card opened beneath the row: what runs it, on which
|
||||
// nodes, and how many replicas it runs.
|
||||
Detail *heldDetail
|
||||
// Kind is what a single GGUF file is — "adapter" or "model" — and Arch
|
||||
// its architecture; empty when not read.
|
||||
Kind, Arch string
|
||||
// Open is the row's ?open= value.
|
||||
Open string
|
||||
|
||||
@@ -299,6 +312,7 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setModel))
|
||||
mux.HandleFunc("POST /admin/mirror/adapters", p.mirrorAction(m, p.setAdapters))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
|
||||
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
@@ -306,6 +320,16 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// The CPU while GPUs sit free is a choice to make on purpose:
|
||||
// made by default, it was a model running at a tenth of the
|
||||
// speed with nothing saying why.
|
||||
if model.Profiles[0].CPUOnly() && r.FormValue("cpu_ok") == "" {
|
||||
if n := freeGPUs(p.fleet); n > 0 {
|
||||
return "", fmt.Errorf("this fleet has %s free and %s was set to run on the CPU; "+
|
||||
"choose a GPU placement, or tick \"run on the CPU anyway\"",
|
||||
plural(n, "GPU", "GPUs"), model.Name)
|
||||
}
|
||||
}
|
||||
// The same check `llmgw ctl model add` makes. This page builds
|
||||
// the source out of a row the mirror reported, so it is right
|
||||
// far more often than something typed — but a file removed
|
||||
@@ -321,8 +345,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
// Placed on the next pass, not now: saying it is serving would
|
||||
// be wrong for however long the weights take to reach a node.
|
||||
return fmt.Sprintf(
|
||||
"%s added to the catalog, %s; it is placed on the next pass",
|
||||
model.Name, plural(model.Scaling.Min, "replica", "replicas")), nil
|
||||
"%s added to the catalog, %s: %s; it is placed on the next pass",
|
||||
model.Name, plural(model.Scaling.Min, "replica", "replicas"), describeServe(model)), nil
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -469,6 +493,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
|
||||
Min: m.Scaling.Min, Max: m.Scaling.Max,
|
||||
Class: m.Class, EffClass: m.EffectiveClass(), Owner: m.Owner,
|
||||
Adapters: m.Adapters,
|
||||
Declared: p.catalog.Declared(m.Name),
|
||||
}
|
||||
if len(m.Profiles) > 0 {
|
||||
@@ -487,6 +512,8 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
}
|
||||
|
||||
p.heldKinds(r.Context(), d.Rows)
|
||||
|
||||
// The one row whose card is open. From the form too, so a change made in
|
||||
// the card comes back with the card still open to show it.
|
||||
open := r.FormValue("open")
|
||||
@@ -494,6 +521,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
|
||||
if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
|
||||
d.Rows[i].Detail = p.heldDetailFor(r.Context(), d.Rows[i])
|
||||
d.Rows[i].Detail.addAdapterChoices(d.Rows[i], d.Rows)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -753,8 +781,59 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
|
||||
rs.Sources = append(rs.Sources,
|
||||
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
|
||||
}
|
||||
// Safetensors are not a model on their own: vLLM loads a directory,
|
||||
// and reads its config and tokenizer before any weight. Fetched as
|
||||
// part of the set, or the set arrives as shards nothing can load.
|
||||
if strings.HasSuffix(strings.ToLower(set.Name), ".safetensors") {
|
||||
for _, f := range hf.Companions(got.Files) {
|
||||
rs.Sources = append(rs.Sources,
|
||||
repoSource{URL: got.DownloadURL("", f.Path), Path: f.Path})
|
||||
rs.Companions = append(rs.Companions, f.Path)
|
||||
rs.Bytes += f.Bytes
|
||||
}
|
||||
}
|
||||
d.RepoSets = append(d.RepoSets, rs)
|
||||
}
|
||||
p.probeKinds(r.Context(), d.RepoSets)
|
||||
}
|
||||
|
||||
// kindOf reads what a GGUF file is; a variable so tests need no registry.
|
||||
var kindOf = func(ctx context.Context, url string) (gguf.Header, error) {
|
||||
return gguf.KindFromURL(ctx, nil, url)
|
||||
}
|
||||
|
||||
// probeKinds reads the start of each single-file GGUF set, to tell LoRA
|
||||
// adapters from models: a repository tagged "lora" holds a model with the
|
||||
// adapter merged in as often as an adapter, and only the file says which.
|
||||
// A few at once and a few in all, since each is a request to the registry.
|
||||
func (p *AccountPages) probeKinds(ctx context.Context, sets []repoSet) {
|
||||
const most, together = 12, 6
|
||||
sem := make(chan struct{}, together)
|
||||
var wg sync.WaitGroup
|
||||
n := 0
|
||||
for i := range sets {
|
||||
s := &sets[i]
|
||||
if len(s.Sources) != 1 || !gguf.IsGGUF(s.Sources[0].Path) || n == most {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{}
|
||||
defer func() { <-sem }()
|
||||
cctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
if h, err := kindOf(cctx, s.Sources[0].URL); err == nil {
|
||||
s.Arch = h.Architecture
|
||||
s.Kind = "model"
|
||||
if h.IsAdapter() {
|
||||
s.Kind = "adapter"
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// defaultModelName is the catalog name a repository suggests.
|
||||
|
||||
@@ -710,3 +710,37 @@ func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
|
||||
t.Errorf("a started download should be shown where it is followed: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASafetensorsSetIsFetchedWithWhatTheEngineReadsBesideIt(t *testing.T) {
|
||||
// The field: a four-shard AWQ model mirrored as its shards alone, and
|
||||
// vLLM failing at start for want of config.json.
|
||||
reg := &fakeRegistry{repo: hf.Repo{
|
||||
ID: "Qwen/Qwen2.5-32B-Instruct-AWQ", Revision: "abc123def456",
|
||||
Files: []hf.File{
|
||||
{Path: "model-00001-of-00002.safetensors", Bytes: 4_000_000_000},
|
||||
{Path: "model-00002-of-00002.safetensors", Bytes: 3_000_000_000},
|
||||
{Path: "model.safetensors.index.json", Bytes: 90_000},
|
||||
{Path: "config.json", Bytes: 900},
|
||||
{Path: "tokenizer.json", Bytes: 7_000_000},
|
||||
{Path: "tokenizer_config.json", Bytes: 7_000},
|
||||
{Path: "README.md", Bytes: 9_000},
|
||||
{Path: "modeling_custom.py", Bytes: 40_000},
|
||||
{Path: "onnx/config.json", Bytes: 900},
|
||||
},
|
||||
}}
|
||||
body := mirrorGet(t, mirrorPageWith(t, &fakeMirror{}, reg, true),
|
||||
"/admin/mirror?repo=Qwen/Qwen2.5-32B-Instruct-AWQ").Body.String()
|
||||
for _, want := range []string{"config.json", "tokenizer.json", "tokenizer_config.json", "model.safetensors.index.json"} {
|
||||
if !strings.Contains(body, `name="file" value="`+want+`"`) {
|
||||
t.Errorf("%s is not fetched with the set", want)
|
||||
}
|
||||
}
|
||||
for _, not := range []string{"README.md", "modeling_custom.py", "onnx/config.json"} {
|
||||
if strings.Contains(body, `name="file" value="`+not+`"`) {
|
||||
t.Errorf("%s was swept in with the set", not)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(body, "with 4 files the engine reads beside them") {
|
||||
t.Errorf("the companions are not said; got %s", body)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// NodeEngineAdmin chooses an engine's build for a node: "auto", a build
|
||||
// such as "cu129", or "none".
|
||||
type NodeEngineAdmin interface {
|
||||
SetNodeEngine(ctx context.Context, node, engine, build, by string) error
|
||||
}
|
||||
|
||||
// RegisterNodeEngines mounts the choice of engine builds on the cluster
|
||||
// page: for an administrator, with the session's CSRF token, and answered by
|
||||
// going back to the page with what happened.
|
||||
func (p *AccountPages) RegisterNodeEngines(mux *http.ServeMux, a NodeEngineAdmin) {
|
||||
mux.HandleFunc("POST /admin/nodes/{id}/engines", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
sess, _, _ := p.session(r)
|
||||
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
node, engine, build := r.PathValue("id"), r.FormValue("engine"), r.FormValue("build")
|
||||
q := url.Values{}
|
||||
if err := a.SetNodeEngine(r.Context(), node, engine, build, admin.Email); err != nil {
|
||||
p.log.Warn("an engine build was not chosen", "admin", admin.Email, "node", node, "err", err)
|
||||
q.Set("problem", node+": "+err.Error())
|
||||
} else {
|
||||
p.log.Info("an engine build was chosen", "admin", admin.Email, "node", node, "engine", engine, "build", build)
|
||||
q.Set("notice", node+" installs "+engine+" "+buildWords(build)+
|
||||
"; the node is told now, and installs it in the background")
|
||||
}
|
||||
http.Redirect(w, r, "/admin?"+q.Encode(), http.StatusSeeOther)
|
||||
}))
|
||||
}
|
||||
|
||||
func buildWords(b string) string {
|
||||
switch b {
|
||||
case "auto":
|
||||
return "in the build its hardware runs"
|
||||
case "none":
|
||||
return "no longer"
|
||||
}
|
||||
return "built for " + map[string]string{"cu13": "CUDA 13", "cu129": "CUDA 12.9"}[b]
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// buildFleet is one GPU node whose vLLM build was chosen, and one CPU node.
|
||||
type buildFleet struct{ fakeFleet }
|
||||
|
||||
func (buildFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{
|
||||
{ID: "gpu07", State: "ready", GPUTotal: 8, GPUFree: 8,
|
||||
EngineBuilds: map[string]string{"vllm": "cu129"},
|
||||
Provision: []inventory.EngineInstall{{Engine: "vllm", Version: "0.30.0", Variant: "cu129", State: "installing"}}},
|
||||
{ID: "cpu01", State: "ready"},
|
||||
}
|
||||
}
|
||||
|
||||
type recordedChoice struct{ node, engine, build, by string }
|
||||
|
||||
type fakeEngineAdmin struct{ got []recordedChoice }
|
||||
|
||||
func (f *fakeEngineAdmin) SetNodeEngine(_ context.Context, node, engine, build, by string) error {
|
||||
f.got = append(f.got, recordedChoice{node, engine, build, by})
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /admin", p.AdminPage(Dashboard(buildFleet{}, nil, "v1", time.Second, false, NavExtras{}, nil)))
|
||||
fe := &fakeEngineAdmin{}
|
||||
p.RegisterNodeEngines(mux, fe)
|
||||
|
||||
body := mirrorGet(t, mux, "/admin").Body.String()
|
||||
if !strings.Contains(body, `action="/admin/nodes/gpu07/engines"`) ||
|
||||
!strings.Contains(body, `<option value="cu129" selected>CUDA 12.9</option>`) {
|
||||
t.Fatalf("no choice of build for the GPU node, with its current one selected: %s", body)
|
||||
}
|
||||
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
|
||||
t.Error("a node with no GPU was offered a vLLM build")
|
||||
}
|
||||
if !strings.Contains(body, "CUDA 12.9 installing") {
|
||||
t.Error("what is installed is not shown beside the choice")
|
||||
}
|
||||
|
||||
// The read-only page on the admin listener has no session, so no form.
|
||||
_, ro := get(t, Dashboard(buildFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(ro, "/engines\"") {
|
||||
t.Error("the admin listener's page offers a change it cannot sign")
|
||||
}
|
||||
|
||||
w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"tok"}, "engine": {"vllm"}, "build": {"none"}})
|
||||
if w.Code != http.StatusSeeOther || len(fe.got) != 1 || fe.got[0] != (recordedChoice{"gpu07", "vllm", "none", "a@example.com"}) {
|
||||
t.Fatalf("status %d, chose %+v", w.Code, fe.got)
|
||||
}
|
||||
if loc := w.Header().Get("Location"); !strings.Contains(loc, "notice=") {
|
||||
t.Errorf("redirected to %s, saying nothing", loc)
|
||||
}
|
||||
if w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"bad"}, "engine": {"vllm"}, "build": {"none"}}); w.Code != http.StatusBadRequest || len(fe.got) != 1 {
|
||||
t.Errorf("a forged form was accepted: %d", w.Code)
|
||||
}
|
||||
}
|
||||
+183
-6
@@ -27,6 +27,7 @@ type CatalogAdmin interface {
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// serveForm is the offer to put one mirrored version into service.
|
||||
@@ -42,8 +43,20 @@ type serveForm struct {
|
||||
Engine string
|
||||
// Profile is the profile name, which doubles as how a node_selector would
|
||||
// be written later. Named for the hardware rather than the model, because
|
||||
// a profile is about where a model runs.
|
||||
// a profile is about where a model runs. Derived from the placement, not
|
||||
// typed: a name is only a name, and a field for it read as though calling
|
||||
// a profile "gpu" would put it on one.
|
||||
Profile string
|
||||
// Placements are where it can run: each GPU class the fleet has, and for
|
||||
// llama.cpp the CPU. One is selected.
|
||||
Placements []placeChoice
|
||||
// GPUs is how many cards a replica takes, when placed on a GPU class.
|
||||
GPUs int
|
||||
// Parallel is how many requests one replica serves at once.
|
||||
Parallel int
|
||||
// FreeGPUs is how many cards no replica holds, across the fleet: what
|
||||
// choosing the CPU leaves unused.
|
||||
FreeGPUs int
|
||||
// Files is how many files the version holds, above one for a set.
|
||||
Files int
|
||||
// MinMemoryGB is the host RAM to ask for, estimated from the weights.
|
||||
@@ -125,6 +138,13 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
|
||||
default:
|
||||
f.Profile = "gpu"
|
||||
}
|
||||
f.GPUs, f.Parallel = 1, defaultParallel
|
||||
f.Placements, f.FreeGPUs = placementsFor(fleet, f.Engine)
|
||||
for _, pc := range f.Placements {
|
||||
if pc.Selected {
|
||||
f.Profile = pc.profile()
|
||||
}
|
||||
}
|
||||
// Whether any node can actually run it. Agents report their installed
|
||||
// engines now, so this is a fact rather than the guess it started as —
|
||||
// which was cards, and would have passed a GPU node with no vLLM on it.
|
||||
@@ -273,12 +293,38 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
if engine == "" {
|
||||
return m, errors.New("which engine should run it?")
|
||||
}
|
||||
p := catalog.Profile{
|
||||
Name: strings.TrimSpace(get("profile")), Engine: engine,
|
||||
Weight: 1, Priority: 100,
|
||||
p := catalog.Profile{Engine: engine, Weight: 1, Priority: 100}
|
||||
gpus := 1
|
||||
if v := strings.TrimSpace(get("gpus")); v != "" {
|
||||
if gpus, err = positive(v, "the GPUs per replica"); err != nil {
|
||||
return m, err
|
||||
}
|
||||
gpus = max(gpus, 1)
|
||||
}
|
||||
if p.Name == "" {
|
||||
p.Name = engine
|
||||
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
|
||||
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
|
||||
case onGPU && class != "":
|
||||
// Placement is decided by gpu_class, which is what the selector
|
||||
// sets; the count is gpus_per_node for llama.cpp, which spreads
|
||||
// layers over a node's cards, and tp for the others.
|
||||
p.GPUClass = class
|
||||
if engine == "llamacpp" {
|
||||
p.GPUsPerNode = gpus
|
||||
} else {
|
||||
p.TP = gpus
|
||||
}
|
||||
place.GPUs = gpus
|
||||
case place.Value == "" || place.Value == "cpu":
|
||||
if engine != "llamacpp" {
|
||||
return m, fmt.Errorf("%s runs on GPUs; choose a GPU class to place it on", engine)
|
||||
}
|
||||
place.Value = "cpu"
|
||||
default:
|
||||
return m, fmt.Errorf("%q is not a placement", place.Value)
|
||||
}
|
||||
p.Name = place.profile()
|
||||
if v := strings.TrimSpace(get("profile")); v != "" {
|
||||
p.Name = v
|
||||
}
|
||||
// The pool, not the per-request limit: llama.cpp divides it across its
|
||||
// parallel slots, so the number a client may use is smaller and is what
|
||||
@@ -289,6 +335,15 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
}
|
||||
p.Args = []string{"--ctx-size", v}
|
||||
}
|
||||
if v := strings.TrimSpace(get("parallel")); v != "" && engine == "llamacpp" {
|
||||
n, err := positive(v, "the parallel requests")
|
||||
if err != nil {
|
||||
return m, err
|
||||
}
|
||||
if n > 0 {
|
||||
p.Args = append(p.Args, "--parallel", strconv.Itoa(n))
|
||||
}
|
||||
}
|
||||
if v := strings.TrimSpace(get("min_memory_gb")); v != "" {
|
||||
if p.MinMemoryGB, err = positive(v, "the memory requirement"); err != nil {
|
||||
return m, err
|
||||
@@ -407,3 +462,125 @@ func modeOf(sc catalog.Scaling) string {
|
||||
// defaultCtxPool is the llama.cpp context pool the serve form starts with.
|
||||
// --parallel divides it among slots; it does not multiply it.
|
||||
const defaultCtxPool = 16384
|
||||
|
||||
// defaultParallel is the requests a llama.cpp replica serves at once when
|
||||
// the form is not changed: the agent's own default.
|
||||
const defaultParallel = 4
|
||||
|
||||
// placeChoice is one place a model can run: a GPU class, or the CPU.
|
||||
type placeChoice struct {
|
||||
// Value is "gpu:<class>" or "cpu", as the form submits it.
|
||||
Value string
|
||||
Label string
|
||||
// GPUs is how many cards a replica takes, for naming the profile.
|
||||
GPUs int
|
||||
Selected bool
|
||||
}
|
||||
|
||||
// profile names the profile for where it runs: the class, with the count when
|
||||
// it is more than one card ("t4", "t4-x4"), or "cpu".
|
||||
func (c placeChoice) profile() string {
|
||||
class, ok := strings.CutPrefix(c.Value, "gpu:")
|
||||
if !ok || class == "" {
|
||||
return "cpu"
|
||||
}
|
||||
if c.GPUs > 1 {
|
||||
return fmt.Sprintf("%s-x%d", class, c.GPUs)
|
||||
}
|
||||
return class
|
||||
}
|
||||
|
||||
// placementsFor lists where engine can run on this fleet, with one selected,
|
||||
// and how many GPUs are free.
|
||||
//
|
||||
// A GPU when the fleet has one: a GGUF model was put on the CPU by this form
|
||||
// on a cluster of GPUs, ran at a tenth of their speed, and nothing on the way
|
||||
// said why — the form had no choice to offer and named the profile "cpu". The
|
||||
// class selected is the one with the most cards free.
|
||||
func placementsFor(fleet Source, engine string) ([]placeChoice, int) {
|
||||
var out []placeChoice
|
||||
free, best := 0, -1
|
||||
if fleet != nil {
|
||||
for _, c := range fleet.Summary().GPUs {
|
||||
n := c.Free()
|
||||
free += n
|
||||
out = append(out, placeChoice{
|
||||
Value: "gpu:" + c.Class,
|
||||
Label: fmt.Sprintf("GPU: %s, %d of %d free", c.Class, n, c.Total),
|
||||
})
|
||||
if best < 0 || n > fleet.Summary().GPUs[best].Free() {
|
||||
best = len(out) - 1
|
||||
}
|
||||
}
|
||||
}
|
||||
if best >= 0 {
|
||||
out[best].Selected = true
|
||||
}
|
||||
if engine == "llamacpp" {
|
||||
label := "CPU only: no GPU, many times slower"
|
||||
if len(out) == 0 {
|
||||
label = "CPU: this fleet has no GPU"
|
||||
}
|
||||
out = append(out, placeChoice{Value: "cpu", Label: label, Selected: best < 0})
|
||||
}
|
||||
return out, free
|
||||
}
|
||||
|
||||
// freeGPUs is how many cards no replica holds across the fleet.
|
||||
func freeGPUs(fleet Source) int {
|
||||
if fleet == nil {
|
||||
return 0
|
||||
}
|
||||
n := 0
|
||||
for _, c := range fleet.Summary().GPUs {
|
||||
n += c.Free()
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// describeServe says, in one line, what a model will run as: the summary
|
||||
// shown before it is added and in the message after.
|
||||
func describeServe(m catalog.Model) string {
|
||||
if len(m.Profiles) == 0 {
|
||||
return ""
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
engine := map[string]string{"llamacpp": "llama.cpp", "vllm": "vLLM", "sglang": "SGLang"}[p.Engine]
|
||||
if engine == "" {
|
||||
engine = p.Engine
|
||||
}
|
||||
where := "on the CPU"
|
||||
if p.GPUClass != "" {
|
||||
n := max(p.GPUsPerNode, p.TP, 1)
|
||||
where = fmt.Sprintf("on %s %s", plural(n, "GPU", "GPUs"), p.GPUClass)
|
||||
if n > 1 {
|
||||
where = fmt.Sprintf("on %d %s GPUs of one node", n, p.GPUClass)
|
||||
}
|
||||
}
|
||||
parts := []string{engine + " " + where}
|
||||
if p.MinMemoryGB > 0 {
|
||||
parts = append(parts, fmt.Sprintf("%d GB of host memory", p.MinMemoryGB))
|
||||
}
|
||||
if p.GPUClass == "" && p.CPUCores > 0 {
|
||||
parts = append(parts, plural(p.CPUCores, "core", "cores"))
|
||||
}
|
||||
pool, parallel := argInt(p.Args, "--ctx-size"), argInt(p.Args, "--parallel")
|
||||
switch {
|
||||
case pool > 0 && parallel > 0:
|
||||
parts = append(parts, fmt.Sprintf("context pool %d over %d parallel requests (%d each)", pool, parallel, pool/parallel))
|
||||
case pool > 0:
|
||||
parts = append(parts, fmt.Sprintf("context pool %d", pool))
|
||||
}
|
||||
return strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// argInt is the number after flag in args, or 0.
|
||||
func argInt(args []string, flag string) int {
|
||||
for i := 0; i+1 < len(args); i++ {
|
||||
if args[i] == flag {
|
||||
n, _ := strconv.Atoi(args[i+1])
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -16,15 +16,16 @@ import (
|
||||
)
|
||||
|
||||
type fakeCatalog struct {
|
||||
added []catalog.Model
|
||||
declared string
|
||||
sources map[string][]catalog.Model
|
||||
scaled []string
|
||||
lastScale catalog.Scaling
|
||||
err error
|
||||
classes []string
|
||||
owners []string
|
||||
resources []string
|
||||
added []catalog.Model
|
||||
declared string
|
||||
sources map[string][]catalog.Model
|
||||
scaled []string
|
||||
lastScale catalog.Scaling
|
||||
err error
|
||||
classes []string
|
||||
owners []string
|
||||
resources []string
|
||||
adapterSets [][]catalog.Adapter
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
|
||||
@@ -39,6 +40,11 @@ func (f *fakeCatalog) Sources(context.Context) (map[string][]catalog.Model, erro
|
||||
return f.sources, nil
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) SetAdapters(_ context.Context, name string, as []catalog.Adapter, _ string) (catalog.Model, error) {
|
||||
f.adapterSets = append(f.adapterSets, as)
|
||||
return catalog.Model{Name: name, Adapters: as}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
|
||||
f.classes = append(f.classes, name+"="+class)
|
||||
return catalog.Model{Name: name, Class: class}, f.err
|
||||
@@ -65,6 +71,8 @@ type fakeFleet struct {
|
||||
gpus int
|
||||
engines []string
|
||||
memoryGB int
|
||||
// used is how many of the cards replicas hold.
|
||||
used int
|
||||
}
|
||||
|
||||
func (f fakeFleet) Nodes() []adminapi.NodeView {
|
||||
@@ -75,7 +83,12 @@ func (f fakeFleet) Nodes() []adminapi.NodeView {
|
||||
func (f fakeFleet) Models() []adminapi.ModelView { return nil }
|
||||
func (f fakeFleet) Replicas(string) []adminapi.ReplicaView { return nil }
|
||||
func (f fakeFleet) Explain(string) adminapi.ExplainView { return adminapi.ExplainView{} }
|
||||
func (f fakeFleet) Summary() adminapi.Summary { return adminapi.Summary{} }
|
||||
func (f fakeFleet) Summary() adminapi.Summary {
|
||||
if f.gpus == 0 {
|
||||
return adminapi.Summary{}
|
||||
}
|
||||
return adminapi.Summary{GPUs: []adminapi.ClassUse{{Class: "t4", Total: f.gpus, Used: f.used}}}
|
||||
}
|
||||
|
||||
func held(model, version, file string) mirror.Status {
|
||||
return mirror.Status{Model: model, Version: version, File: file,
|
||||
@@ -446,3 +459,90 @@ func TestServingOnDemandLoadsWhenAskedFor(t *testing.T) {
|
||||
t.Errorf("on demand was added as %+v", sc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGGUFOnAFleetWithGPUsIsOfferedTheGPUFirst(t *testing.T) {
|
||||
gguf := []mirror.Status{held("coder", "v1", "coder-q4_k_m.gguf")}
|
||||
body := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 4, "llamacpp"),
|
||||
"/admin/mirror?serve=coder:v1").Body.String()
|
||||
// Found in the field: no choice offered, a profile called "cpu", and a
|
||||
// model on the CPU of a GPU cluster at a tenth of the speed.
|
||||
if !strings.Contains(body, `<option value="gpu:t4" selected>GPU: t4, 4 of 4 free</option>`) {
|
||||
t.Errorf("the free GPU class should be offered and chosen; got %s", body)
|
||||
}
|
||||
if !strings.Contains(body, `<option value="cpu">CPU only`) {
|
||||
t.Error("the CPU should still be on offer, unselected")
|
||||
}
|
||||
if !strings.Contains(body, `name="cpu_ok"`) {
|
||||
t.Error("choosing the CPU with GPUs free needs saying so")
|
||||
}
|
||||
|
||||
cpuOnly := mirrorGet(t, servePage(t, gguf, &fakeCatalog{}, 0, "llamacpp"),
|
||||
"/admin/mirror?serve=coder:v1").Body.String()
|
||||
if !strings.Contains(cpuOnly, `<option value="cpu" selected>CPU: this fleet has no GPU</option>`) ||
|
||||
strings.Contains(cpuOnly, `name="cpu_ok"`) {
|
||||
t.Errorf("a fleet without GPUs has only the CPU, and nothing to confirm; got %s", cpuOnly)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheChosenPlacementDecidesWhereItRuns(t *testing.T) {
|
||||
form := func(engine, place, gpus string) url.Values {
|
||||
return url.Values{
|
||||
"name": {"m"}, "source": {"http://x/m"}, "engine": {engine}, "replicas": {"1"},
|
||||
"context_length": {"4096"}, "ctx_size": {"16384"}, "parallel": {"4"},
|
||||
"placement": {place}, "gpus": {gpus}, "min_memory_gb": {"8"},
|
||||
}
|
||||
}
|
||||
m, err := buildModel(form("llamacpp", "gpu:t4", "4").Get)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
if p.GPUClass != "t4" || p.GPUsPerNode != 4 || p.TP > 1 || p.Name != "t4-x4" {
|
||||
t.Errorf("llama.cpp on four t4s: %+v", p)
|
||||
}
|
||||
if got := describeServe(m); got != "llama.cpp on 4 t4 GPUs of one node, 8 GB of host memory, "+
|
||||
"context pool 16384 over 4 parallel requests (4096 each)" {
|
||||
t.Errorf("summary %q", got)
|
||||
}
|
||||
if m, _ := buildModel(form("vllm", "gpu:t4", "2").Get); m.Profiles[0].TP != 2 || m.Profiles[0].GPUsPerNode != 0 {
|
||||
t.Errorf("vLLM's GPU count is tp: %+v", m.Profiles[0])
|
||||
}
|
||||
if m, _ := buildModel(form("llamacpp", "cpu", "1").Get); m.Profiles[0].GPUClass != "" || m.Profiles[0].Name != "cpu" {
|
||||
t.Errorf("the CPU: %+v", m.Profiles[0])
|
||||
}
|
||||
if _, err := buildModel(form("vllm", "cpu", "1").Get); err == nil {
|
||||
t.Error("vLLM on the CPU was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCPUWithGPUsFreeIsAChoiceMadeOnPurpose(t *testing.T) {
|
||||
values := func(extra ...string) url.Values {
|
||||
v := url.Values{
|
||||
"csrf": {"tok"}, "name": {"coder"}, "source": {reachableSource(t)}, "engine": {"llamacpp"},
|
||||
"replicas": {"1"}, "context_length": {"4096"}, "placement": {"cpu"}, "min_memory_gb": {"8"},
|
||||
}
|
||||
for i := 0; i+1 < len(extra); i += 2 {
|
||||
v.Set(extra[i], extra[i+1])
|
||||
}
|
||||
return v
|
||||
}
|
||||
cat := &fakeCatalog{}
|
||||
page := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
|
||||
Fleet: fakeFleet{gpus: 4, used: 1, engines: []string{"llamacpp"}}}, true)
|
||||
body := mirrorPost(t, page, "/admin/mirror/serve", values()).Body.String()
|
||||
if len(cat.added) != 0 || !strings.Contains(body, "3 GPUs free") {
|
||||
t.Fatalf("the CPU with three GPUs free went through unasked: %s", body)
|
||||
}
|
||||
mirrorPost(t, page, "/admin/mirror/serve", values("cpu_ok", "yes"))
|
||||
if len(cat.added) != 1 {
|
||||
t.Fatal("ticking the box should add it")
|
||||
}
|
||||
// Every card held: nothing to ask.
|
||||
cat = &fakeCatalog{}
|
||||
full := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat,
|
||||
Fleet: fakeFleet{gpus: 4, used: 4, engines: []string{"llamacpp"}}}, true)
|
||||
mirrorPost(t, full, "/admin/mirror/serve", values())
|
||||
if len(cat.added) != 1 {
|
||||
t.Error("with no GPU free the CPU is the only choice and should not be questioned")
|
||||
}
|
||||
}
|
||||
@@ -19,13 +19,16 @@ type DashboardData struct {
|
||||
// of a GPU table that would only say there are none.
|
||||
CPU cpuCapacity
|
||||
// Menu is the shared top menu; Nav below is what the cluster page adds.
|
||||
Menu Nav
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
Menu Nav
|
||||
// Notice and Problem are what the last change made from this page did,
|
||||
// carried over the redirect back to it.
|
||||
Notice, Problem string
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
// Fabrics is the physical fabrics and what disagrees about them, from a
|
||||
// source that can say.
|
||||
Fabrics *fabrics.Report
|
||||
@@ -170,6 +173,9 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
Now: time.Now().UTC().Format(time.RFC3339),
|
||||
AuthEnabled: usage != nil,
|
||||
}
|
||||
if d.Menu.Admin {
|
||||
d.Notice, d.Problem = r.URL.Query().Get("notice"), r.URL.Query().Get("problem")
|
||||
}
|
||||
if src != nil {
|
||||
d.Nodes = src.Nodes()
|
||||
d.CPU = cpuCapacityOf(d.Nodes)
|
||||
|
||||
@@ -4,13 +4,19 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"html/template"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -37,9 +43,15 @@ type heldDetail struct {
|
||||
// heldModel is one catalog model running a held version.
|
||||
type heldModel struct {
|
||||
modelUse
|
||||
View adminapi.ModelView
|
||||
Replicas []heldReplica
|
||||
Headroom *adminapi.HeadroomView
|
||||
// Adapters are the model's own; AdapterChoices the held adapter files
|
||||
// of its architecture not attached yet; FindAdapters the explorer's
|
||||
// list of adapters made for it.
|
||||
Adapters []catalog.Adapter
|
||||
AdapterChoices []adapterChoice
|
||||
FindAdapters template.URL
|
||||
View adminapi.ModelView
|
||||
Replicas []heldReplica
|
||||
Headroom *adminapi.HeadroomView
|
||||
// MostNow is how many replicas could always run, Current plus the room
|
||||
// found; zero when the room was not measured or had no end in sight.
|
||||
MostNow int
|
||||
@@ -74,7 +86,7 @@ func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDe
|
||||
}
|
||||
if p.fleet == nil {
|
||||
for _, u := range row.UsedBy {
|
||||
d.Models = append(d.Models, heldModel{modelUse: u})
|
||||
d.Models = append(d.Models, heldModel{modelUse: u, Adapters: u.Adapters})
|
||||
}
|
||||
return d
|
||||
}
|
||||
@@ -88,7 +100,7 @@ func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDe
|
||||
}
|
||||
hs, _ := p.fleet.(HeadroomSource)
|
||||
for _, u := range row.UsedBy {
|
||||
hm := heldModel{modelUse: u, View: views[u.Name]}
|
||||
hm := heldModel{modelUse: u, View: views[u.Name], Adapters: u.Adapters}
|
||||
for _, rep := range p.fleet.Replicas(u.Name) {
|
||||
// Built field by field: a ReplicaView carries the engine's key,
|
||||
// which is for the admin listener and never for a page.
|
||||
@@ -282,3 +294,140 @@ func (p *AccountPages) setModel(ctx context.Context, admin store.User, r *http.R
|
||||
}
|
||||
return name + " " + strings.Join(done, "; "), nil
|
||||
}
|
||||
|
||||
// heldKinds reads what each held single-file GGUF is — an adapter or a model —
|
||||
// once per file, remembered after.
|
||||
func (p *AccountPages) heldKinds(ctx context.Context, rows []mirrorRow) {
|
||||
var wg sync.WaitGroup
|
||||
sem := make(chan struct{}, 6)
|
||||
for i := range rows {
|
||||
row := &rows[i]
|
||||
if row.State != mirror.Present || row.Files != 1 || !gguf.IsGGUF(row.File) {
|
||||
continue
|
||||
}
|
||||
if h, ok := p.kinds.Load(row.URL); ok {
|
||||
row.Kind, row.Arch = kindName(h.(gguf.Header)), h.(gguf.Header).Architecture
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{}
|
||||
defer func() { <-sem }()
|
||||
cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
if h, err := kindOf(cctx, row.URL); err == nil {
|
||||
p.kinds.Store(row.URL, h)
|
||||
row.Kind, row.Arch = kindName(h), h.Architecture
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func kindName(h gguf.Header) string {
|
||||
if h.IsAdapter() {
|
||||
return "adapter"
|
||||
}
|
||||
return "model"
|
||||
}
|
||||
|
||||
// adapterChoice is a held adapter file that could be attached to a model.
|
||||
type adapterChoice struct {
|
||||
URL, Label, Suggest string
|
||||
}
|
||||
|
||||
// addAdapterChoices offers, on each model of a held model's card, the held
|
||||
// adapter files of the same architecture, and a way to find more.
|
||||
func (d *heldDetail) addAdapterChoices(row mirrorRow, rows []mirrorRow) {
|
||||
for i := range d.Models {
|
||||
hm := &d.Models[i]
|
||||
hm.FindAdapters = findAdaptersHref(row.Source)
|
||||
if row.Kind == "adapter" {
|
||||
continue
|
||||
}
|
||||
for _, r := range rows {
|
||||
if r.Kind != "adapter" || (row.Arch != "" && r.Arch != row.Arch) {
|
||||
continue
|
||||
}
|
||||
attached := false
|
||||
for _, a := range hm.Adapters {
|
||||
attached = attached || a.Source == r.URL
|
||||
}
|
||||
if !attached {
|
||||
hm.AdapterChoices = append(hm.AdapterChoices, adapterChoice{URL: r.URL,
|
||||
Label: r.Model + " " + r.Version + " · " + r.File, Suggest: hm.Name + "-" + r.Model})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// findAdaptersHref is the explorer's list of adapters for a held model: those
|
||||
// made for the model its repository was made from, when it came from the
|
||||
// registry, and every adapter otherwise.
|
||||
func findAdaptersHref(source string) template.URL {
|
||||
const base = "/admin/mirror?tab=explore&view=adapters"
|
||||
u, err := url.Parse(source)
|
||||
if err != nil || u.Host != "huggingface.co" {
|
||||
return base
|
||||
}
|
||||
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
|
||||
if len(parts) < 2 {
|
||||
return base
|
||||
}
|
||||
return template.URL(base + "&from=" + url.QueryEscape(parts[0]+"/"+parts[1]))
|
||||
}
|
||||
|
||||
// setAdapters attaches or detaches one adapter on a model, from its card.
|
||||
// An adapter is checked before it is attached: a file that is not an adapter,
|
||||
// or one made for another architecture, is a replica that fails to start.
|
||||
func (p *AccountPages) setAdapters(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
name := r.FormValue("model")
|
||||
current, err := p.currentModel(ctx, name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
adapters := slices.Clone(current.Adapters)
|
||||
switch r.FormValue("op") {
|
||||
case "remove":
|
||||
gone := r.FormValue("name")
|
||||
adapters = slices.DeleteFunc(adapters, func(a catalog.Adapter) bool { return a.Name == gone })
|
||||
if len(adapters) == len(current.Adapters) {
|
||||
return "", fmt.Errorf("%s has no adapter %q", name, gone)
|
||||
}
|
||||
if _, err := p.catalog.SetAdapters(ctx, name, adapters, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprintf("%s: adapter %s removed; replicas drop it when restarted (llmgw ctl restart --model %s)",
|
||||
name, gone, name), nil
|
||||
case "add":
|
||||
an := strings.TrimSpace(r.FormValue("name"))
|
||||
src := strings.TrimSpace(r.FormValue("source"))
|
||||
scale := 1.0
|
||||
if v := strings.TrimSpace(r.FormValue("scale")); v != "" {
|
||||
if scale, err = strconv.ParseFloat(v, 64); err != nil || scale <= 0 {
|
||||
return "", errors.New("the scale is a positive number")
|
||||
}
|
||||
}
|
||||
if an == "" || src == "" {
|
||||
return "", errors.New("an adapter needs a name and a file")
|
||||
}
|
||||
h, err := kindOf(ctx, src)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the adapter file could not be read: %w", err)
|
||||
}
|
||||
if !h.IsAdapter() {
|
||||
return "", errors.New("that file is a model, not a LoRA adapter")
|
||||
}
|
||||
if base, err := kindOf(ctx, current.Source); err == nil && base.Architecture != h.Architecture {
|
||||
return "", fmt.Errorf("that adapter is for %s models and %s is %s", h.Architecture, name, base.Architecture)
|
||||
}
|
||||
adapters = append(adapters, catalog.Adapter{Name: an, Source: src, Scale: scale})
|
||||
if _, err := p.catalog.SetAdapters(ctx, name, adapters, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return fmt.Sprintf("%s: adapter %s added; ask for it by that name once the replicas are restarted "+
|
||||
"(llmgw ctl restart --model %s)", name, an, name), nil
|
||||
}
|
||||
return "", errors.New("add or remove an adapter")
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
@@ -9,6 +10,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -269,3 +271,65 @@ func TestRoomAHigherClassWouldMakeIsOfferedAndSaid(t *testing.T) {
|
||||
t.Errorf("three is past even what preemption makes: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdapterHeldInTheMirrorIsAttachedFromItsModelsCard(t *testing.T) {
|
||||
prev := kindOf
|
||||
t.Cleanup(func() { kindOf = prev })
|
||||
kindOf = func(_ context.Context, u string) (gguf.Header, error) {
|
||||
switch {
|
||||
case strings.Contains(u, "/sql/"):
|
||||
return gguf.Header{Architecture: "qwen2", Type: "adapter"}, nil
|
||||
case strings.Contains(u, "/llama-lora/"):
|
||||
return gguf.Header{Architecture: "llama", Type: "adapter"}, nil
|
||||
}
|
||||
return gguf.Header{Architecture: "qwen2"}, nil
|
||||
}
|
||||
cat := coderCatalog()
|
||||
coder := &cat.sources["http://mirror:8081/coder/v1/c.gguf"][0]
|
||||
coder.Source = "http://mirror:8081/coder/v1/c.gguf"
|
||||
coder.Profiles = []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12}}
|
||||
m := &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1,
|
||||
Source: "https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/abc/c.gguf"},
|
||||
{Model: "sql", Version: "v1", File: "sql.gguf", State: mirror.Present, BytesDone: 1},
|
||||
{Model: "llama-lora", Version: "v1", File: "l.gguf", State: mirror.Present, BytesDone: 1},
|
||||
}}
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Catalog: cat, Fleet: &roomyFleet{more: 1}}, true)
|
||||
|
||||
list := mirrorGet(t, mux, "/admin/mirror").Body.String()
|
||||
if strings.Count(list, ">LoRA adapter</span>") != 2 {
|
||||
t.Error("the two adapter files are not marked as adapters in the list")
|
||||
}
|
||||
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
|
||||
if !strings.Contains(card, `<option value="http://mirror:8081/sql/v1/sql.gguf" data-suggest="coder-sql">`) {
|
||||
t.Error("the qwen2 adapter is not offered for the qwen2 model")
|
||||
}
|
||||
if strings.Contains(card, "llama-lora") && strings.Contains(card, `value="http://mirror:8081/llama-lora/v1/l.gguf"`) {
|
||||
t.Error("an adapter for another architecture is offered")
|
||||
}
|
||||
if !strings.Contains(card, "view=adapters&from=bartowski%2FQwen2.5-Coder-7B-Instruct-GGUF") {
|
||||
t.Error("finding adapters should start from the model's own repository")
|
||||
}
|
||||
|
||||
post := func(v url.Values) string {
|
||||
v.Set("csrf", "tok")
|
||||
v.Set("model", "coder")
|
||||
v.Set("open", "coder:v1")
|
||||
return mirrorPost(t, mux, "/admin/mirror/adapters", v).Body.String()
|
||||
}
|
||||
body := post(url.Values{"op": {"add"}, "name": {"coder-sql"}, "source": {"http://mirror:8081/sql/v1/sql.gguf"}, "scale": {"0.8"}})
|
||||
if len(cat.adapterSets) != 1 || len(cat.adapterSets[0]) != 1 || cat.adapterSets[0][0].Scale != 0.8 {
|
||||
t.Fatalf("attached %+v: %s", cat.adapterSets, body)
|
||||
}
|
||||
if !strings.Contains(body, "adapter coder-sql added") {
|
||||
t.Errorf("the attach was not said: %s", body)
|
||||
}
|
||||
for name, src := range map[string]string{"another architecture": "http://mirror:8081/llama-lora/v1/l.gguf",
|
||||
"a model": "http://mirror:8081/coder/v1/c.gguf"} {
|
||||
n := len(cat.adapterSets)
|
||||
out := post(url.Values{"op": {"add"}, "name": {"x"}, "source": {src}})
|
||||
if len(cat.adapterSets) != n || !strings.Contains(out, `class="error"`) {
|
||||
t.Errorf("%s was attached", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,9 @@ const (
|
||||
viewTrending = "trending"
|
||||
viewNew = "new"
|
||||
viewDownloads = "downloads"
|
||||
// viewAdapters lists LoRA adapters, all of them or those made for one
|
||||
// model.
|
||||
viewAdapters = "adapters"
|
||||
)
|
||||
|
||||
// newWithin is how recent a repository has to be to count as new.
|
||||
@@ -31,6 +34,7 @@ const exploreShown = 25
|
||||
|
||||
var exploreViews = []struct{ key, label string }{
|
||||
{viewTrending, "Trending"}, {viewNew, "New this month"}, {viewDownloads, "Most downloaded"},
|
||||
{viewAdapters, "LoRA adapters"},
|
||||
}
|
||||
|
||||
var exploreTasks = []struct{ key, label, tag string }{
|
||||
@@ -48,6 +52,9 @@ var exploreSizes = []struct {
|
||||
// open: a listing, its filters, and links that keep them.
|
||||
type explorer struct {
|
||||
View, Task, Size string
|
||||
// Of is the model whose adapters are listed, in the adapters view; empty
|
||||
// lists adapters for any model.
|
||||
Of string
|
||||
// Here keeps to what the nodes' llama.cpp can load. On by default: a
|
||||
// trending list is mostly things this cluster cannot run, and finding
|
||||
// that out one repository at a time is what the explorer is for avoiding.
|
||||
@@ -123,11 +130,12 @@ func trimZero(s string) string { return strings.TrimSuffix(s, ".0") }
|
||||
// — the page opened from the menu — it is trending chat models this cluster
|
||||
// can run.
|
||||
func exploreState(q url.Values) explorer {
|
||||
e := explorer{View: q.Get("view"), Task: q.Get("task"), Size: q.Get("size")}
|
||||
e := explorer{View: q.Get("view"), Task: q.Get("task"), Size: q.Get("size"),
|
||||
Of: strings.Trim(strings.TrimSpace(q.Get("of")), "/")}
|
||||
// "x" marks a submitted form, so an unticked box reads as off rather than
|
||||
// as not yet chosen.
|
||||
e.Here = q.Get("x") != "1" || q.Get("here") == "1"
|
||||
if !oneOf(e.View, viewTrending, viewNew, viewDownloads) {
|
||||
if !oneOf(e.View, viewTrending, viewNew, viewDownloads, viewAdapters) {
|
||||
e.View = viewTrending
|
||||
// A search with no view asks for what a name usually means, which is
|
||||
// the one everybody downloads.
|
||||
@@ -165,6 +173,10 @@ func (e explorer) href(query, key, value string) template.URL {
|
||||
if key != "" {
|
||||
v.Set(key, value)
|
||||
}
|
||||
// Whose adapters, kept while it is adapters being looked at.
|
||||
if e.Of != "" && v.Get("view") == viewAdapters {
|
||||
v.Set("of", e.Of)
|
||||
}
|
||||
return template.URL("/admin/mirror?" + v.Encode())
|
||||
}
|
||||
|
||||
@@ -181,6 +193,16 @@ func (e explorer) hfQuery(search string) hf.Query {
|
||||
// Trending, then kept to the recent: sorted by creation date alone,
|
||||
// the newest repositories are mostly empty uploads nobody has used.
|
||||
q.Limit = 100
|
||||
case viewAdapters:
|
||||
// Adapters rarely carry a task or a parameter count, so neither is
|
||||
// asked for; what makes one useful is the model it was made for.
|
||||
q.Sort = hf.SortDownloads
|
||||
if e.Of != "" {
|
||||
q.Filters = []string{"base_model:adapter:" + e.Of}
|
||||
} else {
|
||||
q.Filters = []string{"lora"}
|
||||
}
|
||||
return q
|
||||
}
|
||||
for _, t := range exploreTasks {
|
||||
if t.key == e.Task {
|
||||
@@ -198,6 +220,15 @@ func (e explorer) hfQuery(search string) hf.Query {
|
||||
// exploreList fills in the listing.
|
||||
func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
|
||||
e := exploreState(r.URL.Query())
|
||||
// From a held model's card: its repository, whose adapters are those of
|
||||
// the model it was made from — a GGUF repository is a quantisation, and
|
||||
// adapters are made for the original.
|
||||
if from := strings.TrimSpace(r.URL.Query().Get("from")); e.View == viewAdapters && e.Of == "" && from != "" {
|
||||
e.Of = from
|
||||
if repo, err := p.registry.Repo(r.Context(), from); err == nil && len(repo.BaseModel) > 0 {
|
||||
e.Of = repo.BaseModel[0]
|
||||
}
|
||||
}
|
||||
for _, v := range exploreViews {
|
||||
e.Views = append(e.Views, exploreLink{v.label, e.href(d.Query, "view", v.key), v.key == e.View})
|
||||
}
|
||||
@@ -225,6 +256,14 @@ func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
|
||||
if a := m.Architecture(); a != "" && p.llamaVersion != "" {
|
||||
row.Verdict = gguf.LlamaCpp(p.llamaVersion, a)
|
||||
}
|
||||
// Adapters are small and say little about their architecture in a
|
||||
// listing; whether one fits is read from its file when it is opened.
|
||||
if e.View == viewAdapters {
|
||||
if len(e.Rows) < exploreShown {
|
||||
e.Rows = append(e.Rows, row)
|
||||
}
|
||||
continue
|
||||
}
|
||||
// An adapter or a fragment, not a model: tens of megabytes of LoRA
|
||||
// weights listed among the models they need a model to run on.
|
||||
if e.Here && m.Params() > 0 && m.Params() < minParams {
|
||||
|
||||
@@ -1,15 +1,18 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"html"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
)
|
||||
|
||||
@@ -195,3 +198,52 @@ func TestAFetchFromTheListAnswersWithItsOutcomeAlone(t *testing.T) {
|
||||
t.Errorf("a refusal: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// noProbe answers every file as an unreadable one, for tests that are not
|
||||
// about what the files are.
|
||||
func TestMain(m *testing.M) {
|
||||
kindOf = func(context.Context, string) (gguf.Header, error) {
|
||||
return gguf.Header{}, errors.New("not probed in tests")
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func TestAdaptersAreListedForAModelAndToldFromModelsByTheirFiles(t *testing.T) {
|
||||
prev := kindOf
|
||||
t.Cleanup(func() { kindOf = prev })
|
||||
kindOf = func(_ context.Context, url string) (gguf.Header, error) {
|
||||
if strings.Contains(url, "adapter") {
|
||||
return gguf.Header{Architecture: "qwen2", Type: "adapter"}, nil
|
||||
}
|
||||
return gguf.Header{Architecture: "qwen2"}, nil
|
||||
}
|
||||
reg := &fakeRegistry{
|
||||
results: []hf.Model{{ID: "ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF", Downloads: 900}},
|
||||
repo: hf.Repo{ID: "ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF", Revision: "776530df0000",
|
||||
BaseModel: []string{"Qwen/Qwen2.5-Coder-7B-Instruct"},
|
||||
Files: []hf.File{{Path: "lora-adapter-f16.gguf", Bytes: 726 << 20}, {Path: "merged-Q4_K_M.gguf", Bytes: 4 << 30}}},
|
||||
}
|
||||
mux := mirrorPageWith(t, &fakeMirror{}, reg, true)
|
||||
|
||||
// From a held model's card: its repository is looked up, and the
|
||||
// adapters listed are those of the model it was made from.
|
||||
body := mirrorGet(t, mux, "/admin/mirror?tab=explore&view=adapters&from=bartowski/Qwen2.5-Coder-7B-Instruct-GGUF").Body.String()
|
||||
want := "base_model:adapter:Qwen/Qwen2.5-Coder-7B-Instruct"
|
||||
if len(reg.query.Filters) != 1 || reg.query.Filters[0] != want {
|
||||
t.Errorf("asked the registry for %v, expected %s", reg.query.Filters, want)
|
||||
}
|
||||
if !strings.Contains(body, "LoRA adapters made for <code>Qwen/Qwen2.5-Coder-7B-Instruct</code>") {
|
||||
t.Error("the page does not say whose adapters these are")
|
||||
}
|
||||
if strings.Contains(body, `aria-label="Task"`) {
|
||||
t.Error("task and size filters shown for adapters, which rarely carry either")
|
||||
}
|
||||
|
||||
repo := mirrorGet(t, mux, "/admin/mirror?repo=ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF").Body.String()
|
||||
if !strings.Contains(repo, "LoRA adapter · qwen2") {
|
||||
t.Error("the adapter file is not marked as one")
|
||||
}
|
||||
if !strings.Contains(repo, `<span class="opt">model · qwen2</span>`) {
|
||||
t.Error("the merged file is not marked as a model")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// signupAccounts is an account service whose sign-up answers with err.
|
||||
type signupAccounts struct {
|
||||
fakeAccounts
|
||||
err error
|
||||
}
|
||||
|
||||
func (s signupAccounts) SignUp(context.Context, string, string, string) error { return s.err }
|
||||
func (signupAccounts) SignupOpen() bool { return true }
|
||||
|
||||
func signupBody(t *testing.T, err error) string {
|
||||
t.Helper()
|
||||
mux := http.NewServeMux()
|
||||
NewAccountPages(signupAccounts{err: err}, nil, nil, "v0", slog.New(slog.DiscardHandler)).Register(mux)
|
||||
form := url.Values{"email": {"a@b.c"}, "name": {"A"}, "password": {"long enough password"}}
|
||||
req := httptest.NewRequest(http.MethodPost, "/signup", strings.NewReader(form.Encode()))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec.Body.String()
|
||||
}
|
||||
|
||||
func TestASignupThatFailsSaysSoWithAReferenceAndNoDatabaseWords(t *testing.T) {
|
||||
body := signupBody(t, errors.New(`duplicate key value violates unique constraint "users_email_key"`))
|
||||
if strings.Contains(body, "users_email_key") || strings.Contains(body, "constraint") {
|
||||
t.Errorf("the database's own words reached the page: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "was not created") || !strings.Contains(body, "reference: err-") {
|
||||
t.Errorf("no plain failure with a reference to give an operator: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASignupSaysWhatToDoIfTheAddressHadAnAccount(t *testing.T) {
|
||||
// The service answers a taken address as it does a new one, so nobody
|
||||
// can find out from this page who has an account; what the page can do
|
||||
// is say what to do in either case.
|
||||
body := signupBody(t, nil)
|
||||
if !strings.Contains(body, "already had an account") || !strings.Contains(body, `href="/login"`) ||
|
||||
!strings.Contains(body, "reset it") {
|
||||
t.Errorf("the done page leaves someone who already had an account stuck: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -858,3 +858,7 @@ table.rows td.warn { color: var(--warn); cursor: help; }
|
||||
.held-card .replicas-form select, .capacity-form select { display: block; margin-top: .25rem; }
|
||||
.held-card .replicas-form .field-row { flex-wrap: wrap; gap: .75rem 1rem; }
|
||||
.quota-tree td.q-act form.inline { margin-left: .75rem; }
|
||||
.held-card .adapters { margin: .75rem 0; }
|
||||
.held-card .adapters h4 { font-size: .9rem; margin: 0 0 .35rem; display: flex; gap: .75rem; align-items: baseline; }
|
||||
.held-card .adapter-form .field-row { align-items: flex-end; flex-wrap: wrap; gap: .5rem 1rem; }
|
||||
.held-card .adapter-form select { display: block; margin-top: .25rem; max-width: 28rem; }
|
||||
@@ -11,6 +11,11 @@
|
||||
<p class="note">You will not get an email from this gateway — it does not send any.
|
||||
Tell whoever you asked for access that you have signed up.</p>
|
||||
<p><a href="/login">Sign in</a> once you have been approved.</p>
|
||||
{{/* The same page whether or not the address already had an account, so
|
||||
it says what to do in either case without saying which this was. */}}
|
||||
<p class="note">If this address already had an account, nothing new was made:
|
||||
<a href="/login">sign in</a> with it, or, if you have forgotten its password, ask
|
||||
an operator to reset it.</p>
|
||||
{{else}}
|
||||
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
|
||||
<form method="post" autocomplete="on">
|
||||
|
||||
@@ -47,7 +47,8 @@ reachable from this machine.</p>
|
||||
{{range .Rows}}{{$row0 := .}}<tr{{if $grouped}} class="model-version"{{end}}{{if .Detail}} class="open"{{end}}>
|
||||
<td>{{if eq (printf "%s" .State) "present"}}<a class="open-held" aria-expanded="{{if .Detail}}true{{else}}false{{end}}"
|
||||
href="/admin/mirror?tab=held{{if not .Detail}}&open={{.Open}}{{end}}"><strong>{{.Model}}</strong></a>
|
||||
{{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span></td>
|
||||
{{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span>
|
||||
{{if eq .Kind "adapter"}}<span class="tag new" title="a LoRA adapter for {{.Arch}} models: attach it on a model's card">LoRA adapter</span>{{end}}</td>
|
||||
<td class="file" title="{{.File}}{{if .SHA256}} · sha256 {{.SHA256}}{{end}}">
|
||||
{{if gt .Files 1}}{{.Files}} files{{else}}<code>{{.File}}</code>{{end}}</td>
|
||||
<td class="num">
|
||||
@@ -130,7 +131,7 @@ reachable from this machine.</p>
|
||||
<p class="error">Cannot be served as it stands: {{.Refusal}}.</p>
|
||||
{{else}}
|
||||
{{if .Warning}}<p class="note error-row">{{.Warning}}.</p>{{end}}
|
||||
<form method="post" action="/admin/mirror/serve">
|
||||
<form method="post" action="/admin/mirror/serve" class="serve-form" data-engine="{{.Engine}}" data-free-gpus="{{.FreeGPUs}}">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="source" value="{{.Source}}">
|
||||
<input type="hidden" name="engine" value="{{.Engine}}">
|
||||
@@ -162,8 +163,23 @@ reachable from this machine.</p>
|
||||
<label>Engine context pool <span class="opt">divided across slots</span>
|
||||
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
|
||||
</div>
|
||||
{{/* Where it runs is a choice with consequences, so it is a choice: the
|
||||
GPU classes the fleet has, and the CPU for llama.cpp. A profile name
|
||||
field stood here once, and reading as though naming a profile "gpu"
|
||||
would put it on one, it put GGUF models on the CPU of GPU clusters. */}}
|
||||
<div class="field-row">
|
||||
<label>Runs on
|
||||
<select name="placement">
|
||||
{{range .Placements}}<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>{{end}}
|
||||
</select></label>
|
||||
<label>GPUs per replica <span class="opt">on one node</span>
|
||||
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>
|
||||
{{if eq .Engine "llamacpp"}}<label>Parallel requests <span class="opt">per replica</span>
|
||||
<input type="text" name="parallel" value="{{.Parallel}}" size="3"></label>{{end}}
|
||||
</div>
|
||||
{{if .FreeGPUs}}<label class="cpu-ok"><input type="checkbox" name="cpu_ok" value="yes">
|
||||
Run on the CPU anyway: {{.FreeGPUs}} GPUs are free and would sit unused</label>{{end}}
|
||||
<div class="field-row">
|
||||
<label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
|
||||
<label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
|
||||
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}" required></label>
|
||||
<label>CPU cores <span class="opt">reserved; the engine's threads</span>
|
||||
@@ -172,8 +188,44 @@ reachable from this machine.</p>
|
||||
<input type="text" name="selector_key" placeholder="role" size="8">
|
||||
<input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
|
||||
</div>
|
||||
<p class="sub serve-summary" hidden></p>
|
||||
<button type="submit">Add to the catalog</button>
|
||||
</form>
|
||||
<script>
|
||||
// What it will run as, in one line, kept up to date as the form changes:
|
||||
// engine, where, how many cards, memory, context and how it is shared.
|
||||
(function () {
|
||||
const f = document.currentScript.previousElementSibling;
|
||||
if (!f || !f.classList.contains("serve-form")) return;
|
||||
const out = f.querySelector(".serve-summary"), ok = f.querySelector(".cpu-ok");
|
||||
const v = (n) => (f.elements[n] ? f.elements[n].value.trim() : "");
|
||||
const engine = { llamacpp: "llama.cpp", vllm: "vLLM", sglang: "SGLang" }[f.dataset.engine] || f.dataset.engine;
|
||||
function draw() {
|
||||
const place = v("placement"), gpus = parseInt(v("gpus"), 10) || 1;
|
||||
const onCPU = place === "cpu" || place === "";
|
||||
let where = "on the CPU";
|
||||
if (!onCPU) {
|
||||
const cls = place.replace(/^gpu:/, "");
|
||||
where = gpus > 1 ? "on " + gpus + " " + cls + " GPUs of one node" : "on 1 GPU " + cls;
|
||||
}
|
||||
const parts = [engine + " " + where];
|
||||
if (v("min_memory_gb")) parts.push(v("min_memory_gb") + " GB of host memory");
|
||||
if (onCPU && v("cpu_cores")) parts.push(v("cpu_cores") + " cores");
|
||||
const pool = parseInt(v("ctx_size"), 10), par = parseInt(v("parallel"), 10);
|
||||
if (pool && par) parts.push("context pool " + pool + " over " + par + " parallel requests (" + Math.floor(pool / par) + " each)");
|
||||
else if (pool) parts.push("context pool " + pool);
|
||||
out.textContent = "Will run as: " + parts.join(", ") + ".";
|
||||
const warn = onCPU && +f.dataset.freeGpus > 0;
|
||||
out.classList.toggle("error", warn);
|
||||
if (warn) out.textContent += " " + f.dataset.freeGpus + " GPUs are free; on the CPU this runs many times slower.";
|
||||
if (ok) ok.hidden = !onCPU;
|
||||
out.hidden = false;
|
||||
}
|
||||
f.addEventListener("input", draw);
|
||||
f.addEventListener("change", draw);
|
||||
draw();
|
||||
})();
|
||||
</script>
|
||||
<p class="note">The context pool is what the engine is given; llama.cpp divides
|
||||
it across its parallel slots, so the number one request may use is the smaller
|
||||
one above.</p>
|
||||
@@ -215,16 +267,25 @@ document.addEventListener('click', function (e) {
|
||||
{{if and (eq .Tab "explore") .HasRegistry}}
|
||||
{{if not .Repo}}{{with .Explore}}
|
||||
<nav class="pills" aria-label="Which models">{{range .Views}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}</a>{{end}}</nav>
|
||||
{{if eq .View "adapters"}}
|
||||
<p class="note">{{if .Of}}LoRA adapters made for <code>{{.Of}}</code> ·
|
||||
<a href="/admin/mirror?tab=explore&view=adapters">all adapters</a>{{else}}LoRA adapters for any model.
|
||||
From a held model's card, <em>find adapters</em> lists those made for it.{{end}}
|
||||
An adapter runs on the model it was made for, and only as a GGUF adapter file;
|
||||
open one to see which of its files are.</p>
|
||||
{{else}}
|
||||
<div class="explore-filters">
|
||||
<nav class="pills small" aria-label="Task">{{range .Tasks}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
|
||||
<nav class="pills small" aria-label="Size">{{range .Sizes}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
|
||||
</div>
|
||||
{{end}}
|
||||
{{end}}
|
||||
<form method="get" action="/admin/mirror" class="explore-search">
|
||||
<input type="hidden" name="tab" value="explore">
|
||||
{{with .Explore}}<input type="hidden" name="view" value="{{.View}}">
|
||||
<input type="hidden" name="task" value="{{.Task}}">
|
||||
<input type="hidden" name="size" value="{{.Size}}">
|
||||
{{if .Of}}<input type="hidden" name="of" value="{{.Of}}">{{end}}
|
||||
<input type="hidden" name="x" value="1">{{end}}
|
||||
<input type="search" name="q" value="{{.Query}}" placeholder="Search Hugging Face — qwen coder, gemma, bge…" aria-label="Search Hugging Face">
|
||||
<label class="check"><input type="checkbox" name="here" value="1" onchange="this.form.submit()"{{if .Explore}}{{if .Explore.Here}} checked{{end}}{{end}}> only what these nodes can run</label>
|
||||
@@ -405,7 +466,10 @@ version and the old weights stay where they are.</p>
|
||||
<tr><th>What</th><th class="num">Size</th><th>Published checksum</th><th></th></tr>
|
||||
{{range .RepoSets}}<tr>
|
||||
<td><code>{{.Name}}</code>
|
||||
{{if .Shards}}<span class="opt">{{.Shards}} shards, fetched together</span>{{end}}</td>
|
||||
{{if .Shards}}<span class="opt">{{.Shards}} shards, fetched together</span>{{end}}
|
||||
{{with .Companions}}<span class="opt" title="{{join .}}">with {{len .}} files the engine reads beside them</span>{{end}}
|
||||
{{if eq .Kind "adapter"}}<span class="tag new" title="a LoRA adapter: it runs on a {{.Arch}} model it was made for, not on its own">LoRA adapter · {{.Arch}}</span>
|
||||
{{else if eq .Kind "model"}}<span class="opt">model{{with .Arch}} · {{.}}{{end}}</span>{{end}}</td>
|
||||
<td class="num">{{bytes .Bytes}}</td>
|
||||
<td>{{with .SHA256}}<code title="{{.}}">{{slice . 0 12}}</code>{{else}}—{{end}}</td>
|
||||
<td>
|
||||
@@ -429,7 +493,8 @@ version and the old weights stay where they are.</p>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<p class="note">Only the weights are listed; a repository is mostly a README and
|
||||
a tokenizer. Shards of one file are one row and one button — a fifteen-shard
|
||||
a tokenizer. A safetensors set is fetched with the files vLLM reads beside it —
|
||||
config.json, the tokenizer, the shard index — and never with code. Shards of one file are one row and one button — a fifteen-shard
|
||||
model is one decision, not fifteen. Two separate files are two rows because they
|
||||
are usually alternatives: one quantisation or the other, not both.</p>
|
||||
<p class="note">The source URLs are pinned to
|
||||
@@ -489,6 +554,43 @@ computes once it has it.</p>
|
||||
<ul>{{range .Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if not .Declared}}{{$mname := .Name}}
|
||||
<div class="adapters">
|
||||
<h4>LoRA adapters <a class="note" href="{{.FindAdapters}}">find adapters on Hugging Face →</a></h4>
|
||||
{{if .Adapters}}<table class="compact rows">
|
||||
<tr><th>Asked for as</th><th>File</th><th class="num">Scale</th><th></th></tr>
|
||||
{{range .Adapters}}{{$a := .}}<tr>
|
||||
<td><code>{{.Name}}</code></td>
|
||||
<td class="clip" title="{{.Source}}">{{.Source}}</td>
|
||||
<td class="num">{{.EffectiveScale}}</td>
|
||||
<td class="actions"><form method="post" action="/admin/mirror/adapters" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$mname}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
<input type="hidden" name="op" value="remove">
|
||||
<input type="hidden" name="name" value="{{$a.Name}}">
|
||||
<button type="submit" class="link danger">remove</button></form></td>
|
||||
</tr>{{end}}</table>
|
||||
{{else}}<p class="note">None. A request for an adapter's name is served by this model's replicas with it applied.</p>{{end}}
|
||||
{{if .AdapterChoices}}{{$m := .}}
|
||||
<form method="post" action="/admin/mirror/adapters" class="adapter-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
<input type="hidden" name="op" value="add">
|
||||
<div class="field-row">
|
||||
<label>Adapter file <select name="source" required
|
||||
onchange="this.form.elements.name.value = this.selectedOptions[0].dataset.suggest">
|
||||
{{range .AdapterChoices}}<option value="{{.URL}}" data-suggest="{{.Suggest}}">{{.Label}}</option>{{end}}
|
||||
</select></label>
|
||||
<label>Asked for as <input type="text" name="name" value="{{(index .AdapterChoices 0).Suggest}}" required></label>
|
||||
<label>Scale <input type="text" name="scale" value="1" size="4"></label>
|
||||
<button type="submit">Attach</button>
|
||||
</div>
|
||||
</form>{{else}}<p class="note">No held adapter file for this model's architecture. Fetch one from Hugging Face first.</p>{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if .Declared}}
|
||||
<p class="note">Set in server.yaml, which is read again on every start: change its replicas there.</p>
|
||||
{{else}}
|
||||
|
||||
@@ -50,8 +50,9 @@
|
||||
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
|
||||
|
||||
<h2>Nodes</h2>
|
||||
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
|
||||
{{if .Nodes}}
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyGPUs}}<th>vLLM build</th>{{end}}{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
{{range .Nodes}}<tr>
|
||||
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
|
||||
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
|
||||
@@ -66,6 +67,23 @@
|
||||
was never asked about its disk must not read as a full one. */}}
|
||||
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
|
||||
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
|
||||
{{/* Which vLLM build the node installs: left to its hardware, or chosen.
|
||||
What is installed beside it, so a choice and its outcome are read
|
||||
together. Only a GPU node has one; changing it is for an
|
||||
administrator, on the page reached through a session. */}}
|
||||
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{if and $.Menu.CSRF .EngineBuilds}}
|
||||
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
|
||||
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
|
||||
<input type="hidden" name="engine" value="vllm">
|
||||
<select name="build" title="the vLLM build this node installs">
|
||||
{{range engineBuilds "vllm"}}<option value="{{.}}"{{if eq . (index $node.EngineBuilds "vllm")}} selected{{end}}>{{buildLabel .}}</option>{{end}}
|
||||
</select>
|
||||
<noscript><button type="submit">Save</button></noscript>
|
||||
</form>
|
||||
{{end}}
|
||||
{{else}}—{{end}}</td>{{end}}
|
||||
{{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
|
||||
{{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
|
||||
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
|
||||
@@ -75,6 +93,13 @@
|
||||
{{range .Provision}}{{if and (ne .State "ready") (ne .State "skipped")}}<tr><td colspan="12" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}</td></tr>{{end}}{{end}}
|
||||
{{range .Warnings}}<tr><td colspan="12" class="note bad">{{.}}</td></tr>{{end}}
|
||||
{{end}}</table>
|
||||
<script>
|
||||
// A build chosen is saved as it is chosen, with the page reloaded on the
|
||||
// answer: one select, no separate button to forget.
|
||||
document.querySelectorAll("form.build-choice select").forEach(function (s) {
|
||||
s.addEventListener("change", function () { s.form.submit(); });
|
||||
});
|
||||
</script>
|
||||
{{else}}<p class="empty">No agents have registered.</p>{{end}}
|
||||
|
||||
<h2>Models</h2>
|
||||
@@ -89,7 +114,7 @@
|
||||
<td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
|
||||
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
|
||||
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
|
||||
<td>{{join .Profiles}}</td>
|
||||
<td>{{if .External}}<span class="tag" title="served outside the cluster; its prompts leave it">external</span> {{.External}}{{else}}{{join .Profiles}}{{end}}</td>
|
||||
{{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
|
||||
{{/* Cut to the column, the whole sentence on hover: a row of its own for
|
||||
each description made the table twice as long as the catalog. */}}
|
||||
|
||||
@@ -82,9 +82,9 @@ the adapter came up: a number that grows is the one to look at.</p>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<table class="rows compact gpu-list">
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">Compute</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
{{range .Node.GPUs}}<tr>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.NUMA}}</td>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.ComputeCapability}}</td><td class="num">{{.NUMA}}</td>
|
||||
<td>{{if .NearestNIC}}<code>{{.NearestNIC}}</code> <span class="note">{{.NICPath}}</span>{{else}}—{{end}}</td>
|
||||
<td>{{if .Domain}}<span class="state-ready">joined</span>{{if and .Bandwidth (ne .Bandwidth "Full")}} <span class="bad">{{.Bandwidth}} bandwidth</span>{{end}}{{else if .FabricState}}<span class="bad">{{.FabricState}}</span>{{else}}<span class="note">—</span>{{end}}</td>
|
||||
</tr>{{end}}
|
||||
|
||||
+20
-1
@@ -26,6 +26,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -118,7 +119,25 @@ var funcs = template.FuncMap{
|
||||
},
|
||||
// shortNode is a node's name without what every node's name starts with.
|
||||
"shortNode": strings.TrimPrefix,
|
||||
"age": age,
|
||||
// engineBuilds are the builds an engine can be given on a node, and
|
||||
// buildLabel says one in words.
|
||||
"engineBuilds": control.EngineBuilds,
|
||||
"buildLabel": func(b string) string {
|
||||
switch b {
|
||||
case "auto":
|
||||
return "auto: what the hardware runs"
|
||||
case "cu13":
|
||||
return "CUDA 13"
|
||||
case "cu129":
|
||||
return "CUDA 12.9"
|
||||
case "cu12":
|
||||
return "CUDA 12"
|
||||
case "none":
|
||||
return "none"
|
||||
}
|
||||
return b
|
||||
},
|
||||
"age": age,
|
||||
// pctbar is a percentage as the minibar template takes it: whole, and
|
||||
// held to 0..100 so a measurement a little over does not draw past its box.
|
||||
"pctbar": func(v float64) struct{ P int } {
|
||||
|
||||
+7
-3
@@ -1459,7 +1459,8 @@ The launch path is now built too:
|
||||
hits. It does nothing when the working set fits in one replica's cache,
|
||||
which is worth knowing rather than assuming.
|
||||
- [x] Fallback models
|
||||
- [ ] Pool spillover
|
||||
- [x] Pool spillover: a saturated pool passes requests to the next one that
|
||||
is not, by admission's own measure of saturation
|
||||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||||
@@ -1496,8 +1497,11 @@ The launch path is now built too:
|
||||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||||
checked against their digest, and reclaiming what nothing uses
|
||||
- [ ] Prefill/decode disaggregation (§12)
|
||||
- [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
|
||||
- [ ] External provider support (as a fallback)
|
||||
- [x] LoRA adapters on llama.cpp: every replica loads a model's adapters and
|
||||
applies one per request, so any replica serves any adapter
|
||||
- [ ] LoRA affinity for engines that load adapters on demand (vLLM)
|
||||
- [x] External provider support: a catalog model served by an OpenAI-compatible
|
||||
API elsewhere, usable as a fallback, under the same allowlists and budgets
|
||||
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
|
||||
placement failures, usage) and a public landing page
|
||||
- [x] `/chat`: an in-browser playground for signed-in people, dispatched through
|
||||
|
||||
Reference in new issue
Block a user