Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
990d82d801 | ||
|
|
26151ec25a | ||
|
|
235bf33dcd | ||
|
|
7099043c75 | ||
|
|
a32ea0e5ab | ||
|
|
d7f1c1d545 | ||
|
|
2ae12ae1fe | ||
|
|
62657f4c2a |
No files matched your search
+120
-2
@@ -22,8 +22,103 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
Changes since v0.10.3, published as release candidates (v0.10.4-rc.N) until
|
||||
there are enough for v0.10.4.
|
||||
Changes since v0.10.4, published as release candidates (v0.10.5-rc.N) until
|
||||
there are enough for v0.10.5.
|
||||
|
||||
### CPU cores are reserved, and memory has to be declared
|
||||
|
||||
The first step of resource allocation for larger clusters: what a replica
|
||||
holds of a node is reserved, not assumed.
|
||||
|
||||
- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
|
||||
memory is, and is what its engine computes on (llama.cpp's `--threads`). A
|
||||
node holds as many CPU replicas as it has cores for, and two on one node no
|
||||
longer spin against each other. Agents report their CPU count; a node that
|
||||
does not, and a profile that declares no cores, constrain nothing.
|
||||
- **Memory is required of new CPU models.** A model with no GPU is all host
|
||||
memory, and one declaring none looked to placement as if it fitted
|
||||
everywhere, any number of times. Adding one without `min_memory_gb` is now
|
||||
refused. Models stored before this still load, and say what is missing.
|
||||
- **Declared against measured.** Each model shows what one replica reserves
|
||||
and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
|
||||
cluster page — with a warning when nothing is declared or when it uses more
|
||||
than a tenth over what it reserves.
|
||||
- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
|
||||
(and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
|
||||
`ctl model serve` and the mirror's serve form fill in cores — half the
|
||||
smallest node the model fits on — beside the memory they already estimate.
|
||||
Running replicas keep their old figures until restarted.
|
||||
|
||||
### The mirror page in tabs
|
||||
|
||||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||||
explorer and the fetch form, one under another, with every visit waiting on
|
||||
Hugging Face for a list it might not show. It is now three tabs — Held (with
|
||||
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
|
||||
URL — each with its own address. Only the tab shown does its work, and only
|
||||
Held reloads while something downloads. A download started from the explorer
|
||||
links to Held, where its progress is; a fetch the mirror refused comes back on
|
||||
the form it was sent from.
|
||||
|
||||
### A held model opens to where it runs, and how many
|
||||
|
||||
Clicking a model under Held opens a card beneath it: each replica with its
|
||||
node, state, age and the CPU and memory its engine uses, and how many more
|
||||
replicas the cluster has room for now, on which nodes, and why not more. The
|
||||
replicas it always runs and the most it grows to under load are changed there.
|
||||
Room is found by trying the scheduler itself on a copy of the inventory — the
|
||||
same decisions the next passes would make, not an estimate beside them — and a
|
||||
number of always-running replicas it cannot place is refused with the reason,
|
||||
rather than accepted and left short. A ceiling above the room is taken, and
|
||||
said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
|
||||
API.
|
||||
|
||||
Each row also says how its model runs without being opened: replicas ready of
|
||||
those it should have, how many are still starting, and the nodes they are on —
|
||||
in amber when short, in red when parked or when it does not fit. It is one line
|
||||
per model beside its mode, and nodes drop the part of the name every node
|
||||
shares (node05 for llmgw-node05, the full name on hover).
|
||||
|
||||
### One line per record, in every table
|
||||
|
||||
The same rule across the admin pages. Headers do not wrap; a record is one
|
||||
line; free text is cut to its column with the whole of it on hover rather than
|
||||
given a row of its own; a row across the whole table is only a note or an open
|
||||
form. On the cluster page that means a node is one line (its address and logs
|
||||
beside its name, not under it), a model's description is a column rather than a
|
||||
second row, node names drop the prefix they all share, and the GPUs, classes,
|
||||
fabrics and fallbacks columns appear only when some row has something in them —
|
||||
on a CPU-only cluster they were three columns of dashes pushing the table past
|
||||
the page. On People, a person in many teams shows two and a count, the rest on
|
||||
hover.
|
||||
|
||||
## v0.10.4 — 2026-09-30
|
||||
|
||||
Published first as release candidates v0.10.4-rc.1 to rc.9.
|
||||
|
||||
After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
|
||||
thread count below only when they start.
|
||||
|
||||
### CPU replicas no longer fight over the CPUs
|
||||
|
||||
llama.cpp computes on every CPU of the machine unless told otherwise, and the
|
||||
agent did not tell it. Two busy CPU replicas on one node then spin against each
|
||||
other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
|
||||
0.8 each with both busy, in two rounds of the same test. The agent now starts a
|
||||
CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
|
||||
make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
|
||||
node — all of them suits a node that only ever runs one CPU replica — and a
|
||||
profile's own `--threads` still wins. Measured on VMs whose host is
|
||||
oversubscribed; the proportions, not the numbers, are the point.
|
||||
|
||||
### Tracing
|
||||
|
||||
With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
|
||||
request is traced over OTLP/HTTP: a span for the request, carrying its
|
||||
`X-Request-Id`, with admission, scheduling and each attempt at an engine under
|
||||
it. A caller's own `traceparent` is continued, and the engine is sent one for
|
||||
its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
|
||||
without a decision. Off, it costs nothing.
|
||||
|
||||
### Restarting replicas
|
||||
|
||||
@@ -157,6 +252,27 @@ half again plus two" could not see.
|
||||
links.** Archives are unpacked through `os.Root`, and a link may only point
|
||||
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
|
||||
follows that rule.
|
||||
- **Onboarding made every node trust the cluster's root for everything.** It
|
||||
was added to the system store, so a certificate the control plane signed
|
||||
would pass for any site with every TLS client on the node. The agent trusts
|
||||
it through its own configuration only; onboarding no longer adds it, and
|
||||
takes it out of a node that an older version onboarded.
|
||||
- **Any node could manage a mirror running mutual TLS**, since one root signs
|
||||
every node's certificate. `llmgw mirror serve --api-client NAME` answers only
|
||||
the certificate with that common name, and the mirror warns when mutual TLS
|
||||
runs without one.
|
||||
- **The administrator's password stayed in `init.log`.** It is shown once, on
|
||||
the screen, and the log keeps a placeholder.
|
||||
- **Auth off on a public address was silent.** The server now warns at start
|
||||
when the API listens beyond loopback without keys.
|
||||
- **The actor recorded with a change was whatever the caller sent**, line
|
||||
breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
|
||||
records the person who ran sudo rather than root.
|
||||
- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
|
||||
reaches it. It is off. The head itself still has no authentication, and
|
||||
SGLang still takes its key as a flag; docs/install.md says what to do about
|
||||
both. The monitoring README says how to keep Prometheus's queries to the
|
||||
control plane.
|
||||
|
||||
### Fixes
|
||||
|
||||
@@ -165,6 +281,8 @@ half again plus two" could not see.
|
||||
loaded runner.
|
||||
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
|
||||
`ctl restart` did on its first real use.
|
||||
- The status each failure is answered with — 499, 502, 503 with and without
|
||||
Retry-After, 504, 429 — is now tested case by case.
|
||||
|
||||
### Releases
|
||||
|
||||
|
||||
@@ -534,6 +534,10 @@ type Register struct {
|
||||
// meant "an agent too old to say" and constrained nothing — so a freshly
|
||||
// built node with no engine installed yet was a candidate for every model.
|
||||
EnginesProbed bool `protobuf:"varint,14,opt,name=engines_probed,json=enginesProbed,proto3" json:"engines_probed,omitempty"`
|
||||
// cpu_count is the CPUs the agent may run on. Placement reserves cores
|
||||
// for CPU replicas out of it, the way it reserves memory out of
|
||||
// memory_total_mb; zero, from an older agent, reserves nothing.
|
||||
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
|
||||
// imex is the service that lets a multi-node NVLink domain share memory.
|
||||
Imex *IMEX `protobuf:"bytes,15,opt,name=imex,proto3" json:"imex,omitempty"`
|
||||
// installable_engines are the engines this agent knows how to install
|
||||
@@ -679,6 +683,13 @@ func (x *Register) GetEnginesProbed() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Register) GetCpuCount() int32 {
|
||||
if x != nil {
|
||||
return x.CpuCount
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *Register) GetImex() *IMEX {
|
||||
if x != nil {
|
||||
return x.Imex
|
||||
@@ -2017,8 +2028,10 @@ type Heartbeat struct {
|
||||
// gpu_fabric is each GPU's NVLink domain membership as it stands now, sent
|
||||
// once a minute: a fabric manager restarting drops every card out of its
|
||||
// domain, and the node has to stop being offered work that spans it.
|
||||
GpuFabric []*GPUFabricState `protobuf:"bytes,16,rep,name=gpu_fabric,json=gpuFabric,proto3" json:"gpu_fabric,omitempty"`
|
||||
Imex *IMEX `protobuf:"bytes,17,opt,name=imex,proto3" json:"imex,omitempty"`
|
||||
GpuFabric []*GPUFabricState `protobuf:"bytes,16,rep,name=gpu_fabric,json=gpuFabric,proto3" json:"gpu_fabric,omitempty"`
|
||||
Imex *IMEX `protobuf:"bytes,17,opt,name=imex,proto3" json:"imex,omitempty"`
|
||||
// cpu_count as in Register, for a VM given more CPUs while it runs.
|
||||
CpuCount int32 `protobuf:"varint,18,opt,name=cpu_count,json=cpuCount,proto3" json:"cpu_count,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -2172,6 +2185,13 @@ func (x *Heartbeat) GetImex() *IMEX {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Heartbeat) GetCpuCount() int32 {
|
||||
if x != nil {
|
||||
return x.CpuCount
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
// versions. Nothing more.
|
||||
//
|
||||
@@ -2561,7 +2581,10 @@ type StartShard struct {
|
||||
// api_key is what the engine demands of every request, derived by the
|
||||
// control plane from the replica id. The router sends it; nothing else
|
||||
// that can reach the node's ports has it.
|
||||
ApiKey string `protobuf:"bytes,16,opt,name=api_key,json=apiKey,proto3" json:"api_key,omitempty"`
|
||||
ApiKey string `protobuf:"bytes,16,opt,name=api_key,json=apiKey,proto3" json:"api_key,omitempty"`
|
||||
// cpu_threads is how many CPU threads a shard with no GPU computes on:
|
||||
// the cores placement reserved for it. Zero leaves it to the agent.
|
||||
CpuThreads int32 `protobuf:"varint,17,opt,name=cpu_threads,json=cpuThreads,proto3" json:"cpu_threads,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -2708,6 +2731,13 @@ func (x *StartShard) GetApiKey() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *StartShard) GetCpuThreads() int32 {
|
||||
if x != nil {
|
||||
return x.CpuThreads
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
type EngineProfile struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
|
||||
@@ -3655,7 +3685,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"fabricTest\x122\n" +
|
||||
"\aupgrade\x18\n" +
|
||||
" \x01(\v2\x16.llmgw.v1.UpgradeAgentH\x00R\aupgradeB\x05\n" +
|
||||
"\x03msg\"\xe3\x06\n" +
|
||||
"\x03msg\"\x80\a\n" +
|
||||
"\bRegister\x12\x17\n" +
|
||||
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12\x1a\n" +
|
||||
"\bhostname\x18\x02 \x01(\tR\bhostname\x12\x1b\n" +
|
||||
@@ -3671,7 +3701,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\aweights\x18\v \x03(\v2\x16.llmgw.v1.LocalWeightsR\aweights\x12+\n" +
|
||||
"\astorage\x18\f \x03(\v2\x11.llmgw.v1.StorageR\astorage\x12\"\n" +
|
||||
"\x04host\x18\r \x01(\v2\x0e.llmgw.v1.HostR\x04host\x12%\n" +
|
||||
"\x0eengines_probed\x18\x0e \x01(\bR\renginesProbed\x12\"\n" +
|
||||
"\x0eengines_probed\x18\x0e \x01(\bR\renginesProbed\x12\x1b\n" +
|
||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x12\"\n" +
|
||||
"\x04imex\x18\x0f \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12/\n" +
|
||||
"\x13installable_engines\x18\x10 \x03(\tR\x12installableEngines\x12\x1a\n" +
|
||||
"\bfeatures\x18\x11 \x03(\tR\bfeatures\x1a9\n" +
|
||||
@@ -3811,7 +3842,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
|
||||
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
||||
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xd9\x06\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xf6\x06\n" +
|
||||
"\tHeartbeat\x12\x17\n" +
|
||||
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
|
||||
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
|
||||
@@ -3831,7 +3862,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\afabrics\x18\x0f \x03(\v2\x1a.llmgw.v1.FabricAttachmentR\afabrics\x127\n" +
|
||||
"\n" +
|
||||
"gpu_fabric\x18\x10 \x03(\v2\x18.llmgw.v1.GPUFabricStateR\tgpuFabric\x12\"\n" +
|
||||
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x1aA\n" +
|
||||
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
|
||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x1aA\n" +
|
||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||
@@ -3872,7 +3904,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"bytes_done\x18\x02 \x01(\x03R\tbytesDone\x12\x1f\n" +
|
||||
"\vbytes_total\x18\x03 \x01(\x03R\n" +
|
||||
"bytesTotal\"\xf2\x03\n" +
|
||||
"bytesTotal\"\x93\x04\n" +
|
||||
"\n" +
|
||||
"StartShard\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -3896,7 +3928,9 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x06fabric\x18\x0e \x01(\tR\x06fabric\x12\x1d\n" +
|
||||
"\n" +
|
||||
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
|
||||
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\"\xfd\x02\n" +
|
||||
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" +
|
||||
"\vcpu_threads\x18\x11 \x01(\x05R\n" +
|
||||
"cpuThreads\"\xfd\x02\n" +
|
||||
"\rEngineProfile\x12\x12\n" +
|
||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||
"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
|
||||
|
||||
@@ -120,6 +120,10 @@ message Register {
|
||||
// meant "an agent too old to say" and constrained nothing — so a freshly
|
||||
// built node with no engine installed yet was a candidate for every model.
|
||||
bool engines_probed = 14;
|
||||
// cpu_count is the CPUs the agent may run on. Placement reserves cores
|
||||
// for CPU replicas out of it, the way it reserves memory out of
|
||||
// memory_total_mb; zero, from an older agent, reserves nothing.
|
||||
int32 cpu_count = 18;
|
||||
// imex is the service that lets a multi-node NVLink domain share memory.
|
||||
IMEX imex = 15;
|
||||
|
||||
@@ -420,6 +424,8 @@ message Heartbeat {
|
||||
// domain, and the node has to stop being offered work that spans it.
|
||||
repeated GPUFabricState gpu_fabric = 16;
|
||||
IMEX imex = 17;
|
||||
// cpu_count as in Register, for a VM given more CPUs while it runs.
|
||||
int32 cpu_count = 18;
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
@@ -530,6 +536,9 @@ message StartShard {
|
||||
// control plane from the replica id. The router sends it; nothing else
|
||||
// that can reach the node's ports has it.
|
||||
string api_key = 16;
|
||||
// cpu_threads is how many CPU threads a shard with no GPU computes on:
|
||||
// the cores placement reserved for it. Zero leaves it to the agent.
|
||||
int32 cpu_threads = 17;
|
||||
}
|
||||
|
||||
message EngineProfile {
|
||||
|
||||
@@ -122,9 +122,10 @@ TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
"$BINARY" ctl cert issue-client "$NODE" --dir "$PKI" --out "$TMP" >/dev/null
|
||||
|
||||
# 2 and 3. Identity, and trusting the root so future tooling on that machine
|
||||
# does not treat our certificates as strangers.
|
||||
say "installing the certificate and trusting the root"
|
||||
# 2 and 3. Identity. The root is trusted by the agent's configuration only,
|
||||
# never system-wide: it signs whatever the control plane asks, and every
|
||||
# TLS client on the machine would take it for any site.
|
||||
say "installing the certificate"
|
||||
scp -q $SSHOPTS "$PKI/ca.pem" "$TMP/$NODE.pem" "$TMP/$NODE-key.pem" "$SSH_USER@$ADDR:/tmp/"
|
||||
$SSH "set -e
|
||||
sudo install -d -m 0755 /etc/llmgw
|
||||
@@ -132,8 +133,10 @@ $SSH "set -e
|
||||
sudo install -m 0644 /tmp/$NODE.pem /etc/llmgw/agent.pem
|
||||
sudo install -m 0600 /tmp/$NODE-key.pem /etc/llmgw/agent-key.pem
|
||||
sudo chown root:root /etc/llmgw/*
|
||||
sudo install -m 0644 /tmp/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt
|
||||
sudo update-ca-certificates >/dev/null 2>&1 || true
|
||||
if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
|
||||
sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt
|
||||
sudo update-ca-certificates --fresh >/dev/null 2>&1 || true
|
||||
fi
|
||||
rm -f /tmp/ca.pem /tmp/$NODE.pem /tmp/$NODE-key.pem"
|
||||
|
||||
# 4. The binary, the configuration and the unit.
|
||||
|
||||
@@ -52,6 +52,13 @@ install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
|
||||
# client; one Alertmanager needs no cluster port.
|
||||
echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
|
||||
> /etc/default/prometheus-alertmanager
|
||||
# Prometheus itself is read-only as installed: its admin and lifecycle APIs
|
||||
# stay off (no --web.enable-admin-api, no --web.enable-lifecycle). It still
|
||||
# answers any query from anyone who reaches :9090, which says which models and
|
||||
# nodes there are and how busy each is. Its readers are the control plane
|
||||
# (monitoring.prometheus) and Grafana beside it; to keep it to those, allow
|
||||
# the control plane and drop the rest, e.g. with nftables:
|
||||
# nft add rule inet filter input tcp dport 9090 ip saddr != { 127.0.0.1, CONTROL_PLANE_IP } drop
|
||||
promtool check config /etc/prometheus/prometheus.yml
|
||||
systemctl restart prometheus prometheus-alertmanager
|
||||
|
||||
|
||||
@@ -1124,3 +1124,11 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
has them. Single-node placement on real GPUs is the well-trodden path; if you
|
||||
are the first to run a TP group across real cards, run `llmgw doctor --json`
|
||||
on each node first and keep the reports.
|
||||
- **Two engine setups are weaker than the rest.** An engine's key is passed
|
||||
in its environment, where only root and its own user can read it, except
|
||||
SGLang's, which takes it only as a flag: other users on the node can see it
|
||||
in the process list. Mount `/proc` with `hidepid=2` on nodes where that
|
||||
matters. vLLM's `launch_mode: ray` starts a Ray head without
|
||||
authentication, so anyone who can reach its port can run code on the
|
||||
node. The dashboard is off, but firewall the port to the other nodes in the
|
||||
replica, or use the default `mp` launch mode, which needs no Ray at all.
|
||||
@@ -81,6 +81,7 @@ rather than repeated.
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `api` | `string` | [Mirror](#mirror) |
|
||||
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
|
||||
| `registry` | `string` | [Mirror](#mirror) |
|
||||
@@ -128,6 +129,7 @@ rather than repeated.
|
||||
| `args` | `[]string` | [Profile](#profile) |
|
||||
| `env` | `map[string]string` | [Profile](#profile) |
|
||||
| `min_memory_gb` | `int` | [Profile](#profile) |
|
||||
| `cpu_cores` | `int` | [Profile](#profile) |
|
||||
| `min_disk_gb` | `int` | [Profile](#profile) |
|
||||
| `node_selector` | `map[string]string` | [Profile](#profile) |
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
@@ -167,6 +169,7 @@ rather than repeated.
|
||||
| `landing` | `Landing` | [Server](#server) |
|
||||
| `agent_tls` | `tlsconf.Config` | [Server](#server) |
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
@@ -184,6 +187,8 @@ rather than repeated.
|
||||
| `first_token` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
| `idle` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
| `total` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
| `endpoint` | `string` | [Tracing](#tracing) |
|
||||
| `sample_ratio` | `float64` | [Tracing](#tracing) |
|
||||
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
@@ -634,6 +639,20 @@ Parallel is how many requests a server handles at once, when a profile
|
||||
does not say. llama.cpp defaults to one, which would make every second
|
||||
request wait behind the first no matter what the router decided.
|
||||
|
||||
### `threads`
|
||||
|
||||
`int`
|
||||
|
||||
Threads is how many CPU threads a shard with no GPU computes on. Zero
|
||||
takes half this machine's CPUs.
|
||||
|
||||
Not all of them, which is llama.cpp's own default: two busy servers
|
||||
that each spin on every CPU fight for them, and on an 8-vCPU node
|
||||
that made 7B generation fall from 10 tokens a second to under 1 each.
|
||||
At half, one server alone is about a third slower and two together
|
||||
keep 5.5 each. A node that only ever runs one CPU replica does better
|
||||
with all of them; say so here.
|
||||
|
||||
## Mirror
|
||||
|
||||
### `api`
|
||||
@@ -948,6 +967,20 @@ this. A node that does not report its memory is not excluded either —
|
||||
an older agent must not have its nodes quietly emptied by an upgrade of
|
||||
the control plane.
|
||||
|
||||
### `cpu_cores`
|
||||
|
||||
`int`
|
||||
|
||||
CPUCores is the cores a replica of a profile with no GPU is given:
|
||||
reserved out of its node's CPUs by placement, as memory is, and what
|
||||
its engine computes on (llama.cpp's --threads).
|
||||
|
||||
Reserved because an engine takes every core it is told about, and two
|
||||
on one node that each think they have all of them spin against each
|
||||
other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
|
||||
to under one each. Zero reserves nothing and leaves the thread count to
|
||||
the node, which is every profile from before this.
|
||||
|
||||
### `min_disk_gb`
|
||||
|
||||
`int`
|
||||
@@ -1246,6 +1279,16 @@ Monitoring is where the dashboard reads its history from.
|
||||
|
||||
Takes a [Monitoring](#monitoring) block.
|
||||
|
||||
### `tracing`
|
||||
|
||||
`Tracing`
|
||||
|
||||
Tracing sends a trace of each inference request to an OpenTelemetry
|
||||
collector. Off unless an endpoint is set here or in
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT.
|
||||
|
||||
Takes a [Tracing](#tracing) block.
|
||||
|
||||
### `mirror`
|
||||
|
||||
`Mirror`
|
||||
@@ -1366,6 +1409,22 @@ about a hundred tokens on a slow model.
|
||||
|
||||
`time.Duration`
|
||||
|
||||
## Tracing
|
||||
|
||||
### `endpoint`
|
||||
|
||||
`string`
|
||||
|
||||
Endpoint is the collector, for example http://otel.internal:4318.
|
||||
Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
|
||||
|
||||
### `sample_ratio`
|
||||
|
||||
`float64`
|
||||
|
||||
SampleRatio is the share of requests traced, 0 to 1, when the caller
|
||||
did not decide it with a sampled traceparent of its own. Zero takes 1.
|
||||
|
||||
## VLLMConfig
|
||||
|
||||
### `bin`
|
||||
|
||||
@@ -19,6 +19,10 @@ require (
|
||||
github.com/prometheus/common v0.71.0
|
||||
github.com/spf13/cobra v1.10.2
|
||||
github.com/yuin/goldmark v1.8.6
|
||||
go.opentelemetry.io/otel v1.46.0
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
|
||||
go.opentelemetry.io/otel/sdk v1.46.0
|
||||
go.opentelemetry.io/otel/trace v1.46.0
|
||||
golang.org/x/crypto v0.57.0
|
||||
golang.org/x/net v0.59.0
|
||||
golang.org/x/sync v0.23.0
|
||||
@@ -29,6 +33,7 @@ require (
|
||||
require (
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||
github.com/charmbracelet/colorprofile v0.4.1 // indirect
|
||||
github.com/charmbracelet/x/ansi v0.11.6 // indirect
|
||||
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
|
||||
@@ -39,8 +44,11 @@ require (
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
||||
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
|
||||
github.com/go-logr/logr v1.4.4 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
|
||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
@@ -61,9 +69,16 @@ require (
|
||||
github.com/rivo/uniseg v0.4.7 // indirect
|
||||
github.com/spf13/pflag v1.0.9 // indirect
|
||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.46.0 // indirect
|
||||
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
golang.org/x/sys v0.48.0 // indirect
|
||||
golang.org/x/text v0.42.0 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
|
||||
google.golang.org/grpc v1.83.1 // indirect
|
||||
modernc.org/libc v1.75.7 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
modernc.org/memory v1.12.1 // indirect
|
||||
|
||||
@@ -6,6 +6,8 @@ github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiE
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
||||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
|
||||
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
|
||||
@@ -34,14 +36,23 @@ github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
|
||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
|
||||
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
|
||||
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
|
||||
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||
@@ -117,6 +128,24 @@ github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavM
|
||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
|
||||
github.com/yuin/goldmark v1.8.6 h1:d0VcaP1sx9GkFVkoW+KtggpGi2KZ965i14b0+bDQST4=
|
||||
github.com/yuin/goldmark v1.8.6/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
|
||||
go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzbxyuo6/6HyBlnlN1CWEJmBVcw=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0 h1:KrC1YrQeSt46ITMWAbgQx1M1eV1/1TKzttrBzymPmss=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0/go.mod h1:zDSEzoEqsOrgBeGvH66KRgxh90VonFyJqBHA0Pk3+rM=
|
||||
go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8=
|
||||
go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o=
|
||||
go.opentelemetry.io/otel/sdk v1.46.0 h1:h5CNQQjEbuQXY/JfZtgt3i7HVFV3aHPO2OAwO2eTYPI=
|
||||
go.opentelemetry.io/otel/sdk v1.46.0/go.mod h1:GAERFXFt5SYCEB+YiKUbMBeza6UaDH7GmGOZEfh2gSM=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.46.0 h1:0piZ26EG4RBfebb2jhDH6ERCYHoVWduc3kLgPCwSnSE=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.46.0/go.mod h1:I1PbKrdVc8Qu8HYVDNtqVIwLwjNrhsV/uFuxfwg8mO4=
|
||||
go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c=
|
||||
go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI=
|
||||
go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
|
||||
go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
||||
@@ -141,6 +170,14 @@ golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
|
||||
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
|
||||
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
|
||||
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 h1:ax2KzoSRIZU/M0cIxri3pKxy99vniH1PVxWC6si/eZI=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688/go.mod h1:1RJ9BQGyNdZwkGc1eTqkErfRZ6RJyYPHZo73BZ1vQqI=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
|
||||
google.golang.org/grpc v1.83.1 h1:HIO0+BEtBP6soyqvqC8sNUjZ7bTs+0hFQuFF+RAy++Y=
|
||||
google.golang.org/grpc v1.83.1/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
|
||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
|
||||
+1
-859
@@ -8,21 +8,16 @@ import (
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"io"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"math/rand/v2"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -36,278 +31,9 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
// Config configures the agent.
|
||||
type Config struct {
|
||||
// NodeID defaults to the hostname. It must equal the common name of the
|
||||
// agent's certificate, so set it only where the two differ.
|
||||
NodeID string `koanf:"node_id"`
|
||||
// Server is the control plane, and Servers is the same thing when there is
|
||||
// more than one. Both are accepted: every configuration written before
|
||||
// there could be a second control plane names one address, and none of
|
||||
// them should have to change.
|
||||
//
|
||||
// The order is tried in order, and a refusal moves to the next. Only the
|
||||
// leader accepts an agent, so a standby is skipped in the time it takes to
|
||||
// be told so — which is what makes failover a reconnect rather than an
|
||||
// edit to every node's configuration.
|
||||
Server string `koanf:"server"`
|
||||
Servers []string `koanf:"servers"`
|
||||
// Hostname defaults to the machine's.
|
||||
Hostname string `koanf:"hostname"`
|
||||
// MgmtAddr is the address engines are reached on. It defaults to this
|
||||
// machine's address on the route to the control plane, which is the one
|
||||
// the control plane and the router can reach.
|
||||
MgmtAddr string `koanf:"mgmt_addr"`
|
||||
// Labels are free-form facts about a node for a profile's node_selector.
|
||||
// Nothing needs them: memory, GPUs, engines and disk are measured, and a
|
||||
// label restating one of those goes stale the day the machine changes.
|
||||
Labels map[string]string `koanf:"labels"`
|
||||
|
||||
// MgmtInterface is the ethernet interface NCCL bootstraps over. It
|
||||
// defaults to the interface holding MgmtAddr, then eth0.
|
||||
MgmtInterface string `koanf:"mgmt_interface"`
|
||||
// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
|
||||
// default subnet prefix is the same on every fabric, so nothing can be
|
||||
// inferred from the wire (plan §5.2).
|
||||
Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
|
||||
|
||||
HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
|
||||
// ReconnectBase is the first reconnect delay; it backs off from there.
|
||||
ReconnectBase time.Duration `koanf:"reconnect_base"`
|
||||
ReconnectMax time.Duration `koanf:"reconnect_max"`
|
||||
|
||||
// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
|
||||
// health check before the agent stops waiting on it.
|
||||
//
|
||||
// Shorter than a startup timeout on purpose: this engine is already
|
||||
// running, so it is not loading a model, it is either serving or it is not.
|
||||
AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
|
||||
|
||||
// TLS authenticates both ends of the control-plane stream. Leaving it unset
|
||||
// keeps the plaintext behaviour, which only belongs on a trusted network.
|
||||
TLS tlsconf.Config `koanf:"tls"`
|
||||
|
||||
// ModelDirs are the directories this node keeps model weights in, beyond
|
||||
// WeightsCache, which is always looked in. Storage differs at every installation,
|
||||
// so llmgw observes it rather than owning it: a Lustre mount, an image
|
||||
// exported over the fabric, local NVMe and a copy made by hand all look
|
||||
// identical from here, and placement only needs to know whether the file
|
||||
// is present.
|
||||
ModelDirs []string `koanf:"model_dirs"`
|
||||
|
||||
// EnginesDir is where the agent installs engines, one directory per
|
||||
// build. Defaults to /opt/llmgw/engines.
|
||||
EnginesDir string `koanf:"engines_dir"`
|
||||
|
||||
// WeightsCache is where a remote source is downloaded to. Defaults to the
|
||||
// first entry of ModelDirs, because a node that says where its models live
|
||||
// has already answered this question, and otherwise to
|
||||
// /var/lib/llmgw-weights.
|
||||
//
|
||||
// Fetching is the simple arrangement: every node keeps its own copy. The
|
||||
// exotic ones need no code — weights on a shared filer or an image exported
|
||||
// over the fabric are simply present, and present means nothing is fetched.
|
||||
WeightsCache string `koanf:"weights_cache"`
|
||||
|
||||
Discovery discovery.Options `koanf:"discovery"`
|
||||
Engines engines.Config `koanf:"engines"`
|
||||
Superv supervisor.Config `koanf:"supervisor"`
|
||||
|
||||
Version string
|
||||
}
|
||||
|
||||
// DefaultConfig returns sensible defaults.
|
||||
func DefaultConfig() Config {
|
||||
// No MgmtInterface: normalize finds the one holding the management
|
||||
// address, and falls back to eth0 only when it cannot. Defaulted here, it
|
||||
// was never found — every node said eth0 unless told otherwise, and a node
|
||||
// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
|
||||
return Config{
|
||||
HeartbeatInterval: 5 * time.Second,
|
||||
ReconnectBase: time.Second,
|
||||
ReconnectMax: 30 * time.Second,
|
||||
AdoptReadyTimeout: 60 * time.Second,
|
||||
Superv: supervisor.DefaultConfig(),
|
||||
// NVML gives NVLink topology, so prefer it when this build has it.
|
||||
Discovery: discovery.Options{PreferNVML: true},
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) normalize() error {
|
||||
d := DefaultConfig()
|
||||
if err := c.TLS.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.normalizeServerURL(); err != nil {
|
||||
return err
|
||||
}
|
||||
// Everything below is found rather than written. A file that has to say
|
||||
// which machine it is on, and at what address, is a file that differs on
|
||||
// every node — and a copy taken from one node to the next registers the
|
||||
// second as the first.
|
||||
host, _ := os.Hostname()
|
||||
if c.NodeID == "" {
|
||||
c.NodeID = host
|
||||
}
|
||||
if c.NodeID == "" {
|
||||
return errors.New("node_id is not set and the hostname could not be read")
|
||||
}
|
||||
if c.Hostname == "" {
|
||||
c.Hostname = host
|
||||
}
|
||||
if c.MgmtAddr == "" {
|
||||
c.MgmtAddr = addrToward(c.Server)
|
||||
}
|
||||
if c.MgmtInterface == "" {
|
||||
c.MgmtInterface = interfaceOf(c.MgmtAddr)
|
||||
}
|
||||
if c.MgmtInterface == "" {
|
||||
c.MgmtInterface = "eth0"
|
||||
}
|
||||
// The directory downloads land in is always one that is looked in. When
|
||||
// they were two settings, a node could fill its disk with fetched weights
|
||||
// that placement never saw — which is what every node here was doing.
|
||||
if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
|
||||
c.WeightsCache = c.ModelDirs[0]
|
||||
}
|
||||
if c.WeightsCache == "" {
|
||||
c.WeightsCache = DefaultWeightsCache
|
||||
}
|
||||
if !slices.Contains(c.ModelDirs, c.WeightsCache) {
|
||||
c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
|
||||
}
|
||||
if c.HeartbeatInterval <= 0 {
|
||||
c.HeartbeatInterval = d.HeartbeatInterval
|
||||
}
|
||||
if c.ReconnectBase <= 0 {
|
||||
c.ReconnectBase = d.ReconnectBase
|
||||
}
|
||||
if c.AdoptReadyTimeout <= 0 {
|
||||
c.AdoptReadyTimeout = d.AdoptReadyTimeout
|
||||
}
|
||||
if c.ReconnectMax < c.ReconnectBase {
|
||||
c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// normalizeServerURL makes the configured address a URL connect can use, and
|
||||
// refuses the combination that would quietly undo the certificates: a scheme
|
||||
// that says plaintext while TLS is configured. Silently connecting anyway would
|
||||
// mean a cluster believing it has mutual authentication when it has none.
|
||||
func (c *Config) normalizeServerURL() error {
|
||||
// One list, however it was written. Server first when both are given,
|
||||
// because an operator adding a standby to an existing file expects the
|
||||
// one already there to stay preferred.
|
||||
all := make([]string, 0, 1+len(c.Servers))
|
||||
if c.Server != "" {
|
||||
all = append(all, c.Server)
|
||||
}
|
||||
all = append(all, c.Servers...)
|
||||
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]string, 0, len(all))
|
||||
for _, s := range all {
|
||||
fixed, err := normalizeOne(s, c.TLS.Enabled())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// A duplicate would be tried twice in a row on failover, which reads
|
||||
// as the rotation not working.
|
||||
if _, dup := seen[fixed]; dup {
|
||||
continue
|
||||
}
|
||||
seen[fixed] = struct{}{}
|
||||
out = append(out, fixed)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return errors.New("server is required")
|
||||
}
|
||||
c.Servers = out
|
||||
c.Server = out[0]
|
||||
return nil
|
||||
}
|
||||
|
||||
// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
|
||||
const DefaultWeightsCache = "/var/lib/llmgw-weights"
|
||||
|
||||
// addrToward is this machine's address on the route to the control plane.
|
||||
//
|
||||
// The address engines are reached on is whichever one the control plane can
|
||||
// reach, and the kernel already knows which that is: a UDP "connection" picks
|
||||
// the source address for the route without sending a packet. Asking the
|
||||
// interfaces instead would have to guess among docker bridges, VPNs and
|
||||
// link-local addresses.
|
||||
func addrToward(server string) string {
|
||||
u, err := url.Parse(server)
|
||||
if err != nil || u.Host == "" {
|
||||
return ""
|
||||
}
|
||||
host := u.Hostname()
|
||||
port := u.Port()
|
||||
if port == "" {
|
||||
port = "443"
|
||||
}
|
||||
conn, err := net.Dial("udp", net.JoinHostPort(host, port))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
defer conn.Close()
|
||||
if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
|
||||
return a.IP.String()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// interfaceOf names the interface that holds an address, or "" when none does.
|
||||
func interfaceOf(addr string) string {
|
||||
ip := net.ParseIP(addr)
|
||||
if ip == nil {
|
||||
return ""
|
||||
}
|
||||
ifaces, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
for _, ifc := range ifaces {
|
||||
addrs, err := ifc.Addrs()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
|
||||
return ifc.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// normalizeOne applies the scheme rules to a single address.
|
||||
func normalizeOne(addr string, secure bool) (string, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(addr, "https://"):
|
||||
if !secure {
|
||||
return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
|
||||
}
|
||||
case strings.HasPrefix(addr, "http://"):
|
||||
if secure {
|
||||
return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
|
||||
}
|
||||
case strings.Contains(addr, "://"):
|
||||
return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
|
||||
case secure:
|
||||
return "https://" + addr, nil
|
||||
default:
|
||||
return "http://" + addr, nil
|
||||
}
|
||||
return addr, nil
|
||||
}
|
||||
|
||||
// Agent runs on a GPU node.
|
||||
type Agent struct {
|
||||
cfg Config
|
||||
@@ -588,131 +314,6 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
"shard", o.ShardID, "replica", o.ReplicaID)
|
||||
}
|
||||
|
||||
// usableFabrics reports the fabrics this node can actually carry traffic on.
|
||||
//
|
||||
// Loud about what it drops. A fabric silently missing from a registration looks
|
||||
// identical to one never configured, and the operator who typed the adapter
|
||||
// name is the only person who can fix it — so the reason is logged here, on the
|
||||
// machine where both halves are visible, rather than inferred later from a
|
||||
// replica that will not converge.
|
||||
func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
|
||||
out := a.checkFabrics(inv.HCAs, true)
|
||||
a.mu.Lock()
|
||||
a.fabricsSent = fabricsFingerprint(out)
|
||||
a.mu.Unlock()
|
||||
return out
|
||||
}
|
||||
|
||||
// checkFabrics is the declared fabrics this node can carry traffic on, from
|
||||
// the adapters as they are now. Loud logs every problem; the heartbeat checks
|
||||
// quietly and speaks only when the answer changes.
|
||||
func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
|
||||
declared := map[string][]string{}
|
||||
for _, f := range a.cfg.Fabrics {
|
||||
declared[f.Fabric] = f.HCAs
|
||||
}
|
||||
|
||||
var out []inventory.FabricAttachment
|
||||
for _, c := range discovery.CheckFabrics(declared, hcas) {
|
||||
if loud {
|
||||
for _, p := range c.Problems {
|
||||
a.log.Error("a declared fabric adapter cannot carry traffic",
|
||||
"fabric", c.Fabric, "problem", p)
|
||||
}
|
||||
}
|
||||
if !c.OK() {
|
||||
if loud {
|
||||
a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
|
||||
"fabric", c.Fabric,
|
||||
"consequence", "no multi-node replica will be placed here")
|
||||
}
|
||||
continue
|
||||
}
|
||||
out = append(out, inventory.FabricAttachment{
|
||||
Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
|
||||
// changing.
|
||||
func fabricsFingerprint(fs []inventory.FabricAttachment) string {
|
||||
var b strings.Builder
|
||||
for _, f := range fs {
|
||||
fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// sampleIB re-reads the adapters for the heartbeat: every port's state and
|
||||
// counters, and the fabrics again when they no longer match what the control
|
||||
// plane was told.
|
||||
//
|
||||
// Every heartbeat, because it is a few small files per port, and because a
|
||||
// port that went down used to be invisible until the agent restarted — the
|
||||
// node went on being offered multi-node work that would hang on its first
|
||||
// collective.
|
||||
func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
|
||||
a.mu.Lock()
|
||||
had := len(a.inv.HCAs) > 0
|
||||
a.mu.Unlock()
|
||||
if !had && len(a.cfg.Fabrics) == 0 {
|
||||
return
|
||||
}
|
||||
hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
a.attachSM(hcas)
|
||||
for _, h := range hcas {
|
||||
for _, p := range h.Ports {
|
||||
ip := &llmgwv1.IBPort{
|
||||
Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
|
||||
RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
|
||||
NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
|
||||
}
|
||||
a.mu.Lock()
|
||||
if p.SM != nil {
|
||||
ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
|
||||
} else {
|
||||
ip.SmError = a.smErr[smKey(h.Name, p.Num)]
|
||||
}
|
||||
for _, r := range a.sms[smKey(h.Name, p.Num)] {
|
||||
ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
|
||||
Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
|
||||
}
|
||||
ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
|
||||
a.mu.Unlock()
|
||||
hb.IbPorts = append(hb.IbPorts, ip)
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.inv.HCAs = hcas // a reconnect registers what is true now
|
||||
a.mu.Unlock()
|
||||
if len(a.cfg.Fabrics) == 0 {
|
||||
return
|
||||
}
|
||||
now := a.checkFabrics(hcas, false)
|
||||
fp := fabricsFingerprint(now)
|
||||
a.mu.Lock()
|
||||
changed := fp != a.fabricsSent
|
||||
a.fabricsSent = fp
|
||||
a.mu.Unlock()
|
||||
if !changed {
|
||||
return
|
||||
}
|
||||
// Said once, with the reasons, the moment it happens.
|
||||
a.checkFabrics(hcas, true)
|
||||
a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
|
||||
hb.FabricsChanged = true
|
||||
for _, f := range now {
|
||||
hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
|
||||
Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// weights builds the fetcher lazily, so an agent that never starts a shard
|
||||
// never needs a cache directory.
|
||||
func (a *Agent) weights() *weights.Fetcher {
|
||||
@@ -949,6 +550,7 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
||||
// and a node that grew from 7 GB to 32 GB has to be seen to have grown
|
||||
// without anybody restarting anything.
|
||||
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
|
||||
hb.CpuCount = int32(runtime.NumCPU())
|
||||
|
||||
a.reprobeEngines()
|
||||
a.mu.Lock()
|
||||
@@ -1414,215 +1016,6 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
|
||||
return &http.Client{Transport: &http2.Transport{TLSClientConfig: tc}}, nil
|
||||
}
|
||||
|
||||
// engines is what this node can run: what the agent installed and saw run,
|
||||
// and what the configuration names and answers. Nothing merely found on the
|
||||
// machine — a llama-server on PATH is somebody's, not necessarily one that
|
||||
// works, and placing on it is how a replica fails on a node nobody watches.
|
||||
func (a *Agent) engines(ctx context.Context) map[string]string {
|
||||
a.engineOnce.Do(func() {
|
||||
vers := a.reg.ProbeConfigured(ctx)
|
||||
a.mu.Lock()
|
||||
a.configuredVers, a.lastEngineProbe = vers, time.Now()
|
||||
a.mu.Unlock()
|
||||
if len(vers) > 0 {
|
||||
a.log.Info("engines named in the configuration",
|
||||
"engines", engines.ProbedNames(vers))
|
||||
}
|
||||
})
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
vers := a.engineReportLocked()
|
||||
// Whatever registration sends is what the control plane now knows, so a
|
||||
// heartbeat repeats it only once it differs.
|
||||
a.enginesSent = enginesFingerprint(vers)
|
||||
return vers
|
||||
}
|
||||
|
||||
// engineReportLocked merges the two sources. Installed wins: it is the one
|
||||
// that was seen to run. Called with mu held.
|
||||
func (a *Agent) engineReportLocked() map[string]string {
|
||||
out := maps.Clone(a.configuredVers)
|
||||
if out == nil {
|
||||
out = map[string]string{}
|
||||
}
|
||||
maps.Copy(out, a.installedVers)
|
||||
return out
|
||||
}
|
||||
|
||||
// provisionEvery is how often installed engines are checked again. A library
|
||||
// removed by an unrelated upgrade is found here rather than by the next
|
||||
// replica to start.
|
||||
const provisionEvery = 10 * time.Minute
|
||||
|
||||
// setProvision takes the control plane's list of wanted engines.
|
||||
func (a *Agent) setProvision(p *llmgwv1.Provision) {
|
||||
wants := []provision.Want{}
|
||||
for _, e := range p.GetEngines() {
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.provWants = wants
|
||||
a.mu.Unlock()
|
||||
select {
|
||||
case a.provKick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// provisionLoop installs and re-checks the wanted engines, one pass at a time.
|
||||
func (a *Agent) provisionLoop(ctx context.Context) {
|
||||
// After a failure the next pass comes sooner, and backs off: a lock held
|
||||
// by a new machine's own updates clears in a minute or two, while ten
|
||||
// minutes' wait left a node idle for no reason.
|
||||
next := provisionEvery
|
||||
retry := time.Minute
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-a.provKick:
|
||||
case <-time.After(next):
|
||||
}
|
||||
a.mu.Lock()
|
||||
wants := slices.Clone(a.provWants)
|
||||
managed := a.provWants != nil
|
||||
a.mu.Unlock()
|
||||
if !managed {
|
||||
continue
|
||||
}
|
||||
|
||||
statuses := a.prov.Converge(ctx, wants)
|
||||
next = provisionEvery
|
||||
for _, st := range statuses {
|
||||
if st.State == provision.Failed {
|
||||
next, retry = retry, min(retry*2, provisionEvery)
|
||||
break
|
||||
}
|
||||
}
|
||||
if next == provisionEvery {
|
||||
retry = time.Minute
|
||||
}
|
||||
installed := map[string]string{}
|
||||
keep := map[string]bool{}
|
||||
for _, st := range statuses {
|
||||
keep[st.Engine] = true
|
||||
if st.State == provision.Ready {
|
||||
a.reg.UseInstalled(st.Engine, st.Binary)
|
||||
installed[st.Engine] = st.Version
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
|
||||
a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
|
||||
}
|
||||
a.installedVers = installed
|
||||
// An engine no longer asked for is no longer reported on.
|
||||
for e := range a.provStatus {
|
||||
if !keep[e] {
|
||||
delete(a.provStatus, e)
|
||||
}
|
||||
}
|
||||
a.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// provisionStatus records a change in one engine's installation.
|
||||
func (a *Agent) provisionStatus(st provision.Status) {
|
||||
a.mu.Lock()
|
||||
prev, had := a.provStatus[st.Engine]
|
||||
a.provStatus[st.Engine] = st
|
||||
a.mu.Unlock()
|
||||
if !had || prev.State != st.State || prev.Detail != st.Detail {
|
||||
lvl := slog.LevelInfo
|
||||
if st.State == provision.Failed || st.State == provision.Unsupported {
|
||||
lvl = slog.LevelWarn
|
||||
}
|
||||
a.log.Log(context.Background(), lvl, "engine installation",
|
||||
"engine", st.Engine, "version", st.Version, "variant", st.Variant,
|
||||
"state", st.State, "detail", st.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func provisionFingerprint(m map[string]provision.Status) string {
|
||||
var b strings.Builder
|
||||
for _, k := range slices.Sorted(maps.Keys(m)) {
|
||||
st := m[k]
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// engineReprobe is how often the installed engines are looked for again.
|
||||
//
|
||||
// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
|
||||
// minute's delay in noticing is invisible, while SGLang's probe starts a
|
||||
// Python interpreter that imports it.
|
||||
const engineReprobe = time.Minute
|
||||
|
||||
// reprobeEngines looks for installed engines again when it is time, off the
|
||||
// heartbeat goroutine so a slow probe never delays a heartbeat.
|
||||
func (a *Agent) reprobeEngines() {
|
||||
a.mu.Lock()
|
||||
due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
|
||||
time.Since(a.lastEngineProbe) >= engineReprobe
|
||||
if due {
|
||||
a.probingEngines = true
|
||||
}
|
||||
a.mu.Unlock()
|
||||
if !due {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
vers := a.reg.ProbeConfigured(ctx)
|
||||
|
||||
a.mu.Lock()
|
||||
changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
|
||||
a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
|
||||
a.mu.Unlock()
|
||||
if changed {
|
||||
a.log.Info("engines named in the configuration changed",
|
||||
"engines", engines.ProbedNames(vers))
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func enginesFingerprint(vers map[string]string) string {
|
||||
var b strings.Builder
|
||||
for _, k := range slices.Sorted(maps.Keys(vers)) {
|
||||
fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// logsReply answers a request for one of this node's logs.
|
||||
func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
|
||||
n := int(req.GetLines())
|
||||
if n <= 0 {
|
||||
n = 200
|
||||
}
|
||||
n = min(n, agentLogLines)
|
||||
shards := a.sup.IDs()
|
||||
slices.Sort(shards)
|
||||
out := &llmgwv1.LogsReply{
|
||||
RequestId: req.GetRequestId(),
|
||||
Sources: append([]string{"agent"}, shards...),
|
||||
}
|
||||
switch src := req.GetSource(); src {
|
||||
case "", "agent":
|
||||
out.Lines = a.logRing.tail(n)
|
||||
default:
|
||||
sh, err := a.sup.Get(src)
|
||||
if err != nil {
|
||||
out.Error = "no engine " + src + " runs on this node"
|
||||
return out
|
||||
}
|
||||
out.Lines = sh.Logs(n)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// localizeSpec fills in what only this node knows.
|
||||
//
|
||||
// The interface NCCL bootstraps over is this node's, not the cluster's: the
|
||||
@@ -1673,254 +1066,3 @@ func ResolveManagement(c Config) (addr, iface string) {
|
||||
}
|
||||
return addr, iface
|
||||
}
|
||||
|
||||
func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
|
||||
|
||||
// scanFabric is discovery.ScanFabric, replaced in tests.
|
||||
var scanFabric = discovery.ScanFabric
|
||||
|
||||
// querySMs is discovery.QuerySMs, replaced in tests.
|
||||
var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
|
||||
return discovery.QuerySMs(context.Background(), hca, port)
|
||||
}
|
||||
|
||||
func splitSMKey(k string) (string, int) {
|
||||
i := strings.LastIndexByte(k, '/')
|
||||
port, _ := strconv.Atoi(k[i+1:])
|
||||
return k[:i], port
|
||||
}
|
||||
|
||||
// querySM is discovery.QuerySM, replaced in tests.
|
||||
var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
|
||||
return discovery.QuerySM(context.Background(), hca, port)
|
||||
}
|
||||
|
||||
// attachSM puts on each active InfiniBand port the subnet manager it sees,
|
||||
// asking again once a minute and otherwise reusing the last answer.
|
||||
func (a *Agent) attachSM(hcas []discovery.HCA) {
|
||||
a.mu.Lock()
|
||||
stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
|
||||
a.mu.Unlock()
|
||||
if stale {
|
||||
sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
|
||||
_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
|
||||
got, err := querySM(hca, port)
|
||||
if err != nil {
|
||||
errs[smKey(hca, port)] = err.Error()
|
||||
return nil, err
|
||||
}
|
||||
sm[smKey(hca, port)] = got
|
||||
return got, nil
|
||||
})
|
||||
// Every subnet manager on each fabric, through one port per master:
|
||||
// the same list through every port of one fabric is one query too
|
||||
// many for each.
|
||||
sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
|
||||
asked := map[string]bool{}
|
||||
for k, v := range sm {
|
||||
if asked[v.GUID] {
|
||||
continue
|
||||
}
|
||||
asked[v.GUID] = true
|
||||
hca, port := splitSMKey(k)
|
||||
if recs, err := querySMs(hca, port); err != nil {
|
||||
smsErr[k] = err.Error()
|
||||
} else {
|
||||
sms[k] = recs
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
for k, v := range sm {
|
||||
if old := a.sm[k]; old != nil && old.GUID != v.GUID {
|
||||
a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
|
||||
}
|
||||
}
|
||||
a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
|
||||
a.sms, a.smsErr = sms, smsErr
|
||||
a.mu.Unlock()
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
for i := range hcas {
|
||||
for j := range hcas[i].Ports {
|
||||
if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
|
||||
hcas[i].Ports[j].SM = sm
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ensureDiags installs infiniband-diags on a node with adapters and without
|
||||
// it. sminfo is what tells which physical fabric a port is on; without it,
|
||||
// two separate fabrics given one name cannot be caught. Installed the way
|
||||
// engine dependencies are, from the machine's own package sources.
|
||||
func (a *Agent) ensureDiags(ctx context.Context) {
|
||||
a.mu.Lock()
|
||||
hasIB := len(a.inv.HCAs) > 0
|
||||
a.mu.Unlock()
|
||||
if !hasIB || a.prov == nil {
|
||||
return
|
||||
}
|
||||
if _, err := exec.LookPath("sminfo"); err == nil {
|
||||
return
|
||||
}
|
||||
a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
|
||||
pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
|
||||
if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
|
||||
a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
|
||||
"err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// FeatureFabricTest is the agent answering FabricTestRequest.
|
||||
const FeatureFabricTest = "fabric-test"
|
||||
|
||||
// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
|
||||
// installed; both replaced in tests.
|
||||
var (
|
||||
runWriteBW = discovery.RunWriteBW
|
||||
haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
|
||||
)
|
||||
|
||||
// fabricTest runs this node's side of a bandwidth test with another.
|
||||
func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
|
||||
start := time.Now()
|
||||
reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
|
||||
defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
|
||||
|
||||
w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
|
||||
TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
|
||||
if req.GetRole() == "client" {
|
||||
w.Peer = req.GetPeer()
|
||||
}
|
||||
a.mu.Lock()
|
||||
found := false
|
||||
for _, h := range a.inv.HCAs {
|
||||
for _, p := range h.Ports {
|
||||
if h.Name == w.HCA && p.Num == w.Port {
|
||||
found = true
|
||||
// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
|
||||
if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
|
||||
w.GIDIndex = p.RoCEv2GID
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
a.mu.Unlock()
|
||||
if !found {
|
||||
reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
|
||||
return reply
|
||||
}
|
||||
if !haveWriteBW() && a.prov != nil {
|
||||
a.log.Info("installing perftest, for a bandwidth test with another node")
|
||||
pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
|
||||
if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
|
||||
reply.Error = "installing perftest: " + err.Error()
|
||||
return reply
|
||||
}
|
||||
}
|
||||
gbps, out, err := runWriteBW(ctx, w)
|
||||
reply.Gbps = gbps
|
||||
reply.Output = tailLines(out, 12)
|
||||
if err != nil {
|
||||
reply.Error = err.Error()
|
||||
}
|
||||
return reply
|
||||
}
|
||||
|
||||
func tailLines(s string, n int) string {
|
||||
lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// discoverGPUFabric and discoverIMEX are replaced in tests.
|
||||
var (
|
||||
discoverGPUFabric = discovery.DiscoverGPUFabric
|
||||
discoverIMEX = discovery.DiscoverIMEX
|
||||
)
|
||||
|
||||
// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
|
||||
// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
|
||||
// for every heartbeat, and a fabric that was joined an hour ago rarely
|
||||
// changes in five seconds — but when a fabric manager restarts it does, for
|
||||
// every card at once.
|
||||
func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
|
||||
a.mu.Lock()
|
||||
gpus := slices.Clone(a.inv.GPUs)
|
||||
due := time.Since(a.nvlAsked) > time.Minute
|
||||
if due {
|
||||
a.nvlAsked = time.Now()
|
||||
}
|
||||
a.mu.Unlock()
|
||||
if len(gpus) == 0 || !due {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
|
||||
if err != nil || len(fab) == 0 {
|
||||
return
|
||||
}
|
||||
discovery.ApplyFabric(gpus, fab)
|
||||
for _, g := range gpus {
|
||||
if g.Fabric == nil {
|
||||
continue
|
||||
}
|
||||
hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
|
||||
Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
|
||||
ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
|
||||
})
|
||||
}
|
||||
m := discoverIMEX(ctx)
|
||||
hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
|
||||
}
|
||||
|
||||
// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
|
||||
// Restart=on-failure starts the new binary, and distinct, so the journal says
|
||||
// why the agent went.
|
||||
const upgradeExit = 75
|
||||
|
||||
// exitForUpgrade ends the process; a variable so a test can see it called
|
||||
// rather than have its own process end.
|
||||
var exitForUpgrade = func() { os.Exit(upgradeExit) }
|
||||
|
||||
// upgradeTarget is the binary an upgrade replaces; a variable so a test
|
||||
// replaces a file of its own rather than the test binary running it.
|
||||
var upgradeTarget = os.Executable
|
||||
|
||||
// upgrade replaces this agent's binary with a release and restarts into it.
|
||||
//
|
||||
// Engines are not touched: an agent restart leaves them running and adopts
|
||||
// them again, which is what makes upgrading a node something that can be done
|
||||
// while it serves.
|
||||
func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
|
||||
reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
|
||||
send := func() {
|
||||
select {
|
||||
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}
|
||||
a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
|
||||
self, err := upgradeTarget()
|
||||
if err == nil {
|
||||
dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
|
||||
cancel()
|
||||
}
|
||||
if err != nil {
|
||||
reply.Error = err.Error()
|
||||
a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
|
||||
send()
|
||||
return
|
||||
}
|
||||
send()
|
||||
a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
|
||||
// A moment for the reply to leave before the stream goes.
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
exitForUpgrade()
|
||||
}
|
||||
@@ -0,0 +1,286 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/url"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
// Config configures the agent.
|
||||
type Config struct {
|
||||
// NodeID defaults to the hostname. It must equal the common name of the
|
||||
// agent's certificate, so set it only where the two differ.
|
||||
NodeID string `koanf:"node_id"`
|
||||
// Server is the control plane, and Servers is the same thing when there is
|
||||
// more than one. Both are accepted: every configuration written before
|
||||
// there could be a second control plane names one address, and none of
|
||||
// them should have to change.
|
||||
//
|
||||
// The order is tried in order, and a refusal moves to the next. Only the
|
||||
// leader accepts an agent, so a standby is skipped in the time it takes to
|
||||
// be told so — which is what makes failover a reconnect rather than an
|
||||
// edit to every node's configuration.
|
||||
Server string `koanf:"server"`
|
||||
Servers []string `koanf:"servers"`
|
||||
// Hostname defaults to the machine's.
|
||||
Hostname string `koanf:"hostname"`
|
||||
// MgmtAddr is the address engines are reached on. It defaults to this
|
||||
// machine's address on the route to the control plane, which is the one
|
||||
// the control plane and the router can reach.
|
||||
MgmtAddr string `koanf:"mgmt_addr"`
|
||||
// Labels are free-form facts about a node for a profile's node_selector.
|
||||
// Nothing needs them: memory, GPUs, engines and disk are measured, and a
|
||||
// label restating one of those goes stale the day the machine changes.
|
||||
Labels map[string]string `koanf:"labels"`
|
||||
|
||||
// MgmtInterface is the ethernet interface NCCL bootstraps over. It
|
||||
// defaults to the interface holding MgmtAddr, then eth0.
|
||||
MgmtInterface string `koanf:"mgmt_interface"`
|
||||
// Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
|
||||
// default subnet prefix is the same on every fabric, so nothing can be
|
||||
// inferred from the wire (plan §5.2).
|
||||
Fabrics []inventory.FabricAttachment `koanf:"fabrics"`
|
||||
|
||||
HeartbeatInterval time.Duration `koanf:"heartbeat_interval"`
|
||||
// ReconnectBase is the first reconnect delay; it backs off from there.
|
||||
ReconnectBase time.Duration `koanf:"reconnect_base"`
|
||||
ReconnectMax time.Duration `koanf:"reconnect_max"`
|
||||
|
||||
// AdoptReadyTimeout bounds how long an adopted engine is given to answer a
|
||||
// health check before the agent stops waiting on it.
|
||||
//
|
||||
// Shorter than a startup timeout on purpose: this engine is already
|
||||
// running, so it is not loading a model, it is either serving or it is not.
|
||||
AdoptReadyTimeout time.Duration `koanf:"adopt_ready_timeout"`
|
||||
|
||||
// TLS authenticates both ends of the control-plane stream. Leaving it unset
|
||||
// keeps the plaintext behaviour, which only belongs on a trusted network.
|
||||
TLS tlsconf.Config `koanf:"tls"`
|
||||
|
||||
// ModelDirs are the directories this node keeps model weights in, beyond
|
||||
// WeightsCache, which is always looked in. Storage differs at every installation,
|
||||
// so llmgw observes it rather than owning it: a Lustre mount, an image
|
||||
// exported over the fabric, local NVMe and a copy made by hand all look
|
||||
// identical from here, and placement only needs to know whether the file
|
||||
// is present.
|
||||
ModelDirs []string `koanf:"model_dirs"`
|
||||
|
||||
// EnginesDir is where the agent installs engines, one directory per
|
||||
// build. Defaults to /opt/llmgw/engines.
|
||||
EnginesDir string `koanf:"engines_dir"`
|
||||
|
||||
// WeightsCache is where a remote source is downloaded to. Defaults to the
|
||||
// first entry of ModelDirs, because a node that says where its models live
|
||||
// has already answered this question, and otherwise to
|
||||
// /var/lib/llmgw-weights.
|
||||
//
|
||||
// Fetching is the simple arrangement: every node keeps its own copy. The
|
||||
// exotic ones need no code — weights on a shared filer or an image exported
|
||||
// over the fabric are simply present, and present means nothing is fetched.
|
||||
WeightsCache string `koanf:"weights_cache"`
|
||||
|
||||
Discovery discovery.Options `koanf:"discovery"`
|
||||
Engines engines.Config `koanf:"engines"`
|
||||
Superv supervisor.Config `koanf:"supervisor"`
|
||||
|
||||
Version string
|
||||
}
|
||||
|
||||
// DefaultConfig returns sensible defaults.
|
||||
func DefaultConfig() Config {
|
||||
// No MgmtInterface: normalize finds the one holding the management
|
||||
// address, and falls back to eth0 only when it cannot. Defaulted here, it
|
||||
// was never found — every node said eth0 unless told otherwise, and a node
|
||||
// on ens3 or bond0 had NCCL bootstrap over an interface it does not have.
|
||||
return Config{
|
||||
HeartbeatInterval: 5 * time.Second,
|
||||
ReconnectBase: time.Second,
|
||||
ReconnectMax: 30 * time.Second,
|
||||
AdoptReadyTimeout: 60 * time.Second,
|
||||
Superv: supervisor.DefaultConfig(),
|
||||
// NVML gives NVLink topology, so prefer it when this build has it.
|
||||
Discovery: discovery.Options{PreferNVML: true},
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Config) normalize() error {
|
||||
d := DefaultConfig()
|
||||
if err := c.TLS.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.normalizeServerURL(); err != nil {
|
||||
return err
|
||||
}
|
||||
// Everything below is found rather than written. A file that has to say
|
||||
// which machine it is on, and at what address, is a file that differs on
|
||||
// every node — and a copy taken from one node to the next registers the
|
||||
// second as the first.
|
||||
host, _ := os.Hostname()
|
||||
if c.NodeID == "" {
|
||||
c.NodeID = host
|
||||
}
|
||||
if c.NodeID == "" {
|
||||
return errors.New("node_id is not set and the hostname could not be read")
|
||||
}
|
||||
if c.Hostname == "" {
|
||||
c.Hostname = host
|
||||
}
|
||||
if c.MgmtAddr == "" {
|
||||
c.MgmtAddr = addrToward(c.Server)
|
||||
}
|
||||
if c.MgmtInterface == "" {
|
||||
c.MgmtInterface = interfaceOf(c.MgmtAddr)
|
||||
}
|
||||
if c.MgmtInterface == "" {
|
||||
c.MgmtInterface = "eth0"
|
||||
}
|
||||
// The directory downloads land in is always one that is looked in. When
|
||||
// they were two settings, a node could fill its disk with fetched weights
|
||||
// that placement never saw — which is what every node here was doing.
|
||||
if c.WeightsCache == "" && len(c.ModelDirs) > 0 {
|
||||
c.WeightsCache = c.ModelDirs[0]
|
||||
}
|
||||
if c.WeightsCache == "" {
|
||||
c.WeightsCache = DefaultWeightsCache
|
||||
}
|
||||
if !slices.Contains(c.ModelDirs, c.WeightsCache) {
|
||||
c.ModelDirs = append(c.ModelDirs, c.WeightsCache)
|
||||
}
|
||||
if c.HeartbeatInterval <= 0 {
|
||||
c.HeartbeatInterval = d.HeartbeatInterval
|
||||
}
|
||||
if c.ReconnectBase <= 0 {
|
||||
c.ReconnectBase = d.ReconnectBase
|
||||
}
|
||||
if c.AdoptReadyTimeout <= 0 {
|
||||
c.AdoptReadyTimeout = d.AdoptReadyTimeout
|
||||
}
|
||||
if c.ReconnectMax < c.ReconnectBase {
|
||||
c.ReconnectMax = max(c.ReconnectBase, d.ReconnectMax)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// normalizeServerURL makes the configured address a URL connect can use, and
|
||||
// refuses the combination that would quietly undo the certificates: a scheme
|
||||
// that says plaintext while TLS is configured. Silently connecting anyway would
|
||||
// mean a cluster believing it has mutual authentication when it has none.
|
||||
func (c *Config) normalizeServerURL() error {
|
||||
// One list, however it was written. Server first when both are given,
|
||||
// because an operator adding a standby to an existing file expects the
|
||||
// one already there to stay preferred.
|
||||
all := make([]string, 0, 1+len(c.Servers))
|
||||
if c.Server != "" {
|
||||
all = append(all, c.Server)
|
||||
}
|
||||
all = append(all, c.Servers...)
|
||||
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]string, 0, len(all))
|
||||
for _, s := range all {
|
||||
fixed, err := normalizeOne(s, c.TLS.Enabled())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// A duplicate would be tried twice in a row on failover, which reads
|
||||
// as the rotation not working.
|
||||
if _, dup := seen[fixed]; dup {
|
||||
continue
|
||||
}
|
||||
seen[fixed] = struct{}{}
|
||||
out = append(out, fixed)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return errors.New("server is required")
|
||||
}
|
||||
c.Servers = out
|
||||
c.Server = out[0]
|
||||
return nil
|
||||
}
|
||||
|
||||
// DefaultWeightsCache is where fetched weights go when nothing says otherwise.
|
||||
const DefaultWeightsCache = "/var/lib/llmgw-weights"
|
||||
|
||||
// addrToward is this machine's address on the route to the control plane.
|
||||
//
|
||||
// The address engines are reached on is whichever one the control plane can
|
||||
// reach, and the kernel already knows which that is: a UDP "connection" picks
|
||||
// the source address for the route without sending a packet. Asking the
|
||||
// interfaces instead would have to guess among docker bridges, VPNs and
|
||||
// link-local addresses.
|
||||
func addrToward(server string) string {
|
||||
u, err := url.Parse(server)
|
||||
if err != nil || u.Host == "" {
|
||||
return ""
|
||||
}
|
||||
host := u.Hostname()
|
||||
port := u.Port()
|
||||
if port == "" {
|
||||
port = "443"
|
||||
}
|
||||
conn, err := net.Dial("udp", net.JoinHostPort(host, port))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
defer conn.Close()
|
||||
if a, ok := conn.LocalAddr().(*net.UDPAddr); ok {
|
||||
return a.IP.String()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// interfaceOf names the interface that holds an address, or "" when none does.
|
||||
func interfaceOf(addr string) string {
|
||||
ip := net.ParseIP(addr)
|
||||
if ip == nil {
|
||||
return ""
|
||||
}
|
||||
ifaces, err := net.Interfaces()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
for _, ifc := range ifaces {
|
||||
addrs, err := ifc.Addrs()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, a := range addrs {
|
||||
if n, ok := a.(*net.IPNet); ok && n.IP.Equal(ip) {
|
||||
return ifc.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// normalizeOne applies the scheme rules to a single address.
|
||||
func normalizeOne(addr string, secure bool) (string, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(addr, "https://"):
|
||||
if !secure {
|
||||
return "", fmt.Errorf("%s is an https:// address but no tls certificates are configured", addr)
|
||||
}
|
||||
case strings.HasPrefix(addr, "http://"):
|
||||
if secure {
|
||||
return "", fmt.Errorf("tls is configured but %s is an http:// address; use https://", addr)
|
||||
}
|
||||
case strings.Contains(addr, "://"):
|
||||
return "", fmt.Errorf("server %q must be an http:// or https:// address", addr)
|
||||
case secure:
|
||||
return "https://" + addr, nil
|
||||
default:
|
||||
return "http://" + addr, nil
|
||||
}
|
||||
return addr, nil
|
||||
}
|
||||
@@ -76,6 +76,10 @@ type ShardSpec struct {
|
||||
// and accounting. Given to the engine in its environment rather than on
|
||||
// its command line, which every user on the node can read.
|
||||
APIKey string `json:"api_key,omitempty"`
|
||||
// Threads is the CPU cores placement reserved for a shard with no GPU,
|
||||
// and so what its engine computes on. Zero leaves it to the engine's own
|
||||
// configuration.
|
||||
Threads int `json:"threads,omitempty"`
|
||||
|
||||
// Pool and Fabric are control plane concepts the agent cannot derive. They
|
||||
// are carried so the agent can report them back after a restart; Fabric
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -21,8 +22,21 @@ type LlamaCppConfig struct {
|
||||
// does not say. llama.cpp defaults to one, which would make every second
|
||||
// request wait behind the first no matter what the router decided.
|
||||
Parallel int `koanf:"parallel"`
|
||||
// Threads is how many CPU threads a shard with no GPU computes on. Zero
|
||||
// takes half this machine's CPUs.
|
||||
//
|
||||
// Not all of them, which is llama.cpp's own default: two busy servers
|
||||
// that each spin on every CPU fight for them, and on an 8-vCPU node
|
||||
// that made 7B generation fall from 10 tokens a second to under 1 each.
|
||||
// At half, one server alone is about a third slower and two together
|
||||
// keep 5.5 each. A node that only ever runs one CPU replica does better
|
||||
// with all of them; say so here.
|
||||
Threads int `koanf:"threads"`
|
||||
}
|
||||
|
||||
// numCPU is the CPUs this process may run on; a variable for tests.
|
||||
var numCPU = runtime.NumCPU
|
||||
|
||||
// LlamaCpp builds llama.cpp launch commands.
|
||||
//
|
||||
// It is the engine that makes a cluster without GPUs useful: llama.cpp runs a
|
||||
@@ -118,6 +132,18 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
|
||||
args = append(args, "--n-gpu-layers", "999")
|
||||
}
|
||||
if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
|
||||
// What placement reserved first: it is what the node's other
|
||||
// replicas were placed around.
|
||||
n := s.Threads
|
||||
if n <= 0 {
|
||||
n = l.cfg.Threads
|
||||
}
|
||||
if n <= 0 {
|
||||
n = max(numCPU()/2, 1)
|
||||
}
|
||||
args = append(args, "--threads", strconv.Itoa(n))
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
|
||||
env := baseEnv(s)
|
||||
|
||||
@@ -261,3 +261,48 @@ func TestLlamaCppIsGivenItsKeyOutOfSightOfTheProcessList(t *testing.T) {
|
||||
t.Error("llama-server was not given its key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
|
||||
prev := numCPU
|
||||
numCPU = func() int { return 8 }
|
||||
t.Cleanup(func() { numCPU = prev })
|
||||
|
||||
threads := func(l *LlamaCpp, s ShardSpec) (string, bool) {
|
||||
cmds, err := l.Commands(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return flagValue(cmds[0].Args, "--threads")
|
||||
}
|
||||
if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "4" {
|
||||
t.Errorf("--threads %q on 8 CPUs, expected 4", v)
|
||||
}
|
||||
if v, _ := threads(&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}, llamaSpec(nil)); v != "8" {
|
||||
t.Errorf("--threads %q with 8 configured", v)
|
||||
}
|
||||
if _, ok := threads(&LlamaCpp{}, llamaSpec([]int{0})); ok {
|
||||
t.Error("a GPU shard was given a thread count")
|
||||
}
|
||||
s := llamaSpec(nil)
|
||||
s.Profile.Args = []string{"-t", "2"}
|
||||
cmds, _ := (&LlamaCpp{}).Commands(s)
|
||||
if slices.Contains(cmds[0].Args, "--threads") {
|
||||
t.Error("the profile's own -t was overridden")
|
||||
}
|
||||
numCPU = func() int { return 1 }
|
||||
if v, _ := threads(&LlamaCpp{}, llamaSpec(nil)); v != "1" {
|
||||
t.Errorf("--threads %q on one CPU", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
|
||||
s := llamaSpec(nil)
|
||||
s.Threads = 3
|
||||
cmds, err := (&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}).Commands(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, _ := flagValue(cmds[0].Args, "--threads"); v != "3" {
|
||||
t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
|
||||
}
|
||||
}
|
||||
@@ -169,7 +169,10 @@ func (v *VLLM) rayCommand(s ShardSpec) (Command, bool) {
|
||||
|
||||
args := []string{"start", "--block"}
|
||||
if s.Leader() {
|
||||
args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort))
|
||||
// Ray's head has no authentication: whoever reaches its port can run
|
||||
// code on the node. Its dashboard, and the job API behind it, is the
|
||||
// easiest way in, and nothing here uses it.
|
||||
args = append(args, "--head", "--port", strconv.Itoa(s.LeaderPort), "--include-dashboard=false")
|
||||
} else {
|
||||
args = append(args, "--address", fmt.Sprintf("%s:%d", s.LeaderAddr, s.LeaderPort))
|
||||
}
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// usableFabrics reports the fabrics this node can actually carry traffic on.
|
||||
//
|
||||
// Loud about what it drops. A fabric silently missing from a registration looks
|
||||
// identical to one never configured, and the operator who typed the adapter
|
||||
// name is the only person who can fix it — so the reason is logged here, on the
|
||||
// machine where both halves are visible, rather than inferred later from a
|
||||
// replica that will not converge.
|
||||
func (a *Agent) usableFabrics(inv discovery.Inventory) []inventory.FabricAttachment {
|
||||
out := a.checkFabrics(inv.HCAs, true)
|
||||
a.mu.Lock()
|
||||
a.fabricsSent = fabricsFingerprint(out)
|
||||
a.mu.Unlock()
|
||||
return out
|
||||
}
|
||||
|
||||
// checkFabrics is the declared fabrics this node can carry traffic on, from
|
||||
// the adapters as they are now. Loud logs every problem; the heartbeat checks
|
||||
// quietly and speaks only when the answer changes.
|
||||
func (a *Agent) checkFabrics(hcas []discovery.HCA, loud bool) []inventory.FabricAttachment {
|
||||
declared := map[string][]string{}
|
||||
for _, f := range a.cfg.Fabrics {
|
||||
declared[f.Fabric] = f.HCAs
|
||||
}
|
||||
|
||||
var out []inventory.FabricAttachment
|
||||
for _, c := range discovery.CheckFabrics(declared, hcas) {
|
||||
if loud {
|
||||
for _, p := range c.Problems {
|
||||
a.log.Error("a declared fabric adapter cannot carry traffic",
|
||||
"fabric", c.Fabric, "problem", p)
|
||||
}
|
||||
}
|
||||
if !c.OK() {
|
||||
if loud {
|
||||
a.log.Error("fabric not advertised; no declared adapter on it can carry traffic",
|
||||
"fabric", c.Fabric,
|
||||
"consequence", "no multi-node replica will be placed here")
|
||||
}
|
||||
continue
|
||||
}
|
||||
out = append(out, inventory.FabricAttachment{
|
||||
Fabric: c.Fabric, HCAs: c.Usable, RateGbps: c.RateGbps,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// fabricsFingerprint is enough to notice a fabric, an adapter or a rate
|
||||
// changing.
|
||||
func fabricsFingerprint(fs []inventory.FabricAttachment) string {
|
||||
var b strings.Builder
|
||||
for _, f := range fs {
|
||||
fmt.Fprintf(&b, "%s:%s:%d;", f.Fabric, strings.Join(f.HCAs, ","), f.RateGbps)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// sampleIB re-reads the adapters for the heartbeat: every port's state and
|
||||
// counters, and the fabrics again when they no longer match what the control
|
||||
// plane was told.
|
||||
//
|
||||
// Every heartbeat, because it is a few small files per port, and because a
|
||||
// port that went down used to be invisible until the agent restarted — the
|
||||
// node went on being offered multi-node work that would hang on its first
|
||||
// collective.
|
||||
func (a *Agent) sampleIB(hb *llmgwv1.Heartbeat) {
|
||||
a.mu.Lock()
|
||||
had := len(a.inv.HCAs) > 0
|
||||
a.mu.Unlock()
|
||||
if !had && len(a.cfg.Fabrics) == 0 {
|
||||
return
|
||||
}
|
||||
hcas, err := discovery.DiscoverIB(a.cfg.Discovery.IBRoot)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
a.attachSM(hcas)
|
||||
for _, h := range hcas {
|
||||
for _, p := range h.Ports {
|
||||
ip := &llmgwv1.IBPort{
|
||||
Hca: h.Name, Port: int32(p.Num), State: p.State, PhysState: p.PhysState,
|
||||
RateGbps: int32(p.RateGbps), LinkLayer: string(p.LinkLayer), Counters: p.Counters,
|
||||
NodeGuid: h.NodeGUID, PortGuid: p.PortGUID, Network: p.Network, GidPrefix: p.GIDPrefix,
|
||||
}
|
||||
a.mu.Lock()
|
||||
if p.SM != nil {
|
||||
ip.SmGuid, ip.SmLid, ip.SmPriority, ip.SmState = p.SM.GUID, int32(p.SM.LID), int32(p.SM.Priority), p.SM.State
|
||||
} else {
|
||||
ip.SmError = a.smErr[smKey(h.Name, p.Num)]
|
||||
}
|
||||
for _, r := range a.sms[smKey(h.Name, p.Num)] {
|
||||
ip.SubnetManagers = append(ip.SubnetManagers, &llmgwv1.SubnetManager{
|
||||
Guid: r.GUID, Lid: int32(r.LID), Priority: int32(r.Priority), State: r.State})
|
||||
}
|
||||
ip.SubnetManagersError = a.smsErr[smKey(h.Name, p.Num)]
|
||||
a.mu.Unlock()
|
||||
hb.IbPorts = append(hb.IbPorts, ip)
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.inv.HCAs = hcas // a reconnect registers what is true now
|
||||
a.mu.Unlock()
|
||||
if len(a.cfg.Fabrics) == 0 {
|
||||
return
|
||||
}
|
||||
now := a.checkFabrics(hcas, false)
|
||||
fp := fabricsFingerprint(now)
|
||||
a.mu.Lock()
|
||||
changed := fp != a.fabricsSent
|
||||
a.fabricsSent = fp
|
||||
a.mu.Unlock()
|
||||
if !changed {
|
||||
return
|
||||
}
|
||||
// Said once, with the reasons, the moment it happens.
|
||||
a.checkFabrics(hcas, true)
|
||||
a.log.Warn("the fabrics this node can carry traffic on changed", "now", fp)
|
||||
hb.FabricsChanged = true
|
||||
for _, f := range now {
|
||||
hb.Fabrics = append(hb.Fabrics, &llmgwv1.FabricAttachment{
|
||||
Fabric: f.Fabric, Hcas: f.HCAs, RateGbps: int32(f.RateGbps),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func smKey(hca string, port int) string { return fmt.Sprintf("%s/%d", hca, port) }
|
||||
|
||||
// scanFabric is discovery.ScanFabric, replaced in tests.
|
||||
var scanFabric = discovery.ScanFabric
|
||||
|
||||
// querySMs is discovery.QuerySMs, replaced in tests.
|
||||
var querySMs = func(hca string, port int) ([]discovery.SMRecord, error) {
|
||||
return discovery.QuerySMs(context.Background(), hca, port)
|
||||
}
|
||||
|
||||
func splitSMKey(k string) (string, int) {
|
||||
i := strings.LastIndexByte(k, '/')
|
||||
port, _ := strconv.Atoi(k[i+1:])
|
||||
return k[:i], port
|
||||
}
|
||||
|
||||
// querySM is discovery.QuerySM, replaced in tests.
|
||||
var querySM = func(hca string, port int) (*discovery.SMInfo, error) {
|
||||
return discovery.QuerySM(context.Background(), hca, port)
|
||||
}
|
||||
|
||||
// attachSM puts on each active InfiniBand port the subnet manager it sees,
|
||||
// asking again once a minute and otherwise reusing the last answer.
|
||||
func (a *Agent) attachSM(hcas []discovery.HCA) {
|
||||
a.mu.Lock()
|
||||
stale := time.Since(a.smAsked) > time.Minute || a.sm == nil
|
||||
a.mu.Unlock()
|
||||
if stale {
|
||||
sm, errs := map[string]*discovery.SMInfo{}, map[string]string{}
|
||||
_ = discovery.AttachSM(hcas, func(hca string, port int) (*discovery.SMInfo, error) {
|
||||
got, err := querySM(hca, port)
|
||||
if err != nil {
|
||||
errs[smKey(hca, port)] = err.Error()
|
||||
return nil, err
|
||||
}
|
||||
sm[smKey(hca, port)] = got
|
||||
return got, nil
|
||||
})
|
||||
// Every subnet manager on each fabric, through one port per master:
|
||||
// the same list through every port of one fabric is one query too
|
||||
// many for each.
|
||||
sms, smsErr := map[string][]discovery.SMRecord{}, map[string]string{}
|
||||
asked := map[string]bool{}
|
||||
for k, v := range sm {
|
||||
if asked[v.GUID] {
|
||||
continue
|
||||
}
|
||||
asked[v.GUID] = true
|
||||
hca, port := splitSMKey(k)
|
||||
if recs, err := querySMs(hca, port); err != nil {
|
||||
smsErr[k] = err.Error()
|
||||
} else {
|
||||
sms[k] = recs
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
for k, v := range sm {
|
||||
if old := a.sm[k]; old != nil && old.GUID != v.GUID {
|
||||
a.log.Warn("the master subnet manager changed", "port", k, "from", old.GUID, "to", v.GUID)
|
||||
}
|
||||
}
|
||||
a.sm, a.smErr, a.smAsked = sm, errs, time.Now()
|
||||
a.sms, a.smsErr = sms, smsErr
|
||||
a.mu.Unlock()
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
for i := range hcas {
|
||||
for j := range hcas[i].Ports {
|
||||
if sm := a.sm[smKey(hcas[i].Name, hcas[i].Ports[j].Num)]; sm != nil && hcas[i].Ports[j].Active() {
|
||||
hcas[i].Ports[j].SM = sm
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ensureDiags installs infiniband-diags on a node with adapters and without
|
||||
// it. sminfo is what tells which physical fabric a port is on; without it,
|
||||
// two separate fabrics given one name cannot be caught. Installed the way
|
||||
// engine dependencies are, from the machine's own package sources.
|
||||
func (a *Agent) ensureDiags(ctx context.Context) {
|
||||
a.mu.Lock()
|
||||
hasIB := len(a.inv.HCAs) > 0
|
||||
a.mu.Unlock()
|
||||
if !hasIB || a.prov == nil {
|
||||
return
|
||||
}
|
||||
if _, err := exec.LookPath("sminfo"); err == nil {
|
||||
return
|
||||
}
|
||||
a.log.Info("installing infiniband-diags, to tell which fabric each port is on")
|
||||
pkgs := map[string][]string{"deb": {"infiniband-diags"}, "rpm": {"infiniband-diags"}}
|
||||
if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
|
||||
a.log.Warn("could not install infiniband-diags; fabrics cannot be told apart by their subnet manager",
|
||||
"err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// FeatureFabricTest is the agent answering FabricTestRequest.
|
||||
const FeatureFabricTest = "fabric-test"
|
||||
|
||||
// runWriteBW is discovery.RunWriteBW, and haveWriteBW whether it is
|
||||
// installed; both replaced in tests.
|
||||
var (
|
||||
runWriteBW = discovery.RunWriteBW
|
||||
haveWriteBW = func() bool { _, err := exec.LookPath("ib_write_bw"); return err == nil }
|
||||
)
|
||||
|
||||
// fabricTest runs this node's side of a bandwidth test with another.
|
||||
func (a *Agent) fabricTest(ctx context.Context, req *llmgwv1.FabricTestRequest) *llmgwv1.FabricTestReply {
|
||||
start := time.Now()
|
||||
reply := &llmgwv1.FabricTestReply{RequestId: req.GetRequestId()}
|
||||
defer func() { reply.TookMs = time.Since(start).Milliseconds() }()
|
||||
|
||||
w := discovery.WriteBW{HCA: req.GetHca(), Port: int(req.GetPort()), GIDIndex: -1,
|
||||
TCPPort: int(req.GetTcpPort()), Seconds: int(req.GetSeconds())}
|
||||
if req.GetRole() == "client" {
|
||||
w.Peer = req.GetPeer()
|
||||
}
|
||||
a.mu.Lock()
|
||||
found := false
|
||||
for _, h := range a.inv.HCAs {
|
||||
for _, p := range h.Ports {
|
||||
if h.Name == w.HCA && p.Num == w.Port {
|
||||
found = true
|
||||
// RoCE needs its v2 GID named, as NCCL does; InfiniBand uses the LID.
|
||||
if p.LinkLayer == discovery.LinkEthernet && p.RoCEv2GID >= 0 {
|
||||
w.GIDIndex = p.RoCEv2GID
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
a.mu.Unlock()
|
||||
if !found {
|
||||
reply.Error = fmt.Sprintf("this node has no port %s/%d", w.HCA, w.Port)
|
||||
return reply
|
||||
}
|
||||
if !haveWriteBW() && a.prov != nil {
|
||||
a.log.Info("installing perftest, for a bandwidth test with another node")
|
||||
pkgs := map[string][]string{"deb": {"perftest"}, "rpm": {"perftest"}}
|
||||
if err := a.prov.InstallPackages(ctx, pkgs); err != nil {
|
||||
reply.Error = "installing perftest: " + err.Error()
|
||||
return reply
|
||||
}
|
||||
}
|
||||
gbps, out, err := runWriteBW(ctx, w)
|
||||
reply.Gbps = gbps
|
||||
reply.Output = tailLines(out, 12)
|
||||
if err != nil {
|
||||
reply.Error = err.Error()
|
||||
}
|
||||
return reply
|
||||
}
|
||||
|
||||
func tailLines(s string, n int) string {
|
||||
lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// discoverGPUFabric and discoverIMEX are replaced in tests.
|
||||
var (
|
||||
discoverGPUFabric = discovery.DiscoverGPUFabric
|
||||
discoverIMEX = discovery.DiscoverIMEX
|
||||
)
|
||||
|
||||
// sampleNVLink reads, once a minute, which NVLink domain each GPU is joined
|
||||
// to and whether IMEX is up. nvidia-smi -q takes a second or two, too long
|
||||
// for every heartbeat, and a fabric that was joined an hour ago rarely
|
||||
// changes in five seconds — but when a fabric manager restarts it does, for
|
||||
// every card at once.
|
||||
func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
|
||||
a.mu.Lock()
|
||||
gpus := slices.Clone(a.inv.GPUs)
|
||||
due := time.Since(a.nvlAsked) > time.Minute
|
||||
if due {
|
||||
a.nvlAsked = time.Now()
|
||||
}
|
||||
a.mu.Unlock()
|
||||
if len(gpus) == 0 || !due {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
fab, err := discoverGPUFabric(ctx, a.cfg.Discovery.NvidiaSMI)
|
||||
if err != nil || len(fab) == 0 {
|
||||
return
|
||||
}
|
||||
discovery.ApplyFabric(gpus, fab)
|
||||
for _, g := range gpus {
|
||||
if g.Fabric == nil {
|
||||
continue
|
||||
}
|
||||
hb.GpuFabric = append(hb.GpuFabric, &llmgwv1.GPUFabricState{
|
||||
Index: int32(g.Index), State: g.Fabric.State, Status: g.Fabric.Status,
|
||||
ClusterUuid: g.Fabric.ClusterUUID, CliqueId: g.Fabric.CliqueID, Bandwidth: g.Fabric.Bandwidth,
|
||||
})
|
||||
}
|
||||
m := discoverIMEX(ctx)
|
||||
hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// logsReply answers a request for one of this node's logs.
|
||||
func (a *Agent) logsReply(req *llmgwv1.LogsRequest) *llmgwv1.LogsReply {
|
||||
n := int(req.GetLines())
|
||||
if n <= 0 {
|
||||
n = 200
|
||||
}
|
||||
n = min(n, agentLogLines)
|
||||
shards := a.sup.IDs()
|
||||
slices.Sort(shards)
|
||||
out := &llmgwv1.LogsReply{
|
||||
RequestId: req.GetRequestId(),
|
||||
Sources: append([]string{"agent"}, shards...),
|
||||
}
|
||||
switch src := req.GetSource(); src {
|
||||
case "", "agent":
|
||||
out.Lines = a.logRing.tail(n)
|
||||
default:
|
||||
sh, err := a.sup.Get(src)
|
||||
if err != nil {
|
||||
out.Error = "no engine " + src + " runs on this node"
|
||||
return out
|
||||
}
|
||||
out.Lines = sh.Logs(n)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
)
|
||||
|
||||
// engines is what this node can run: what the agent installed and saw run,
|
||||
// and what the configuration names and answers. Nothing merely found on the
|
||||
// machine — a llama-server on PATH is somebody's, not necessarily one that
|
||||
// works, and placing on it is how a replica fails on a node nobody watches.
|
||||
func (a *Agent) engines(ctx context.Context) map[string]string {
|
||||
a.engineOnce.Do(func() {
|
||||
vers := a.reg.ProbeConfigured(ctx)
|
||||
a.mu.Lock()
|
||||
a.configuredVers, a.lastEngineProbe = vers, time.Now()
|
||||
a.mu.Unlock()
|
||||
if len(vers) > 0 {
|
||||
a.log.Info("engines named in the configuration",
|
||||
"engines", engines.ProbedNames(vers))
|
||||
}
|
||||
})
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
vers := a.engineReportLocked()
|
||||
// Whatever registration sends is what the control plane now knows, so a
|
||||
// heartbeat repeats it only once it differs.
|
||||
a.enginesSent = enginesFingerprint(vers)
|
||||
return vers
|
||||
}
|
||||
|
||||
// engineReportLocked merges the two sources. Installed wins: it is the one
|
||||
// that was seen to run. Called with mu held.
|
||||
func (a *Agent) engineReportLocked() map[string]string {
|
||||
out := maps.Clone(a.configuredVers)
|
||||
if out == nil {
|
||||
out = map[string]string{}
|
||||
}
|
||||
maps.Copy(out, a.installedVers)
|
||||
return out
|
||||
}
|
||||
|
||||
// provisionEvery is how often installed engines are checked again. A library
|
||||
// removed by an unrelated upgrade is found here rather than by the next
|
||||
// replica to start.
|
||||
const provisionEvery = 10 * time.Minute
|
||||
|
||||
// setProvision takes the control plane's list of wanted engines.
|
||||
func (a *Agent) setProvision(p *llmgwv1.Provision) {
|
||||
wants := []provision.Want{}
|
||||
for _, e := range p.GetEngines() {
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.provWants = wants
|
||||
a.mu.Unlock()
|
||||
select {
|
||||
case a.provKick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// provisionLoop installs and re-checks the wanted engines, one pass at a time.
|
||||
func (a *Agent) provisionLoop(ctx context.Context) {
|
||||
// After a failure the next pass comes sooner, and backs off: a lock held
|
||||
// by a new machine's own updates clears in a minute or two, while ten
|
||||
// minutes' wait left a node idle for no reason.
|
||||
next := provisionEvery
|
||||
retry := time.Minute
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-a.provKick:
|
||||
case <-time.After(next):
|
||||
}
|
||||
a.mu.Lock()
|
||||
wants := slices.Clone(a.provWants)
|
||||
managed := a.provWants != nil
|
||||
a.mu.Unlock()
|
||||
if !managed {
|
||||
continue
|
||||
}
|
||||
|
||||
statuses := a.prov.Converge(ctx, wants)
|
||||
next = provisionEvery
|
||||
for _, st := range statuses {
|
||||
if st.State == provision.Failed {
|
||||
next, retry = retry, min(retry*2, provisionEvery)
|
||||
break
|
||||
}
|
||||
}
|
||||
if next == provisionEvery {
|
||||
retry = time.Minute
|
||||
}
|
||||
installed := map[string]string{}
|
||||
keep := map[string]bool{}
|
||||
for _, st := range statuses {
|
||||
keep[st.Engine] = true
|
||||
if st.State == provision.Ready {
|
||||
a.reg.UseInstalled(st.Engine, st.Binary)
|
||||
installed[st.Engine] = st.Version
|
||||
}
|
||||
}
|
||||
a.mu.Lock()
|
||||
if enginesFingerprint(installed) != enginesFingerprint(a.installedVers) {
|
||||
a.log.Info("engines installed and running", "engines", engines.ProbedNames(installed))
|
||||
}
|
||||
a.installedVers = installed
|
||||
// An engine no longer asked for is no longer reported on.
|
||||
for e := range a.provStatus {
|
||||
if !keep[e] {
|
||||
delete(a.provStatus, e)
|
||||
}
|
||||
}
|
||||
a.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// provisionStatus records a change in one engine's installation.
|
||||
func (a *Agent) provisionStatus(st provision.Status) {
|
||||
a.mu.Lock()
|
||||
prev, had := a.provStatus[st.Engine]
|
||||
a.provStatus[st.Engine] = st
|
||||
a.mu.Unlock()
|
||||
if !had || prev.State != st.State || prev.Detail != st.Detail {
|
||||
lvl := slog.LevelInfo
|
||||
if st.State == provision.Failed || st.State == provision.Unsupported {
|
||||
lvl = slog.LevelWarn
|
||||
}
|
||||
a.log.Log(context.Background(), lvl, "engine installation",
|
||||
"engine", st.Engine, "version", st.Version, "variant", st.Variant,
|
||||
"state", st.State, "detail", st.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func provisionFingerprint(m map[string]provision.Status) string {
|
||||
var b strings.Builder
|
||||
for _, k := range slices.Sorted(maps.Keys(m)) {
|
||||
st := m[k]
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%s\n", st.Engine, st.Version, st.Variant, st.State, st.Detail)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// engineReprobe is how often the installed engines are looked for again.
|
||||
//
|
||||
// Minutes, not seconds: an engine is installed by somebody at a keyboard, so a
|
||||
// minute's delay in noticing is invisible, while SGLang's probe starts a
|
||||
// Python interpreter that imports it.
|
||||
const engineReprobe = time.Minute
|
||||
|
||||
// reprobeEngines looks for installed engines again when it is time, off the
|
||||
// heartbeat goroutine so a slow probe never delays a heartbeat.
|
||||
func (a *Agent) reprobeEngines() {
|
||||
a.mu.Lock()
|
||||
due := !a.probingEngines && !a.lastEngineProbe.IsZero() &&
|
||||
time.Since(a.lastEngineProbe) >= engineReprobe
|
||||
if due {
|
||||
a.probingEngines = true
|
||||
}
|
||||
a.mu.Unlock()
|
||||
if !due {
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
vers := a.reg.ProbeConfigured(ctx)
|
||||
|
||||
a.mu.Lock()
|
||||
changed := enginesFingerprint(vers) != enginesFingerprint(a.configuredVers)
|
||||
a.configuredVers, a.lastEngineProbe, a.probingEngines = vers, time.Now(), false
|
||||
a.mu.Unlock()
|
||||
if changed {
|
||||
a.log.Info("engines named in the configuration changed",
|
||||
"engines", engines.ProbedNames(vers))
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func enginesFingerprint(vers map[string]string) string {
|
||||
var b strings.Builder
|
||||
for _, k := range slices.Sorted(maps.Keys(vers)) {
|
||||
fmt.Fprintf(&b, "%s=%s\n", k, vers[k])
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
|
||||
// Restart=on-failure starts the new binary, and distinct, so the journal says
|
||||
// why the agent went.
|
||||
const upgradeExit = 75
|
||||
|
||||
// exitForUpgrade ends the process; a variable so a test can see it called
|
||||
// rather than have its own process end.
|
||||
var exitForUpgrade = func() { os.Exit(upgradeExit) }
|
||||
|
||||
// upgradeTarget is the binary an upgrade replaces; a variable so a test
|
||||
// replaces a file of its own rather than the test binary running it.
|
||||
var upgradeTarget = os.Executable
|
||||
|
||||
// upgrade replaces this agent's binary with a release and restarts into it.
|
||||
//
|
||||
// Engines are not touched: an agent restart leaves them running and adopts
|
||||
// them again, which is what makes upgrading a node something that can be done
|
||||
// while it serves.
|
||||
func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
|
||||
reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
|
||||
send := func() {
|
||||
select {
|
||||
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}
|
||||
a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
|
||||
self, err := upgradeTarget()
|
||||
if err == nil {
|
||||
dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
|
||||
cancel()
|
||||
}
|
||||
if err != nil {
|
||||
reply.Error = err.Error()
|
||||
a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
|
||||
send()
|
||||
return
|
||||
}
|
||||
send()
|
||||
a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
|
||||
// A moment for the reply to leave before the stream goes.
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
exitForUpgrade()
|
||||
}
|
||||
+34
-4
@@ -139,6 +139,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newStatusCmd(client, &asJSON),
|
||||
newModelCmd(client),
|
||||
newScaleCmd(client),
|
||||
newResourcesCmd(client),
|
||||
newRestartCmd(client),
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
@@ -323,17 +324,27 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
}
|
||||
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tALIASES\tPROFILES\tFALLBACKS")
|
||||
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
|
||||
var warnings []string
|
||||
for _, m := range models {
|
||||
serving := "no"
|
||||
if m.Ready > 0 {
|
||||
serving = "yes"
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\n",
|
||||
m.Name, serving, m.Ready, m.Min, modelStatus(m),
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\n",
|
||||
m.Name, serving, m.Ready, m.Min, modelStatus(m), reserves(m),
|
||||
join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
|
||||
for _, x := range m.Warnings {
|
||||
warnings = append(warnings, m.Name+": "+x)
|
||||
}
|
||||
}
|
||||
return w.Flush()
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, x := range warnings {
|
||||
fmt.Println("warning:", x)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -511,3 +522,22 @@ func dash(s string) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// reserves is what one replica of a model holds of a node, with what it was
|
||||
// measured using beside it: "6G 2c (5.1G used)".
|
||||
func reserves(m adminapi.ModelView) string {
|
||||
if m.MemoryGB == 0 && m.CPUCores == 0 {
|
||||
return "-"
|
||||
}
|
||||
out := fmt.Sprintf("%dG", m.MemoryGB)
|
||||
if m.CPUCores > 0 {
|
||||
out += fmt.Sprintf(" %dc", m.CPUCores)
|
||||
}
|
||||
if m.MeasuredMemoryGB > 0 {
|
||||
out += fmt.Sprintf(" (%.1fG used)", m.MeasuredMemoryGB)
|
||||
}
|
||||
if len(m.Warnings) > 0 {
|
||||
out += " !"
|
||||
}
|
||||
return out
|
||||
}
|
||||
+31
-2
@@ -8,6 +8,7 @@ import (
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -232,7 +233,7 @@ func newMirrorRemoveCmd(dir *string) *cobra.Command {
|
||||
func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
var listen, apiListen, base string
|
||||
var tlsc tlsconf.Config
|
||||
var allow []string
|
||||
var allow, clients []string
|
||||
var allowPrivate bool
|
||||
|
||||
c := &cobra.Command{
|
||||
@@ -259,6 +260,9 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
if err := tlsc.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(clients) > 0 && !tlsc.Enabled() {
|
||||
return fmt.Errorf("--api-client names a certificate, and needs --tls-ca, --tls-cert and --tls-key")
|
||||
}
|
||||
|
||||
out := cmd.OutOrStdout()
|
||||
log := slog.Default()
|
||||
@@ -282,7 +286,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
}
|
||||
admin := &http.Server{
|
||||
Addr: apiListen,
|
||||
Handler: apiGate(tlsc.Enabled(), allowed, api.Handler()),
|
||||
Handler: onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
|
||||
@@ -294,6 +298,12 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
switch {
|
||||
case tlsc.Enabled():
|
||||
fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
|
||||
if len(clients) == 0 {
|
||||
// The same root signs every node's certificate, so without a
|
||||
// name any node could delete models here.
|
||||
log.Warn("the management API takes any certificate from " + tlsc.CA +
|
||||
", every node's included; name the control plane's with --api-client")
|
||||
}
|
||||
case len(allowed) > 0:
|
||||
fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
|
||||
default:
|
||||
@@ -317,6 +327,8 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
"the address nodes use to reach this mirror, as it should appear in catalog sources")
|
||||
c.Flags().StringArrayVar(&allow, "api-allow", nil,
|
||||
"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
|
||||
c.Flags().StringArrayVar(&clients, "api-client", nil,
|
||||
"with mutual TLS, the common name of a certificate the management API answers; repeatable (default: any from --tls-ca)")
|
||||
c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
|
||||
"let downloads reach loopback, private and link-local addresses")
|
||||
c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
|
||||
@@ -452,6 +464,23 @@ func apiGate(mtls bool, allowed []netip.Prefix, next http.Handler) http.Handler
|
||||
return onlyFrom(allowed, next)
|
||||
}
|
||||
|
||||
// onlyClients answers only certificates with one of these common names, when
|
||||
// any are given. The handshake has already checked the certificate against
|
||||
// the root; this narrows which of the root's certificates may manage.
|
||||
func onlyClients(names []string, next http.Handler) http.Handler {
|
||||
if len(names) == 0 {
|
||||
return next
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.TLS == nil || len(r.TLS.VerifiedChains) == 0 ||
|
||||
!slices.Contains(names, r.TLS.VerifiedChains[0][0].Subject.CommonName) {
|
||||
http.Error(w, "this mirror is managed by its control plane only", http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// onlyFrom answers only this machine and the addresses allowed: the management
|
||||
// API can delete models and start downloads, and a mirror on its own machine
|
||||
// is reached over the network by one control plane and no one else.
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
@@ -37,3 +40,27 @@ func TestTheMirrorAPIAnswersOnlyThisMachineUnlessTold(t *testing.T) {
|
||||
t.Errorf("mTLS without an allowlist refused a client: %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyANamedCertificateManagesTheMirror(t *testing.T) {
|
||||
ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {})
|
||||
with := func(cn string) *http.Request {
|
||||
r := httptest.NewRequest("GET", "/", nil)
|
||||
if cn != "" {
|
||||
r.TLS = &tls.ConnectionState{VerifiedChains: [][]*x509.Certificate{{{Subject: pkix.Name{CommonName: cn}}}}}
|
||||
}
|
||||
return r
|
||||
}
|
||||
h := onlyClients([]string{"control-plane"}, ok)
|
||||
for cn, want := range map[string]int{"control-plane": 200, "node01": 403, "": 403} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, with(cn))
|
||||
if w.Code != want {
|
||||
t.Errorf("%q: %d, expected %d", cn, w.Code, want)
|
||||
}
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
onlyClients(nil, ok).ServeHTTP(w, with("node01"))
|
||||
if w.Code != 200 {
|
||||
t.Error("no names given narrowed it anyway")
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"time"
|
||||
|
||||
@@ -127,6 +128,10 @@ func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
|
||||
|
||||
// actor names who made a change, for the audit trail on a stored model.
|
||||
func actor() string {
|
||||
// Under sudo, the person who ran it rather than root.
|
||||
if u := os.Getenv("SUDO_USER"); u != "" && os.Geteuid() == 0 {
|
||||
return u
|
||||
}
|
||||
if u, err := user.Current(); err == nil && u.Username != "" {
|
||||
return u.Username
|
||||
}
|
||||
@@ -187,3 +192,44 @@ func newScaleCmd(client func() *ctlClient) *cobra.Command {
|
||||
"how long an on-demand model stays up unasked-for; zero takes the cluster default")
|
||||
return c
|
||||
}
|
||||
|
||||
func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
var memory, cores int
|
||||
var profile string
|
||||
c := &cobra.Command{
|
||||
Use: "resources <model> --memory GB [--cores N]",
|
||||
Short: "Set what a model reserves of a node: memory, and CPU cores",
|
||||
Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
|
||||
"weights, KV cache and engine together. Placement reserves it on the node,\n" +
|
||||
"so a node holds only as many replicas as it has memory for.\n\n" +
|
||||
"--cores is the CPU cores a replica with no GPU computes on. They are\n" +
|
||||
"reserved the same way, and become the engine's thread count, so two\n" +
|
||||
"replicas on one node stop competing for the same cores. Zero reserves\n" +
|
||||
"none and leaves the thread count to the node.\n\n" +
|
||||
"Running replicas keep what they were started with; `llmgw ctl restart\n" +
|
||||
"--model M` moves them to the new figures.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if !cmd.Flags().Changed("memory") {
|
||||
return fmt.Errorf("--memory is required")
|
||||
}
|
||||
body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var m catalog.Model
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/resources", body, &m); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s/%s: %d GB memory, %d cores\n", m.Name, p.Name, p.MinMemoryGB, p.CPUCores)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Running replicas keep their old figures until restarted: llmgw ctl restart --model %s\n", m.Name)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
|
||||
c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
|
||||
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
|
||||
return c
|
||||
}
|
||||
@@ -75,6 +75,7 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
|
||||
f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
|
||||
f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
|
||||
f.IntVar(&req.CPUCores, "cores", 0, "CPU cores a replica computes on; 0 takes half the smallest node it fits")
|
||||
f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
|
||||
return c
|
||||
}
|
||||
@@ -35,6 +35,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -90,6 +91,21 @@ func newServerCmd() *cobra.Command {
|
||||
"or give --endpoint for a router-only run")
|
||||
}
|
||||
|
||||
// Before the router, so its tracer is the configured one.
|
||||
shutdownTracing, err := obs.SetupTracing(cmd.Context(), obs.TracingOptions{
|
||||
Endpoint: cfg.Tracing.Endpoint, SampleRatio: cfg.Tracing.SampleRatio,
|
||||
Service: "llmgw-server", Version: version.Version,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
// Bounded: a collector that is gone must not hold up a shutdown.
|
||||
sctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = shutdownTracing(sctx)
|
||||
}()
|
||||
|
||||
rt, err := router.New(cfg, log)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
@@ -389,17 +389,22 @@ func (in *serverInit) runInitTUI(ctx context.Context, out io.Writer) error {
|
||||
return fmt.Errorf("stopped; run init again to finish what was started")
|
||||
}
|
||||
|
||||
// The whole run is kept, root's only, since it holds the password. On a
|
||||
// failure it is printed too, for the lines the screen had no room for;
|
||||
// otherwise the screen's last frame says it, and only what must outlive
|
||||
// the screen is printed again: the password, and what to do next.
|
||||
// The whole run is kept, root's only, with the password taken out: it is
|
||||
// shown once, on the screen, and a file on disk would outlive that. On a
|
||||
// failure the run is printed too, for the lines the screen had no room
|
||||
// for; otherwise the screen's last frame says it, and only what must
|
||||
// outlive the screen is printed again: the password, and what to do next.
|
||||
logMu.Lock()
|
||||
defer logMu.Unlock()
|
||||
whole := strings.Join(log, "\n") + "\n"
|
||||
kept := whole
|
||||
if fm.password != "" {
|
||||
kept = strings.ReplaceAll(kept, fm.password, "[password shown once, not kept]")
|
||||
}
|
||||
saved := ""
|
||||
if in.stateDir != "" {
|
||||
p := filepath.Join(in.stateDir, "init.log")
|
||||
if err := os.WriteFile(p, []byte(whole), 0o600); err == nil {
|
||||
if err := os.WriteFile(p, []byte(kept), 0o600); err == nil {
|
||||
saved = p
|
||||
}
|
||||
}
|
||||
|
||||
@@ -67,6 +67,10 @@ type Server struct {
|
||||
|
||||
// Monitoring is where the dashboard reads its history from.
|
||||
Monitoring Monitoring `koanf:"monitoring"`
|
||||
// Tracing sends a trace of each inference request to an OpenTelemetry
|
||||
// collector. Off unless an endpoint is set here or in
|
||||
// OTEL_EXPORTER_OTLP_ENDPOINT.
|
||||
Tracing Tracing `koanf:"tracing"`
|
||||
|
||||
// Mirror points the control plane at a model mirror it can administer.
|
||||
// Empty means there is none, and /admin/mirror stays unmounted.
|
||||
@@ -120,6 +124,21 @@ type Monitoring struct {
|
||||
Timeout time.Duration `koanf:"timeout"`
|
||||
}
|
||||
|
||||
// Tracing is where request traces go, over OTLP/HTTP.
|
||||
//
|
||||
// One span per inference request, with admission, scheduling and each attempt
|
||||
// at an engine beneath it, and the trace carried on to the engine in a
|
||||
// traceparent header. The request's X-Request-Id is on the span, so a trace
|
||||
// and `llmgw ctl request` name the same request.
|
||||
type Tracing struct {
|
||||
// Endpoint is the collector, for example http://otel.internal:4318.
|
||||
// Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
|
||||
Endpoint string `koanf:"endpoint"`
|
||||
// SampleRatio is the share of requests traced, 0 to 1, when the caller
|
||||
// did not decide it with a sampled traceparent of its own. Zero takes 1.
|
||||
SampleRatio float64 `koanf:"sample_ratio"`
|
||||
}
|
||||
|
||||
// Mirror is a model mirror this control plane administers over its API.
|
||||
//
|
||||
// The mirror is a separate machine with a large disk, not part of the gateway,
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func TestAClaimedActorIsOneShortName(t *testing.T) {
|
||||
for claimed, want := range map[string]string{
|
||||
"ilker": "ilker",
|
||||
"": "unknown",
|
||||
"ilker\nlevel=ERROR msg=x": "ilkerlevelERRORmsgx",
|
||||
strings.Repeat("a", 500): strings.Repeat("a", 64),
|
||||
"ops@example.org": "ops@example.org",
|
||||
"\x1b[31m": "31m",
|
||||
} {
|
||||
r := httptest.NewRequest("POST", "/", nil)
|
||||
r.Header.Set("X-Llmgw-Actor", claimed)
|
||||
if got := actorOf(r); got != want {
|
||||
t.Errorf("%q: %q, expected %q", claimed, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReservationThatPlacementCannotTrustIsSaid(t *testing.T) {
|
||||
cpu := func(mem int) catalog.Model {
|
||||
return catalog.Model{Name: "m", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: mem}}}
|
||||
}
|
||||
if w := reservationWarnings(cpu(0), 0); len(w) != 1 || !strings.Contains(w[0], "declares no memory") {
|
||||
t.Errorf("no memory declared: %v", w)
|
||||
}
|
||||
if w := reservationWarnings(cpu(6), 8.2); len(w) != 1 || !strings.Contains(w[0], "uses 8.2 GB and reserves 6") {
|
||||
t.Errorf("under-declared: %v", w)
|
||||
}
|
||||
if w := reservationWarnings(cpu(6), 6.4); len(w) != 0 {
|
||||
t.Errorf("within a tenth is mapped weights, not a wrong figure: %v", w)
|
||||
}
|
||||
gpu := catalog.Model{Name: "g", Profiles: []catalog.Profile{{Name: "tp1", Engine: "vllm", GPUClass: "h100-80g"}}}
|
||||
if w := reservationWarnings(gpu, 3); len(w) != 0 {
|
||||
t.Errorf("a GPU model needs no host memory figure: %v", w)
|
||||
}
|
||||
}
|
||||
@@ -15,8 +15,10 @@ import (
|
||||
"io"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
|
||||
@@ -55,6 +57,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /api/v1/models", a.addModel)
|
||||
mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
|
||||
mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
|
||||
mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
|
||||
@@ -90,6 +93,15 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
|
||||
mux.HandleFunc("GET /api/v1/models/{name}/headroom", func(w http.ResponseWriter, r *http.Request) {
|
||||
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
|
||||
v, ok := a.Headroom(r.PathValue("name"), limit)
|
||||
if !ok {
|
||||
http.Error(w, "no such model", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
})
|
||||
}
|
||||
|
||||
// --- responses ---
|
||||
@@ -108,6 +120,8 @@ type NodeView struct {
|
||||
// what decides whether a CPU profile can be placed here, so it belongs
|
||||
// where someone looks when a model will not schedule.
|
||||
MemoryGB int `json:"memory_gb,omitempty"`
|
||||
// CPUCount is the machine's CPUs, 0 when the agent did not report them.
|
||||
CPUCount int `json:"cpu_count,omitempty"`
|
||||
// Engines are the inference engines installed here, as the agent found
|
||||
// them. Absent means the agent is older than the probe and said nothing,
|
||||
// which is not the same as having none — placement treats it that way too,
|
||||
@@ -259,6 +273,15 @@ type ModelView struct {
|
||||
Unplaceable bool `json:"unplaceable,omitempty"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty"`
|
||||
Profiles []string `json:"profiles"`
|
||||
// MemoryGB and CPUCores are what one replica reserves, from its first
|
||||
// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
|
||||
// when none is measured.
|
||||
MemoryGB int `json:"memory_gb,omitempty"`
|
||||
CPUCores int `json:"cpu_cores,omitempty"`
|
||||
MeasuredMemoryGB float64 `json:"measured_memory_gb,omitempty"`
|
||||
// Warnings are what makes its reservation wrong: nothing declared where
|
||||
// placement needs a figure, or less declared than it uses.
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
// ReplicaView is a replica as `ctl replicas` shows it.
|
||||
@@ -335,6 +358,22 @@ type ExplainView struct {
|
||||
Rejections []RejectionView `json:"rejections,omitempty"`
|
||||
}
|
||||
|
||||
// HeadroomView is how many more replicas of a model fit now, and where.
|
||||
type HeadroomView struct {
|
||||
Model string `json:"model"`
|
||||
// Current is the replicas holding capacity: starting or ready.
|
||||
Current int `json:"current"`
|
||||
// More is how many more placement would find room for now.
|
||||
More int `json:"more"`
|
||||
// Nodes is where each of those would go.
|
||||
Nodes [][]string `json:"nodes,omitempty"`
|
||||
// Limited is set when More stopped at the limit asked rather than at the
|
||||
// cluster's edge: there may be room for more.
|
||||
Limited bool `json:"limited,omitempty"`
|
||||
// Blocked says why one more does not fit, a line per node considered.
|
||||
Blocked []string `json:"blocked,omitempty"`
|
||||
}
|
||||
|
||||
// RejectionView is one rejected candidate.
|
||||
type RejectionView struct {
|
||||
Profile string `json:"profile"`
|
||||
@@ -369,8 +408,8 @@ func (a *API) Nodes() []NodeView {
|
||||
ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr,
|
||||
State: n.State.String(), Labels: n.Labels,
|
||||
GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
|
||||
MemoryGB: n.MemoryTotalMB / 1024,
|
||||
Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
|
||||
MemoryGB: n.MemoryTotalMB / 1024, CPUCount: n.CPUCount,
|
||||
Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
|
||||
Replicas: replicasOn[n.ID],
|
||||
}
|
||||
if free, total := n.LocalDisk(); total > 0 {
|
||||
@@ -470,6 +509,15 @@ func (a *API) Models() []ModelView {
|
||||
for _, h := range a.ctl.Reconciler().ModelHealth() {
|
||||
health[h.Model] = h
|
||||
}
|
||||
// The most one replica of each model uses, summed over its shards.
|
||||
measured := map[string]int64{}
|
||||
for _, rep := range a.Replicas("") {
|
||||
var b int64
|
||||
for _, sh := range rep.Shards {
|
||||
b += sh.MemoryBytes
|
||||
}
|
||||
measured[rep.Model] = max(measured[rep.Model], b)
|
||||
}
|
||||
out := make([]ModelView, 0)
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
h := health[m.Name]
|
||||
@@ -484,6 +532,11 @@ func (a *API) Models() []ModelView {
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
}
|
||||
if len(m.Profiles) > 0 {
|
||||
v.MemoryGB, v.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
|
||||
}
|
||||
v.MeasuredMemoryGB = float64(measured[m.Name]) / (1 << 30)
|
||||
v.Warnings = reservationWarnings(m, v.MeasuredMemoryGB)
|
||||
out = append(out, v)
|
||||
}
|
||||
return out
|
||||
@@ -554,6 +607,15 @@ type Models interface {
|
||||
Add(ctx context.Context, m catalog.Model, by string) error
|
||||
Remove(ctx context.Context, name string) error
|
||||
Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// ResourcesRequest sets what a model's profiles reserve of a node.
|
||||
type ResourcesRequest struct {
|
||||
// Profile is one profile's name; empty is every profile.
|
||||
Profile string `json:"profile,omitempty"`
|
||||
MemoryGB int `json:"memory_gb"`
|
||||
CPUCores int `json:"cpu_cores"`
|
||||
}
|
||||
|
||||
// Usage reports what each group of the tenancy tree spent. The store
|
||||
@@ -685,7 +747,7 @@ func (a *API) putCapacity(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
if err := a.capacity.PutCapacity(r.Context(), store.StoredCapacity{
|
||||
Model: c.Model, Profile: c.Profile, Measurement: body,
|
||||
MeasuredAt: c.MeasuredAt, MeasuredBy: r.Header.Get("X-Llmgw-Actor"),
|
||||
MeasuredAt: c.MeasuredAt, MeasuredBy: actorOf(r),
|
||||
}); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
@@ -736,7 +798,7 @@ func (a *API) addModel(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "the model definition is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := a.models.Add(r.Context(), m, r.Header.Get("X-Llmgw-Actor")); err != nil {
|
||||
if err := a.models.Add(r.Context(), m, actorOf(r)); err != nil {
|
||||
// Every failure here is the definition being wrong, which is the
|
||||
// caller's to fix — a collision, a missing fallback, a profile that
|
||||
// cannot be placed. None of them are server faults.
|
||||
@@ -785,7 +847,7 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
m, err := a.models.Scale(r.Context(), name,
|
||||
catalog.Scaling{Min: req.Min, Max: req.Max, IdleTimeout: req.IdleTimeout},
|
||||
r.Header.Get("X-Llmgw-Actor"))
|
||||
actorOf(r))
|
||||
if err != nil {
|
||||
// Like addModel: every refusal here is the request being wrong — a
|
||||
// model declared in configuration, one that was never added, a max
|
||||
@@ -799,6 +861,26 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
|
||||
})
|
||||
}
|
||||
|
||||
// setResources sets a model's reserved memory and cores. Placed replicas keep
|
||||
// what they were started with until they are restarted.
|
||||
func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req ResourcesRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
m, err := a.models.Resources(r.Context(), r.PathValue("name"), req.Profile, req.MemoryGB, req.CPUCores, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, m)
|
||||
}
|
||||
|
||||
func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
@@ -835,6 +917,20 @@ func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
|
||||
// It does not check that the model exists; the handler above does that so it
|
||||
// can answer 404, while the dashboard only ever asks about models it just read
|
||||
// out of the catalog.
|
||||
// Headroom tries placing up to limit more replicas of a model, on a copy of
|
||||
// the inventory, and says how many fit. A limit of zero or less takes 16.
|
||||
func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
|
||||
if limit <= 0 || limit > 64 {
|
||||
limit = 16
|
||||
}
|
||||
h, ok := a.ctl.Reconciler().Headroom(model, limit)
|
||||
if !ok {
|
||||
return HeadroomView{}, false
|
||||
}
|
||||
return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
|
||||
Limited: h.Limited, Blocked: h.Blocked}, true
|
||||
}
|
||||
|
||||
func (a *API) Explain(model string) ExplainView {
|
||||
v := ExplainView{Model: model}
|
||||
for _, rep := range a.ctl.Reconciler().Replicas() {
|
||||
@@ -1224,7 +1320,7 @@ func (a *API) serveModel(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
req.By = r.Header.Get("X-Llmgw-Actor")
|
||||
req.By = actorOf(r)
|
||||
job, err := a.serve.Start(req)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
@@ -1245,3 +1341,61 @@ func (a *API) job(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
writeJSON(w, http.StatusOK, job)
|
||||
}
|
||||
|
||||
// actorOf is who a change is recorded as made by: what the caller says, since
|
||||
// the admin API answers only this machine and trusts whoever reaches it. Kept
|
||||
// to a short printable name, so a claim cannot pass for more than one — no
|
||||
// line breaks to forge a second log line, no length to bury the rest.
|
||||
func actorOf(r *http.Request) string {
|
||||
claimed := r.Header.Get("X-Llmgw-Actor")
|
||||
var b strings.Builder
|
||||
for _, c := range claimed {
|
||||
if b.Len() >= 64 {
|
||||
break
|
||||
}
|
||||
if unicode.IsLetter(c) || unicode.IsDigit(c) || strings.ContainsRune("._-@", c) {
|
||||
b.WriteRune(c)
|
||||
}
|
||||
}
|
||||
if b.Len() == 0 {
|
||||
return "unknown"
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// NodeCPUs is the CPU counts of ready nodes with at least this much memory:
|
||||
// where a replica needing it could go.
|
||||
func (a *API) NodeCPUs(minMemoryGB int) []int {
|
||||
var out []int
|
||||
for _, n := range a.ctl.Inventory().Nodes() {
|
||||
if n.State.String() != "ready" || n.CPUCount == 0 {
|
||||
continue
|
||||
}
|
||||
if n.MemoryTotalMB > 0 && n.MemoryTotalMB/1024 < minMemoryGB {
|
||||
continue
|
||||
}
|
||||
out = append(out, n.CPUCount)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// reservationWarnings says what is wrong with what a model reserves.
|
||||
//
|
||||
// Resident memory includes the weights mapped from disk, which the kernel can
|
||||
// take back, so it overstates what a model needs by the part of the weights
|
||||
// not in use. A tenth over is let pass for that; more is a figure too low for
|
||||
// placement to be trusted with.
|
||||
func reservationWarnings(m catalog.Model, measuredGB float64) []string {
|
||||
var out []string
|
||||
for _, p := range m.Profiles {
|
||||
if p.CPUOnly() && p.MinMemoryGB <= 0 {
|
||||
out = append(out, fmt.Sprintf("profile %s declares no memory: placement cannot tell when a node is full "+
|
||||
"(llmgw ctl model resources %s --memory GB)", p.Name, m.Name))
|
||||
}
|
||||
}
|
||||
if len(m.Profiles) == 1 && m.Profiles[0].MinMemoryGB > 0 && measuredGB > float64(m.Profiles[0].MinMemoryGB)*1.1 {
|
||||
out = append(out, fmt.Sprintf("uses %.1f GB and reserves %d: raise it, or nodes will be filled past their memory",
|
||||
measuredGB, m.Profiles[0].MinMemoryGB))
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -258,6 +258,13 @@ type fakeModels struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeModels) Resources(_ context.Context, name, profile string, mem, cores int, _ string) (catalog.Model, error) {
|
||||
if f.err != nil {
|
||||
return catalog.Model{}, f.err
|
||||
}
|
||||
return catalog.Model{Name: name, Profiles: []catalog.Profile{{Name: "p", MinMemoryGB: mem, CPUCores: cores}}}, nil
|
||||
}
|
||||
|
||||
func (f *fakeModels) Scale(_ context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error) {
|
||||
if f.err != nil {
|
||||
return catalog.Model{}, f.err
|
||||
|
||||
@@ -4,6 +4,7 @@ package agentsvc
|
||||
import (
|
||||
"fmt"
|
||||
"maps"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
@@ -27,6 +28,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
|
||||
EngineVersions: r.GetEngineVersions(),
|
||||
EnginesProbed: r.GetEnginesProbed(),
|
||||
MemoryTotalMB: int(r.GetMemoryTotalMb()),
|
||||
CPUCount: int(r.GetCpuCount()),
|
||||
}
|
||||
if h := r.GetHost(); h != nil {
|
||||
n.Host = discovery.Host{
|
||||
@@ -90,6 +92,7 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
|
||||
NodeId: nodeID, Hostname: hostname, MgmtAddr: mgmtAddr,
|
||||
Labels: labels, AgentVersion: agentVersion,
|
||||
MemoryTotalMb: int64(inv.MemoryTotalMB),
|
||||
CpuCount: int32(runtime.NumCPU()),
|
||||
}
|
||||
for _, w := range inv.Weights {
|
||||
r.Weights = append(r.Weights, &llmgwv1.LocalWeights{
|
||||
@@ -192,6 +195,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
|
||||
Fabric: cmd.Fabric,
|
||||
ProfileFp: cmd.ProfileFP,
|
||||
ApiKey: cmd.Spec.APIKey,
|
||||
CpuThreads: int32(cmd.Spec.Threads),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -213,6 +217,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
|
||||
Fabric: m.GetFabric(),
|
||||
ProfileFP: m.GetProfileFp(),
|
||||
APIKey: m.GetApiKey(),
|
||||
Threads: int(m.GetCpuThreads()),
|
||||
Profile: engines.Profile{
|
||||
Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
|
||||
TP: int(p.GetTp()), PP: int(p.GetPp()),
|
||||
|
||||
@@ -345,8 +345,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
problems := hb.GetNewProblems()
|
||||
storage := hb.GetStorage()
|
||||
memMB := int(hb.GetMemoryTotalMb())
|
||||
cpus := int(hb.GetCpuCount())
|
||||
if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
|
||||
!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
|
||||
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
|
||||
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
|
||||
!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
|
||||
return
|
||||
@@ -406,6 +407,12 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
"from_mb", n.MemoryTotalMB, "to_mb", memMB)
|
||||
n.MemoryTotalMB = memMB
|
||||
}
|
||||
if cpus > 0 && cpus != n.CPUCount {
|
||||
if n.CPUCount > 0 {
|
||||
s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
|
||||
}
|
||||
n.CPUCount = cpus
|
||||
}
|
||||
if hb.GetEnginesChanged() {
|
||||
vers := maps.Clone(hb.GetEngineVersions())
|
||||
if !maps.Equal(vers, n.EngineVersions) {
|
||||
|
||||
@@ -74,6 +74,17 @@ type Profile struct {
|
||||
// the control plane.
|
||||
MinMemoryGB int `json:"min_memory_gb,omitempty" koanf:"min_memory_gb"`
|
||||
|
||||
// CPUCores is the cores a replica of a profile with no GPU is given:
|
||||
// reserved out of its node's CPUs by placement, as memory is, and what
|
||||
// its engine computes on (llama.cpp's --threads).
|
||||
//
|
||||
// Reserved because an engine takes every core it is told about, and two
|
||||
// on one node that each think they have all of them spin against each
|
||||
// other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
|
||||
// to under one each. Zero reserves nothing and leaves the thread count to
|
||||
// the node, which is every profile from before this.
|
||||
CPUCores int `json:"cpu_cores,omitempty" koanf:"cpu_cores"`
|
||||
|
||||
// MinDiskGB is the free space a node needs before it can hold these
|
||||
// weights.
|
||||
//
|
||||
@@ -380,6 +391,7 @@ func (m Model) PlacementProfiles() []placement.Profile {
|
||||
NodeSelector: p.NodeSelector,
|
||||
MinMemoryGB: p.MinMemoryGB,
|
||||
MinDiskGB: p.MinDiskGB,
|
||||
CPUCores: p.CPUCores,
|
||||
// The model's source, so placement can prefer nodes that already
|
||||
// hold the weights rather than one that must first move them.
|
||||
Source: m.Source,
|
||||
@@ -475,3 +487,19 @@ func (p Profile) Fingerprint() string {
|
||||
sum := sha256.Sum256(b)
|
||||
return hex.EncodeToString(sum[:8])
|
||||
}
|
||||
|
||||
// SuggestCores is the cores to give a replica with no GPU on nodes of these
|
||||
// CPU counts: half of the smallest, so two replicas share any of them without
|
||||
// competing. Zero when no node says, which reserves nothing.
|
||||
func SuggestCores(cpuCounts []int) int {
|
||||
smallest := 0
|
||||
for _, c := range cpuCounts {
|
||||
if c > 0 && (smallest == 0 || c < smallest) {
|
||||
smallest = c
|
||||
}
|
||||
}
|
||||
if smallest == 0 {
|
||||
return 0
|
||||
}
|
||||
return max(smallest/2, 1)
|
||||
}
|
||||
@@ -79,6 +79,9 @@ type Node struct {
|
||||
// it, which is treated as no constraint — the same as every node before
|
||||
// this field existed.
|
||||
MemoryTotalMB int `json:"memory_total_mb,omitempty"`
|
||||
// CPUCount is the CPUs the agent may run on. Zero is an agent too old to
|
||||
// say, which, like memory, constrains nothing.
|
||||
CPUCount int `json:"cpu_count,omitempty"`
|
||||
|
||||
// Weights are the model files this node can see, however they got there.
|
||||
// Empty means the node was not asked to look, which is not the same as
|
||||
@@ -373,6 +376,9 @@ type Allocation struct {
|
||||
// Zero for a profile that declared nothing, which constrains nothing and
|
||||
// is counted as nothing.
|
||||
MemoryGB int
|
||||
// CPUCores is the cores this shard was given to compute on, for a shard
|
||||
// with no GPU; zero for one that shares whatever is free.
|
||||
CPUCores int
|
||||
}
|
||||
|
||||
// Store holds the cluster inventory. It is safe for concurrent use.
|
||||
@@ -392,6 +398,23 @@ func NewStore() *Store {
|
||||
return &Store{nodes: map[string]*Node{}, allocs: map[string][]Allocation{}}
|
||||
}
|
||||
|
||||
// Clone is a copy to try decisions on without making them: allocations made
|
||||
// on it are its own. Nodes are copied by value; the slices inside them are
|
||||
// shared, and a clone must not write through them.
|
||||
func (s *Store) Clone() *Store {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
c := NewStore()
|
||||
for id, n := range s.nodes {
|
||||
cp := *n
|
||||
c.nodes[id] = &cp
|
||||
}
|
||||
for id, list := range s.allocs {
|
||||
c.allocs[id] = slices.Clone(list)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Upsert records a node's registration, preserving operator-set state such as
|
||||
// a cordon: an agent reconnecting must not silently undo an operator's decision.
|
||||
// RecordSMChange keeps a port's change of master subnet manager, the last
|
||||
|
||||
@@ -91,6 +91,9 @@ type Request struct {
|
||||
// use. They default to 16384 and 4096, what four parallel slots divide.
|
||||
CtxPool int `json:"ctx_pool,omitempty"`
|
||||
ContextLength int `json:"context_length,omitempty"`
|
||||
// CPUCores is what a replica computes on; zero takes half the smallest
|
||||
// node it fits on.
|
||||
CPUCores int `json:"cpu_cores,omitempty"`
|
||||
// By is who asked, for the catalog's record.
|
||||
By string `json:"-"`
|
||||
}
|
||||
@@ -278,13 +281,20 @@ func (s *Server) run(j *Job) {
|
||||
}
|
||||
s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
|
||||
|
||||
cores := req.CPUCores
|
||||
if cores <= 0 {
|
||||
if nc, ok := s.deps.Fleet.(interface{ NodeCPUs(int) []int }); ok {
|
||||
cores = catalog.SuggestCores(nc.NodeCPUs(memGB))
|
||||
}
|
||||
}
|
||||
|
||||
// 4. The catalog.
|
||||
m := catalog.Model{
|
||||
Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
|
||||
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
|
||||
Profiles: []catalog.Profile{{
|
||||
Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
|
||||
Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB,
|
||||
Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB, CPUCores: cores,
|
||||
}},
|
||||
Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ package modelsvc
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sync"
|
||||
@@ -104,6 +105,16 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
|
||||
if s.Declared(m.Name) {
|
||||
return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
|
||||
}
|
||||
// A model with no GPU is all host memory, and one that declares none
|
||||
// fits everywhere as far as placement can tell — however many of it
|
||||
// there already are. Required of what is added now; models stored before
|
||||
// this still load, and say what they are missing.
|
||||
for _, p := range m.Profiles {
|
||||
if p.CPUOnly() && p.MinMemoryGB <= 0 {
|
||||
return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
|
||||
"placement could not tell when a node is full", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
if s.check != nil {
|
||||
if err := s.check(ctx, m); err != nil {
|
||||
return err
|
||||
@@ -189,36 +200,72 @@ func (s *Service) Remove(ctx context.Context, name string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Scale changes how many replicas a model runs, without a restart.
|
||||
//
|
||||
// Only the scaling block moves. Everything else about the model — its source,
|
||||
// its profiles, the engine arguments somebody tuned — is read back from what
|
||||
// was stored and written out again untouched, so changing a number cannot
|
||||
// quietly become a redefinition.
|
||||
//
|
||||
// A model declared in configuration is refused, for the same reason Remove
|
||||
// refuses one: the YAML is read on every start, so the new number would hold
|
||||
// until the next restart and then silently revert. A cluster that disagrees
|
||||
// with its own config file is worse than an error message saying where to edit.
|
||||
// Scale changes how many replicas a model added at runtime runs.
|
||||
func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
|
||||
m.Scaling = sc
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model scaled", "model", name,
|
||||
"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// Resources sets what a model's profile reserves of a node: host memory, and
|
||||
// the CPU cores a replica without a GPU computes on. An empty profile name
|
||||
// sets every profile.
|
||||
func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error) {
|
||||
if memoryGB < 0 || cpuCores < 0 {
|
||||
return catalog.Model{}, errors.New("memory and cores cannot be negative")
|
||||
}
|
||||
final, err := s.update(ctx, name, by, "change its resources", func(m *catalog.Model) error {
|
||||
found := false
|
||||
for i := range m.Profiles {
|
||||
if profile != "" && m.Profiles[i].Name != profile {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
m.Profiles[i].MinMemoryGB = memoryGB
|
||||
m.Profiles[i].CPUCores = cpuCores
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("model %q has no profile %q", name, profile)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model resources set", "model", name, "profile", profile,
|
||||
"memory_gb", memoryGB, "cpu_cores", cpuCores, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// update changes one model added at runtime and keeps the change.
|
||||
//
|
||||
// Built before anything is written, like Add: catalog.New is what applies the
|
||||
// defaults and refuses what does not validate, so the model that comes back
|
||||
// is the one that will run rather than the one that was asked for.
|
||||
func (s *Service) update(ctx context.Context, name, by, what string, change func(*catalog.Model) error) (catalog.Model, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
if s.Declared(name) {
|
||||
return catalog.Model{}, fmt.Errorf(
|
||||
"model %q is declared in configuration; change its scaling there, or it reverts on the next restart", name)
|
||||
"model %q is declared in configuration; %s there, or it reverts on the next restart", name, what)
|
||||
}
|
||||
|
||||
stored, err := s.storedModels(ctx)
|
||||
if err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
|
||||
next := append([]catalog.Model{}, s.declared...)
|
||||
found := false
|
||||
for _, e := range stored {
|
||||
if e.Name == name {
|
||||
e.Scaling = sc
|
||||
if err := change(&e); err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
found = true
|
||||
}
|
||||
next = append(next, e)
|
||||
@@ -226,20 +273,14 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
|
||||
if !found {
|
||||
return catalog.Model{}, fmt.Errorf("model %q was not added at runtime", name)
|
||||
}
|
||||
|
||||
// Built before anything is written, like Add: catalog.New is what applies
|
||||
// the defaults (max follows min when it is left out) and what refuses a
|
||||
// max below a min, so the model that comes back is the one that will run
|
||||
// rather than the one that was asked for.
|
||||
built, err := catalog.New(next)
|
||||
if err != nil {
|
||||
return catalog.Model{}, fmt.Errorf("scaling %q that way would make the catalog invalid: %w", name, err)
|
||||
return catalog.Model{}, fmt.Errorf("that would make the catalog invalid: %w", err)
|
||||
}
|
||||
final, ok := built.Resolve(name)
|
||||
if !ok {
|
||||
return catalog.Model{}, fmt.Errorf("model %q vanished from the catalog it was just built into", name)
|
||||
}
|
||||
|
||||
def, err := json.Marshal(final)
|
||||
if err != nil {
|
||||
return catalog.Model{}, err
|
||||
@@ -249,10 +290,7 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
|
||||
}); err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
|
||||
s.apply.ReplaceCatalog(ctx, built)
|
||||
s.log.Info("model scaled", "model", name,
|
||||
"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
|
||||
return final, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -48,7 +48,7 @@ func model(name string) catalog.Model {
|
||||
return catalog.Model{
|
||||
Name: name, Source: "/models/" + name,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1}},
|
||||
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1, MinMemoryGB: 2}},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -337,3 +337,43 @@ func TestMinZeroWithAMaxIsOnDemand(t *testing.T) {
|
||||
t.Errorf("the idle timeout was lost; got %v", got.Scaling.IdleTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACPUModelAddedNowMustSayHowMuchMemoryItNeeds(t *testing.T) {
|
||||
s, st, _ := newService(t)
|
||||
m := model("bare")
|
||||
m.Profiles[0].MinMemoryGB = 0
|
||||
err := s.Add(t.Context(), m, "tester")
|
||||
if err == nil || !strings.Contains(err.Error(), "declares no memory") {
|
||||
t.Fatalf("a CPU model with no memory was added: %v", err)
|
||||
}
|
||||
if len(st.rows) != 0 {
|
||||
t.Error("it was stored anyway")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcesAreSetOnAStoredModelAndSurviveARestart(t *testing.T) {
|
||||
s, _, _ := newService(t)
|
||||
if err := s.Add(t.Context(), model("m"), "tester"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := s.Resources(t.Context(), "m", "", 6, 3, "tester")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if p := got.Profiles[0]; p.MinMemoryGB != 6 || p.CPUCores != 3 {
|
||||
t.Errorf("profile %+v", p)
|
||||
}
|
||||
if _, err := s.Resources(t.Context(), "m", "nope", 6, 3, "tester"); err == nil {
|
||||
t.Error("a profile the model does not have was accepted")
|
||||
}
|
||||
if _, err := s.Resources(t.Context(), "m", "", -1, 0, "tester"); err == nil {
|
||||
t.Error("negative memory was accepted")
|
||||
}
|
||||
cat, err := s.Load(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m, _ := cat.Resolve("m"); m.Profiles[0].CPUCores != 3 {
|
||||
t.Error("the cores were lost on reload")
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package placement
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
@@ -100,3 +101,38 @@ func TestCPUProfileRespectsAvoid(t *testing.T) {
|
||||
t.Errorf("placed on %q despite being told to avoid it", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCPUCoresAreReservedAsMemoryIs(t *testing.T) {
|
||||
n := node("n1", "", 0)
|
||||
n.CPUCount = 8
|
||||
st := store(n)
|
||||
s := New(st)
|
||||
p := cpuProfile("cpu")
|
||||
p.CPUCores = 3
|
||||
|
||||
for i := range 2 {
|
||||
res := s.Place([]Profile{p}, nil)
|
||||
if res.Placement == nil {
|
||||
t.Fatalf("replica %d did not fit in 8 cores:\n%s", i+1, res.Explain())
|
||||
}
|
||||
st.Allocate("n1", inventory.Allocation{ShardID: string(rune('a' + i)), CPUCores: 3})
|
||||
}
|
||||
// Six of eight given away: a third would need three.
|
||||
res := s.Place([]Profile{p}, nil)
|
||||
if res.Placement != nil {
|
||||
t.Fatal("a third replica was given cores the node does not have")
|
||||
}
|
||||
if got := res.Explain(); !strings.Contains(got, "has 8 CPUs with 2 free, needs 3") {
|
||||
t.Errorf("the refusal should say how many cores are left:\n%s", got)
|
||||
}
|
||||
|
||||
// Unknown counts constrain nothing: an older agent, or a profile that
|
||||
// reserves no cores.
|
||||
old := node("n2", "", 0)
|
||||
if res := New(store(old)).Place([]Profile{p}, nil); res.Placement == nil {
|
||||
t.Error("a node that does not report its CPUs was excluded")
|
||||
}
|
||||
if res := s.Place([]Profile{cpuProfile("cpu")}, nil); res.Placement == nil {
|
||||
t.Error("a profile reserving no cores was refused")
|
||||
}
|
||||
}
|
||||
@@ -39,6 +39,9 @@ type Profile struct {
|
||||
// MinDiskGB is the free space needed to hold the weights, checked only on
|
||||
// nodes that do not already have them. Zero is unconstrained.
|
||||
MinDiskGB int
|
||||
// CPUCores is the cores a profile with no GPU reserves. Zero is
|
||||
// unconstrained.
|
||||
CPUCores int
|
||||
// Source is the model's weights, as the engine would be given it. Used to
|
||||
// prefer nodes that already hold them; empty asks for no preference.
|
||||
Source string
|
||||
@@ -582,6 +585,18 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
}
|
||||
}
|
||||
|
||||
// Cores, for a replica that computes on them: what is left after the
|
||||
// cores already given to the node's other replicas. Same exemptions
|
||||
// as memory: zero declares nothing, and a node that does not report
|
||||
// its CPUs is an older agent, not one with none.
|
||||
if p.CPUOnly() && p.CPUCores > 0 && n.CPUCount > 0 {
|
||||
free := n.CPUCount - s.claimedCores(n.ID)
|
||||
if free < p.CPUCores {
|
||||
reject(fmt.Sprintf("has %d CPUs with %d free, needs %d", n.CPUCount, free, p.CPUCores))
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if p.CPUOnly() {
|
||||
// Every ready node can run it; there is no card to be free.
|
||||
out = append(out, candidate{node: n})
|
||||
@@ -723,6 +738,15 @@ func (s *Scheduler) claimedMemory(nodeID string) int {
|
||||
return total
|
||||
}
|
||||
|
||||
// claimedCores is the CPU cores already given to replicas on a node.
|
||||
func (s *Scheduler) claimedCores(nodeID string) int {
|
||||
total := 0
|
||||
for _, a := range s.inv.Allocations(nodeID) {
|
||||
total += a.CPUCores
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// installing says why an engine is missing, when the node has said.
|
||||
func installing(n inventory.Node, engine string) string {
|
||||
for _, p := range n.Provision {
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// Adopt rebuilds replica state from shards an agent reports it is already
|
||||
// running (plan §6.5).
|
||||
//
|
||||
// This is what makes a control plane restart survivable. Without it the
|
||||
// reconciler sees an empty cluster, places duplicates, and the engines already
|
||||
// running keep holding GPUs the inventory believes are free — so the duplicates
|
||||
// fail to allocate, with an error naming the new process rather than the old.
|
||||
//
|
||||
// Shards whose model or profile is no longer in the catalog are returned for
|
||||
// the caller to stop: nothing will ever claim them.
|
||||
func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
|
||||
defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
|
||||
|
||||
byReplica := map[string][]AdoptedShard{}
|
||||
for _, s := range shards {
|
||||
byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
for replicaID, group := range byReplica {
|
||||
if existing, known := r.replicas[replicaID]; known {
|
||||
// The other half of a replica we are still assembling. Merge rather
|
||||
// than choose: these shards belong to the same replica as the ones
|
||||
// already adopted, they are simply on a different node whose agent
|
||||
// registered a moment later. Treating them as a competing claim is
|
||||
// what tore multi-node replicas apart on every control plane
|
||||
// restart — one half adopted and failed for being short, the other
|
||||
// stopped as unclaimed when it finally arrived.
|
||||
if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
|
||||
r.mergeAdopted(existing, group)
|
||||
continue
|
||||
}
|
||||
// Usually this is an agent reconnecting rather than a control plane
|
||||
// restart, and its status reports keep the state current.
|
||||
//
|
||||
// Not always. Replica ids are derived from the model name and a
|
||||
// counter, so a control plane that restarts and places before the
|
||||
// agents reconnect produces a *different* replica under the same
|
||||
// id, on a freshly allocated port. Skipping the adoption then
|
||||
// leaves the cluster pointing at a port nothing listens on, while
|
||||
// the engine that is actually serving is treated as a stray.
|
||||
//
|
||||
// The fleet is authoritative about what is running (plan §6.5), so
|
||||
// what we merely placed loses to what an agent reports — unless our
|
||||
// copy has a shard that has actually come up, in which case two
|
||||
// things really are running and the one we did not ask for is the
|
||||
// stray.
|
||||
if existing.hasLiveShard() {
|
||||
// Only what we do not recognise. A reconnecting agent reports
|
||||
// the shards we already know, and disowning those would tell it
|
||||
// to stop the very replica that is serving.
|
||||
known := map[string]struct{}{}
|
||||
for _, sh := range existing.Shards {
|
||||
known[sh.ID] = struct{}{}
|
||||
}
|
||||
var strays []AdoptedShard
|
||||
for _, s := range group {
|
||||
if _, ok := known[s.ShardID]; !ok {
|
||||
strays = append(strays, s)
|
||||
}
|
||||
}
|
||||
unclaimed = append(unclaimed, strays...)
|
||||
r.reservePorts(strays)
|
||||
continue
|
||||
}
|
||||
r.log.Warn("a running replica reported by the fleet replaces one we placed",
|
||||
"replica", replicaID, "placed_endpoint", existing.Endpoint)
|
||||
for _, s := range existing.Shards {
|
||||
r.inv.Release(s.ID)
|
||||
}
|
||||
delete(r.replicas, replicaID)
|
||||
}
|
||||
|
||||
first := group[0]
|
||||
m, ok := r.cat.Load().Resolve(first.Model)
|
||||
if !ok {
|
||||
unclaimed = append(unclaimed, group...)
|
||||
r.reservePorts(group)
|
||||
continue
|
||||
}
|
||||
prof, ok := m.ProfileByName(first.Profile)
|
||||
if !ok {
|
||||
unclaimed = append(unclaimed, group...)
|
||||
r.reservePorts(group)
|
||||
continue
|
||||
}
|
||||
|
||||
rep := &Replica{
|
||||
ID: replicaID, Model: m.Name, Profile: prof.Name,
|
||||
Fabric: first.Fabric, Pool: first.Pool,
|
||||
// From the agent, which was told it when the shard started. Empty
|
||||
// from an agent too old to report it, which reads as unknown.
|
||||
ProfileFP: first.ProfileFP,
|
||||
Weight: prof.Weight, CreatedAt: r.now(),
|
||||
Endpoint: first.Endpoint,
|
||||
// From the profile, not from what was reported: the point is to
|
||||
// notice when what was reported is short.
|
||||
WantShards: max(prof.Nodes, 1),
|
||||
AdoptedAt: r.now(),
|
||||
}
|
||||
r.mergeAdopted(rep, group)
|
||||
r.replicas[replicaID] = rep
|
||||
|
||||
}
|
||||
return unclaimed
|
||||
}
|
||||
|
||||
// mergeAdopted folds shards an agent reports into a replica being reassembled.
|
||||
//
|
||||
// Called both for the first group, which creates the replica, and for every
|
||||
// later group belonging to it, so a replica spanning N nodes is built up over N
|
||||
// registrations. Shard ids already present are skipped: an agent that
|
||||
// reconnects repeats what it is running, and counting those again would carry a
|
||||
// replica past its shard count without a second node ever appearing.
|
||||
func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
|
||||
have := make(map[string]struct{}, len(rep.Shards))
|
||||
for _, sh := range rep.Shards {
|
||||
have[sh.ID] = struct{}{}
|
||||
}
|
||||
for _, s := range group {
|
||||
if _, dup := have[s.ShardID]; dup {
|
||||
continue
|
||||
}
|
||||
have[s.ShardID] = struct{}{}
|
||||
rep.Shards = append(rep.Shards, Shard{
|
||||
ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
|
||||
GPUIndices: s.GPUIndices, HCAs: s.HCAs,
|
||||
HTTPPort: s.HTTPPort, State: s.State,
|
||||
})
|
||||
// Reclaim the GPUs and the memory, or the next placement hands them
|
||||
// out twice. The memory comes from the catalog rather than the agent:
|
||||
// what a profile asked for is not something a running process reports.
|
||||
r.inv.Allocate(s.NodeID, inventory.Allocation{
|
||||
ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
|
||||
MemoryGB: r.profileMemory(rep.Model, rep.Profile),
|
||||
CPUCores: r.adoptedCores(rep.Model, rep.Profile),
|
||||
})
|
||||
// The endpoint comes from the leader, which may be in any of the
|
||||
// groups — the rank 0 node is not necessarily the first to register.
|
||||
if s.Rank == 0 && s.Endpoint != "" {
|
||||
rep.Endpoint = s.Endpoint
|
||||
}
|
||||
}
|
||||
slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
|
||||
rep.deriveState(r.now())
|
||||
|
||||
r.log.Info("adopted replica from the fleet",
|
||||
"replica", rep.ID, "model", rep.Model,
|
||||
"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
|
||||
}
|
||||
|
||||
// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
|
||||
// agent, having just told us everything it is running, did not mention.
|
||||
//
|
||||
// The mirror image of adoption, and the half that was missing. Adoption handles
|
||||
// "the agent has something we do not know about"; this handles "we think the
|
||||
// agent has something it does not". Both follow from the same rule: the fleet is
|
||||
// authoritative about what is running (plan §6.5).
|
||||
//
|
||||
// Without it, an engine that dies while its agent survives leaves the control
|
||||
// plane believing a replica is Ready forever. Observed on a real cluster: the
|
||||
// engine was killed, the agent restarted and registered with nothing running,
|
||||
// and the replica stayed Ready pointing at a dead port for as long as anyone
|
||||
// watched. Traffic survived — the breaker took the endpoint out — but the
|
||||
// cluster ran a third short with no sign of it.
|
||||
//
|
||||
// Safe against the obvious race. A registration follows a fresh connection, so
|
||||
// anything we dispatched over a previous one is not running by definition;
|
||||
// there is no window where a shard is legitimately started but unreported.
|
||||
func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
|
||||
if nodeID == "" {
|
||||
return
|
||||
}
|
||||
reported := make(map[string]struct{}, len(running))
|
||||
for _, s := range running {
|
||||
reported[s.ShardID] = struct{}{}
|
||||
}
|
||||
|
||||
var lost []string
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State == Stopped {
|
||||
continue
|
||||
}
|
||||
changed := false
|
||||
for i := range rep.Shards {
|
||||
sh := &rep.Shards[i]
|
||||
if sh.NodeID != nodeID || sh.State.Terminal() {
|
||||
continue
|
||||
}
|
||||
if _, ok := reported[sh.ID]; ok {
|
||||
continue
|
||||
}
|
||||
sh.State = supervisor.Failed
|
||||
sh.Err = "the agent reports it is not running"
|
||||
r.inv.Release(sh.ID)
|
||||
changed = true
|
||||
lost = append(lost, sh.ID)
|
||||
}
|
||||
if changed {
|
||||
rep.deriveState(r.now())
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
for _, id := range lost {
|
||||
r.log.Warn("a shard we believed was running is not; the agent does not have it",
|
||||
"node", nodeID, "shard", id)
|
||||
}
|
||||
}
|
||||
|
||||
// reservePorts records ports held by shards nothing owns.
|
||||
//
|
||||
// They are being stopped, but stopping is not instant and the engine keeps its
|
||||
// socket until it exits. Handing the port to a new shard in the meantime makes
|
||||
// that shard fail to bind, which fails its replica, which places another one on
|
||||
// the same port — a loop that does not end on its own.
|
||||
//
|
||||
// Called with the lock held.
|
||||
func (r *Reconciler) reservePorts(shards []AdoptedShard) {
|
||||
for _, s := range shards {
|
||||
if s.HTTPPort > 0 {
|
||||
r.reservedPorts[s.HTTPPort] = struct{}{}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// AdoptedShard is one running shard an agent reported at registration.
|
||||
type AdoptedShard struct {
|
||||
NodeID string
|
||||
ReplicaID string
|
||||
ShardID string
|
||||
Rank int
|
||||
GPUIndices []int
|
||||
HCAs []string
|
||||
HTTPPort int
|
||||
State supervisor.State
|
||||
Model string
|
||||
Profile string
|
||||
Pool string
|
||||
Fabric string
|
||||
ProfileFP string
|
||||
// Endpoint is the leader's base URL, set on the rank 0 shard.
|
||||
Endpoint string
|
||||
}
|
||||
|
||||
// adoptedCores is what an adopted shard is taken to hold of its node's CPUs.
|
||||
func (r *Reconciler) adoptedCores(model, profile string) int {
|
||||
p, _ := r.profileOf(model, profile)
|
||||
return cpuCores(p)
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// RecordUsage stamps replicas with when each was last sent a request.
|
||||
//
|
||||
// Replicas absent from the map are left alone rather than zeroed: a router that
|
||||
// has just restarted knows nothing about what happened before it, and treating
|
||||
// that as "never used" would make every replica look abandoned at once —
|
||||
// exactly when a fleet is least able to afford acting on that.
|
||||
func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
|
||||
if len(lastUsed) == 0 {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
for id, at := range lastUsed {
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
// Only forwards. Two routers reporting, or a stale snapshot arriving
|
||||
// late, must not move a replica's last use backwards.
|
||||
if at.After(rep.LastUsedAt) {
|
||||
rep.LastUsedAt = at
|
||||
}
|
||||
}
|
||||
r.noteUsage(lastUsed)
|
||||
}
|
||||
|
||||
// IdleFor reports how long a replica has gone without a request, and whether
|
||||
// that is known at all.
|
||||
//
|
||||
// The second return is not a detail. "Idle for an hour" and "nobody has told me
|
||||
// yet" are different states, and anything that acts on idleness — eviction,
|
||||
// most obviously — has to refuse to act on the second.
|
||||
func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok || rep.LastUsedAt.IsZero() {
|
||||
return 0, false
|
||||
}
|
||||
return r.now().Sub(rep.LastUsedAt), true
|
||||
}
|
||||
|
||||
// Want marks an on-demand model as asked for, so the next pass places it.
|
||||
//
|
||||
// Called by whatever fields the request — the router, in practice. It returns
|
||||
// whether the model is one this applies to, so a caller can tell "being loaded"
|
||||
// from "there is genuinely nothing serving this", which are different answers
|
||||
// to give a client.
|
||||
func (r *Reconciler) Want(model string) bool {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok || !m.Scaling.OnDemand() {
|
||||
return false
|
||||
}
|
||||
r.mu.Lock()
|
||||
r.wanted[m.Name] = r.now()
|
||||
// A placement that failed before this request says nothing about now:
|
||||
// forgotten, so the pass this triggers decides afresh and Unservable
|
||||
// does not answer from a stale result.
|
||||
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
|
||||
delete(r.lastExplain, m.Name)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
// Immediately, not on the next tick. Someone is waiting on this request,
|
||||
// and five seconds of ticker is five seconds they spend staring at nothing.
|
||||
r.Notify()
|
||||
return true
|
||||
}
|
||||
|
||||
// desiredFor is how many replicas a model should have right now.
|
||||
//
|
||||
// For an ordinary model that is its configured minimum. For an on-demand one it
|
||||
// is one while somebody is asking and none once they stop, which is the whole
|
||||
// feature: a cluster with more models than GPUs can hold every model in its
|
||||
// catalog and only the wanted ones in its cards.
|
||||
func (r *Reconciler) desiredFor(m catalog.Model) int {
|
||||
base := r.baseFor(m)
|
||||
if base == 0 {
|
||||
return 0
|
||||
}
|
||||
r.mu.Lock()
|
||||
extra := r.extra[m.Name]
|
||||
r.mu.Unlock()
|
||||
return min(base+extra, max(m.Scaling.Max, base))
|
||||
}
|
||||
|
||||
// baseFor is what a model runs without load: its minimum, or for an on-demand
|
||||
// model one while somebody is asking and none once they stop.
|
||||
func (r *Reconciler) baseFor(m catalog.Model) int {
|
||||
if !m.Scaling.OnDemand() {
|
||||
return m.Scaling.Min
|
||||
}
|
||||
r.mu.Lock()
|
||||
last, asked := r.wanted[m.Name]
|
||||
r.mu.Unlock()
|
||||
if !asked {
|
||||
return 0
|
||||
}
|
||||
if r.now().Sub(last) > r.idleTimeout(m) {
|
||||
return 0
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
// idleTimeout is how long an on-demand model stays up unasked-for.
|
||||
func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
|
||||
if m.Scaling.IdleTimeout > 0 {
|
||||
return m.Scaling.IdleTimeout
|
||||
}
|
||||
return r.cfg.IdleTimeout
|
||||
}
|
||||
|
||||
// noteUsage keeps an on-demand model alive while it is being used.
|
||||
//
|
||||
// Without this a model would be evicted exactly IdleTimeout after the request
|
||||
// that loaded it, however busy it had been since — the timer would measure
|
||||
// time since the first request rather than the last.
|
||||
func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
|
||||
byModel := map[string]time.Time{}
|
||||
for id, at := range lastUsed {
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if at.After(byModel[rep.Model]) {
|
||||
byModel[rep.Model] = at
|
||||
}
|
||||
}
|
||||
for model, at := range byModel {
|
||||
if cur, ok := r.wanted[model]; ok && at.After(cur) {
|
||||
r.wanted[model] = at
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Load is what a model's replicas are doing, as the router sees it.
|
||||
type Load struct {
|
||||
Replicas int
|
||||
Running int
|
||||
Waiting int
|
||||
}
|
||||
|
||||
// RecordLoad adds or takes away replicas between a model's min and max as its
|
||||
// load says.
|
||||
//
|
||||
// Max was a ceiling nothing reached: every model ran exactly its minimum
|
||||
// however long its queue. Now requests waiting at the engines — every slot of
|
||||
// every replica busy — held for ScaleUpAfter add one replica, and a quiet spell
|
||||
// of ScaleDownAfter takes one of those back. One step at a time, and not while
|
||||
// the last one is still starting: on a small CPU fleet each replica is minutes
|
||||
// of weights and gigabytes of memory, and a spike that is over by the time it
|
||||
// loads should not leave three behind.
|
||||
func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
now := r.now()
|
||||
for _, m := range r.cat.Load().Models() {
|
||||
base := r.baseFor(m)
|
||||
r.mu.Lock()
|
||||
if base == 0 || m.Scaling.Max <= base {
|
||||
delete(r.extra, m.Name)
|
||||
delete(r.busySince, m.Name)
|
||||
delete(r.quietSince, m.Name)
|
||||
r.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
starting := false
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name && (rep.State == Placing || rep.State == Starting) {
|
||||
starting = true
|
||||
}
|
||||
}
|
||||
l := loads[m.Name]
|
||||
perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
|
||||
extra := r.extra[m.Name]
|
||||
switch {
|
||||
case perReplica >= r.cfg.ScaleUpWaiting:
|
||||
delete(r.quietSince, m.Name)
|
||||
since, ok := r.busySince[m.Name]
|
||||
if !ok {
|
||||
r.busySince[m.Name] = now
|
||||
break
|
||||
}
|
||||
if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < m.Scaling.Max && !starting {
|
||||
r.extra[m.Name] = extra + 1
|
||||
r.busySince[m.Name] = now
|
||||
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
|
||||
"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
|
||||
}
|
||||
case l.Waiting == 0 && extra > 0:
|
||||
delete(r.busySince, m.Name)
|
||||
since, ok := r.quietSince[m.Name]
|
||||
if !ok {
|
||||
r.quietSince[m.Name] = now
|
||||
break
|
||||
}
|
||||
if now.Sub(since) >= r.cfg.ScaleDownAfter {
|
||||
r.extra[m.Name] = extra - 1
|
||||
r.quietSince[m.Name] = now
|
||||
r.log.Info("scaling down after a quiet spell", "model", m.Name,
|
||||
"replicas", base+extra-1, "quiet_for", now.Sub(since).Round(time.Second))
|
||||
}
|
||||
default:
|
||||
delete(r.busySince, m.Name)
|
||||
delete(r.quietSince, m.Name)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
)
|
||||
|
||||
// Headroom is how many more replicas of a model the cluster has room for now.
|
||||
type Headroom struct {
|
||||
// Current is the replicas the model has that hold capacity: starting or
|
||||
// ready, not those on their way out.
|
||||
Current int
|
||||
// More is how many more could be placed now, at most the limit asked.
|
||||
More int
|
||||
// Nodes is where each of those would go, in order, a replica's nodes
|
||||
// joined when it spans several.
|
||||
Nodes [][]string
|
||||
// Limited is set when More reached the limit asked, so there may be room
|
||||
// for more than that. A CPU profile that declares no memory is always
|
||||
// limited: nothing it asks for runs out.
|
||||
Limited bool
|
||||
// Blocked is why one more than More does not fit, a line per node that
|
||||
// was considered. Empty when Limited.
|
||||
Blocked []string
|
||||
}
|
||||
|
||||
// Headroom tries placing up to limit more replicas of a model on a copy of
|
||||
// the inventory, each one's allocation made on the copy before the next is
|
||||
// tried, exactly as the reconciler would place them one pass at a time.
|
||||
//
|
||||
// The scheduler's own decisions, not an estimate beside them: what the page
|
||||
// offers is then what the next passes will do, and a count that the
|
||||
// arithmetic of a second implementation allowed but placement refuses would
|
||||
// leave a model short with nobody told why.
|
||||
func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return Headroom{}, false
|
||||
}
|
||||
var h Headroom
|
||||
var avoid []string
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name && rep.State.Counts() && rep.RetiringAt.IsZero() {
|
||||
h.Current++
|
||||
avoid = append(avoid, rep.NodeIDs()...)
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
inv := r.inv.Clone()
|
||||
sched := placement.New(inv)
|
||||
for h.More < limit {
|
||||
res := sched.Place(m.PlacementProfiles(), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// As scaleUp does: spreading is a preference, not a rule.
|
||||
res = sched.Place(m.PlacementProfiles(), nil)
|
||||
}
|
||||
if res.Placement == nil {
|
||||
for _, rej := range res.Rejections {
|
||||
h.Blocked = append(h.Blocked, rej.String())
|
||||
}
|
||||
slices.Sort(h.Blocked)
|
||||
h.Blocked = slices.Compact(h.Blocked)
|
||||
return h, true
|
||||
}
|
||||
mem, cores := 0, 0
|
||||
if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
|
||||
mem, cores = prof.MinMemoryGB, cpuCores(prof)
|
||||
}
|
||||
var nodes []string
|
||||
for _, a := range res.Placement.Nodes {
|
||||
inv.Allocate(a.NodeID, inventory.Allocation{
|
||||
ShardID: "headroom", ReplicaID: "headroom", GPUIndices: a.GPUIndices, MemoryGB: mem, CPUCores: cores,
|
||||
})
|
||||
nodes = append(nodes, a.NodeID)
|
||||
}
|
||||
avoid = append(avoid, nodes...)
|
||||
h.Nodes = append(h.Nodes, nodes)
|
||||
h.More++
|
||||
}
|
||||
h.Limited = true
|
||||
return h, true
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestHeadroomIsWhatPlacementWouldDoNext(t *testing.T) {
|
||||
// Two 8-GPU nodes and a 4-GPU profile: four replicas in all, one placed.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
|
||||
got, ok := h.r.Headroom("llama", 10)
|
||||
if !ok {
|
||||
t.Fatal("the model was not found")
|
||||
}
|
||||
if got.Current != 1 || got.More != 3 || got.Limited {
|
||||
t.Fatalf("headroom %+v, expected 1 placed and room for 3 more", got)
|
||||
}
|
||||
// Spread first: the next one goes to the node without one.
|
||||
if first := got.Nodes[0][0]; first == h.r.Replicas()[0].NodeIDs()[0] {
|
||||
t.Errorf("the next replica would stack on %s beside the first", first)
|
||||
}
|
||||
if len(got.Blocked) == 0 || !strings.Contains(strings.Join(got.Blocked, "\n"), "GPU") {
|
||||
t.Errorf("no reason given for the fifth: %v", got.Blocked)
|
||||
}
|
||||
// Tried on a copy: nothing was taken from the real inventory.
|
||||
if free := len(h.inv.FreeGPUs("1")) + len(h.inv.FreeGPUs("2")); free != 12 {
|
||||
t.Errorf("%d GPUs free after asking, expected the 12 the one replica leaves", free)
|
||||
}
|
||||
|
||||
capped, _ := h.r.Headroom("llama", 2)
|
||||
if capped.More != 2 || !capped.Limited || capped.Blocked != nil {
|
||||
t.Errorf("asked for at most 2: %+v", capped)
|
||||
}
|
||||
if _, ok := h.r.Headroom("nope", 5); ok {
|
||||
t.Error("an unknown model has headroom")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeadroomCountsMemoryAlreadyClaimed(t *testing.T) {
|
||||
cpu := func(id string) inventory.Node {
|
||||
return inventory.Node{ID: id, MgmtAddr: "10.0.0." + id, State: inventory.Ready, MemoryTotalMB: 16 * 1024}
|
||||
}
|
||||
m := catalog.Model{Name: "small", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6}}}
|
||||
h := newHarness(t, m, cpu("a"), cpu("b"))
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
// 16 GB each at 6 GB a replica: two per node, one of them taken.
|
||||
got, _ := h.r.Headroom("small", 10)
|
||||
if got.Current != 1 || got.More != 3 {
|
||||
t.Errorf("headroom %+v, expected room for 3 more", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReservedCoresAreTheEnginesThreadsAndBoundTheRoom(t *testing.T) {
|
||||
cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 64 * 1024, CPUCount: 8}
|
||||
m := catalog.Model{Name: "coder", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6, CPUCores: 3}}}
|
||||
h := newHarness(t, m, cpu)
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
started := h.disp.startedShards()
|
||||
if len(started) != 1 || started[0].Spec.Threads != 3 {
|
||||
t.Fatalf("the engine was not told its 3 cores: %+v", started)
|
||||
}
|
||||
if a := h.inv.Allocations("a"); len(a) != 1 || a[0].CPUCores != 3 {
|
||||
t.Errorf("the cores were not reserved: %+v", a)
|
||||
}
|
||||
// Memory would allow nine more; cores allow one: 8 - 3 = 5, then 2.
|
||||
got, _ := h.r.Headroom("coder", 10)
|
||||
if got.More != 1 {
|
||||
t.Errorf("room for %d more, expected 1 by cores: %+v", got.More, got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// parkRecord is why a model stopped being retried.
|
||||
type parkRecord struct {
|
||||
Since time.Time
|
||||
Reason string
|
||||
Failures int
|
||||
// fp is the model's definition when it was parked; a different one
|
||||
// unparks it.
|
||||
fp string
|
||||
}
|
||||
|
||||
type placementNote struct {
|
||||
text string
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// modelFingerprint identifies a model's definition, so a change to it can be
|
||||
// told from a retry of the same thing.
|
||||
func modelFingerprint(m catalog.Model) string {
|
||||
b, _ := json.Marshal(m)
|
||||
sum := sha256.Sum256(b)
|
||||
return hex.EncodeToString(sum[:8])
|
||||
}
|
||||
|
||||
// placementLogEvery is how often an unchanged placement failure is repeated.
|
||||
const placementLogEvery = 10 * time.Minute
|
||||
|
||||
// notePlacementFailure logs why a model could not be placed, when the reason
|
||||
// is new or has not been said for a while. Said on every pass it was 250
|
||||
// identical lines in two days for one model, burying everything else.
|
||||
func (r *Reconciler) notePlacementFailure(model, why string) {
|
||||
r.mu.Lock()
|
||||
last := r.placementLogged[model]
|
||||
say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
|
||||
if say {
|
||||
r.placementLogged[model] = placementNote{text: why, at: r.now()}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if say {
|
||||
r.log.Warn("no placement for model", "model", model, "reasons", why)
|
||||
}
|
||||
}
|
||||
|
||||
// Retry clears a parked model and its backoff, so it is placed on the next
|
||||
// pass.
|
||||
func (r *Reconciler) Retry(model string) bool {
|
||||
r.mu.Lock()
|
||||
_, was := r.parked[model]
|
||||
delete(r.parked, model)
|
||||
r.attempts[model] = 0
|
||||
delete(r.nextAttempt, model)
|
||||
r.mu.Unlock()
|
||||
r.Notify()
|
||||
return was
|
||||
}
|
||||
|
||||
// ModelHealth is how one model stands, for status and metrics.
|
||||
type ModelHealth struct {
|
||||
Model string
|
||||
// Desired is how many replicas it should have now; 0 for a stopped model
|
||||
// or an on-demand one nobody is asking for.
|
||||
Desired int
|
||||
Ready int
|
||||
// Failures counts replicas that failed in a row.
|
||||
Failures int
|
||||
// Parked is set when Failures reached ParkAfter; Reason is the last error.
|
||||
Parked bool
|
||||
ParkedSince time.Time
|
||||
Reason string
|
||||
// Unplaceable is set when it is short of replicas and the last attempt to
|
||||
// place one found nowhere; Placement says why.
|
||||
Unplaceable bool
|
||||
Placement string
|
||||
}
|
||||
|
||||
// ModelHealth reports every model in the catalog.
|
||||
func (r *Reconciler) ModelHealth() []ModelHealth {
|
||||
models := r.cat.Load().Models()
|
||||
desired := map[string]int{}
|
||||
for _, m := range models {
|
||||
desired[m.Name] = r.desiredFor(m)
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
ready, live := map[string]int{}, map[string]int{}
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State == Ready {
|
||||
ready[rep.Model]++
|
||||
}
|
||||
if rep.State.Counts() {
|
||||
live[rep.Model]++
|
||||
}
|
||||
}
|
||||
out := make([]ModelHealth, 0, len(models))
|
||||
for _, m := range models {
|
||||
h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
|
||||
Failures: r.attempts[m.Name]}
|
||||
if p, ok := r.parked[m.Name]; ok {
|
||||
h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
|
||||
}
|
||||
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
|
||||
h.Unplaceable, h.Placement = true, res.Explain()
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Unservable says why a model cannot be served at all, so a request for it is
|
||||
// told now rather than asked to retry something that will never come up.
|
||||
func (r *Reconciler) Unservable(model string) (string, bool) {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if p, ok := r.parked[m.Name]; ok {
|
||||
return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
|
||||
m.Name, p.Failures, p.Reason), true
|
||||
}
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name && rep.State.Counts() {
|
||||
return "", false // something is on its way
|
||||
}
|
||||
}
|
||||
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
|
||||
return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
func firstLine(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if i := strings.IndexByte(s, '\n'); i >= 0 {
|
||||
return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -3,15 +3,10 @@ package reconciler
|
||||
import (
|
||||
"context"
|
||||
crypterand "crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -460,140 +455,6 @@ func (r *Reconciler) fleetHasSettled() bool {
|
||||
return settled
|
||||
}
|
||||
|
||||
// RecordUsage stamps replicas with when each was last sent a request.
|
||||
//
|
||||
// Replicas absent from the map are left alone rather than zeroed: a router that
|
||||
// has just restarted knows nothing about what happened before it, and treating
|
||||
// that as "never used" would make every replica look abandoned at once —
|
||||
// exactly when a fleet is least able to afford acting on that.
|
||||
func (r *Reconciler) RecordUsage(lastUsed map[string]time.Time) {
|
||||
if len(lastUsed) == 0 {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
for id, at := range lastUsed {
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
// Only forwards. Two routers reporting, or a stale snapshot arriving
|
||||
// late, must not move a replica's last use backwards.
|
||||
if at.After(rep.LastUsedAt) {
|
||||
rep.LastUsedAt = at
|
||||
}
|
||||
}
|
||||
r.noteUsage(lastUsed)
|
||||
}
|
||||
|
||||
// IdleFor reports how long a replica has gone without a request, and whether
|
||||
// that is known at all.
|
||||
//
|
||||
// The second return is not a detail. "Idle for an hour" and "nobody has told me
|
||||
// yet" are different states, and anything that acts on idleness — eviction,
|
||||
// most obviously — has to refuse to act on the second.
|
||||
func (r *Reconciler) IdleFor(id string) (time.Duration, bool) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok || rep.LastUsedAt.IsZero() {
|
||||
return 0, false
|
||||
}
|
||||
return r.now().Sub(rep.LastUsedAt), true
|
||||
}
|
||||
|
||||
// Want marks an on-demand model as asked for, so the next pass places it.
|
||||
//
|
||||
// Called by whatever fields the request — the router, in practice. It returns
|
||||
// whether the model is one this applies to, so a caller can tell "being loaded"
|
||||
// from "there is genuinely nothing serving this", which are different answers
|
||||
// to give a client.
|
||||
func (r *Reconciler) Want(model string) bool {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok || !m.Scaling.OnDemand() {
|
||||
return false
|
||||
}
|
||||
r.mu.Lock()
|
||||
r.wanted[m.Name] = r.now()
|
||||
// A placement that failed before this request says nothing about now:
|
||||
// forgotten, so the pass this triggers decides afresh and Unservable
|
||||
// does not answer from a stale result.
|
||||
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
|
||||
delete(r.lastExplain, m.Name)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
// Immediately, not on the next tick. Someone is waiting on this request,
|
||||
// and five seconds of ticker is five seconds they spend staring at nothing.
|
||||
r.Notify()
|
||||
return true
|
||||
}
|
||||
|
||||
// desiredFor is how many replicas a model should have right now.
|
||||
//
|
||||
// For an ordinary model that is its configured minimum. For an on-demand one it
|
||||
// is one while somebody is asking and none once they stop, which is the whole
|
||||
// feature: a cluster with more models than GPUs can hold every model in its
|
||||
// catalog and only the wanted ones in its cards.
|
||||
func (r *Reconciler) desiredFor(m catalog.Model) int {
|
||||
base := r.baseFor(m)
|
||||
if base == 0 {
|
||||
return 0
|
||||
}
|
||||
r.mu.Lock()
|
||||
extra := r.extra[m.Name]
|
||||
r.mu.Unlock()
|
||||
return min(base+extra, max(m.Scaling.Max, base))
|
||||
}
|
||||
|
||||
// baseFor is what a model runs without load: its minimum, or for an on-demand
|
||||
// model one while somebody is asking and none once they stop.
|
||||
func (r *Reconciler) baseFor(m catalog.Model) int {
|
||||
if !m.Scaling.OnDemand() {
|
||||
return m.Scaling.Min
|
||||
}
|
||||
r.mu.Lock()
|
||||
last, asked := r.wanted[m.Name]
|
||||
r.mu.Unlock()
|
||||
if !asked {
|
||||
return 0
|
||||
}
|
||||
if r.now().Sub(last) > r.idleTimeout(m) {
|
||||
return 0
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
// idleTimeout is how long an on-demand model stays up unasked-for.
|
||||
func (r *Reconciler) idleTimeout(m catalog.Model) time.Duration {
|
||||
if m.Scaling.IdleTimeout > 0 {
|
||||
return m.Scaling.IdleTimeout
|
||||
}
|
||||
return r.cfg.IdleTimeout
|
||||
}
|
||||
|
||||
// noteUsage keeps an on-demand model alive while it is being used.
|
||||
//
|
||||
// Without this a model would be evicted exactly IdleTimeout after the request
|
||||
// that loaded it, however busy it had been since — the timer would measure
|
||||
// time since the first request rather than the last.
|
||||
func (r *Reconciler) noteUsage(lastUsed map[string]time.Time) {
|
||||
byModel := map[string]time.Time{}
|
||||
for id, at := range lastUsed {
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if at.After(byModel[rep.Model]) {
|
||||
byModel[rep.Model] = at
|
||||
}
|
||||
}
|
||||
for model, at := range byModel {
|
||||
if cur, ok := r.wanted[model]; ok && at.After(cur) {
|
||||
r.wanted[model] = at
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// expireStuckReplicas fails replicas that never finished coming up.
|
||||
//
|
||||
// Every other path out of Starting depends on the agent reporting something. A
|
||||
@@ -884,110 +745,6 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.retire(ctx, retiringIDs, readyLive, len(liveIDs), desired)
|
||||
}
|
||||
|
||||
// retire finishes restarting replicas.
|
||||
//
|
||||
// A retiring replica keeps serving until its replacement is ready, and is then
|
||||
// drained like any other. When no replacement could be placed at all —
|
||||
// nowhere else has the memory, which on a small CPU fleet is the usual case —
|
||||
// it is stopped after DrainGrace so its replacement can take its place: a
|
||||
// restart in place, with the model unserved while it loads if it had no other
|
||||
// replica. Said in the log, since that is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
|
||||
for _, id := range ids {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var since time.Time
|
||||
if ok {
|
||||
since = rep.RetiringAt
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case readyLive >= desired:
|
||||
r.drainForRestart(ctx, id)
|
||||
case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no room for its replacement beside it",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var shards []Shard
|
||||
if ok {
|
||||
rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
|
||||
shards = slices.Clone(rep.Shards)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
for _, s := range shards {
|
||||
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
|
||||
r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// nextRestartLocked pops the next queued replica of a model that is still
|
||||
// ready to be restarted. Called with mu held.
|
||||
func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
|
||||
q := r.restartQueue[model]
|
||||
for len(q) > 0 {
|
||||
id := q[0]
|
||||
q = q[1:]
|
||||
if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
|
||||
r.restartQueue[model] = q
|
||||
return id, true
|
||||
}
|
||||
}
|
||||
delete(r.restartQueue, model)
|
||||
return "", false
|
||||
}
|
||||
|
||||
// ErrNoSuchReplica means no ready replica has that id.
|
||||
var ErrNoSuchReplica = errors.New("no ready replica with that id")
|
||||
|
||||
// Restart queues replicas to be replaced, each by a new one started beside it
|
||||
// before it is stopped — to pick up an edited profile, an engine key, a new
|
||||
// engine version. Replicas of one model are replaced one at a time.
|
||||
func (r *Reconciler) Restart(ids ...string) error {
|
||||
r.mu.Lock()
|
||||
for _, id := range ids {
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok || rep.State != Ready {
|
||||
r.mu.Unlock()
|
||||
return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
|
||||
}
|
||||
if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
|
||||
r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
r.Notify()
|
||||
return nil
|
||||
}
|
||||
|
||||
// RestartState says where a replica's restart stands: "queued", "replacing",
|
||||
// or "" when none was asked for.
|
||||
func (r *Reconciler) RestartState(id string) string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
rep, ok := r.replicas[id]
|
||||
switch {
|
||||
case !ok:
|
||||
return ""
|
||||
case !rep.RetiringAt.IsZero():
|
||||
return "replacing"
|
||||
case slices.Contains(r.restartQueue[rep.Model], id):
|
||||
return "queued"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// scaleUp places and starts one replica per pass.
|
||||
//
|
||||
// One at a time on purpose: placement decisions depend on what is already
|
||||
@@ -1068,7 +825,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
|
||||
})
|
||||
r.inv.Allocate(a.NodeID, inventory.Allocation{
|
||||
ShardID: shardID, ReplicaID: id, GPUIndices: a.GPUIndices,
|
||||
MemoryGB: prof.MinMemoryGB,
|
||||
MemoryGB: prof.MinMemoryGB, CPUCores: cpuCores(prof),
|
||||
})
|
||||
}
|
||||
rep.Endpoint = fmt.Sprintf("http://%s:%d", addrOf(leaderNode), basePort)
|
||||
@@ -1112,6 +869,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
|
||||
HTTPPort: s.HTTPPort,
|
||||
Profile: espec,
|
||||
APIKey: r.engineKey(id),
|
||||
Threads: cpuCores(prof),
|
||||
},
|
||||
StartupTimeout: prof.StartupTimeout,
|
||||
Pool: pl.Pool,
|
||||
@@ -1229,253 +987,6 @@ func (r *Reconciler) stopReplica(ctx context.Context, id string) {
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
// Adopt rebuilds replica state from shards an agent reports it is already
|
||||
// running (plan §6.5).
|
||||
//
|
||||
// This is what makes a control plane restart survivable. Without it the
|
||||
// reconciler sees an empty cluster, places duplicates, and the engines already
|
||||
// running keep holding GPUs the inventory believes are free — so the duplicates
|
||||
// fail to allocate, with an error naming the new process rather than the old.
|
||||
//
|
||||
// Shards whose model or profile is no longer in the catalog are returned for
|
||||
// the caller to stop: nothing will ever claim them.
|
||||
func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []AdoptedShard) {
|
||||
defer r.forgetShardsTheAgentDoesNotHave(nodeID, shards)
|
||||
|
||||
byReplica := map[string][]AdoptedShard{}
|
||||
for _, s := range shards {
|
||||
byReplica[s.ReplicaID] = append(byReplica[s.ReplicaID], s)
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
for replicaID, group := range byReplica {
|
||||
if existing, known := r.replicas[replicaID]; known {
|
||||
// The other half of a replica we are still assembling. Merge rather
|
||||
// than choose: these shards belong to the same replica as the ones
|
||||
// already adopted, they are simply on a different node whose agent
|
||||
// registered a moment later. Treating them as a competing claim is
|
||||
// what tore multi-node replicas apart on every control plane
|
||||
// restart — one half adopted and failed for being short, the other
|
||||
// stopped as unclaimed when it finally arrived.
|
||||
if !existing.AdoptedAt.IsZero() && len(existing.Shards) < existing.WantShards {
|
||||
r.mergeAdopted(existing, group)
|
||||
continue
|
||||
}
|
||||
// Usually this is an agent reconnecting rather than a control plane
|
||||
// restart, and its status reports keep the state current.
|
||||
//
|
||||
// Not always. Replica ids are derived from the model name and a
|
||||
// counter, so a control plane that restarts and places before the
|
||||
// agents reconnect produces a *different* replica under the same
|
||||
// id, on a freshly allocated port. Skipping the adoption then
|
||||
// leaves the cluster pointing at a port nothing listens on, while
|
||||
// the engine that is actually serving is treated as a stray.
|
||||
//
|
||||
// The fleet is authoritative about what is running (plan §6.5), so
|
||||
// what we merely placed loses to what an agent reports — unless our
|
||||
// copy has a shard that has actually come up, in which case two
|
||||
// things really are running and the one we did not ask for is the
|
||||
// stray.
|
||||
if existing.hasLiveShard() {
|
||||
// Only what we do not recognise. A reconnecting agent reports
|
||||
// the shards we already know, and disowning those would tell it
|
||||
// to stop the very replica that is serving.
|
||||
known := map[string]struct{}{}
|
||||
for _, sh := range existing.Shards {
|
||||
known[sh.ID] = struct{}{}
|
||||
}
|
||||
var strays []AdoptedShard
|
||||
for _, s := range group {
|
||||
if _, ok := known[s.ShardID]; !ok {
|
||||
strays = append(strays, s)
|
||||
}
|
||||
}
|
||||
unclaimed = append(unclaimed, strays...)
|
||||
r.reservePorts(strays)
|
||||
continue
|
||||
}
|
||||
r.log.Warn("a running replica reported by the fleet replaces one we placed",
|
||||
"replica", replicaID, "placed_endpoint", existing.Endpoint)
|
||||
for _, s := range existing.Shards {
|
||||
r.inv.Release(s.ID)
|
||||
}
|
||||
delete(r.replicas, replicaID)
|
||||
}
|
||||
|
||||
first := group[0]
|
||||
m, ok := r.cat.Load().Resolve(first.Model)
|
||||
if !ok {
|
||||
unclaimed = append(unclaimed, group...)
|
||||
r.reservePorts(group)
|
||||
continue
|
||||
}
|
||||
prof, ok := m.ProfileByName(first.Profile)
|
||||
if !ok {
|
||||
unclaimed = append(unclaimed, group...)
|
||||
r.reservePorts(group)
|
||||
continue
|
||||
}
|
||||
|
||||
rep := &Replica{
|
||||
ID: replicaID, Model: m.Name, Profile: prof.Name,
|
||||
Fabric: first.Fabric, Pool: first.Pool,
|
||||
// From the agent, which was told it when the shard started. Empty
|
||||
// from an agent too old to report it, which reads as unknown.
|
||||
ProfileFP: first.ProfileFP,
|
||||
Weight: prof.Weight, CreatedAt: r.now(),
|
||||
Endpoint: first.Endpoint,
|
||||
// From the profile, not from what was reported: the point is to
|
||||
// notice when what was reported is short.
|
||||
WantShards: max(prof.Nodes, 1),
|
||||
AdoptedAt: r.now(),
|
||||
}
|
||||
r.mergeAdopted(rep, group)
|
||||
r.replicas[replicaID] = rep
|
||||
|
||||
}
|
||||
return unclaimed
|
||||
}
|
||||
|
||||
// mergeAdopted folds shards an agent reports into a replica being reassembled.
|
||||
//
|
||||
// Called both for the first group, which creates the replica, and for every
|
||||
// later group belonging to it, so a replica spanning N nodes is built up over N
|
||||
// registrations. Shard ids already present are skipped: an agent that
|
||||
// reconnects repeats what it is running, and counting those again would carry a
|
||||
// replica past its shard count without a second node ever appearing.
|
||||
func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
|
||||
have := make(map[string]struct{}, len(rep.Shards))
|
||||
for _, sh := range rep.Shards {
|
||||
have[sh.ID] = struct{}{}
|
||||
}
|
||||
for _, s := range group {
|
||||
if _, dup := have[s.ShardID]; dup {
|
||||
continue
|
||||
}
|
||||
have[s.ShardID] = struct{}{}
|
||||
rep.Shards = append(rep.Shards, Shard{
|
||||
ID: s.ShardID, NodeID: s.NodeID, Rank: s.Rank,
|
||||
GPUIndices: s.GPUIndices, HCAs: s.HCAs,
|
||||
HTTPPort: s.HTTPPort, State: s.State,
|
||||
})
|
||||
// Reclaim the GPUs and the memory, or the next placement hands them
|
||||
// out twice. The memory comes from the catalog rather than the agent:
|
||||
// what a profile asked for is not something a running process reports.
|
||||
r.inv.Allocate(s.NodeID, inventory.Allocation{
|
||||
ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
|
||||
MemoryGB: r.profileMemory(rep.Model, rep.Profile),
|
||||
})
|
||||
// The endpoint comes from the leader, which may be in any of the
|
||||
// groups — the rank 0 node is not necessarily the first to register.
|
||||
if s.Rank == 0 && s.Endpoint != "" {
|
||||
rep.Endpoint = s.Endpoint
|
||||
}
|
||||
}
|
||||
slices.SortFunc(rep.Shards, func(a, b Shard) int { return a.Rank - b.Rank })
|
||||
rep.deriveState(r.now())
|
||||
|
||||
r.log.Info("adopted replica from the fleet",
|
||||
"replica", rep.ID, "model", rep.Model,
|
||||
"shards", len(rep.Shards), "want", rep.WantShards, "state", rep.State)
|
||||
}
|
||||
|
||||
// forgetShardsTheAgentDoesNotHave fails shards we believe are on a node that the
|
||||
// agent, having just told us everything it is running, did not mention.
|
||||
//
|
||||
// The mirror image of adoption, and the half that was missing. Adoption handles
|
||||
// "the agent has something we do not know about"; this handles "we think the
|
||||
// agent has something it does not". Both follow from the same rule: the fleet is
|
||||
// authoritative about what is running (plan §6.5).
|
||||
//
|
||||
// Without it, an engine that dies while its agent survives leaves the control
|
||||
// plane believing a replica is Ready forever. Observed on a real cluster: the
|
||||
// engine was killed, the agent restarted and registered with nothing running,
|
||||
// and the replica stayed Ready pointing at a dead port for as long as anyone
|
||||
// watched. Traffic survived — the breaker took the endpoint out — but the
|
||||
// cluster ran a third short with no sign of it.
|
||||
//
|
||||
// Safe against the obvious race. A registration follows a fresh connection, so
|
||||
// anything we dispatched over a previous one is not running by definition;
|
||||
// there is no window where a shard is legitimately started but unreported.
|
||||
func (r *Reconciler) forgetShardsTheAgentDoesNotHave(nodeID string, running []AdoptedShard) {
|
||||
if nodeID == "" {
|
||||
return
|
||||
}
|
||||
reported := make(map[string]struct{}, len(running))
|
||||
for _, s := range running {
|
||||
reported[s.ShardID] = struct{}{}
|
||||
}
|
||||
|
||||
var lost []string
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State == Stopped {
|
||||
continue
|
||||
}
|
||||
changed := false
|
||||
for i := range rep.Shards {
|
||||
sh := &rep.Shards[i]
|
||||
if sh.NodeID != nodeID || sh.State.Terminal() {
|
||||
continue
|
||||
}
|
||||
if _, ok := reported[sh.ID]; ok {
|
||||
continue
|
||||
}
|
||||
sh.State = supervisor.Failed
|
||||
sh.Err = "the agent reports it is not running"
|
||||
r.inv.Release(sh.ID)
|
||||
changed = true
|
||||
lost = append(lost, sh.ID)
|
||||
}
|
||||
if changed {
|
||||
rep.deriveState(r.now())
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
for _, id := range lost {
|
||||
r.log.Warn("a shard we believed was running is not; the agent does not have it",
|
||||
"node", nodeID, "shard", id)
|
||||
}
|
||||
}
|
||||
|
||||
// reservePorts records ports held by shards nothing owns.
|
||||
//
|
||||
// They are being stopped, but stopping is not instant and the engine keeps its
|
||||
// socket until it exits. Handing the port to a new shard in the meantime makes
|
||||
// that shard fail to bind, which fails its replica, which places another one on
|
||||
// the same port — a loop that does not end on its own.
|
||||
//
|
||||
// Called with the lock held.
|
||||
func (r *Reconciler) reservePorts(shards []AdoptedShard) {
|
||||
for _, s := range shards {
|
||||
if s.HTTPPort > 0 {
|
||||
r.reservedPorts[s.HTTPPort] = struct{}{}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// AdoptedShard is one running shard an agent reported at registration.
|
||||
type AdoptedShard struct {
|
||||
NodeID string
|
||||
ReplicaID string
|
||||
ShardID string
|
||||
Rank int
|
||||
GPUIndices []int
|
||||
HCAs []string
|
||||
HTTPPort int
|
||||
State supervisor.State
|
||||
Model string
|
||||
Profile string
|
||||
Pool string
|
||||
Fabric string
|
||||
ProfileFP string
|
||||
// Endpoint is the leader's base URL, set on the rank 0 shard.
|
||||
Endpoint string
|
||||
}
|
||||
|
||||
// OnFetchProgress records how far a shard's download has got.
|
||||
//
|
||||
// Separate from OnShardStatus because it is not a transition: the shard is
|
||||
@@ -1598,16 +1109,31 @@ func addrOf(n inventory.Node) string {
|
||||
// claim on restart would place a second large model onto a node already full of
|
||||
// the first.
|
||||
func (r *Reconciler) profileMemory(model, profile string) int {
|
||||
p, _ := r.profileOf(model, profile)
|
||||
return p.MinMemoryGB
|
||||
}
|
||||
|
||||
// profileOf is a replica's profile as the catalog holds it now.
|
||||
func (r *Reconciler) profileOf(model, profile string) (catalog.Profile, bool) {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return 0
|
||||
return catalog.Profile{}, false
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if p.Name == profile {
|
||||
return p.MinMemoryGB
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
return 0
|
||||
return catalog.Profile{}, false
|
||||
}
|
||||
|
||||
// cpuCores is what a profile reserves of a node's CPUs: its cores when it
|
||||
// computes on them, nothing when a GPU does the work.
|
||||
func cpuCores(p catalog.Profile) int {
|
||||
if !p.CPUOnly() {
|
||||
return 0
|
||||
}
|
||||
return p.CPUCores
|
||||
}
|
||||
|
||||
func (r *Reconciler) engineKey(replicaID string) string {
|
||||
@@ -1616,215 +1142,3 @@ func (r *Reconciler) engineKey(replicaID string) string {
|
||||
}
|
||||
return r.cfg.EngineKey(replicaID)
|
||||
}
|
||||
|
||||
// parkRecord is why a model stopped being retried.
|
||||
type parkRecord struct {
|
||||
Since time.Time
|
||||
Reason string
|
||||
Failures int
|
||||
// fp is the model's definition when it was parked; a different one
|
||||
// unparks it.
|
||||
fp string
|
||||
}
|
||||
|
||||
type placementNote struct {
|
||||
text string
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// modelFingerprint identifies a model's definition, so a change to it can be
|
||||
// told from a retry of the same thing.
|
||||
func modelFingerprint(m catalog.Model) string {
|
||||
b, _ := json.Marshal(m)
|
||||
sum := sha256.Sum256(b)
|
||||
return hex.EncodeToString(sum[:8])
|
||||
}
|
||||
|
||||
// placementLogEvery is how often an unchanged placement failure is repeated.
|
||||
const placementLogEvery = 10 * time.Minute
|
||||
|
||||
// notePlacementFailure logs why a model could not be placed, when the reason
|
||||
// is new or has not been said for a while. Said on every pass it was 250
|
||||
// identical lines in two days for one model, burying everything else.
|
||||
func (r *Reconciler) notePlacementFailure(model, why string) {
|
||||
r.mu.Lock()
|
||||
last := r.placementLogged[model]
|
||||
say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
|
||||
if say {
|
||||
r.placementLogged[model] = placementNote{text: why, at: r.now()}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if say {
|
||||
r.log.Warn("no placement for model", "model", model, "reasons", why)
|
||||
}
|
||||
}
|
||||
|
||||
// Retry clears a parked model and its backoff, so it is placed on the next
|
||||
// pass.
|
||||
func (r *Reconciler) Retry(model string) bool {
|
||||
r.mu.Lock()
|
||||
_, was := r.parked[model]
|
||||
delete(r.parked, model)
|
||||
r.attempts[model] = 0
|
||||
delete(r.nextAttempt, model)
|
||||
r.mu.Unlock()
|
||||
r.Notify()
|
||||
return was
|
||||
}
|
||||
|
||||
// ModelHealth is how one model stands, for status and metrics.
|
||||
type ModelHealth struct {
|
||||
Model string
|
||||
// Desired is how many replicas it should have now; 0 for a stopped model
|
||||
// or an on-demand one nobody is asking for.
|
||||
Desired int
|
||||
Ready int
|
||||
// Failures counts replicas that failed in a row.
|
||||
Failures int
|
||||
// Parked is set when Failures reached ParkAfter; Reason is the last error.
|
||||
Parked bool
|
||||
ParkedSince time.Time
|
||||
Reason string
|
||||
// Unplaceable is set when it is short of replicas and the last attempt to
|
||||
// place one found nowhere; Placement says why.
|
||||
Unplaceable bool
|
||||
Placement string
|
||||
}
|
||||
|
||||
// ModelHealth reports every model in the catalog.
|
||||
func (r *Reconciler) ModelHealth() []ModelHealth {
|
||||
models := r.cat.Load().Models()
|
||||
desired := map[string]int{}
|
||||
for _, m := range models {
|
||||
desired[m.Name] = r.desiredFor(m)
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
ready, live := map[string]int{}, map[string]int{}
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State == Ready {
|
||||
ready[rep.Model]++
|
||||
}
|
||||
if rep.State.Counts() {
|
||||
live[rep.Model]++
|
||||
}
|
||||
}
|
||||
out := make([]ModelHealth, 0, len(models))
|
||||
for _, m := range models {
|
||||
h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
|
||||
Failures: r.attempts[m.Name]}
|
||||
if p, ok := r.parked[m.Name]; ok {
|
||||
h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
|
||||
}
|
||||
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
|
||||
h.Unplaceable, h.Placement = true, res.Explain()
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Unservable says why a model cannot be served at all, so a request for it is
|
||||
// told now rather than asked to retry something that will never come up.
|
||||
func (r *Reconciler) Unservable(model string) (string, bool) {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if p, ok := r.parked[m.Name]; ok {
|
||||
return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
|
||||
m.Name, p.Failures, p.Reason), true
|
||||
}
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name && rep.State.Counts() {
|
||||
return "", false // something is on its way
|
||||
}
|
||||
}
|
||||
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
|
||||
return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
func firstLine(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if i := strings.IndexByte(s, '\n'); i >= 0 {
|
||||
return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Load is what a model's replicas are doing, as the router sees it.
|
||||
type Load struct {
|
||||
Replicas int
|
||||
Running int
|
||||
Waiting int
|
||||
}
|
||||
|
||||
// RecordLoad adds or takes away replicas between a model's min and max as its
|
||||
// load says.
|
||||
//
|
||||
// Max was a ceiling nothing reached: every model ran exactly its minimum
|
||||
// however long its queue. Now requests waiting at the engines — every slot of
|
||||
// every replica busy — held for ScaleUpAfter add one replica, and a quiet spell
|
||||
// of ScaleDownAfter takes one of those back. One step at a time, and not while
|
||||
// the last one is still starting: on a small CPU fleet each replica is minutes
|
||||
// of weights and gigabytes of memory, and a spike that is over by the time it
|
||||
// loads should not leave three behind.
|
||||
func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
now := r.now()
|
||||
for _, m := range r.cat.Load().Models() {
|
||||
base := r.baseFor(m)
|
||||
r.mu.Lock()
|
||||
if base == 0 || m.Scaling.Max <= base {
|
||||
delete(r.extra, m.Name)
|
||||
delete(r.busySince, m.Name)
|
||||
delete(r.quietSince, m.Name)
|
||||
r.mu.Unlock()
|
||||
continue
|
||||
}
|
||||
starting := false
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name && (rep.State == Placing || rep.State == Starting) {
|
||||
starting = true
|
||||
}
|
||||
}
|
||||
l := loads[m.Name]
|
||||
perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
|
||||
extra := r.extra[m.Name]
|
||||
switch {
|
||||
case perReplica >= r.cfg.ScaleUpWaiting:
|
||||
delete(r.quietSince, m.Name)
|
||||
since, ok := r.busySince[m.Name]
|
||||
if !ok {
|
||||
r.busySince[m.Name] = now
|
||||
break
|
||||
}
|
||||
if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < m.Scaling.Max && !starting {
|
||||
r.extra[m.Name] = extra + 1
|
||||
r.busySince[m.Name] = now
|
||||
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
|
||||
"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
|
||||
}
|
||||
case l.Waiting == 0 && extra > 0:
|
||||
delete(r.busySince, m.Name)
|
||||
since, ok := r.quietSince[m.Name]
|
||||
if !ok {
|
||||
r.quietSince[m.Name] = now
|
||||
break
|
||||
}
|
||||
if now.Sub(since) >= r.cfg.ScaleDownAfter {
|
||||
r.extra[m.Name] = extra - 1
|
||||
r.quietSince[m.Name] = now
|
||||
r.log.Info("scaling down after a quiet spell", "model", m.Name,
|
||||
"replicas", base+extra-1, "quiet_for", now.Sub(since).Round(time.Second))
|
||||
}
|
||||
default:
|
||||
delete(r.busySince, m.Name)
|
||||
delete(r.quietSince, m.Name)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"time"
|
||||
)
|
||||
|
||||
// retire finishes restarting replicas.
|
||||
//
|
||||
// A retiring replica keeps serving until its replacement is ready, and is then
|
||||
// drained like any other. When no replacement could be placed at all —
|
||||
// nowhere else has the memory, which on a small CPU fleet is the usual case —
|
||||
// it is stopped after DrainGrace so its replacement can take its place: a
|
||||
// restart in place, with the model unserved while it loads if it had no other
|
||||
// replica. Said in the log, since that is the one case with an outage.
|
||||
func (r *Reconciler) retire(ctx context.Context, ids []string, readyLive, live, desired int) {
|
||||
for _, id := range ids {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var since time.Time
|
||||
if ok {
|
||||
since = rep.RetiringAt
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case readyLive >= desired:
|
||||
r.drainForRestart(ctx, id)
|
||||
case live < desired && r.now().Sub(since) > r.cfg.DrainGrace:
|
||||
r.log.Warn("restarting a replica in place: no room for its replacement beside it",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
r.drainForRestart(ctx, id)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Reconciler) drainForRestart(ctx context.Context, id string) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var shards []Shard
|
||||
if ok {
|
||||
rep.State, rep.DrainingAt, rep.RetiringAt = Draining, r.now(), time.Time{}
|
||||
shards = slices.Clone(rep.Shards)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
for _, s := range shards {
|
||||
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
|
||||
r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// nextRestartLocked pops the next queued replica of a model that is still
|
||||
// ready to be restarted. Called with mu held.
|
||||
func (r *Reconciler) nextRestartLocked(model string) (string, bool) {
|
||||
q := r.restartQueue[model]
|
||||
for len(q) > 0 {
|
||||
id := q[0]
|
||||
q = q[1:]
|
||||
if rep, ok := r.replicas[id]; ok && rep.State == Ready && rep.RetiringAt.IsZero() {
|
||||
r.restartQueue[model] = q
|
||||
return id, true
|
||||
}
|
||||
}
|
||||
delete(r.restartQueue, model)
|
||||
return "", false
|
||||
}
|
||||
|
||||
// ErrNoSuchReplica means no ready replica has that id.
|
||||
var ErrNoSuchReplica = errors.New("no ready replica with that id")
|
||||
|
||||
// Restart queues replicas to be replaced, each by a new one started beside it
|
||||
// before it is stopped — to pick up an edited profile, an engine key, a new
|
||||
// engine version. Replicas of one model are replaced one at a time.
|
||||
func (r *Reconciler) Restart(ids ...string) error {
|
||||
r.mu.Lock()
|
||||
for _, id := range ids {
|
||||
rep, ok := r.replicas[id]
|
||||
if !ok || rep.State != Ready {
|
||||
r.mu.Unlock()
|
||||
return fmt.Errorf("%w: %s", ErrNoSuchReplica, id)
|
||||
}
|
||||
if !slices.Contains(r.restartQueue[rep.Model], id) && rep.RetiringAt.IsZero() {
|
||||
r.restartQueue[rep.Model] = append(r.restartQueue[rep.Model], id)
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
r.Notify()
|
||||
return nil
|
||||
}
|
||||
|
||||
// RestartState says where a replica's restart stands: "queued", "replacing",
|
||||
// or "" when none was asked for.
|
||||
func (r *Reconciler) RestartState(id string) string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
rep, ok := r.replicas[id]
|
||||
switch {
|
||||
case !ok:
|
||||
return ""
|
||||
case !rep.RetiringAt.IsZero():
|
||||
return "replacing"
|
||||
case slices.Contains(r.restartQueue[rep.Model], id):
|
||||
return "queued"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
// a new configuration file is a failure here rather than a silent omission.
|
||||
var Sources = []string{
|
||||
"internal/config/config.go",
|
||||
"internal/agent/agent.go",
|
||||
"internal/agent/config.go",
|
||||
"internal/agent/discovery/discover.go",
|
||||
"internal/agent/engines/engine.go",
|
||||
"internal/agent/engines/llamacpp.go",
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
package obs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
"go.opentelemetry.io/otel/sdk/resource"
|
||||
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
||||
)
|
||||
|
||||
// TracingOptions says where traces go.
|
||||
type TracingOptions struct {
|
||||
Endpoint string // http(s)://host:4318; empty reads OTEL_EXPORTER_OTLP_ENDPOINT
|
||||
SampleRatio float64 // of traces the caller did not already decide; zero takes 1
|
||||
Service string
|
||||
Version string
|
||||
}
|
||||
|
||||
// SetupTracing installs the global tracer provider and the W3C propagator,
|
||||
// and returns what flushes it on the way out. With no endpoint it installs
|
||||
// nothing, and every span is the no-op one: tracing that is off costs a
|
||||
// function call per span.
|
||||
func SetupTracing(ctx context.Context, o TracingOptions) (func(context.Context) error, error) {
|
||||
noop := func(context.Context) error { return nil }
|
||||
endpoint := o.Endpoint
|
||||
if endpoint == "" {
|
||||
endpoint = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
|
||||
}
|
||||
if endpoint == "" {
|
||||
return noop, nil
|
||||
}
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return noop, fmt.Errorf("tracing endpoint %q is not an http(s) address", endpoint)
|
||||
}
|
||||
opts := []otlptracehttp.Option{otlptracehttp.WithEndpoint(u.Host)}
|
||||
if u.Scheme == "http" {
|
||||
opts = append(opts, otlptracehttp.WithInsecure())
|
||||
}
|
||||
if u.Path != "" && u.Path != "/" {
|
||||
opts = append(opts, otlptracehttp.WithURLPath(u.Path))
|
||||
}
|
||||
exp, err := otlptracehttp.New(ctx, opts...)
|
||||
if err != nil {
|
||||
return noop, err
|
||||
}
|
||||
ratio := o.SampleRatio
|
||||
if ratio <= 0 || ratio > 1 {
|
||||
ratio = 1
|
||||
}
|
||||
res := resource.NewSchemaless(
|
||||
attribute.String("service.name", o.Service),
|
||||
attribute.String("service.version", o.Version),
|
||||
)
|
||||
tp := sdktrace.NewTracerProvider(
|
||||
sdktrace.WithBatcher(exp, sdktrace.WithBatchTimeout(5*time.Second)),
|
||||
sdktrace.WithResource(res),
|
||||
// A caller that sent a sampled traceparent has already decided; the
|
||||
// ratio is for requests that arrive without one.
|
||||
sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(ratio))),
|
||||
)
|
||||
otel.SetTracerProvider(tp)
|
||||
otel.SetTextMapPropagator(propagation.TraceContext{})
|
||||
return tp.Shutdown, nil
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
package obs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTracingIsOffUntilGivenSomewhereToSend(t *testing.T) {
|
||||
t.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "")
|
||||
stop, err := SetupTracing(context.Background(), TracingOptions{Service: "t"})
|
||||
if err != nil || stop(context.Background()) != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, bad := range []string{"otel:4318", "ftp://otel:4318", "http://"} {
|
||||
if _, err := SetupTracing(context.Background(), TracingOptions{Endpoint: bad}); err == nil {
|
||||
t.Errorf("%q accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -498,14 +498,17 @@ func (r *Runner) certificate(ctx context.Context, res *Result) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Trusting the root too, so other tools on the machine do not treat the
|
||||
// cluster's certificates as strangers. Best effort: either family's way.
|
||||
_, _ = r.Remote.Run(ctx, `if [ -d /usr/local/share/ca-certificates ]; then
|
||||
sudo install -m 0644 /etc/llmgw/ca.pem /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates >/dev/null 2>&1
|
||||
elif [ -d /etc/pki/ca-trust/source/anchors ]; then
|
||||
sudo install -m 0644 /etc/llmgw/ca.pem /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
|
||||
// The agent trusts the cluster's root through its own configuration only.
|
||||
// Not system-wide: that root signs whatever the control plane asks it to,
|
||||
// and every TLS client on the machine would then take it for any site.
|
||||
// A node onboarded by an older version had it added; take it back out.
|
||||
_, _ = r.Remote.Run(ctx, `if [ -f /usr/local/share/ca-certificates/llmgw-ca.crt ]; then
|
||||
sudo rm -f /usr/local/share/ca-certificates/llmgw-ca.crt && sudo update-ca-certificates --fresh >/dev/null 2>&1
|
||||
fi
|
||||
if [ -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem ]; then
|
||||
sudo rm -f /etc/pki/ca-trust/source/anchors/llmgw-ca.pem && sudo update-ca-trust >/dev/null 2>&1
|
||||
fi; true`, nil)
|
||||
r.emit(StepCert, Done, "certificate for "+res.Node+" installed, the cluster's root trusted", "")
|
||||
r.emit(StepCert, Done, "certificate for "+res.Node+" installed", "")
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
)
|
||||
|
||||
// What a client is told for each way a dispatch can end. Each status is
|
||||
// something a client acts on: 503 with Retry-After is "try again", 503 alone
|
||||
// is "not here", 502 is "the engine failed", and a stream already under way
|
||||
// cannot be given a status at all.
|
||||
func TestEachWayADispatchEndsIsToldToTheClientAsItShouldBe(t *testing.T) {
|
||||
rt := newTestRouter(t, baseConfig())
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
res dispatchResult
|
||||
status int
|
||||
body bool // an error body was written
|
||||
retryAfter bool
|
||||
}{
|
||||
{"served", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}}, 200, false, false},
|
||||
{"engine refused, relayed", dispatchResult{outcome: proxy.Outcome{Status: 400, Wrote: true}}, 400, false, false},
|
||||
{"client went away", dispatchResult{err: proxy.ErrClientGone}, 499, false, false},
|
||||
{"stream cut short", dispatchResult{outcome: proxy.Outcome{Status: 200, Wrote: true}, err: errors.New("eof")}, 200, false, false},
|
||||
{"stream cut before a status", dispatchResult{outcome: proxy.Outcome{Wrote: true}, err: errors.New("eof")}, 200, false, false},
|
||||
{"still loading", dispatchResult{err: fmt.Errorf("%w: retry shortly", errLoading)}, 503, true, true},
|
||||
{"nothing serves it", dispatchResult{err: fmt.Errorf("%w: no replica", errNoCapacity)}, 503, true, false},
|
||||
{"engine failed", dispatchResult{err: errors.New("connection reset")}, 502, true, false},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
w := httptest.NewRecorder()
|
||||
got := rt.finish(w, &parsedRequest{Model: "m"}, c.res, time.Now())
|
||||
if got != c.status {
|
||||
t.Errorf("status %d, expected %d", got, c.status)
|
||||
}
|
||||
if c.body {
|
||||
if w.Code != c.status {
|
||||
t.Errorf("written %d, expected %d", w.Code, c.status)
|
||||
}
|
||||
var e struct {
|
||||
Error struct {
|
||||
Message string `json:"message"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil || e.Error.Message == "" {
|
||||
t.Errorf("error body %q", w.Body.String())
|
||||
}
|
||||
} else if w.Body.Len() != 0 {
|
||||
t.Errorf("wrote %q over a response that was not ours to write", w.Body.String())
|
||||
}
|
||||
if (w.Header().Get("Retry-After") != "") != c.retryAfter {
|
||||
t.Errorf("Retry-After %q", w.Header().Get("Retry-After"))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUpstreamStatusIsKeptWhenTheStreamBreaks(t *testing.T) {
|
||||
rt := newTestRouter(t, baseConfig())
|
||||
w := httptest.NewRecorder()
|
||||
res := dispatchResult{outcome: proxy.Outcome{Status: http.StatusTooManyRequests, Wrote: true}, err: errors.New("eof")}
|
||||
if got := rt.finish(w, &parsedRequest{Model: "m"}, res, time.Now()); got != 429 {
|
||||
t.Errorf("status %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
|
||||
rt := newTestRouter(t, baseConfig())
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
refuse func(http.ResponseWriter)
|
||||
status int
|
||||
retryAfter string
|
||||
}{
|
||||
{"queue full", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueFull, time.Now())
|
||||
}, 503, ""},
|
||||
{"queued too long", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
|
||||
}, 504, ""},
|
||||
{"shutting down", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
|
||||
}, 503, ""},
|
||||
{"client left the queue", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, context.Canceled, time.Now())
|
||||
}, 200, ""}, // nothing written: nobody is listening
|
||||
{"rate limited", func(w http.ResponseWriter) {
|
||||
rt.rejectLimit(w, &parsedRequest{Model: "m"},
|
||||
&accounting.LimitError{Err: accounting.ErrRateLimited, RetryAfter: 3 * time.Second}, time.Now())
|
||||
}, 429, "3"},
|
||||
{"budget spent", func(w http.ResponseWriter) {
|
||||
rt.rejectLimit(w, &parsedRequest{Model: "m"}, &accounting.LimitError{Err: accounting.ErrBudgetSpent}, time.Now())
|
||||
}, 429, ""},
|
||||
{"not a limit at all", func(w http.ResponseWriter) {
|
||||
rt.rejectLimit(w, &parsedRequest{Model: "m"}, errors.New("store down"), time.Now())
|
||||
}, 500, ""},
|
||||
} {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
w := httptest.NewRecorder()
|
||||
c.refuse(w)
|
||||
if w.Code != c.status {
|
||||
t.Errorf("status %d, expected %d", w.Code, c.status)
|
||||
}
|
||||
if got := w.Header().Get("Retry-After"); got != c.retryAfter {
|
||||
t.Errorf("Retry-After %q, expected %q", got, c.retryAfter)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,9 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
|
||||
@@ -288,9 +291,13 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
// somebody asks what happened to it.
|
||||
rid := requestID(req.Header.Get("X-Request-Id"))
|
||||
w.Header().Set("X-Request-Id", rid)
|
||||
ctx, span := startInference(req, path, rid)
|
||||
defer span.End()
|
||||
req = req.WithContext(ctx)
|
||||
|
||||
parsed, err := r.parse(req, path)
|
||||
if err == nil {
|
||||
span.SetAttributes(attribute.String("llmgw.model", parsed.Model))
|
||||
parsed.RequestID = rid
|
||||
if parsed.Header == nil {
|
||||
parsed.Header = http.Header{}
|
||||
@@ -355,7 +362,9 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
}
|
||||
|
||||
// Step 5 of the lifecycle: "now?" before "where?" (plan §7.1).
|
||||
_, aspan := tracer().Start(req.Context(), "admission")
|
||||
lease, err := r.admit(req.Context(), parsed)
|
||||
endSpan(aspan, err)
|
||||
if err != nil {
|
||||
r.rejectAdmission(w, parsed, err, start)
|
||||
return
|
||||
@@ -381,7 +390,13 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
}
|
||||
|
||||
res := r.dispatch(req.Context(), w, parsed)
|
||||
r.finish(w, parsed, res, start)
|
||||
status := r.finish(w, parsed, res, start)
|
||||
span.SetAttributes(attribute.Int("http.response.status_code", status),
|
||||
attribute.String("llmgw.served_model", res.model),
|
||||
attribute.String("llmgw.endpoint", res.endpointID))
|
||||
if status >= 500 {
|
||||
span.SetStatus(codes.Error, http.StatusText(status))
|
||||
}
|
||||
// After the client has been served. A conversation that failed to
|
||||
// record is a worse outcome than a slow reply, but only just, and the
|
||||
// caller has their answer either way.
|
||||
@@ -413,7 +428,8 @@ func (r *Router) permits(p auth.Principal, requested string) bool {
|
||||
}
|
||||
|
||||
// finish records metrics and, when nothing was written, produces the error body.
|
||||
func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) {
|
||||
// It returns the status the request ended with.
|
||||
func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) int {
|
||||
status := res.outcome.Status
|
||||
switch {
|
||||
case res.err == nil:
|
||||
@@ -447,6 +463,7 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
|
||||
r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
|
||||
r.account(req, res, status, time.Since(start))
|
||||
r.accessLog(req, res, status, time.Since(start))
|
||||
return status
|
||||
}
|
||||
|
||||
// account settles a finished request against its key.
|
||||
|
||||
@@ -9,22 +9,26 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"log/slog"
|
||||
"math"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/account"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
@@ -231,6 +235,11 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
Timeout: cfg.Router.ScrapeTimeout,
|
||||
}, log)
|
||||
|
||||
if !cfg.Auth.Enabled && !loopbackOnly(cfg.Listen) {
|
||||
log.Warn("auth is off and the API listens beyond this machine: anyone who can reach "+
|
||||
cfg.Listen+" can use every model; set auth.enabled, or listen on 127.0.0.1",
|
||||
"listen", cfg.Listen)
|
||||
}
|
||||
if cfg.Auth.Enabled {
|
||||
st, err := storepkg.Open(context.Background(), cfg.Auth.Store)
|
||||
if err != nil {
|
||||
@@ -682,7 +691,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
if len(candidates) == 0 {
|
||||
break
|
||||
}
|
||||
_, sspan := tracer().Start(ctx, "schedule", trace.WithAttributes(
|
||||
attribute.String("llmgw.model", model), attribute.Int("llmgw.candidates", len(candidates))))
|
||||
dec, err := profile.Schedule(sctx, candidates)
|
||||
if err == nil {
|
||||
sspan.SetAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID))
|
||||
}
|
||||
endSpan(sspan, err)
|
||||
if err != nil {
|
||||
last = dispatchResult{err: fmt.Errorf("%w: %v", errNoCapacity, err)}
|
||||
break
|
||||
@@ -696,15 +711,20 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
continue
|
||||
}
|
||||
|
||||
out, perr := r.px.Do(ctx, w, proxy.Request{
|
||||
uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
|
||||
trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
|
||||
attribute.String("url.full", dec.Endpoint.URL+req.Path)))
|
||||
out, perr := r.px.Do(uctx, w, proxy.Request{
|
||||
URL: dec.Endpoint.URL + req.Path,
|
||||
APIKey: dec.Endpoint.APIKey,
|
||||
Body: req.UpstreamBody,
|
||||
Header: req.Header,
|
||||
Header: withTraceparent(uctx, req.Header),
|
||||
Stream: req.Stream,
|
||||
SuppressUsageChunk: req.SuppressUsageChunk,
|
||||
CaptureReply: req.captureBytes(),
|
||||
})
|
||||
uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
|
||||
endSpan(uspan, perr)
|
||||
// A retryable failure is the endpoint's fault; a client disconnect is not.
|
||||
release(perr == nil || errors.Is(perr, proxy.ErrClientGone))
|
||||
|
||||
@@ -764,3 +784,17 @@ func personalLimits(p config.Personal) *storepkg.Limits {
|
||||
}
|
||||
return &l
|
||||
}
|
||||
|
||||
// loopbackOnly reports whether a listen address is reachable from this
|
||||
// machine alone. An empty host (":8080") is every interface.
|
||||
func loopbackOnly(listen string) bool {
|
||||
host, _, err := net.SplitHostPort(listen)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if host == "localhost" {
|
||||
return true
|
||||
}
|
||||
a, err := netip.ParseAddr(host)
|
||||
return err == nil && a.IsLoopback()
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package router
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestOnlyALoopbackListenCountsAsPrivate(t *testing.T) {
|
||||
for addr, want := range map[string]bool{
|
||||
"127.0.0.1:8080": true, "[::1]:8080": true, "localhost:8080": true,
|
||||
":8080": false, "0.0.0.0:8080": false, "10.0.0.140:8080": false, "junk": false,
|
||||
} {
|
||||
if got := loopbackOnly(addr); got != want {
|
||||
t.Errorf("%s: %v", addr, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -36,6 +36,8 @@ type fakeEngine struct {
|
||||
status atomic.Int64
|
||||
// last is the body of the last completion asked for.
|
||||
last atomic.Value
|
||||
// traceparent is the one the last completion came with.
|
||||
traceparent atomic.Value
|
||||
}
|
||||
|
||||
func newFakeEngine(t *testing.T, model string) *fakeEngine {
|
||||
@@ -49,6 +51,7 @@ func newFakeEngine(t *testing.T, model string) *fakeEngine {
|
||||
})
|
||||
mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, r *http.Request) {
|
||||
f.calls.Add(1)
|
||||
f.traceparent.Store(r.Header.Get("traceparent"))
|
||||
if b, err := io.ReadAll(r.Body); err == nil {
|
||||
f.last.Store(b)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
// tracer is looked up per use rather than kept, so a provider installed after
|
||||
// the router was built — as tests do — is the one spans go to.
|
||||
func tracer() trace.Tracer { return otel.Tracer("gitea.mynodes.xyz/ilker/llmgw/internal/router") }
|
||||
|
||||
// startInference opens the span for one inference request, continuing the
|
||||
// caller's trace when it sent a traceparent.
|
||||
func startInference(req *http.Request, path, requestID string) (context.Context, trace.Span) {
|
||||
ctx := otel.GetTextMapPropagator().Extract(req.Context(), propagation.HeaderCarrier(req.Header))
|
||||
return tracer().Start(ctx, "inference "+path,
|
||||
trace.WithSpanKind(trace.SpanKindServer),
|
||||
trace.WithAttributes(
|
||||
attribute.String("llmgw.request_id", requestID),
|
||||
attribute.String("http.route", path),
|
||||
))
|
||||
}
|
||||
|
||||
// endSpan closes a span, marking it failed when err is set.
|
||||
func endSpan(span trace.Span, err error) {
|
||||
if err != nil {
|
||||
span.RecordError(err)
|
||||
span.SetStatus(codes.Error, err.Error())
|
||||
}
|
||||
span.End()
|
||||
}
|
||||
|
||||
// withTraceparent is h with the current span's context in it, for the engine.
|
||||
// A copy: the request's header is sent on every attempt, each its own span.
|
||||
func withTraceparent(ctx context.Context, h http.Header) http.Header {
|
||||
if !trace.SpanContextFromContext(ctx).IsValid() {
|
||||
return h
|
||||
}
|
||||
out := h.Clone()
|
||||
if out == nil {
|
||||
out = http.Header{}
|
||||
}
|
||||
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(out))
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
||||
"go.opentelemetry.io/otel/sdk/trace/tracetest"
|
||||
"go.opentelemetry.io/otel/trace/noop"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
func TestARequestIsTracedIntoTheEngine(t *testing.T) {
|
||||
rec := tracetest.NewSpanRecorder()
|
||||
prevTP, prevProp := otel.GetTracerProvider(), otel.GetTextMapPropagator()
|
||||
otel.SetTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec)))
|
||||
otel.SetTextMapPropagator(propagation.TraceContext{})
|
||||
t.Cleanup(func() { otel.SetTracerProvider(prevTP); otel.SetTextMapPropagator(prevProp) })
|
||||
|
||||
e := newFakeEngine(t, "m")
|
||||
rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
|
||||
// The caller's own trace, continued rather than replaced.
|
||||
const caller = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
|
||||
res := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}},
|
||||
map[string]string{"traceparent": caller, "X-Request-Id": "req-1"})
|
||||
if res.Code != 200 {
|
||||
t.Fatalf("status %d: %s", res.Code, res.Body.String())
|
||||
}
|
||||
|
||||
spans := map[string]sdktrace.ReadOnlySpan{}
|
||||
for _, s := range rec.Ended() {
|
||||
spans[s.Name()] = s
|
||||
}
|
||||
root, ok := spans["inference /v1/chat/completions"]
|
||||
if !ok {
|
||||
t.Fatalf("no inference span among %v", spans)
|
||||
}
|
||||
if got := root.SpanContext().TraceID().String(); got != "4bf92f3577b34da6a3ce929d0e0e4736" {
|
||||
t.Errorf("the caller's trace was not continued: %s", got)
|
||||
}
|
||||
attrs := map[string]string{}
|
||||
for _, a := range root.Attributes() {
|
||||
attrs[string(a.Key)] = a.Value.Emit()
|
||||
}
|
||||
if attrs["llmgw.request_id"] != "req-1" || attrs["llmgw.model"] != "m" ||
|
||||
attrs["http.response.status_code"] != "200" || attrs["llmgw.endpoint"] != "a" {
|
||||
t.Errorf("inference span attributes %v", attrs)
|
||||
}
|
||||
for _, name := range []string{"admission", "schedule", "upstream"} {
|
||||
s, ok := spans[name]
|
||||
if !ok {
|
||||
t.Errorf("no %s span", name)
|
||||
continue
|
||||
}
|
||||
if s.Parent().SpanID() != root.SpanContext().SpanID() {
|
||||
t.Errorf("%s is not under the request's span", name)
|
||||
}
|
||||
}
|
||||
up := spans["upstream"]
|
||||
tp, _ := e.traceparent.Load().(string)
|
||||
if want := "00-" + up.SpanContext().TraceID().String() + "-" + up.SpanContext().SpanID().String() + "-01"; tp != want {
|
||||
t.Errorf("the engine got traceparent %q, expected %q", tp, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithoutTracingTheEngineGetsNoTraceparent(t *testing.T) {
|
||||
// What an unconfigured process has. Set explicitly, since the default
|
||||
// global delegates for good to the first real provider another test set.
|
||||
prevTP, prevProp := otel.GetTracerProvider(), otel.GetTextMapPropagator()
|
||||
otel.SetTracerProvider(noop.NewTracerProvider())
|
||||
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator())
|
||||
t.Cleanup(func() { otel.SetTracerProvider(prevTP); otel.SetTextMapPropagator(prevProp) })
|
||||
e := newFakeEngine(t, "m")
|
||||
rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
|
||||
if res := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil); res.Code != 200 {
|
||||
t.Fatalf("status %d", res.Code)
|
||||
}
|
||||
if tp, _ := e.traceparent.Load().(string); tp != "" {
|
||||
t.Errorf("traceparent %q sent with tracing off", tp)
|
||||
}
|
||||
}
|
||||
@@ -7,11 +7,13 @@ import (
|
||||
"fmt"
|
||||
"html/template"
|
||||
"net/http"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
@@ -67,6 +69,18 @@ type modelUse struct {
|
||||
Name string
|
||||
Mode string // always, ondemand or stopped
|
||||
Replicas int
|
||||
// Min and Max are its scaling as the catalog holds it.
|
||||
Min, Max int
|
||||
// Ready, Wanted and Nodes are how it runs now, for the row itself:
|
||||
// replicas ready of those it should have, and the nodes they are on.
|
||||
// Fleet is false where nothing could be asked, so the row says nothing
|
||||
// rather than "0 of 0".
|
||||
Fleet bool
|
||||
Ready, Wanted int
|
||||
Starting int
|
||||
Nodes []nodeName
|
||||
Parked bool
|
||||
Unplaceable bool
|
||||
// Declared models are read from server.yaml on every start, so a mode
|
||||
// changed here would revert; the page says where to change it instead.
|
||||
Declared bool
|
||||
@@ -83,6 +97,11 @@ type mirrorRow struct {
|
||||
// operator clicked. Nil on a row that is still arriving: a catalog entry
|
||||
// pointing at a partial file is the thing versioned paths exist to prevent.
|
||||
Serve *serveForm
|
||||
// Detail is the card opened beneath the row: what runs it, on which
|
||||
// nodes, and how many replicas it runs.
|
||||
Detail *heldDetail
|
||||
// Open is the row's ?open= value.
|
||||
Open string
|
||||
|
||||
// HasPercent is false when the upstream never sent a length. The bar is
|
||||
// then not drawn at all: a progress bar that has to invent its own total is
|
||||
@@ -100,7 +119,11 @@ type mirrorRow struct {
|
||||
}
|
||||
|
||||
type adminMirrorData struct {
|
||||
Nav Nav
|
||||
Nav Nav
|
||||
// Tab is the part of the page shown: held, explore or fetch. Tabs are the
|
||||
// links to each.
|
||||
Tab string
|
||||
Tabs []mirrorTab
|
||||
Addr string
|
||||
Base string
|
||||
Rows []mirrorRow
|
||||
@@ -270,6 +293,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
return name + " now runs all the time", nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setReplicas))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
|
||||
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
model, err := buildModel(r.FormValue)
|
||||
@@ -361,7 +386,14 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
|
||||
d.HasCatalog = p.catalog != nil
|
||||
p.explore(r, &d)
|
||||
d.HasRegistry = p.registry != nil
|
||||
d.Tab = mirrorTabFor(r, errMsg)
|
||||
// Only the tab shown does its work: the registry is a server across the
|
||||
// internet, and asking it for sixty repositories to show somebody who came
|
||||
// to look at a download's progress made every visit wait on it.
|
||||
if d.Tab == tabExplore {
|
||||
p.explore(r, &d)
|
||||
}
|
||||
|
||||
got, err := m.List(r.Context())
|
||||
if err != nil {
|
||||
@@ -370,6 +402,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
if d.Error == "" {
|
||||
d.Error = err.Error()
|
||||
}
|
||||
d.Tabs = mirrorTabs(d)
|
||||
render(w, "admin_mirror", d)
|
||||
return
|
||||
}
|
||||
@@ -412,12 +445,46 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
d.Rows = append(d.Rows, row)
|
||||
}
|
||||
var views map[string]adminapi.ModelView
|
||||
prefix := ""
|
||||
if p.fleet != nil && len(sources) > 0 {
|
||||
views = map[string]adminapi.ModelView{}
|
||||
for _, v := range p.fleet.Models() {
|
||||
views[v.Name] = v
|
||||
}
|
||||
var ids []string
|
||||
for _, n := range p.fleet.Nodes() {
|
||||
ids = append(ids, n.ID)
|
||||
}
|
||||
prefix = sharedPrefix(ids)
|
||||
}
|
||||
for i := range d.Rows {
|
||||
for _, m := range sources[d.Rows[i].URL] {
|
||||
d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, modelUse{
|
||||
u := modelUse{
|
||||
Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
|
||||
Min: m.Scaling.Min, Max: m.Scaling.Max,
|
||||
Declared: p.catalog.Declared(m.Name),
|
||||
})
|
||||
}
|
||||
if v, ok := views[m.Name]; ok {
|
||||
u.Fleet, u.Ready, u.Wanted = true, v.Ready, v.Desired
|
||||
u.Parked, u.Unplaceable = v.Parked, v.Unplaceable
|
||||
var nodes []string
|
||||
nodes, u.Starting = runningOn(p.fleet.Replicas(m.Name))
|
||||
for _, n := range nodes {
|
||||
u.Nodes = append(u.Nodes, nodeName{Full: n, Short: strings.TrimPrefix(n, prefix)})
|
||||
}
|
||||
}
|
||||
d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, u)
|
||||
}
|
||||
}
|
||||
|
||||
// The one row whose card is open. From the form too, so a change made in
|
||||
// the card comes back with the card still open to show it.
|
||||
open := r.FormValue("open")
|
||||
for i := range d.Rows {
|
||||
d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
|
||||
if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
|
||||
d.Rows[i].Detail = p.heldDetailFor(d.Rows[i])
|
||||
}
|
||||
}
|
||||
|
||||
@@ -462,12 +529,78 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
d.Groups = groupByModel(d.Rows)
|
||||
|
||||
if d.Fetching > 0 {
|
||||
// Only on the list: a reload anywhere else takes with it a search, an
|
||||
// open repository or a URL half typed.
|
||||
if d.Fetching > 0 && d.Tab == tabHeld && open == "" {
|
||||
d.RefreshSec = 5
|
||||
}
|
||||
d.Tabs = mirrorTabs(d)
|
||||
render(w, "admin_mirror", d)
|
||||
}
|
||||
|
||||
const (
|
||||
tabHeld = "held"
|
||||
tabExplore = "explore"
|
||||
tabFetch = "fetch"
|
||||
)
|
||||
|
||||
// mirrorTab is one of the page's tabs.
|
||||
type mirrorTab struct {
|
||||
Label string
|
||||
Count string // beside the label, empty for none
|
||||
Href string
|
||||
Active bool
|
||||
}
|
||||
|
||||
// mirrorTabFor is the tab a request is for.
|
||||
//
|
||||
// Named in the address when it is chosen, and otherwise read from what the
|
||||
// address carries, so links made before the page had tabs — a search, an
|
||||
// open repository — still land where they meant. After a change the list is
|
||||
// shown, where its effect is, unless it was refused: then the form it came
|
||||
// from, to try again.
|
||||
func mirrorTabFor(r *http.Request, errMsg string) string {
|
||||
valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch }
|
||||
if r.Method == http.MethodPost {
|
||||
if t := r.PostFormValue("tab"); errMsg != "" && valid(t) {
|
||||
return t
|
||||
}
|
||||
return tabHeld
|
||||
}
|
||||
q := r.URL.Query()
|
||||
if t := q.Get("tab"); valid(t) {
|
||||
return t
|
||||
}
|
||||
for _, k := range []string{"q", "repo", "view", "task", "size", "x"} {
|
||||
if q.Has(k) {
|
||||
return tabExplore
|
||||
}
|
||||
}
|
||||
return tabHeld
|
||||
}
|
||||
|
||||
func mirrorTabs(d adminMirrorData) []mirrorTab {
|
||||
held := mirrorTab{Label: "Held", Href: "/admin/mirror?tab=held", Active: d.Tab == tabHeld}
|
||||
switch {
|
||||
case d.Unreachable:
|
||||
held.Count = "unreachable"
|
||||
case d.Fetching > 0:
|
||||
held.Count = fmt.Sprintf("%d · %d downloading", d.Present, d.Fetching)
|
||||
default:
|
||||
held.Count = strconv.Itoa(d.Present)
|
||||
}
|
||||
if d.Failed > 0 {
|
||||
held.Count += fmt.Sprintf(" · %d failed", d.Failed)
|
||||
}
|
||||
tabs := []mirrorTab{held}
|
||||
if d.HasRegistry {
|
||||
tabs = append(tabs, mirrorTab{Label: "Explore Hugging Face",
|
||||
Href: "/admin/mirror?tab=explore", Active: d.Tab == tabExplore})
|
||||
}
|
||||
return append(tabs, mirrorTab{Label: "Fetch by URL",
|
||||
Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
|
||||
}
|
||||
|
||||
// mirrorGroup is a family of models the mirror holds — every qwen, every
|
||||
// llama — and their versions.
|
||||
type mirrorGroup struct {
|
||||
@@ -663,3 +796,43 @@ func writeJSONStatus(w http.ResponseWriter, code int, v any) {
|
||||
w.WriteHeader(code)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
// runningOn is the nodes a model's replicas are on, each once and in order,
|
||||
// and how many of its replicas are not ready yet.
|
||||
func runningOn(reps []adminapi.ReplicaView) (nodes []string, starting int) {
|
||||
for _, r := range reps {
|
||||
if r.State != "ready" {
|
||||
starting++
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
if !slices.Contains(nodes, sh.Node) {
|
||||
nodes = append(nodes, sh.Node)
|
||||
}
|
||||
}
|
||||
}
|
||||
slices.Sort(nodes)
|
||||
return nodes, starting
|
||||
}
|
||||
|
||||
// nodeName is a node as a row names it: short, with the full name beside.
|
||||
type nodeName struct{ Short, Full string }
|
||||
|
||||
// sharedPrefix is what every node's name starts with, up to and including a
|
||||
// separator — "llmgw-" of llmgw-node01 and llmgw-node02 — so a row can say
|
||||
// node01 and fit on a line. Nothing when the names share no such part, or
|
||||
// when there is one node and dropping it would leave only a number.
|
||||
func sharedPrefix(ids []string) string {
|
||||
if len(ids) < 2 {
|
||||
return ""
|
||||
}
|
||||
p := ids[0]
|
||||
for _, id := range ids[1:] {
|
||||
for !strings.HasPrefix(id, p) {
|
||||
p = p[:len(p)-1]
|
||||
}
|
||||
}
|
||||
if i := strings.LastIndexAny(p, "-._"); i >= 0 {
|
||||
return p[:i+1]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -341,16 +341,18 @@ func TestAFailedRowOffersRetryRatherThanRemove(t *testing.T) {
|
||||
}
|
||||
|
||||
type fakeRegistry struct {
|
||||
results []hf.Model
|
||||
repo hf.Repo
|
||||
err error
|
||||
asked string
|
||||
query hf.Query
|
||||
results []hf.Model
|
||||
repo hf.Repo
|
||||
err error
|
||||
asked string
|
||||
query hf.Query
|
||||
explored int
|
||||
}
|
||||
|
||||
func (f *fakeRegistry) Explore(_ context.Context, q hf.Query) ([]hf.Model, error) {
|
||||
f.asked = q.Search
|
||||
f.query = q
|
||||
f.explored++
|
||||
return f.results, f.err
|
||||
}
|
||||
|
||||
@@ -441,7 +443,11 @@ func TestWithNoRegistryThereIsNoSearchBox(t *testing.T) {
|
||||
if strings.Contains(body, "Search Hugging Face") {
|
||||
t.Error("the explorer should be absent without a registry")
|
||||
}
|
||||
if !strings.Contains(body, "Fetch a model") {
|
||||
if strings.Contains(body, "Explore Hugging Face") {
|
||||
t.Error("a tab for an explorer that is not there")
|
||||
}
|
||||
fetch := mirrorGet(t, mirrorPage(t, &fakeMirror{}, true), "/admin/mirror?tab=fetch").Body.String()
|
||||
if !strings.Contains(fetch, `name="source" required`) {
|
||||
t.Error("fetching by hand must still be there")
|
||||
}
|
||||
}
|
||||
@@ -464,8 +470,7 @@ func TestAShardedModelIsOneRowAndOneButton(t *testing.T) {
|
||||
"/admin/mirror?repo=Qwen/Qwen2.5-7B-Instruct").Body.String()
|
||||
|
||||
// Fifteen shards is one decision, not fifteen. Four here, and one form.
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
|
||||
// One for the set, one for the manual form at the bottom.
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 1 {
|
||||
t.Errorf("expected the four shards as a single form, found %d forms", n)
|
||||
}
|
||||
if !strings.Contains(body, "4 shards, fetched together") {
|
||||
@@ -534,8 +539,8 @@ func TestAlternativeQuantisationsStaySeparateRows(t *testing.T) {
|
||||
|
||||
// Two GGUFs are alternatives, not halves. Offering "fetch all" here would
|
||||
// spend gigabytes on a copy nobody asked for.
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 3 {
|
||||
t.Errorf("expected a form each plus the manual one, found %d", n)
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
|
||||
t.Errorf("expected a form each, found %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -587,7 +592,7 @@ func TestHowAModelRunsIsChangedWhereItsFileIsShown(t *testing.T) {
|
||||
t.Error("the running model's mode is not shown as chosen")
|
||||
}
|
||||
// Declared in server.yaml: shown, not offered, since it would revert.
|
||||
if !strings.Contains(body, "on demand<br>set in server.yaml") {
|
||||
if !strings.Contains(body, `title="set in server.yaml, which is read again on every start">on demand · yaml</span>`) {
|
||||
t.Error("a declared model should say where its mode is set")
|
||||
}
|
||||
if strings.Count(body, `name="mode" class="mode"`) != 1 {
|
||||
@@ -655,3 +660,53 @@ func privateOK(m *mirror.Manager) *mirror.Manager {
|
||||
m.AllowPrivateSources()
|
||||
return m
|
||||
}
|
||||
|
||||
func TestEachTabDoesOnlyItsOwnWork(t *testing.T) {
|
||||
reg := &fakeRegistry{}
|
||||
mux := mirrorPageWith(t, &fakeMirror{models: []mirror.Status{
|
||||
{Model: "m", Version: "v", File: "m.gguf", State: mirror.Present, BytesDone: 1},
|
||||
{Model: "n", Version: "v", File: "n.gguf", State: mirror.Fetching, BytesDone: 1},
|
||||
}}, reg, true)
|
||||
|
||||
held := mirrorGet(t, mux, "/admin/mirror").Body.String()
|
||||
if reg.explored != 0 {
|
||||
t.Error("the list asked the registry for repositories it does not show")
|
||||
}
|
||||
if !strings.Contains(held, `class="on" aria-current="page">Held <span class="count">1 · 1 downloading</span>`) {
|
||||
t.Errorf("the Held tab should be chosen and count what it holds: %s", held)
|
||||
}
|
||||
if strings.Contains(held, `name="source" required`) || strings.Contains(held, "Search Hugging Face") {
|
||||
t.Error("other tabs' forms are on the list")
|
||||
}
|
||||
if !strings.Contains(held, `http-equiv="refresh"`) {
|
||||
t.Error("the list stopped following a download")
|
||||
}
|
||||
|
||||
explore := mirrorGet(t, mux, "/admin/mirror?tab=explore").Body.String()
|
||||
if reg.explored != 1 || !strings.Contains(explore, "Search Hugging Face") {
|
||||
t.Error("the explorer tab did not explore")
|
||||
}
|
||||
if strings.Contains(explore, `id="held"`) || strings.Contains(explore, `http-equiv="refresh"`) {
|
||||
t.Error("the explorer shows the list, or reloads away a search")
|
||||
}
|
||||
// A link from before there were tabs still lands on the explorer.
|
||||
if old := mirrorGet(t, mux, "/admin/mirror?q=qwen").Body.String(); !strings.Contains(old, "Search Hugging Face") {
|
||||
t.Error("a search address no longer opens the explorer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
|
||||
m := &fakeMirror{fetchErr: errors.New("upstream said 404")}
|
||||
mux := mirrorPage(t, m, true)
|
||||
body := mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
|
||||
"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
|
||||
if !strings.Contains(body, "upstream said 404") || !strings.Contains(body, `name="source" required`) {
|
||||
t.Errorf("a refusal should come back on the form it came from: %s", body)
|
||||
}
|
||||
m.fetchErr = nil
|
||||
body = mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
|
||||
"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
|
||||
if !strings.Contains(body, `id="held"`) {
|
||||
t.Errorf("a started download should be shown where it is followed: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -49,6 +49,9 @@ type serveForm struct {
|
||||
// 3.5 GB for a KV pool. Nothing was wrong with placement — it was never
|
||||
// told what the model needed.
|
||||
MinMemoryGB int
|
||||
// CPUCores is the cores a replica computes on: half the smallest node it
|
||||
// fits on, or zero when no node says how many it has.
|
||||
CPUCores int
|
||||
// MemoryBasis says how MinMemoryGB was arrived at, for the form to show.
|
||||
MemoryBasis string
|
||||
// CtxPool is the engine context pool the form starts with, and the one
|
||||
@@ -152,6 +155,15 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
|
||||
defaultCtxPool, header.BlockCount, kvHeads,
|
||||
float64(int64(1<<30)+weights/20)/(1<<30))
|
||||
}
|
||||
if fleet != nil {
|
||||
var cpus []int
|
||||
for _, n := range fleet.Nodes() {
|
||||
if n.State == "ready" && (n.MemoryGB == 0 || n.MemoryGB >= f.MinMemoryGB) {
|
||||
cpus = append(cpus, n.CPUCount)
|
||||
}
|
||||
}
|
||||
f.CPUCores = catalog.SuggestCores(cpus)
|
||||
}
|
||||
if biggest := biggestNode(fleet); biggest > 0 && f.MinMemoryGB > biggest {
|
||||
f.Warning = strings.TrimSuffix(f.Warning, ".") +
|
||||
fmt.Sprintf("%sthe largest node has %d GB and this needs about %d",
|
||||
@@ -277,6 +289,11 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
return m, err
|
||||
}
|
||||
}
|
||||
if v := strings.TrimSpace(get("cpu_cores")); v != "" {
|
||||
if p.CPUCores, err = positive(v, "the cores"); err != nil {
|
||||
return m, err
|
||||
}
|
||||
}
|
||||
if k, v := strings.TrimSpace(get("selector_key")), strings.TrimSpace(get("selector_value")); k != "" && v != "" {
|
||||
// What keeps a large model off the small nodes. A CPU profile has no
|
||||
// card to be short of, so without this every ready node looks like a
|
||||
|
||||
@@ -45,6 +45,13 @@ type DashboardData struct {
|
||||
|
||||
HasPeople bool
|
||||
|
||||
// NodePrefix is what every node's name starts with, left off in tables
|
||||
// so a row fits on a line; the full name is on hover.
|
||||
NodePrefix string
|
||||
// Columns shown only when some row has something in them: a CPU-only
|
||||
// cluster's nodes table was three columns of dashes wide.
|
||||
AnyGPUs, AnyClasses, AnyFabrics, AnyFallbacks bool
|
||||
|
||||
// Charts are pre-rendered rather than assembled in the template: building
|
||||
// SVG out of template fragments is how a chart ends up unescaped.
|
||||
RequestChart template.HTML
|
||||
@@ -177,6 +184,17 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
}
|
||||
}
|
||||
d.Unplaced = unplaced(src, d.Models)
|
||||
ids := make([]string, 0, len(d.Nodes))
|
||||
for _, n := range d.Nodes {
|
||||
ids = append(ids, n.ID)
|
||||
d.AnyGPUs = d.AnyGPUs || n.GPUTotal > 0
|
||||
d.AnyClasses = d.AnyClasses || len(n.Classes) > 0
|
||||
d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
|
||||
}
|
||||
d.NodePrefix = sharedPrefix(ids)
|
||||
for _, m := range d.Models {
|
||||
d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
|
||||
}
|
||||
d.Summary = src.Summary()
|
||||
if fs, ok := src.(FabricSource); ok {
|
||||
r := fs.Fabrics()
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// HeadroomSource is a fleet that can say how many more replicas of a model
|
||||
// fit now. Optional: without it the card shows what runs and offers no change
|
||||
// it cannot check.
|
||||
type HeadroomSource interface {
|
||||
Headroom(model string, limit int) (adminapi.HeadroomView, bool)
|
||||
}
|
||||
|
||||
// maxReplicas bounds what the form accepts. Not a cluster limit, a typing one:
|
||||
// 300 where 3 was meant would otherwise be taken as a ceiling.
|
||||
const maxReplicas = 64
|
||||
|
||||
// heldDetail is the card opened beneath a held version: what runs it, where,
|
||||
// and room to change it.
|
||||
type heldDetail struct {
|
||||
Models []heldModel
|
||||
}
|
||||
|
||||
// heldModel is one catalog model running a held version.
|
||||
type heldModel struct {
|
||||
modelUse
|
||||
View adminapi.ModelView
|
||||
Replicas []heldReplica
|
||||
Headroom *adminapi.HeadroomView
|
||||
// MostNow is how many replicas could always run, Current plus the room
|
||||
// found; zero when the room was not measured or had no end in sight.
|
||||
MostNow int
|
||||
}
|
||||
|
||||
// heldReplica is one replica, by the nodes it runs on.
|
||||
type heldReplica struct {
|
||||
ID string
|
||||
State string
|
||||
Restart string
|
||||
Age int64
|
||||
Endpoint string
|
||||
Nodes []heldShard
|
||||
}
|
||||
|
||||
type heldShard struct {
|
||||
Node string
|
||||
MemoryGB int // the node's RAM, 0 when unknown
|
||||
CPU float64
|
||||
MemBytes int64
|
||||
}
|
||||
|
||||
// openKey names a row the way ?open= does.
|
||||
func openKey(model, version string) string { return model + ":" + version }
|
||||
|
||||
// heldDetailFor reads, for each model using a row, how it runs and how much
|
||||
// more room it has.
|
||||
func (p *AccountPages) heldDetailFor(row mirrorRow) *heldDetail {
|
||||
d := &heldDetail{}
|
||||
if p.fleet == nil {
|
||||
for _, u := range row.UsedBy {
|
||||
d.Models = append(d.Models, heldModel{modelUse: u})
|
||||
}
|
||||
return d
|
||||
}
|
||||
views := map[string]adminapi.ModelView{}
|
||||
for _, v := range p.fleet.Models() {
|
||||
views[v.Name] = v
|
||||
}
|
||||
memory := map[string]int{}
|
||||
for _, n := range p.fleet.Nodes() {
|
||||
memory[n.ID] = n.MemoryGB
|
||||
}
|
||||
hs, _ := p.fleet.(HeadroomSource)
|
||||
for _, u := range row.UsedBy {
|
||||
hm := heldModel{modelUse: u, View: views[u.Name]}
|
||||
for _, rep := range p.fleet.Replicas(u.Name) {
|
||||
// Built field by field: a ReplicaView carries the engine's key,
|
||||
// which is for the admin listener and never for a page.
|
||||
hr := heldReplica{ID: rep.ID, State: rep.State, Restart: rep.Restart,
|
||||
Age: rep.AgeSec, Endpoint: rep.Endpoint}
|
||||
for _, sh := range rep.Shards {
|
||||
hr.Nodes = append(hr.Nodes, heldShard{Node: sh.Node, MemoryGB: memory[sh.Node],
|
||||
CPU: sh.CPUCores, MemBytes: sh.MemoryBytes})
|
||||
}
|
||||
hm.Replicas = append(hm.Replicas, hr)
|
||||
}
|
||||
if hs != nil {
|
||||
if h, ok := hs.Headroom(u.Name, 16); ok {
|
||||
hm.Headroom = &h
|
||||
if !h.Limited {
|
||||
hm.MostNow = h.Current + h.More
|
||||
}
|
||||
}
|
||||
}
|
||||
d.Models = append(d.Models, hm)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// setReplicas changes how many replicas a model always runs and how far it
|
||||
// may grow, refusing an always-on count the cluster has no room for.
|
||||
//
|
||||
// Refused rather than taken and left short: a minimum placement cannot meet
|
||||
// is a model reported as unplaceable on every pass from then on, which is the
|
||||
// same answer given later and less plainly. A maximum above the room is
|
||||
// accepted and said: it is reached only under load, by which time something
|
||||
// may have made room, and refusing it would make the ceiling depend on what
|
||||
// happened to be running when it was set.
|
||||
func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
name := r.FormValue("model")
|
||||
lo, err1 := strconv.Atoi(strings.TrimSpace(r.FormValue("min")))
|
||||
hi, err2 := strconv.Atoi(strings.TrimSpace(r.FormValue("max")))
|
||||
switch {
|
||||
case name == "":
|
||||
return "", errors.New("which model?")
|
||||
case err1 != nil || err2 != nil:
|
||||
return "", errors.New("replicas are whole numbers")
|
||||
case lo < 0 || hi < lo:
|
||||
return "", fmt.Errorf("always running (%d) cannot be more than the most under load (%d)", lo, hi)
|
||||
case hi > maxReplicas:
|
||||
return "", fmt.Errorf("at most %d replicas", maxReplicas)
|
||||
}
|
||||
current, err := p.currentModel(ctx, name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if p.catalog.Declared(name) {
|
||||
return "", fmt.Errorf("%s is set in server.yaml; change it there", name)
|
||||
}
|
||||
|
||||
note := ""
|
||||
if hs, ok := p.fleet.(HeadroomSource); ok && hi > 0 {
|
||||
h, ok := hs.Headroom(name, hi)
|
||||
if !ok {
|
||||
return "", fmt.Errorf("no model %q", name)
|
||||
}
|
||||
fits := h.Current + h.More
|
||||
if lo > fits && !h.Limited {
|
||||
why := "nothing more fits"
|
||||
if len(h.Blocked) > 0 {
|
||||
why = strings.Join(h.Blocked, "; ")
|
||||
}
|
||||
return "", fmt.Errorf("%s can always run at most %d now (%d running, room for %d more): %s",
|
||||
name, fits, h.Current, h.More, why)
|
||||
}
|
||||
if hi > fits && !h.Limited {
|
||||
note = fmt.Sprintf("; there is room for %d now, so it grows past that only if room is made", fits)
|
||||
}
|
||||
}
|
||||
|
||||
sc := catalog.Scaling{Min: lo, Max: hi, IdleTimeout: current.Scaling.IdleTimeout}
|
||||
if sc.OnDemand() && sc.IdleTimeout <= 0 {
|
||||
sc.IdleTimeout = defaultIdle
|
||||
}
|
||||
if _, err := p.catalog.Scale(ctx, name, sc, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
switch {
|
||||
case hi == 0:
|
||||
return name + " is stopped; its replicas are being taken down", nil
|
||||
case lo == 0:
|
||||
return fmt.Sprintf("%s loads when asked for, up to %d under load%s", name, hi, note), nil
|
||||
case lo == hi:
|
||||
return fmt.Sprintf("%s runs %s", name, plural(lo, "replica", "replicas")), nil
|
||||
}
|
||||
return fmt.Sprintf("%s runs %d, and up to %d under load%s", name, lo, hi, note), nil
|
||||
}
|
||||
|
||||
// currentModel is a model as the catalog holds it now.
|
||||
func (p *AccountPages) currentModel(ctx context.Context, name string) (catalog.Model, error) {
|
||||
sources, err := p.catalog.Sources(ctx)
|
||||
if err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
for _, ms := range sources {
|
||||
for _, m := range ms {
|
||||
if m.Name == name {
|
||||
return m, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return catalog.Model{}, fmt.Errorf("no model %q runs from the mirror", name)
|
||||
}
|
||||
|
||||
// age renders seconds the way the replicas table does.
|
||||
func age(sec int64) string {
|
||||
d := time.Duration(sec) * time.Second
|
||||
switch {
|
||||
case d < time.Minute:
|
||||
return fmt.Sprintf("%ds", sec)
|
||||
case d < time.Hour:
|
||||
return fmt.Sprintf("%dm", int(d.Minutes()))
|
||||
case d < 48*time.Hour:
|
||||
return fmt.Sprintf("%dh%02dm", int(d.Hours()), int(d.Minutes())%60)
|
||||
}
|
||||
return fmt.Sprintf("%dd", int(d.Hours()/24))
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
)
|
||||
|
||||
// roomyFleet runs one replica of "coder" on n2 and has room for `more`.
|
||||
type roomyFleet struct {
|
||||
fakeFleet
|
||||
more int
|
||||
blocked []string
|
||||
asked int
|
||||
}
|
||||
|
||||
func (f *roomyFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{{ID: "n1", MemoryGB: 16}, {ID: "n2", MemoryGB: 32}}
|
||||
}
|
||||
func (f *roomyFleet) Models() []adminapi.ModelView {
|
||||
return []adminapi.ModelView{{Name: "coder", Min: 1, Max: 1, Ready: 1, Total: 1, Desired: 1}}
|
||||
}
|
||||
func (f *roomyFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{{ID: "coder-abc", Model: "coder", State: "ready", AgeSec: 3700,
|
||||
EngineKey: "secret-engine-key",
|
||||
Shards: []adminapi.ShardView{{Node: "n2", CPUCores: 3.5, MemoryBytes: 6 << 30}}}}
|
||||
}
|
||||
func (f *roomyFleet) Headroom(model string, limit int) (adminapi.HeadroomView, bool) {
|
||||
f.asked = limit
|
||||
more := min(f.more, limit)
|
||||
h := adminapi.HeadroomView{Model: model, Current: 1, More: more}
|
||||
for range more {
|
||||
h.Nodes = append(h.Nodes, []string{"n1"})
|
||||
}
|
||||
if more < limit {
|
||||
h.Blocked = f.blocked
|
||||
} else {
|
||||
h.Limited = true
|
||||
}
|
||||
return h, true
|
||||
}
|
||||
|
||||
func detailPage(t *testing.T, fleet *roomyFleet, cat *fakeCatalog) func(string) string {
|
||||
t.Helper()
|
||||
m := &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 5 << 30},
|
||||
}}
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Catalog: cat, Fleet: fleet}, true)
|
||||
return func(path string) string { return mirrorGet(t, mux, path).Body.String() }
|
||||
}
|
||||
|
||||
func coderCatalog() *fakeCatalog {
|
||||
return &fakeCatalog{sources: map[string][]catalog.Model{"http://mirror:8081/coder/v1/c.gguf": {
|
||||
{Name: "coder", Scaling: catalog.Scaling{Min: 1, Max: 1}}}}}
|
||||
}
|
||||
|
||||
func TestAHeldModelOpensToWhereItRunsAndHowMuchRoomIsLeft(t *testing.T) {
|
||||
fleet := &roomyFleet{more: 2, blocked: []string{"n2: 2 GB free of 32, profile needs 6"}}
|
||||
get := detailPage(t, fleet, coderCatalog())
|
||||
|
||||
closed := get("/admin/mirror")
|
||||
if strings.Contains(closed, "held-card") {
|
||||
t.Error("a card is open before anything was clicked")
|
||||
}
|
||||
if !strings.Contains(closed, `href="/admin/mirror?tab=held&open=coder%3av1"`) {
|
||||
t.Error("the model's name does not open its card")
|
||||
}
|
||||
|
||||
body := get("/admin/mirror?tab=held&open=coder:v1")
|
||||
for _, want := range []string{
|
||||
"held-card", `<a href="/admin/nodes/n2">n2</a>`, "32 GB", "coder-abc", "1h01m", "3.5 cores", "6.0 GiB",
|
||||
"Room for 2 more now, on n1, n1.", "n2: 2 GB free of 32, profile needs 6",
|
||||
`name="min" value="1" min="0" max="3"`, `action="/admin/mirror/replicas"`,
|
||||
`name="open" value="coder:v1"`,
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the card is missing %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "secret-engine-key") {
|
||||
t.Error("the engine's key reached the page")
|
||||
}
|
||||
if strings.Contains(body, `http-equiv="refresh"`) {
|
||||
t.Error("a page with an open form reloads under the operator")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaCountTheClusterHasNoRoomForIsRefused(t *testing.T) {
|
||||
fleet := &roomyFleet{more: 1, blocked: []string{"n1: 1 GB free of 16, profile needs 6"}}
|
||||
cat := coderCatalog()
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
|
||||
}}, Catalog: cat, Fleet: fleet}, true)
|
||||
post := func(lo, hi string) string {
|
||||
return mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
|
||||
"model": {"coder"}, "min": {lo}, "max": {hi}, "open": {"coder:v1"}}).Body.String()
|
||||
}
|
||||
|
||||
body := post("3", "3")
|
||||
if len(cat.scaled) != 0 {
|
||||
t.Fatalf("three always-on replicas were set with room for two: %+v", cat.lastScale)
|
||||
}
|
||||
if !strings.Contains(body, "can always run at most 2 now") || !strings.Contains(body, "1 GB free of 16") {
|
||||
t.Errorf("the refusal should say how many fit and why: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "held-card") {
|
||||
t.Error("the card closed on the refusal")
|
||||
}
|
||||
|
||||
body = post("2", "4")
|
||||
if len(cat.scaled) != 1 || cat.lastScale.Min != 2 || cat.lastScale.Max != 4 {
|
||||
t.Fatalf("what fits was not set: %+v", cat.lastScale)
|
||||
}
|
||||
if !strings.Contains(body, "coder runs 2, and up to 4 under load; there is room for 2 now") {
|
||||
t.Errorf("a ceiling above the room should be taken and said: %s", body)
|
||||
}
|
||||
|
||||
post("0", "2")
|
||||
if cat.lastScale.Min != 0 || cat.lastScale.Max != 2 || cat.lastScale.IdleTimeout == 0 {
|
||||
t.Errorf("on demand was set as %+v", cat.lastScale)
|
||||
}
|
||||
for _, bad := range [][2]string{{"3", "2"}, {"-1", "1"}, {"x", "1"}, {"1", "100"}} {
|
||||
n := len(cat.scaled)
|
||||
post(bad[0], bad[1])
|
||||
if len(cat.scaled) != n {
|
||||
t.Errorf("min %s max %s was accepted", bad[0], bad[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelSetInConfigurationIsShownButNotChanged(t *testing.T) {
|
||||
cat := coderCatalog()
|
||||
cat.declared = "coder"
|
||||
body := detailPage(t, &roomyFleet{more: 20}, cat)("/admin/mirror?open=coder:v1")
|
||||
if strings.Contains(body, `action="/admin/mirror/replicas"`) || !strings.Contains(body, "change its replicas there") {
|
||||
t.Error("a declared model offered a change that would revert")
|
||||
}
|
||||
if !strings.Contains(body, "Room for at least 16 more") {
|
||||
t.Error("room that reached the limit asked should read as at least")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHeldRowSaysHowItsModelRunsWithoutOpening(t *testing.T) {
|
||||
body := detailPage(t, &roomyFleet{more: 1}, coderCatalog())("/admin/mirror")
|
||||
if strings.Contains(body, "held-card") {
|
||||
t.Fatal("the card opened by itself")
|
||||
}
|
||||
if !strings.Contains(body, `title="replicas ready / wanted">1/1 · <span class="node" title="n2">n2</span></span>`) {
|
||||
t.Errorf("the row does not say where its replicas run: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunningOnNamesEachNodeOnceAndCountsWhatIsStarting(t *testing.T) {
|
||||
nodes, starting := runningOn([]adminapi.ReplicaView{
|
||||
{State: "ready", Shards: []adminapi.ShardView{{Node: "n2"}}},
|
||||
{State: "starting", Shards: []adminapi.ShardView{{Node: "n1"}}},
|
||||
{State: "ready", Shards: []adminapi.ShardView{{Node: "n2"}}},
|
||||
})
|
||||
if strings.Join(nodes, ",") != "n1,n2" || starting != 1 {
|
||||
t.Errorf("nodes %v, starting %d", nodes, starting)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeNamesLoseOnlyWhatEveryNodeShares(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
ids []string
|
||||
want string
|
||||
}{
|
||||
{[]string{"llmgw-node01", "llmgw-node02", "llmgw-node06"}, "llmgw-"},
|
||||
{[]string{"gpu-a.dc1", "gpu-b.dc1"}, "gpu-"},
|
||||
{[]string{"n1", "n2"}, ""},
|
||||
{[]string{"llmgw-node01"}, ""},
|
||||
{[]string{"alpha", "beta"}, ""},
|
||||
} {
|
||||
if got := sharedPrefix(c.ids); got != c.want {
|
||||
t.Errorf("%v: %q, expected %q", c.ids, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -155,7 +155,7 @@ func oneOf(s string, options ...string) bool {
|
||||
|
||||
// href is the page with these settings and one of them changed.
|
||||
func (e explorer) href(query, key, value string) template.URL {
|
||||
v := url.Values{"view": {e.View}, "task": {e.Task}, "size": {e.Size}, "x": {"1"}}
|
||||
v := url.Values{"tab": {tabExplore}, "view": {e.View}, "task": {e.Task}, "size": {e.Size}, "x": {"1"}}
|
||||
if e.Here {
|
||||
v.Set("here", "1")
|
||||
}
|
||||
|
||||
@@ -27,7 +27,7 @@ func TestTheExplorerOpensOnTrendingModelsTheNodesCanRun(t *testing.T) {
|
||||
ggufModel("odd/Kev-GGUF", "ggmlc", 5e8, now.Add(-2*24*time.Hour)),
|
||||
}}
|
||||
body := mirrorGet(t, mirrorPageLlama(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg}),
|
||||
"/admin/mirror").Body.String()
|
||||
"/admin/mirror?tab=explore").Body.String()
|
||||
|
||||
q := reg.query
|
||||
if q.Sort != hf.SortTrending || q.Task != "text-generation" || !q.GGUF || q.Search != "" {
|
||||
@@ -96,7 +96,7 @@ func TestAnAdapterIsNotListedAsAModel(t *testing.T) {
|
||||
ggufModel("a/Tiny-GGUF", "qwen2", 135e6, time.Now()),
|
||||
}}
|
||||
body := mirrorGet(t, mirrorPageLlama(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg}),
|
||||
"/admin/mirror").Body.String()
|
||||
"/admin/mirror?tab=explore").Body.String()
|
||||
if strings.Contains(body, "Some-LoRA") || !strings.Contains(body, "a/Tiny-GGUF") {
|
||||
t.Errorf("adapters should be left out and small models kept: %s", body)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// cpuFleet is a CPU-only cluster whose nodes share a name prefix.
|
||||
type cpuFleet struct{ fakeSource }
|
||||
|
||||
func (cpuFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{
|
||||
{ID: "llmgw-node01", State: "ready", MgmtAddr: "10.0.0.141", MemoryGB: 8},
|
||||
{ID: "llmgw-node02", State: "ready", MgmtAddr: "10.0.0.142", MemoryGB: 8},
|
||||
}
|
||||
}
|
||||
func (cpuFleet) Summary() adminapi.Summary { return adminapi.Summary{} }
|
||||
func (cpuFleet) Models() []adminapi.ModelView {
|
||||
return []adminapi.ModelView{{Name: "qwen", Min: 1, Max: 1, Ready: 1,
|
||||
Description: "A sentence long enough that it once had a row of its own under the model."}}
|
||||
}
|
||||
func (cpuFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{{ID: "qwen-1", Model: "qwen", State: "ready",
|
||||
Shards: []adminapi.ShardView{{Node: "llmgw-node02"}}}}
|
||||
}
|
||||
|
||||
func TestTheClusterPageKeepsEachRecordOnOneLine(t *testing.T) {
|
||||
h := Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil)
|
||||
_, body := get(t, h, "http://127.0.0.1:9090/", nil)
|
||||
|
||||
for _, gone := range []string{"<th>Classes</th>", "<th>Fabrics</th>", `<th class="num">GPUs</th>`, "<th>Fallbacks</th>"} {
|
||||
if strings.Contains(body, gone) {
|
||||
t.Errorf("%s is shown with nothing in it on any row", gone)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(body, `title="llmgw-node01"><code>node01</code>`) {
|
||||
t.Error("node names should drop what every node shares, keeping the whole on hover")
|
||||
}
|
||||
if !strings.Contains(body, `<code title="llmgw-node02">node02</code>`) {
|
||||
t.Error("the replicas table should name nodes the same way")
|
||||
}
|
||||
if strings.Contains(body, `class="url-row"`) || !strings.Contains(body, `<td class="clip" title="A sentence long enough`) {
|
||||
t.Error("a model's description should be a clipped cell, not a row of its own")
|
||||
}
|
||||
|
||||
// A cluster with GPUs still gets its GPU column.
|
||||
_, gpu := get(t, Dashboard(fakeSource{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if !strings.Contains(gpu, `<th class="num">GPUs</th>`) || !strings.Contains(gpu, "<th>Fabrics</th>") {
|
||||
t.Error("columns with something in them were hidden")
|
||||
}
|
||||
}
|
||||
@@ -60,7 +60,7 @@ li { margin: .25rem 0; }
|
||||
a { color: var(--accent); }
|
||||
.wrap.narrow { max-width: 30rem; }
|
||||
label { display: block; margin: .9rem 0; font-size: .9rem; font-weight: 600; }
|
||||
input[type=text], input[type=email], input[type=password], input[type=search] {
|
||||
input[type=text], input[type=email], input[type=password], input[type=search], input[type=number] {
|
||||
display: block; width: 100%; margin-top: .3rem; font: inherit; font-weight: 400;
|
||||
padding: .5rem .6rem; border: 1px solid var(--line); border-radius: 6px;
|
||||
background: var(--card); color: var(--fg);
|
||||
@@ -444,18 +444,29 @@ h3.limits-for { font-size: .92rem; margin: 1rem 0 .4rem; }
|
||||
table.narrow-table { max-width: 30rem; }
|
||||
.q-act a + a { margin-left: .8rem; }
|
||||
.chip-role { font-size: .68rem; font-weight: 600; color: var(--accent); text-transform: uppercase; letter-spacing: .04em; }
|
||||
/* Runs: one fixed-width selector per model, so they line up down the column
|
||||
whatever the names; a catalog name differing from the mirror's sits above. */
|
||||
table.held .use { width: 9.5rem; margin: .1rem 0; }
|
||||
table.held .use .as { display: block; font-size: .72rem; color: var(--muted);
|
||||
overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
table.held .use .opt { font-size: .75rem; line-height: 1.3; }
|
||||
/* Runs: one line per model — a catalog name that differs from the mirror's,
|
||||
how it runs, and where — with the selector at a fixed width so the
|
||||
replica counts line up down the column. */
|
||||
table.held .use { display: flex; align-items: center; gap: .5rem; margin: .1rem 0; white-space: nowrap; }
|
||||
table.held .use .as { font-size: .72rem; color: var(--muted); max-width: 10rem;
|
||||
overflow: hidden; text-overflow: ellipsis; }
|
||||
table.held .use form.inline { margin: 0; }
|
||||
table.held .use .mode-fixed { width: 7.5rem; font-size: .8rem; color: var(--muted); }
|
||||
table.held td.num { white-space: nowrap; }
|
||||
table.held select.mode { width: 100%; padding: .15rem .3rem; font-size: .8rem; }
|
||||
table.held select.mode { width: 7.5rem; margin: 0; padding: .1rem .3rem; font-size: .8rem; }
|
||||
|
||||
/* Link pills: the same look as .tabs, for choices that are addresses rather
|
||||
than panels on one page — a filtered listing should survive a reload and be
|
||||
worth sending to somebody. */
|
||||
/* A page's own sections, one address each. Underlined rather than pills so
|
||||
* they do not read as one more filter beside the ones inside a section. */
|
||||
.page-tabs { display: flex; flex-wrap: wrap; gap: 0 1.5rem; margin: -1rem 0 1.5rem;
|
||||
border-bottom: 1px solid var(--line); }
|
||||
.page-tabs a { padding: .55rem 0; margin-bottom: -1px; font-weight: 600; color: var(--muted);
|
||||
text-decoration: none; border-bottom: 2px solid transparent; }
|
||||
.page-tabs a:hover { color: var(--fg); }
|
||||
.page-tabs a.on { color: var(--fg); border-bottom-color: var(--accent); }
|
||||
.page-tabs .count { font-weight: 500; font-size: .8rem; color: var(--muted); margin-left: .25rem; }
|
||||
.pills { display: flex; flex-wrap: wrap; gap: .35rem; margin: .5rem 0; }
|
||||
.pills a { font-size: .85rem; font-weight: 600; color: var(--muted); padding: .35rem .85rem;
|
||||
border: 1px solid transparent; border-radius: 999px; text-decoration: none; }
|
||||
@@ -804,3 +815,43 @@ details.chat-settings output { font-variant-numeric: tabular-nums; font-weight:
|
||||
.copyable button { margin-right: .75rem; font-size: .8rem; }
|
||||
.connect-form .field-row label { flex: 1 1 12rem; }
|
||||
table.connect-fields th { white-space: nowrap; }
|
||||
|
||||
/* The card opened beneath a held version on the mirror page. */
|
||||
tr.held-detail > td { background: var(--bg); padding: .75rem 1rem 1rem; }
|
||||
.held-card .held-model { margin-top: .75rem; padding-top: .75rem; border-top: 1px solid var(--line); }
|
||||
.held-card .held-model:first-of-type { border-top: 0; padding-top: 0; }
|
||||
.held-card h3 { font-size: 1rem; margin: 0 0 .5rem; display: flex; gap: .6rem; align-items: baseline; flex-wrap: wrap; }
|
||||
.held-card table.replicas { margin: .25rem 0 .5rem; }
|
||||
.held-card .headroom { margin: .5rem 0 .25rem; font-weight: 600; }
|
||||
.held-card details.blocked { font-size: .85rem; color: var(--muted); margin-bottom: .5rem; }
|
||||
.held-card details.blocked ul { margin: .35rem 0 0 1.1rem; padding: 0; }
|
||||
.held-card .replicas-form .field-row { align-items: flex-end; }
|
||||
.held-card .replicas-form .field-row label { flex: 0 0 auto; }
|
||||
.held-card .replicas-form input[type=number] { width: 8rem; margin: .25rem 0 0; }
|
||||
.held-card .replicas-form button { margin: 0; }
|
||||
a.open-held { text-decoration: none; color: inherit; }
|
||||
a.open-held:hover strong { text-decoration: underline; }
|
||||
a.open-held[aria-expanded=true]::before { content: "▾ "; color: var(--muted); }
|
||||
a.open-held[aria-expanded=false]::before { content: "▸ "; color: var(--muted); }
|
||||
.use .runs-now { font-size: .78rem; color: var(--muted); }
|
||||
.use .runs-now.short { color: var(--warn); }
|
||||
.use .runs-now.bad { color: var(--bad); }
|
||||
.use .runs-now .node { white-space: nowrap; }
|
||||
|
||||
/* One line per record, in every table. A header never wraps; a cell of free
|
||||
* text is cut to its column with the whole of it on hover, instead of a row of
|
||||
* its own or a line break that makes one record look like two. */
|
||||
th { white-space: nowrap; }
|
||||
table.rows td { white-space: nowrap; }
|
||||
td.clip { max-width: 22rem; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
table.rows td.clip { color: var(--muted); font-size: .85rem; }
|
||||
/* A row across the whole table is a note or an open form, not a record: it
|
||||
* wraps as prose does. */
|
||||
table.rows td[colspan] { white-space: normal; }
|
||||
table.rows td.where { max-width: none; }
|
||||
table.rows .more { cursor: help; margin-left: .2rem; }
|
||||
table.explore td:first-child { max-width: 34rem; overflow: hidden; text-overflow: ellipsis; }
|
||||
table.explore .from { margin-left: .5rem; font-size: .78rem; color: var(--muted); }
|
||||
.clip-inline { display: inline-block; max-width: 24rem; overflow: hidden; text-overflow: ellipsis;
|
||||
vertical-align: bottom; font-size: .8rem; color: var(--muted); margin-left: .4rem; }
|
||||
table.rows td.warn { color: var(--warn); cursor: help; }
|
||||
@@ -84,7 +84,7 @@ Tell whoever you asked for access that you have signed up.</p>
|
||||
{{end}}
|
||||
{{if .Fields}}
|
||||
<h3>Settings to fill in</h3>
|
||||
<table class="compact narrow-table connect-fields">{{range .Fields}}
|
||||
<table class="rows compact narrow-table connect-fields">{{range .Fields}}
|
||||
<tr><th>{{.Label}}</th><td><code>{{.Value}}</code></td><td><button type="button" class="link copy" data-value="{{.Value}}">copy</button></td></tr>{{end}}
|
||||
</table>
|
||||
{{end}}
|
||||
@@ -149,7 +149,7 @@ document.addEventListener("click", (e) => {
|
||||
|
||||
<h2>Keys</h2>
|
||||
{{if .Keys}}
|
||||
<table><tr><th>Name</th><th>Key</th><th>Spends against</th><th>Created</th><th>Last used</th><th>State</th><th></th></tr>
|
||||
<table class="rows"><tr><th>Name</th><th>Key</th><th>Spends against</th><th>Created</th><th>Last used</th><th>State</th><th></th></tr>
|
||||
{{range .Keys}}<tr>
|
||||
<td>{{.Name}}</td>
|
||||
<td><code>{{.Prefix}}</code></td>
|
||||
@@ -192,7 +192,7 @@ old one to stop moving, then revoke it.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
{{if .Rows}}
|
||||
<table class="compact narrow-table"><tr><th>Limit</th><th class="num">Value</th></tr>
|
||||
<table class="rows compact narrow-table"><tr><th>Limit</th><th class="num">Value</th></tr>
|
||||
{{range .Rows}}<tr><td>{{.Name}}</td><td class="num">{{.Value}}</td></tr>{{end}}</table>
|
||||
{{else}}<p class="empty">No limits here.</p>{{end}}
|
||||
{{else}}<p class="empty">Nowhere to spend yet.</p>{{end}}
|
||||
|
||||
@@ -8,16 +8,20 @@
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>Model mirror</h1>
|
||||
<p class="sub">
|
||||
{{if .Unreachable}}{{.Addr}} did not answer{{else}}{{.Present}} held ·
|
||||
{{.Fetching}} downloading · {{.Failed}} failed · {{.Addr}}{{end}}
|
||||
</p>
|
||||
<p class="sub">{{.Addr}}{{if .Unreachable}} did not answer{{end}}</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
|
||||
{{/* Links rather than a script's panels: each tab has an address, the back
|
||||
button works, and only the one shown asks anything of the registry. */}}
|
||||
<nav class="page-tabs" aria-label="Mirror">
|
||||
{{range .Tabs}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}{{with .Count}} <span class="count">{{.}}</span>{{end}}</a>
|
||||
{{end}}</nav>
|
||||
|
||||
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
|
||||
{{if .Message}}<p class="ok">{{.Message}}</p>{{end}}
|
||||
|
||||
{{if eq .Tab "held"}}
|
||||
{{if .Unreachable}}
|
||||
{{/* Said outright rather than shown as an empty table. A mirror holding
|
||||
nothing and a mirror that is not there look the same in a list of zero
|
||||
@@ -29,9 +33,8 @@ reachable from this machine.</p>
|
||||
{{else}}
|
||||
|
||||
<div id="held">
|
||||
<h2>What the mirror holds</h2>
|
||||
{{if .Rows}}
|
||||
<table class="compact held">
|
||||
<table class="rows compact held">
|
||||
<tr><th>Model</th><th>Files</th><th class="num">Size</th><th>Runs</th><th></th></tr>
|
||||
{{range .Groups}}{{$grouped := gt .Models 1}}
|
||||
{{if $grouped}}
|
||||
@@ -41,8 +44,10 @@ reachable from this machine.</p>
|
||||
{{if .Failed}}<span class="tag bad">failed</span>{{end}}
|
||||
{{if .InUse}}<span class="tag">in use</span>{{end}}</td></tr>
|
||||
{{end}}
|
||||
{{range .Rows}}<tr{{if $grouped}} class="model-version"{{end}}>
|
||||
<td><strong>{{.Model}}</strong> <span class="opt">{{.Version}}</span></td>
|
||||
{{range .Rows}}{{$row0 := .}}<tr{{if $grouped}} class="model-version"{{end}}{{if .Detail}} class="open"{{end}}>
|
||||
<td>{{if eq (printf "%s" .State) "present"}}<a class="open-held" aria-expanded="{{if .Detail}}true{{else}}false{{end}}"
|
||||
href="/admin/mirror?tab=held{{if not .Detail}}&open={{.Open}}{{end}}"><strong>{{.Model}}</strong></a>
|
||||
{{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span></td>
|
||||
<td class="file" title="{{.File}}{{if .SHA256}} · sha256 {{.SHA256}}{{end}}">
|
||||
{{if gt .Files 1}}{{.Files}} files{{else}}<code>{{.File}}</code>{{end}}</td>
|
||||
<td class="num">
|
||||
@@ -62,7 +67,8 @@ reachable from this machine.</p>
|
||||
{{/* Named only when the catalog calls it something other than the
|
||||
mirror does; otherwise it is the model column again. */}}
|
||||
{{if ne .Name $row.Model}}<span class="as" title="{{.Name}}">as {{.Name}}</span>{{end}}
|
||||
{{if .Declared}}<span class="opt">{{if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}}<br>set in server.yaml</span>
|
||||
{{if .Declared}}<span class="mode-fixed" title="set in server.yaml, which is read again on every start">
|
||||
{{- if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}} · yaml</span>
|
||||
{{else}}
|
||||
{{/* Submitted on change: the choice is the action, and a second click
|
||||
to confirm it would only be a way to forget to. */}}
|
||||
@@ -78,6 +84,12 @@ reachable from this machine.</p>
|
||||
<noscript><button type="submit" class="link">set</button></noscript>
|
||||
</form>
|
||||
{{end}}
|
||||
{{if .Fleet}}<span class="runs-now{{if or .Parked .Unplaceable}} bad{{else if lt .Ready .Wanted}} short{{end}}" title="replicas ready / wanted{{if .Starting}}, + starting{{end}}">
|
||||
{{- if .Parked}}parked
|
||||
{{- else if and (eq .Wanted 0) (eq .Ready 0)}}none running
|
||||
{{- else}}{{.Ready}}/{{.Wanted}}{{if .Starting}} +{{.Starting}}{{end}}{{end}}
|
||||
{{- if .Unplaceable}} · no room{{end}}
|
||||
{{- with .Nodes}} · {{range $i, $n := .}}{{if $i}}, {{end}}<span class="node" title="{{$n.Full}}">{{$n.Short}}</span>{{end}}{{end}}</span>{{end}}
|
||||
</div>{{end}}
|
||||
{{else}}<span class="opt">not served</span>{{end}}
|
||||
</td>
|
||||
@@ -109,6 +121,7 @@ reachable from this machine.</p>
|
||||
</td>
|
||||
</tr>
|
||||
{{if .Err}}<tr><td colspan="5" class="error-row">{{.Err}}</td></tr>{{end}}
|
||||
{{with .Detail}}<tr class="held-detail"><td colspan="5">{{template "mirror_held_detail" dict "D" . "Row" $row0 "CSRF" $.CSRF}}</td></tr>{{end}}
|
||||
|
||||
{{with .Serve}}
|
||||
<tr class="editing"><td colspan="5">
|
||||
@@ -152,7 +165,9 @@ reachable from this machine.</p>
|
||||
<div class="field-row">
|
||||
<label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
|
||||
<label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
|
||||
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}"></label>
|
||||
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}" required></label>
|
||||
<label>CPU cores <span class="opt">reserved; the engine's threads</span>
|
||||
<input type="text" name="cpu_cores" value="{{.CPUCores}}"></label>
|
||||
<label>Only nodes labelled <span class="opt">key / value, optional</span>
|
||||
<input type="text" name="selector_key" placeholder="role" size="8">
|
||||
<input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
|
||||
@@ -190,11 +205,14 @@ document.addEventListener('click', function (e) {
|
||||
} else { window.prompt('Address', b.dataset.url); }
|
||||
});
|
||||
</script>
|
||||
{{else}}<p class="empty">The mirror is reachable and holds nothing yet.</p>{{end}}
|
||||
{{else}}<p class="empty">The mirror is reachable and holds nothing yet.
|
||||
{{if .HasRegistry}}<a href="/admin/mirror?tab=explore">Explore Hugging Face</a> for a model, or
|
||||
{{end}}<a href="/admin/mirror?tab=fetch">fetch one by URL</a>.</p>{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if .HasRegistry}}
|
||||
<h2>Explore models</h2>
|
||||
{{if and (eq .Tab "explore") .HasRegistry}}
|
||||
{{if not .Repo}}{{with .Explore}}
|
||||
<nav class="pills" aria-label="Which models">{{range .Views}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}</a>{{end}}</nav>
|
||||
<div class="explore-filters">
|
||||
@@ -203,6 +221,7 @@ document.addEventListener('click', function (e) {
|
||||
</div>
|
||||
{{end}}
|
||||
<form method="get" action="/admin/mirror" class="explore-search">
|
||||
<input type="hidden" name="tab" value="explore">
|
||||
{{with .Explore}}<input type="hidden" name="view" value="{{.View}}">
|
||||
<input type="hidden" name="task" value="{{.Task}}">
|
||||
<input type="hidden" name="size" value="{{.Size}}">
|
||||
@@ -222,13 +241,13 @@ document.addEventListener('click', function (e) {
|
||||
|
||||
{{if not .Repo}}{{with .Explore}}
|
||||
{{if .Rows}}
|
||||
<table class="explore">
|
||||
<table class="rows explore">
|
||||
<tr><th>Repository</th><th class="num">Size</th><th>Architecture</th><th class="num">Context</th>
|
||||
<th class="num">Downloads</th><th class="num">Likes</th><th class="num">Updated</th></tr>
|
||||
{{range .Rows}}<tr class="explore-row">
|
||||
<td><a href="{{.Href}}" class="open-repo" data-repo="{{.ID}}" aria-expanded="false"><code>{{.ID}}</code></a>
|
||||
{{if .New}}<span class="tag new">new</span>{{end}}{{if .Gated.Is}}<span class="tag">gated</span>{{end}}
|
||||
{{if or .BaseModel .License}}<div class="from">{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</div>{{end}}</td>
|
||||
{{if or .BaseModel .License}}<span class="from" title="{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}">{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</span>{{end}}</td>
|
||||
<td class="num">{{.ParamsLabel}}</td>
|
||||
<td>{{if .Architecture}}<span{{if not .Verdict.Servable}} class="cannot"{{end}} title="{{.Verdict.Reason}}">{{.Architecture}}{{if .Verdict.Checked}}{{if .Verdict.Servable}} ✓{{else}} ✗{{end}}{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{.ContextLabel}}</td>
|
||||
@@ -307,9 +326,12 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
return;
|
||||
}
|
||||
note.classList.remove("bad");
|
||||
note.textContent = "✓ " + out.message + " — its progress is in the table above.";
|
||||
note.textContent = "✓ " + out.message + " — ";
|
||||
const link = document.createElement("a");
|
||||
link.href = "/admin/mirror?tab=held";
|
||||
link.textContent = "follow it under Held";
|
||||
note.appendChild(link);
|
||||
btn.textContent = "Started";
|
||||
followHeld();
|
||||
} catch (err) {
|
||||
note.textContent = "Could not reach the gateway: " + err;
|
||||
note.classList.add("bad");
|
||||
@@ -317,28 +339,6 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
btn.textContent = was;
|
||||
}
|
||||
});
|
||||
|
||||
// Refreshes only the mirror's own table, every five seconds while anything
|
||||
// in it is downloading.
|
||||
let following = false;
|
||||
async function followHeld() {
|
||||
if (following) return;
|
||||
following = true;
|
||||
try {
|
||||
for (;;) {
|
||||
const r = await fetch(location.pathname + location.search, { headers: { Accept: "text/html" } });
|
||||
if (!r.ok) break;
|
||||
const doc = new DOMParser().parseFromString(await r.text(), "text/html");
|
||||
const fresh = doc.getElementById("held"), held = document.getElementById("held");
|
||||
if (!fresh || !held) break;
|
||||
held.replaceWith(fresh);
|
||||
if (!fresh.querySelector(".state-starting")) break;
|
||||
await new Promise((ok) => setTimeout(ok, 5000));
|
||||
}
|
||||
} finally {
|
||||
following = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
{{if .Hidden}}<p class="note">{{.Hidden}} more {{if eq .Hidden 1}}is{{else}}are{{end}} hidden: their architecture is one the nodes' llama.cpp cannot load.
|
||||
Untick “only what these nodes can run” to see them.</p>{{end}}
|
||||
@@ -354,10 +354,11 @@ directory, or give the mirror a token.</p>
|
||||
{{if .Repo}}{{template "mirror_repo" .}}{{end}}
|
||||
{{end}}
|
||||
|
||||
<h2>Fetch a model</h2>
|
||||
{{if eq .Tab "fetch"}}
|
||||
<p class="note">By hand, for anything the search does not reach.</p>
|
||||
<form method="post" action="/admin/mirror/fetch">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<input type="hidden" name="tab" value="fetch">
|
||||
<label>Source URL
|
||||
<input type="text" name="source" required
|
||||
placeholder="https://huggingface.co/…/resolve/main/model.gguf"></label>
|
||||
@@ -373,7 +374,7 @@ directory, or give the mirror a token.</p>
|
||||
</form>
|
||||
<p class="note">The mirror answers as soon as the download is under way, not when
|
||||
it finishes — a model is tens of gigabytes and nothing here holds a connection
|
||||
open for that. The row above will show it arriving. A version is never
|
||||
open for that. The Held tab shows it arriving. A version is never
|
||||
overwritten: fetching into one that exists is refused, so a new build is a new
|
||||
version and the old weights stay where they are.</p>
|
||||
{{end}}
|
||||
@@ -400,7 +401,7 @@ version and the old weights stay where they are.</p>
|
||||
</details>
|
||||
{{end}}
|
||||
{{if .RepoSets}}
|
||||
<table class="repo-files">
|
||||
<table class="rows repo-files">
|
||||
<tr><th>What</th><th class="num">Size</th><th>Published checksum</th><th></th></tr>
|
||||
{{range .RepoSets}}<tr>
|
||||
<td><code>{{.Name}}</code>
|
||||
@@ -410,11 +411,12 @@ version and the old weights stay where they are.</p>
|
||||
<td>
|
||||
{{if not .Verdict.Servable}}
|
||||
<span class="state state-failed" title="{{.Verdict.Reason}}">cannot be served</span>
|
||||
<div class="note">{{.Verdict.Reason}}</div>
|
||||
<span class="clip-inline" title="{{.Verdict.Reason}}">{{.Verdict.Reason}}</span>
|
||||
{{else}}
|
||||
{{if .Verdict.Architecture}}<span class="opt" title="{{.Verdict.Reason}}">{{.Verdict.Architecture}}{{if .Verdict.Checked}} ✓{{end}}</span>{{end}}
|
||||
<form method="post" action="/admin/mirror/fetch" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="tab" value="explore">
|
||||
{{range .Sources}}<input type="hidden" name="source" value="{{.URL}}">
|
||||
<input type="hidden" name="file" value="{{.Path}}">
|
||||
{{end}}
|
||||
@@ -439,3 +441,73 @@ computes once it has it.</p>
|
||||
<p class="empty">No weight files in {{.Repo.ID}}.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{/* The card beneath a held version: what runs it, where, and how many. */}}
|
||||
{{define "mirror_held_detail"}}{{$row := .Row}}{{$csrf := .CSRF}}
|
||||
<div class="held-card">
|
||||
<p class="repo-meta">
|
||||
<span>{{if gt $row.Files 1}}{{$row.Files}} files{{else}}<code>{{$row.File}}</code>{{end}}</span>
|
||||
<span>{{bytes $row.BytesDone}}</span>
|
||||
{{with $row.SHA256}}<span>sha256 <code title="{{.}}">{{slice . 0 12}}</code></span>{{end}}
|
||||
<span><code>{{$row.URL}}</code></span>
|
||||
</p>
|
||||
{{if not .D.Models}}
|
||||
<p class="note">Not in the catalog: nothing runs this version.
|
||||
{{if $row.Serve}}{{else}}Use <strong>serve</strong> on its row to put it into service.{{end}}</p>
|
||||
{{end}}
|
||||
{{range .D.Models}}
|
||||
<section class="held-model">
|
||||
<h3>{{.Name}}
|
||||
{{with .View}}<span class="opt">{{.Ready}} of {{.Desired}} ready{{if ne .Total .Ready}} · {{.Total}} in all{{end}}</span>
|
||||
{{if .Parked}}<span class="tag bad">parked</span>{{else if .Unplaceable}}<span class="tag bad">does not fit</span>{{end}}{{end}}
|
||||
</h3>
|
||||
{{with .View.Reason}}<p class="note error-row">{{.}}</p>{{end}}
|
||||
|
||||
{{if .Replicas}}
|
||||
<table class="rows compact replicas">
|
||||
<tr><th>Replica</th><th>Node</th><th>State</th><th class="num">Age</th><th class="num">CPU</th><th class="num">Memory</th></tr>
|
||||
{{range .Replicas}}{{$rep := .}}{{range $i, $s := .Nodes}}<tr>
|
||||
<td>{{if eq $i 0}}<code>{{$rep.ID}}</code>{{end}}</td>
|
||||
<td><a href="/admin/nodes/{{$s.Node}}">{{$s.Node}}</a>{{if $s.MemoryGB}} <span class="opt">{{$s.MemoryGB}} GB</span>{{end}}</td>
|
||||
<td>{{if eq $i 0}}<span class="state state-{{if eq $rep.State "ready"}}ready{{else if eq $rep.State "failed"}}failed{{else}}starting{{end}}">{{$rep.State}}</span>
|
||||
{{with $rep.Restart}}<span class="opt">restart {{.}}</span>{{end}}{{end}}</td>
|
||||
<td class="num">{{if eq $i 0}}{{age $rep.Age}}{{end}}</td>
|
||||
<td class="num">{{if $s.CPU}}{{printf "%.1f" $s.CPU}} cores{{else}}—{{end}}</td>
|
||||
<td class="num">{{if $s.MemBytes}}{{bytes $s.MemBytes}}{{else}}—{{end}}</td>
|
||||
</tr>{{end}}{{end}}
|
||||
</table>
|
||||
{{else}}<p class="note">No replica runs now{{if eq .Mode "ondemand"}}: it starts when somebody asks for it{{end}}.</p>{{end}}
|
||||
|
||||
{{with .Headroom}}
|
||||
<p class="headroom">
|
||||
{{if .Limited}}Room for at least {{.More}} more replicas.
|
||||
{{else if .More}}Room for {{.More}} more now{{if .Nodes}}, on {{range $i, $n := .Nodes}}{{if $i}}, {{end}}{{join $n}}{{end}}{{end}}.
|
||||
{{else}}No room for another replica now.{{end}}
|
||||
</p>
|
||||
{{if and (not .Limited) .Blocked}}<details class="blocked"><summary>Why not {{if .More}}more{{else}}another{{end}}</summary>
|
||||
<ul>{{range .Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if .Declared}}
|
||||
<p class="note">Set in server.yaml, which is read again on every start: change its replicas there.</p>
|
||||
{{else}}
|
||||
<form method="post" action="/admin/mirror/replicas" class="replicas-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
<div class="field-row">
|
||||
<label>Always running
|
||||
<input type="number" name="min" value="{{.Min}}" min="0"{{if .MostNow}} max="{{.MostNow}}"{{end}} required></label>
|
||||
<label>Up to, under load
|
||||
<input type="number" name="max" value="{{.Max}}" min="0" max="64" required></label>
|
||||
<button type="submit">Update</button>
|
||||
</div>
|
||||
<p class="note">{{if .MostNow}}At most {{.MostNow}} can always run with what is free now. {{end}}0 always running with a
|
||||
higher limit loads it when asked for; 0 and 0 stops it. A replica is added under load when requests wait, and
|
||||
taken away after ten quiet minutes.</p>
|
||||
</form>
|
||||
{{end}}
|
||||
</section>
|
||||
{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
@@ -34,7 +34,7 @@ budgets bind.</p>
|
||||
{{/* The tree is the interface. A quota belongs to a branch, so the form to set
|
||||
one opens on that branch rather than asking somebody to name it again in a
|
||||
box at the bottom. */}}
|
||||
<table class="quota-tree">
|
||||
<table class="rows quota-tree">
|
||||
<tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
|
||||
<th class="num">Budget</th><th>Spent</th><th></th></tr>
|
||||
{{range .Rows}}
|
||||
@@ -119,7 +119,7 @@ up — hover it to see where. A value set here is in normal type.</p>
|
||||
<p class="note">A person's own limit covers everything they do, across all their
|
||||
keys, projects and the chat page. {{.Personal}}</p>
|
||||
{{if .People}}
|
||||
<table class="quota-tree people-limits">
|
||||
<table class="rows quota-tree people-limits">
|
||||
<tr><th>Person</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
|
||||
<th class="num">Budget</th><th>Spent</th><th></th></tr>
|
||||
{{range .People}}
|
||||
|
||||
@@ -47,7 +47,7 @@
|
||||
</div>
|
||||
|
||||
{{if .Rows}}
|
||||
<table>
|
||||
<table class="rows">
|
||||
<tr><th>Group</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
|
||||
{{range .Rows}}
|
||||
<tr>
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
|
||||
{{if .Pending}}
|
||||
<h2>Waiting for approval</h2>
|
||||
<table><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Approve, and put them in</th><th></th></tr>
|
||||
<table class="rows"><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Approve, and put them in</th><th></th></tr>
|
||||
{{range .Pending}}<tr>
|
||||
<td>{{.Email}}</td><td>{{.Name}}</td><td>{{ago .CreatedAt}}</td>
|
||||
<td>
|
||||
@@ -149,14 +149,14 @@ project in the team — an account can only be approved into a project.</p>
|
||||
</div>
|
||||
|
||||
{{with index $.Members (printf "project:%s" $p.ID)}}
|
||||
<table class="compact"><tr><th>Member</th><th>State</th><th>Last sign-in</th><th></th></tr>
|
||||
<table class="rows compact"><tr><th>Member</th><th>State</th><th>Last sign-in</th><th></th></tr>
|
||||
{{range .}}{{template "member-row" (dict "U" . "D" $ "Scope" (printf "project:%s" $p.ID) "Open" $p.ID)}}{{end}}
|
||||
</table>
|
||||
{{else}}<p class="empty">No members of its own. Members of its team or organization can use it too.</p>{{end}}
|
||||
{{template "add-member" (dict "Kind" "project" "ID" $p.ID "D" $ "Open" $p.ID)}}
|
||||
|
||||
{{with index $.Keys $p.ID}}
|
||||
<table class="compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
|
||||
<table class="rows compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
|
||||
{{range .}}<tr>
|
||||
<td>{{.Name}} <code>{{.Prefix}}</code></td>
|
||||
<td>{{or .OwnerEmail "issued by an operator"}}</td>
|
||||
@@ -210,7 +210,7 @@ project in the team — an account can only be approved into a project.</p>
|
||||
|
||||
<h2 id="accounts">All accounts</h2>
|
||||
{{if .Accounts}}
|
||||
<table id="account-list"><tr><th>Person</th><th>State</th><th>Member of</th><th>Last sign-in</th><th>Add to</th><th></th></tr>
|
||||
<table class="rows" id="account-list"><tr><th>Person</th><th>State</th><th>Member of</th><th>Last sign-in</th><th>Add to</th><th></th></tr>
|
||||
{{range .Accounts}}{{template "account-row" (dict "U" . "D" $)}}{{end}}
|
||||
</table>
|
||||
<p class="note">Someone who is a member of nothing uses the gateway on their own:
|
||||
@@ -298,7 +298,7 @@ access than it had, so they need a shell on the control plane —
|
||||
<td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
|
||||
<td class="state state-{{$u.State}}">{{$u.State}}</td>
|
||||
<td class="where">
|
||||
{{range $u.Memberships}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
|
||||
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
|
||||
{{if eq (printf "%s" .Scope) "org"}}
|
||||
<form method="post" action="/admin/users/{{$u.ID}}/role" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
|
||||
@@ -313,8 +313,12 @@ access than it had, so they need a shell on the control plane —
|
||||
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
|
||||
<input type="hidden" name="scope" value="{{.Scope}}:{{.ScopeID}}">
|
||||
<button type="submit" class="chip-x" aria-label="Take out of {{.Path}}" title="Take out of {{.Path}}">×</button>
|
||||
</form></span>
|
||||
</form></span>{{end}}{{end}}
|
||||
{{else}}<span class="opt">on their own</span>{{end}}
|
||||
{{/* Two, then a count: a person in six teams was six chips wrapping the
|
||||
row into a paragraph. The rest are named on hover and in their own
|
||||
scopes' member lists, where taking them out is done. */}}
|
||||
{{if gt (len $u.Memberships) 2}}<span class="opt more" title="{{range $i, $m := $u.Memberships}}{{if ge $i 2}}{{$m.Path}} {{end}}{{end}}">+{{sub (len $u.Memberships) 2}}</span>{{end}}
|
||||
</td>
|
||||
<td>{{ago $u.LastLoginAt}}</td>
|
||||
<td>
|
||||
@@ -365,7 +369,7 @@ access than it had, so they need a shell on the control plane —
|
||||
<details class="edit"><summary>{{len (index $d.Members $k)}} member{{if ne (len (index $d.Members $k)) 1}}s{{end}}</summary>
|
||||
<div class="scope-members">
|
||||
{{with index $d.Members $k}}
|
||||
<table class="compact">
|
||||
<table class="rows compact">
|
||||
{{range .}}{{template "member-row" (dict "U" . "D" $d "Scope" $k "Open" "")}}{{end}}
|
||||
</table>
|
||||
{{else}}<p class="note">Nobody is a member of {{.Name}} itself; a member here could use every project in it.</p>{{end}}
|
||||
|
||||
@@ -51,23 +51,23 @@
|
||||
|
||||
<h2>Nodes</h2>
|
||||
{{if .Nodes}}
|
||||
<table class="nodes"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th><th class="num">GPUs</th><th class="num">Disk</th><th>Engines</th><th>Classes</th><th>Fabrics</th><th class="num">Models</th><th class="num">Replicas</th><th>Last seen</th></tr>
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
{{range .Nodes}}<tr>
|
||||
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}"><code>{{.ID}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
|
||||
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
|
||||
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
|
||||
<td class="state state-{{.State}}">{{.State}}</td>
|
||||
{{/* A dash, not an empty bar, when the agent has not measured it lately:
|
||||
unknown is not idle. */}}
|
||||
<td class="load">{{with .Load}}{{template "minibar" pctbar .CPUPercent}}<span class="load-v">{{printf "%.0f" .CPUPercent}}%{{if .CPUs}} <span class="note">of {{.CPUs}}</span>{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="load">{{with .Load}}{{template "minibar" pctbar .MemPercent}}<span class="load-v">{{bytes .MemUsed}} <span class="note">of {{bytes .MemTotal}}</span></span>{{else}}{{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}}{{end}}</td>
|
||||
<td class="num">{{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}</td>
|
||||
{{if $.AnyGPUs}}<td class="num">{{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}</td>{{end}}
|
||||
{{/* Free out of total, because the first is meaningless without the
|
||||
second, and an em dash where nothing was reported: an agent that
|
||||
was never asked about its disk must not read as a full one. */}}
|
||||
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
|
||||
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
|
||||
<td>{{join .Classes}}</td>
|
||||
<td>{{join .Fabrics}}</td>
|
||||
{{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
|
||||
{{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
|
||||
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
|
||||
<td class="num">{{.Replicas}}</td>
|
||||
<td>{{or .LastSeenAgo "—"}}</td>
|
||||
@@ -80,18 +80,20 @@
|
||||
<h2>Models</h2>
|
||||
|
||||
{{if .Models}}
|
||||
<table><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Profiles</th><th>Fallbacks</th></tr>
|
||||
<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
|
||||
{{range .Models}}<tr>
|
||||
<td><code>{{.Name}}</code></td>
|
||||
<td>{{join .Aliases}}</td>
|
||||
<td class="num">{{.Ready}}</td>
|
||||
<td class="num">{{.Min}}–{{.Max}}</td>
|
||||
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
|
||||
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
|
||||
<td>{{join .Profiles}}</td>
|
||||
<td>{{join .Fallbacks}}</td>
|
||||
{{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
|
||||
{{/* Cut to the column, the whole sentence on hover: a row of its own for
|
||||
each description made the table twice as long as the catalog. */}}
|
||||
<td class="clip"{{with .Description}} title="{{.}}"{{end}}>{{or .Description "—"}}</td>
|
||||
</tr>
|
||||
{{/* Under the row rather than in a column: a sentence in a table cell either
|
||||
wraps every other column into a stripe or gets cut off. */}}
|
||||
{{if .Description}}<tr><td colspan="6" class="url-row">{{.Description}}</td></tr>{{end}}
|
||||
{{end}}</table>
|
||||
{{else}}<p class="empty">The catalog is empty.</p>{{end}}
|
||||
|
||||
@@ -106,7 +108,7 @@
|
||||
|
||||
<h2>Replicas</h2>
|
||||
{{if .Replicas}}
|
||||
<table><tr><th>Replica</th><th>Model</th><th>State</th><th>Pool</th><th>Endpoint</th><th class="num">Age</th><th>Nodes</th></tr>
|
||||
<table class="rows"><tr><th>Replica</th><th>Model</th><th>State</th><th>Pool</th><th>Endpoint</th><th class="num">Age</th><th>Nodes</th></tr>
|
||||
{{range .Replicas}}<tr>
|
||||
<td><code>{{.ID}}</code></td>
|
||||
<td>{{.Model}}</td>
|
||||
@@ -114,7 +116,7 @@
|
||||
<td>{{or .Pool "—"}}</td>
|
||||
<td>{{if .Endpoint}}<code>{{.Endpoint}}</code>{{else}}—{{end}}</td>
|
||||
<td class="num">{{duration .AgeSec}}</td>
|
||||
<td>{{range .Shards}}<code>{{.Node}}</code> {{end}}</td>
|
||||
<td>{{range .Shards}}<code title="{{.Node}}">{{shortNode .Node $.NodePrefix}}</code> {{end}}</td>
|
||||
</tr>{{if .Err}}<tr><td colspan="7" class="note">{{.Err}}</td></tr>{{end}}
|
||||
{{end}}</table>
|
||||
{{else}}<p class="empty">No replicas.</p>{{end}}
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
<h2>Fabric</h2>
|
||||
<p class="sub">{{if .Node.Fabrics}}On {{join .Node.Fabrics}} · <a href="{{if $.OnAdminListener}}/fabrics{{else}}/admin/fabrics{{end}}">the map →</a>{{else}}<span class="bad">On no fabric: no port a declared fabric names can carry traffic, so no multi-node replica is placed here.</span>{{end}}</p>
|
||||
{{if .Node.IBPorts}}
|
||||
<table><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Physically on</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
|
||||
<table class="rows"><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Physically on</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
|
||||
{{range .Node.IBPorts}}<tr>
|
||||
<td><code>{{.HCA}}/{{.Port}}</code>{{if eq .LinkLayer "Ethernet"}} <span class="tag">RoCE</span>{{end}}</td>
|
||||
<td class="{{if .Active}}state-ready{{else}}bad{{end}}">{{.State}}{{with .PhysState}} <span class="note">{{.}}</span>{{end}}</td>
|
||||
@@ -60,13 +60,13 @@ the adapter came up: a number that grows is the one to look at.</p>
|
||||
{{if or .Node.NVLinkDomain .Node.IMEX}} · IMEX {{if eq .Node.IMEX "active"}}<span class="state-ready">active</span>, {{.Node.IMEXChannels}} channel{{if ne .Node.IMEXChannels 1}}s{{end}}{{else}}<span class="bad">{{or .Node.IMEX "not installed"}}</span>{{end}}{{end}}
|
||||
</p>
|
||||
<div class="gpu-wiring">
|
||||
<table class="nvl-matrix" aria-label="NVLink between GPUs">
|
||||
<table class="rows nvl-matrix" aria-label="NVLink between GPUs">
|
||||
<tr><th></th>{{range .Node.GPUs}}<th>{{.Index}}</th>{{end}}</tr>
|
||||
{{range $a := .Node.GPUs}}<tr><th>GPU {{$a.Index}}</th>
|
||||
{{range $b := $.Node.GPUs}}{{if eq $a.Index $b.Index}}<td class="self"></td>{{else if hasInt $a.NVLinkPeers $b.Index}}<td class="nv" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over NVLink">NV</td>{{else}}<td class="pcie" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over PCIe">·</td>{{end}}{{end}}
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<table class="compact gpu-list">
|
||||
<table class="rows compact gpu-list">
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
{{range .Node.GPUs}}<tr>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.NUMA}}</td>
|
||||
@@ -82,7 +82,7 @@ traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS
|
||||
|
||||
<h2>What runs here</h2>
|
||||
{{if .Engines}}
|
||||
<table><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
|
||||
<table class="rows"><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
|
||||
{{range .Engines}}<tr>
|
||||
<td>{{.Model}}</td>
|
||||
<td><code>{{.Replica}}</code></td>
|
||||
|
||||
@@ -42,7 +42,7 @@
|
||||
{{end}}
|
||||
|
||||
{{if .Rows}}
|
||||
<table class="compact">
|
||||
<table class="rows compact">
|
||||
<tr><th>{{if eq .By "user"}}Person{{else if eq .By "project"}}Project{{else}}Team{{end}}</th>
|
||||
<th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
|
||||
{{range .Rows}}<tr>
|
||||
@@ -56,7 +56,7 @@ people do elsewhere — in another organization, or on their own — is not show
|
||||
|
||||
<h2>Teams, projects and limits</h2>
|
||||
{{if .Quotas}}
|
||||
<table class="quota-tree">
|
||||
<table class="rows quota-tree">
|
||||
<tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
|
||||
<th class="num">Budget</th><th>Spent · 30 days</th></tr>
|
||||
{{range .Quotas}}
|
||||
@@ -79,7 +79,7 @@ people do elsewhere — in another organization, or on their own — is not show
|
||||
|
||||
<h2>Members</h2>
|
||||
{{if .Members}}
|
||||
<table class="compact">
|
||||
<table class="rows compact">
|
||||
<tr><th>Person</th><th>Member of</th><th>State</th></tr>
|
||||
{{range .Members}}<tr>
|
||||
<td>{{.Email}}</td>
|
||||
|
||||
@@ -109,6 +109,9 @@ var funcs = template.FuncMap{
|
||||
return p
|
||||
},
|
||||
"ago": ago,
|
||||
// shortNode is a node's name without what every node's name starts with.
|
||||
"shortNode": strings.TrimPrefix,
|
||||
"age": age,
|
||||
// pctbar is a percentage as the minibar template takes it: whole, and
|
||||
// held to 0..100 so a measurement a little over does not draw past its box.
|
||||
"pctbar": func(v float64) struct{ P int } {
|
||||
|
||||
+17
-11
@@ -1349,10 +1349,9 @@ work_dir: /var/lib/llmgw-agent
|
||||
- [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
|
||||
- [x] `EndpointSource` wired from the reconciler to the router
|
||||
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
|
||||
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
|
||||
protocol does not have. The failure path already carries an engine log
|
||||
tail with every status report, so the common case is covered; what is
|
||||
missing is reading a healthy shard's output on demand.
|
||||
- [x] `ctl logs`: a request/response pair on the agent stream (LogsRequest,
|
||||
LogsReply) reads a node's agent log and any shard's engine output on
|
||||
demand, from the CLI and from the node's page under /admin.
|
||||
- **Exit criterion (met against a simulated agent fleet):** shutting down a node
|
||||
moves its replicas elsewhere while the router keeps serving. Measured in
|
||||
`test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is
|
||||
@@ -1452,32 +1451,39 @@ The launch path is now built too:
|
||||
- [x] Model allowlists and key→band mapping. With auth on, the band comes from
|
||||
the key and the header is ignored — it was trusted only while there was
|
||||
no identity to consult, which let any client promote itself.
|
||||
- [ ] Budget fallback
|
||||
- [ ] Routing profiles from configuration plus the `x-llmgw-profile` header
|
||||
- [x] Budget fallback
|
||||
- [x] Routing profiles from configuration plus the `x-llmgw-profile` header
|
||||
- [x] Approximate radix tree (§7.5 tier 1), with its thresholds and the
|
||||
`llmgw_prefix_decisions_total` breakdown. Measured against sim-engines
|
||||
whose caches were too small for the working set: 22.5% → 81.7% cache
|
||||
hits. It does nothing when the working set fits in one replica's cache,
|
||||
which is worth knowing rather than assuming.
|
||||
- [ ] Fallback models and pool spillover
|
||||
- [x] Fallback models
|
||||
- [ ] Pool spillover
|
||||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||||
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
|
||||
rather than printed
|
||||
- [ ] OTel tracing
|
||||
- [x] OTel tracing: an inference span with admission, schedule and each
|
||||
upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
|
||||
set, and the traceparent carried on to the engine
|
||||
- [x] systemd units and `docs/onboarding.md`
|
||||
- [x] Release binaries: static linux/amd64, linux/arm64 and darwin builds
|
||||
attached to each tag, so installing on a node does not need a Go toolchain
|
||||
- [ ] Reference documentation beyond the plan itself
|
||||
- [x] Reference documentation beyond the plan itself: `docs/`, with
|
||||
`docs/reference.md` generated from the configuration structs
|
||||
|
||||
### Phase 6 — v2 ideas
|
||||
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
|
||||
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
||||
- [ ] Admission displacement (dropping requests from lower bands)
|
||||
- [ ] Autoscaling and priority-based preemption
|
||||
- [x] Autoscaling on load: a replica added while requests wait, up to a
|
||||
model's max, and taken back after ten quiet minutes
|
||||
- [ ] Priority-based preemption
|
||||
- [ ] Control plane HA (Raft)
|
||||
- [ ] Weight downloading and cache management
|
||||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||||
checked against their digest, and reclaiming what nothing uses
|
||||
- [ ] Prefill/decode disaggregation (§12)
|
||||
- [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
|
||||
- [ ] External provider support (as a fallback)
|
||||
|
||||
Reference in new issue
Block a user