Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3558dba485 | ||
|
|
481523ca46 | ||
|
|
df6d243ee7 | ||
|
|
ef33e52d3d | ||
|
|
abaf086fd5 | ||
|
|
c5b34978f5 | ||
|
|
77fb48ba32 | ||
|
|
86cfb82f42 | ||
|
|
c5e388807a | ||
|
|
964b659849 | ||
|
|
c2b4985abe | ||
|
|
1d1409faad | ||
|
|
adef25d709 | ||
|
|
ec1918b23b |
No files matched your search
@@ -21,6 +21,66 @@ is only released once its **exit criterion** is met, not when its code lands.
|
|||||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||||
APIs change between minors without a deprecation period.
|
APIs change between minors without a deprecation period.
|
||||||
|
|
||||||
|
## v0.11.4 — 2026-10-05
|
||||||
|
|
||||||
|
Two control planes, run in anger on the reference cluster.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **The console sets up a second control plane**: Do → Add a control plane
|
||||||
|
asks for the machine and a floating address and runs `server add`.
|
||||||
|
- **Who leads, everywhere.** `ctl status` and the console's Overview have a
|
||||||
|
`CONTROL` line: which control plane leads, which stand by, one that does not
|
||||||
|
answer, and whether the floating address is on the leader.
|
||||||
|
- **A standby's console is the leader's.** Its admin API passes `/api/` to the
|
||||||
|
leader over the agent port, with the cluster's certificates both ways, so
|
||||||
|
`sudo llmgw` and `llmgw ctl` show the same cluster on either machine. Control
|
||||||
|
plane certificates are now issued for client use too; `server add` reissues
|
||||||
|
older ones.
|
||||||
|
- **`ctl upgrade` in waves.** Nodes are upgraded one, then two, four… up to
|
||||||
|
`--parallel` (20) at once, so hundreds take minutes; a wave that does not
|
||||||
|
come back stops it. The other control plane is upgraded first, over ssh.
|
||||||
|
- `server add` says when the mirror's `--api-allow` would refuse the new
|
||||||
|
control plane.
|
||||||
|
- **Guardrails in the console**: a view of the policies and what each checks,
|
||||||
|
whether their judge models are serving, where they are attached (attach
|
||||||
|
and detach from it), and what they did in a window. `GET
|
||||||
|
/api/v1/guard/events` answers the same, and `GET /api/v1/guard` now says
|
||||||
|
what each policy does.
|
||||||
|
|
||||||
|
- **A standby's portal is the leader's.** Its client port passes everything
|
||||||
|
but `/health`, `/ready` and `/leader` to the leader, and its admin port the
|
||||||
|
dashboard as well as the API; a proxy left pointing at a standby showed an
|
||||||
|
empty cluster. The leader trusts the control planes' `X-Forwarded-For`.
|
||||||
|
|
||||||
|
- `ctl model serve --tools` says in the catalog that a model makes tool
|
||||||
|
calls, for agents and editors.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Continue's agent did nothing.** The configuration the portal writes told
|
||||||
|
Continue every model uses tools, and nothing of its context: a model whose
|
||||||
|
calls came back as text stalled the agent, and prompts over a CPU
|
||||||
|
replica's 4096 tokens were refused. It now says `tool_use` only for a
|
||||||
|
model whose catalog says tools, and the context and longest answer the
|
||||||
|
gateway allows.
|
||||||
|
- `ctl model serve` gave llama.cpp four slots whatever was asked: a
|
||||||
|
`--context-length 32768 --ctx-pool 32768` model had slots of 8192 and
|
||||||
|
refused what the gateway admitted. Slots are now what the pool holds of
|
||||||
|
the context. The longest answer grows with the context (a quarter, from
|
||||||
|
1024 to 8192) rather than being 1024 for every model.
|
||||||
|
- `ctl upgrade` upgrades the leader last wherever it is run; run on the
|
||||||
|
standby, it upgraded the leader first and the leadership moved twice.
|
||||||
|
- **Failover took 40 seconds; it takes a few.** A stopping control plane held
|
||||||
|
every agent's stream for the thirty seconds of a graceful shutdown; it now
|
||||||
|
gives up its lease first and lets the agents go at once.
|
||||||
|
- A control plane that lost its lease while running kept its agents, leaving
|
||||||
|
the new leader with no nodes. It now lets them go.
|
||||||
|
- An agent's reconnect backoff carried over from one outage to the next, so
|
||||||
|
each failover made the next one slower; it starts again after a session that
|
||||||
|
worked.
|
||||||
|
- The `CONTROL` line named no floating address.
|
||||||
|
|
||||||
## v0.11.2 — 2026-10-05
|
## v0.11.2 — 2026-10-05
|
||||||
|
|
||||||
### Routing by what each vLLM replica's cache holds
|
### Routing by what each vLLM replica's cache holds
|
||||||
|
|||||||
+201
-1
@@ -854,6 +854,43 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"/api/v1/guard/events": {
|
||||||
|
"get": {
|
||||||
|
"operationId": "getGuardEvents",
|
||||||
|
"summary": "What the guardrails did: totals and the newest requests",
|
||||||
|
"tags": [
|
||||||
|
"guard"
|
||||||
|
],
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "since",
|
||||||
|
"in": "query",
|
||||||
|
"description": "How far back: 90m, 24h, 7d; 24h when absent.",
|
||||||
|
"schema": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"responses": {
|
||||||
|
"200": {
|
||||||
|
"description": "OK",
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"$ref": "#/components/schemas/GuardEventsView"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"401": {
|
||||||
|
"$ref": "#/components/responses/Unauthorized"
|
||||||
|
},
|
||||||
|
"default": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"/api/v1/jobs": {
|
"/api/v1/jobs": {
|
||||||
"get": {
|
"get": {
|
||||||
"operationId": "getJobs",
|
"operationId": "getJobs",
|
||||||
@@ -3443,6 +3480,45 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"ControlPlane": {
|
||||||
|
"type": "object",
|
||||||
|
"description": "ControlPlane is one control plane.",
|
||||||
|
"properties": {
|
||||||
|
"name": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"address": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"state": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "State is \"leading\", \"standing by\" or \"not answering\"."
|
||||||
|
},
|
||||||
|
"this": {
|
||||||
|
"type": "boolean",
|
||||||
|
"description": "This is the control plane that answered the summary."
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"ControlPlanes": {
|
||||||
|
"type": "object",
|
||||||
|
"description": "ControlPlanes is every control plane of a cluster that has several, as each answered when asked.",
|
||||||
|
"properties": {
|
||||||
|
"members": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/components/schemas/ControlPlane"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"floating": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "Floating is the address keepalived keeps on the leader (10.0.0.161), and FloatingUp whether it answered as a leader when asked; empty when nothing floats."
|
||||||
|
},
|
||||||
|
"floating_up": {
|
||||||
|
"type": "boolean"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"DomainView": {
|
"DomainView": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"description": "DomainView is one domain: its nodes, its power and whether it is held.",
|
"description": "DomainView is one domain: its nodes, its power and whether it is held.",
|
||||||
@@ -3810,6 +3886,115 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"GuardCountView": {
|
||||||
|
"type": "object",
|
||||||
|
"description": "GuardCountView is how many requests one policy did one thing to.",
|
||||||
|
"properties": {
|
||||||
|
"project": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"action": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"policy": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"category": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"count": {
|
||||||
|
"type": "integer",
|
||||||
|
"format": "int64"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"GuardEventView": {
|
||||||
|
"type": "object",
|
||||||
|
"description": "GuardEventView is one request the guardrails did something to.",
|
||||||
|
"properties": {
|
||||||
|
"at": {
|
||||||
|
"type": "string",
|
||||||
|
"format": "date-time"
|
||||||
|
},
|
||||||
|
"request_id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"project": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"model": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"status": {
|
||||||
|
"type": "integer"
|
||||||
|
},
|
||||||
|
"action": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"policy": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"category": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"guard_ms": {
|
||||||
|
"type": "integer",
|
||||||
|
"format": "int64"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"GuardEventsView": {
|
||||||
|
"type": "object",
|
||||||
|
"description": "GuardEventsView is what the guardrails did since a moment: totals, and the newest requests.",
|
||||||
|
"properties": {
|
||||||
|
"since": {
|
||||||
|
"type": "string",
|
||||||
|
"format": "date-time"
|
||||||
|
},
|
||||||
|
"counts": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/components/schemas/GuardCountView"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"events": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/components/schemas/GuardEventView"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"GuardPolicyView": {
|
||||||
|
"type": "object",
|
||||||
|
"description": "GuardPolicyView is what one policy does, in brief.",
|
||||||
|
"properties": {
|
||||||
|
"name": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"mode": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "Mode is enforce, shadow or off; Fail is open or closed."
|
||||||
|
},
|
||||||
|
"fail": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"no_store": {
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"judge": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "Judge is the judge model, when one reads the text."
|
||||||
|
},
|
||||||
|
"checks": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"description": "Checks are what it looks for, one line each: \"input: pii email → redact\"."
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"GuardView": {
|
"GuardView": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"description": "GuardView is the library and where its policies are attached.",
|
"description": "GuardView is the library and where its policies are attached.",
|
||||||
@@ -3831,6 +4016,13 @@
|
|||||||
"items": {
|
"items": {
|
||||||
"$ref": "#/components/schemas/GuardAttachmentView"
|
"$ref": "#/components/schemas/GuardAttachmentView"
|
||||||
}
|
}
|
||||||
|
},
|
||||||
|
"details": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"$ref": "#/components/schemas/GuardPolicyView"
|
||||||
|
},
|
||||||
|
"description": "Details says what each policy does, in the order of Policies."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -5276,6 +5468,10 @@
|
|||||||
},
|
},
|
||||||
"store_error": {
|
"store_error": {
|
||||||
"type": "string"
|
"type": "string"
|
||||||
|
},
|
||||||
|
"control_planes": {
|
||||||
|
"$ref": "#/components/schemas/ControlPlanes",
|
||||||
|
"description": "ControlPlanes is the cluster's control planes when there is more than one: which leads, which stand by, and the floating address in front."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -6805,11 +7001,15 @@
|
|||||||
},
|
},
|
||||||
"ctx_pool": {
|
"ctx_pool": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide."
|
"description": "CtxPool is llama.cpp's --ctx-size; ContextLength what one request may use. They default to 16384 and 4096, what four parallel slots divide; llama.cpp is given as many slots as the pool holds requests."
|
||||||
},
|
},
|
||||||
"context_length": {
|
"context_length": {
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
},
|
||||||
|
"tools": {
|
||||||
|
"type": "boolean",
|
||||||
|
"description": "Tools says the model makes tool calls the engine returns as calls: it is written into the catalog, where clients and editors read it. Check a running replica with llmgw ctl tool-check before saying so."
|
||||||
|
},
|
||||||
"cpu_cores": {
|
"cpu_cores": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."
|
"description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."
|
||||||
|
|||||||
@@ -184,6 +184,15 @@ asks for, send the results back. Whether a model calls tools well is the
|
|||||||
model's business, not the gateway's; ask your operator which of the served
|
model's business, not the gateway's; ask your operator which of the served
|
||||||
models are set up for it.
|
models are set up for it.
|
||||||
|
|
||||||
|
**Editors' agents need such a model.** Continue's Agent mode, Cline and
|
||||||
|
Codex act only through tool calls: with a model whose calls come back as
|
||||||
|
text — `<tool_call>{...}` or a JSON block in the answer, `tool_calls` empty —
|
||||||
|
they chat but never read a file or run a command. Qwen2.5-Coder is one;
|
||||||
|
Qwen3-Coder is not. The configuration the portal writes for Continue says
|
||||||
|
`tool_use` only for a model whose catalog entry says tools, and gives
|
||||||
|
Continue the model's context and longest answer, so it never sends a request
|
||||||
|
the gateway refuses.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
import json
|
import json
|
||||||
|
|
||||||
|
|||||||
@@ -209,3 +209,14 @@ Each request's usage row has the action, policy, category and the time the
|
|||||||
checks took. The access log has the action, stage and policy. Neither has
|
checks took. The access log has the action, stage and policy. Neither has
|
||||||
the text, a matched value, or the placeholder map: the map exists only in
|
the text, a matched value, or the placeholder map: the map exists only in
|
||||||
the gateway's memory while the request is served.
|
the gateway's memory while the request is served.
|
||||||
|
|
||||||
|
## In the console
|
||||||
|
|
||||||
|
`sudo llmgw` has a **Guardrails** view: each policy with its mode, what it
|
||||||
|
does when it cannot judge, and what it checks; whether each judge model it
|
||||||
|
leans on has replicas ready — a policy that fails closed refuses everything
|
||||||
|
while its judge has none; where policies are attached, with A to attach one
|
||||||
|
and Enter on a row to detach it; and what they did in the last day (W for an
|
||||||
|
hour or a week), totals first and the newest requests after, each with the
|
||||||
|
request id `llmgw ctl request` reads. The same is at `GET /api/v1/guard` and
|
||||||
|
`GET /api/v1/guard/events?since=24h`, and on the portal's Guardrails page.
|
||||||
+68
-23
@@ -1915,15 +1915,53 @@ one trusted machine only; see [development.md](development.md).
|
|||||||
|
|
||||||
## Two control planes
|
## Two control planes
|
||||||
|
|
||||||
llmgw handles the hard half itself: only one places work, and agents find
|
Once the store is replicated (`server init --patroni`, above), one command on
|
||||||
whichever is leading. What it does not do is move client traffic, which
|
the control plane makes another machine a second one — or **Do → Add a
|
||||||
happens in front of both.
|
control plane** in the console, which asks the same two questions:
|
||||||
|
|
||||||
[deploy/ha/](../deploy/ha/) has the pattern and a keepalived example. The one
|
```
|
||||||
thing to get right is the health check: point it at `/ready`, not `/health`.
|
sudo --preserve-env=SSH_AUTH_SOCK llmgw server add ubuntu@10.0.0.150 --vip 10.0.0.161/24
|
||||||
`/health` is 200 on a standby, which is alive and holds no endpoints at all,
|
```
|
||||||
so a load balancer using it would send half the traffic to something that
|
|
||||||
refuses every request.
|
Over ssh, as `node add` works, it gives the machine this llmgw, a certificate
|
||||||
|
from the cluster's authority, this control plane's `server.yaml` and secrets,
|
||||||
|
and lets it into the store; it starts there as a standby. Every node is told
|
||||||
|
of it within half a minute and fails over to it without being touched.
|
||||||
|
`--vip` keeps a floating address on whichever control plane leads, with
|
||||||
|
keepalived on both: that is the address to give clients, or a proxy in front.
|
||||||
|
Without it, put a load balancer in front that checks `/ready` — not
|
||||||
|
`/health`, which is 200 on a standby that would refuse every request.
|
||||||
|
|
||||||
|
- **What a failover costs.** Measured on the reference cluster, stopping the
|
||||||
|
leader: the standby led within five seconds, every node had moved to it
|
||||||
|
and every replica was adopted — none restarted, the engines never stopped
|
||||||
|
serving — and the floating address answered again with it.
|
||||||
|
- **Who leads.** `llmgw ctl status` and the console's Overview have a
|
||||||
|
`CONTROL` line: which leads, which stand by, one that does not answer, and
|
||||||
|
whether the floating address is on the leader. A standby with nothing to
|
||||||
|
take over from is the thing to see before it is needed.
|
||||||
|
- **Either machine, the leader's cluster.** A standby passes its client port
|
||||||
|
— the portal, its admin pages, the API — to the leader's, keeping only
|
||||||
|
`/health`, `/ready` and `/leader` to describe itself, so a proxy pointed at
|
||||||
|
the wrong machine still shows the cluster; the leader trusts a control
|
||||||
|
plane's `X-Forwarded-For`, so clients keep their own address. Its admin
|
||||||
|
API passes everything but `/debug/` and `/metrics` to the leader, over the port control planes already reach each
|
||||||
|
other on and with the cluster's certificates both ways, so `sudo llmgw` and
|
||||||
|
`llmgw ctl` show the same cluster on both — the leader's — and a change made
|
||||||
|
on the standby is made by the leader. A node's certificate is refused
|
||||||
|
there. Control plane certificates from before v0.11.3 cannot present
|
||||||
|
themselves as clients: `server add` reissues them, and the running server
|
||||||
|
picks the new one up when restarted.
|
||||||
|
- **The mirror** answers only the addresses it was given with `--api-allow`.
|
||||||
|
`server add` checks, and says what to add to the mirror's unit if it
|
||||||
|
refuses the new control plane — which would otherwise find out the first
|
||||||
|
time it leads and fetches a model.
|
||||||
|
- **Limits are per process.** Rate limits are counted in memory, and clients
|
||||||
|
reach the leader through the floating address, so a failover starts each
|
||||||
|
key's minute afresh.
|
||||||
|
|
||||||
|
[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
|
||||||
|
their own load balancer in front.
|
||||||
|
|
||||||
## The portal's language
|
## The portal's language
|
||||||
|
|
||||||
@@ -2215,27 +2253,34 @@ joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).
|
|||||||
|
|
||||||
## Upgrading
|
## Upgrading
|
||||||
|
|
||||||
On the control plane:
|
On a control plane:
|
||||||
|
|
||||||
```
|
```
|
||||||
sudo llmgw ctl upgrade v0.10.4
|
sudo --preserve-env=SSH_AUTH_SOCK llmgw ctl upgrade v0.10.4
|
||||||
```
|
```
|
||||||
|
|
||||||
It upgrades this machine first — fetching the release for its platform,
|
With two control planes it upgrades the standby first and the leader last,
|
||||||
checking it against the release's checksums, keeping the old binary as
|
whichever this machine is — the other over ssh as the user who ran sudo
|
||||||
`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
|
(`--ssh-user`), with `llmgw upgrade` there, and this machine —
|
||||||
for every node to reconnect — and then each node in turn. A node's agent is told
|
fetching the release for its platform, checking it against the release's
|
||||||
the version over the stream it already holds, fetches and checks the release
|
checksums, keeping the old binary as `/usr/local/bin/llmgw.bak-pre-<version>`,
|
||||||
itself, swaps it in and restarts; no ssh into the nodes is needed. Release
|
restarting `llmgw-server` and waiting for every node to reconnect. The
|
||||||
candidates (`v0.10.4-rc.2`) work the same way.
|
leader's restart hands over once, to a standby already on the release.
|
||||||
|
|
||||||
|
Then the nodes, in waves: one first, then two, four, and so on up to
|
||||||
|
`--parallel` (20) at once. A node's agent is told the version over the stream
|
||||||
|
it already holds, fetches and checks the release itself, swaps it in and
|
||||||
|
restarts; no ssh into the nodes is needed. A fleet of hundreds takes minutes,
|
||||||
|
not hours. Release candidates (`v0.10.4-rc.2`) work the same way.
|
||||||
|
|
||||||
A node counts as done when it is back on the new version, ready, with as many
|
A node counts as done when it is back on the new version, ready, with as many
|
||||||
ready replicas as before. One that is not within `--timeout` (three minutes)
|
ready replicas as before. A wave not back within `--timeout` (three minutes)
|
||||||
stops the upgrade there, so a bad release reaches one node rather than all of
|
stops the upgrade there — the first wave is one node, so a bad release reaches
|
||||||
them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
|
one node rather than all of them; `llmgw ctl logs <node>` says why, and the node
|
||||||
beside the new one. Nodes already on the version are skipped, so running the
|
keeps its previous binary beside the new one. Nodes already on the version are
|
||||||
command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
|
skipped, so running the command again resumes where it stopped. `--nodes a,b`
|
||||||
and `--skip-control-plane` leaves this machine alone.
|
limits it to some nodes, `--skip-control-plane` leaves the control planes alone,
|
||||||
|
and `--other-control-planes=false` this machine's partner.
|
||||||
|
|
||||||
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
|
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
|
||||||
keep running when the agent restarts, and the agent reports them back on its
|
keep running when the agent restarts, and the agent reports them back on its
|
||||||
|
|||||||
+18
-5
@@ -21,6 +21,7 @@ import (
|
|||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"go.opentelemetry.io/otel/attribute"
|
"go.opentelemetry.io/otel/attribute"
|
||||||
@@ -42,11 +43,14 @@ import (
|
|||||||
|
|
||||||
// Agent runs on a GPU node.
|
// Agent runs on a GPU node.
|
||||||
type Agent struct {
|
type Agent struct {
|
||||||
cfg Config
|
cfg Config
|
||||||
log *slog.Logger
|
// answered says the current session heard from its control plane, which
|
||||||
sup *supervisor.Supervisor
|
// starts the reconnect backoff again when it ends.
|
||||||
reg *engines.Registry
|
answered atomic.Bool
|
||||||
http connectHTTPClient
|
log *slog.Logger
|
||||||
|
sup *supervisor.Supervisor
|
||||||
|
reg *engines.Registry
|
||||||
|
http connectHTTPClient
|
||||||
|
|
||||||
// engineOnce probes the configured engines once before registering.
|
// engineOnce probes the configured engines once before registering.
|
||||||
//
|
//
|
||||||
@@ -215,6 +219,14 @@ func (a *Agent) Run(ctx context.Context) error {
|
|||||||
// never answers — from here they are the same thing, and trying
|
// never answers — from here they are the same thing, and trying
|
||||||
// somewhere else is the answer to all three.
|
// somewhere else is the answer to all three.
|
||||||
next++
|
next++
|
||||||
|
// A session that was answered was a connection that worked, however
|
||||||
|
// it ended: the backoff starts again rather than carrying on from
|
||||||
|
// the last outage. Without this every failover the agent lived
|
||||||
|
// through made the next one slower — the third waited half a minute
|
||||||
|
// on a control plane that had been ready for twenty seconds.
|
||||||
|
if a.answered.Swap(false) {
|
||||||
|
attempt = 0
|
||||||
|
}
|
||||||
attempt++
|
attempt++
|
||||||
delay := a.reconnectDelay(attempt)
|
delay := a.reconnectDelay(attempt)
|
||||||
// No wait when there is somewhere else to try: the delay exists to
|
// No wait when there is somewhere else to try: the delay exists to
|
||||||
@@ -568,6 +580,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
|||||||
// why it is not registering is one an operator cannot fix.
|
// why it is not registering is one an operator cannot fix.
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
a.answered.Store(true)
|
||||||
a.handle(ctx, msg, out)
|
a.handle(ctx, msg, out)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -105,3 +106,49 @@ func TestALeaderStillAcceptsWithNoLeadershipConfigured(t *testing.T) {
|
|||||||
}
|
}
|
||||||
t.Fatal("a service with no leadership configured refused an agent")
|
t.Fatal("a service with no leadership configured refused an agent")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// flipping is a leader that can stop leading partway through a test.
|
||||||
|
type flipping struct{ leading atomic.Bool }
|
||||||
|
|
||||||
|
func (f *flipping) IsLeader() bool { return f.leading.Load() }
|
||||||
|
|
||||||
|
func TestAnAgentLeavesAControlPlaneThatStopsLeading(t *testing.T) {
|
||||||
|
// Leadership lost while the process lives on: the store was slow and the
|
||||||
|
// lease lapsed, or another control plane took it. Agents that stayed on
|
||||||
|
// the old leader would leave the new one with no nodes, so the old one
|
||||||
|
// has to let them go rather than wait for them to leave.
|
||||||
|
was := &flipping{}
|
||||||
|
was.leading.Store(true)
|
||||||
|
wasInv, wasAddr := startPlain(t, was)
|
||||||
|
nowInv, nowAddr := startPlain(t, standby(false))
|
||||||
|
|
||||||
|
ag := runAgent(t, Config{
|
||||||
|
NodeID: "n1", MgmtAddr: "127.0.0.1",
|
||||||
|
Servers: []string{wasAddr, nowAddr},
|
||||||
|
})
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
go ag.Run(ctx)
|
||||||
|
|
||||||
|
waitFor(t, 5*time.Second, "the agent never reached the first leader", func() bool {
|
||||||
|
_, ok := wasInv.Get("n1")
|
||||||
|
return ok
|
||||||
|
})
|
||||||
|
was.leading.Store(false)
|
||||||
|
waitFor(t, 8*time.Second, "the agent stayed on a control plane that stopped leading", func() bool {
|
||||||
|
_, ok := nowInv.Get("n1")
|
||||||
|
return ok
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitFor(t *testing.T, d time.Duration, msg string, ok func() bool) {
|
||||||
|
t.Helper()
|
||||||
|
deadline := time.Now().Add(d)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if ok() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(50 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatal(msg)
|
||||||
|
}
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// floatingPath is where `server add --vip` records the floating address.
|
||||||
|
const floatingPath = "/etc/llmgw/floating"
|
||||||
|
|
||||||
|
// controlPlaneProbe answers the summary's question of which control plane
|
||||||
|
// leads: it asks each one's /leader, on the client port every one of them
|
||||||
|
// listens on, since they share a server.yaml.
|
||||||
|
//
|
||||||
|
// Asked, rather than read from the lease, because the lease names whoever
|
||||||
|
// held it last and says nothing about a standby that has stopped — and a
|
||||||
|
// standby that is not there is the thing worth seeing before it is needed.
|
||||||
|
type controlPlaneProbe struct {
|
||||||
|
st *store.DB
|
||||||
|
self string
|
||||||
|
// floating is the file the floating address is recorded in.
|
||||||
|
floating string
|
||||||
|
client *http.Client
|
||||||
|
// leaderURL is where a host is asked; tests point it elsewhere.
|
||||||
|
leaderURL func(host string) string
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
at time.Time
|
||||||
|
last adminapi.ControlPlanes
|
||||||
|
}
|
||||||
|
|
||||||
|
func newControlPlaneProbe(st *store.DB, listen string) *controlPlaneProbe {
|
||||||
|
_, port, _ := net.SplitHostPort(listen)
|
||||||
|
if port == "" {
|
||||||
|
port = "8080"
|
||||||
|
}
|
||||||
|
self, _ := os.Hostname()
|
||||||
|
return &controlPlaneProbe{st: st, self: self,
|
||||||
|
leaderURL: func(host string) string { return "http://" + net.JoinHostPort(host, port) + "/leader" },
|
||||||
|
floating: floatingPath,
|
||||||
|
client: &http.Client{Timeout: time.Second}}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Status is the answer, kept for a few seconds: the console asks every three,
|
||||||
|
// and each asking is a request to every control plane.
|
||||||
|
func (p *controlPlaneProbe) Status() adminapi.ControlPlanes {
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
if time.Since(p.at) < 2*time.Second {
|
||||||
|
return p.last
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
cps, err := p.st.ControlPlanes(ctx)
|
||||||
|
if err != nil || len(cps) < 2 {
|
||||||
|
// One control plane is not a set of them; nothing to say.
|
||||||
|
p.at, p.last = time.Now(), adminapi.ControlPlanes{}
|
||||||
|
return p.last
|
||||||
|
}
|
||||||
|
// Read each time: server add writes it after this process may have
|
||||||
|
// started.
|
||||||
|
out := adminapi.ControlPlanes{Members: make([]adminapi.ControlPlane, len(cps)), Floating: floatingAddress(p.floating)}
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i, cp := range cps {
|
||||||
|
host := cp.URL
|
||||||
|
if u, err := url.Parse(cp.URL); err == nil && u.Hostname() != "" {
|
||||||
|
host = u.Hostname()
|
||||||
|
}
|
||||||
|
out.Members[i] = adminapi.ControlPlane{Name: cp.Name, Address: host, This: cp.Name == p.self}
|
||||||
|
wg.Go(func() { out.Members[i].State = p.ask(ctx, host) })
|
||||||
|
}
|
||||||
|
if out.Floating != "" {
|
||||||
|
wg.Go(func() { out.FloatingUp = p.ask(ctx, out.Floating) == "leading" })
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
p.at, p.last = time.Now(), out
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// ask is what one address's /leader says.
|
||||||
|
func (p *controlPlaneProbe) ask(ctx context.Context, host string) string {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, p.leaderURL(host), nil)
|
||||||
|
if err != nil {
|
||||||
|
return "not answering"
|
||||||
|
}
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return "not answering"
|
||||||
|
}
|
||||||
|
_ = resp.Body.Close()
|
||||||
|
switch resp.StatusCode {
|
||||||
|
case http.StatusOK:
|
||||||
|
return "leading"
|
||||||
|
case http.StatusServiceUnavailable:
|
||||||
|
return "standing by"
|
||||||
|
}
|
||||||
|
return "not answering"
|
||||||
|
}
|
||||||
|
|
||||||
|
// floatingAddress is the address in the file server add keeps beside the
|
||||||
|
// keepalived configuration, without its prefix; "" when nothing floats.
|
||||||
|
// keepalived's own file holds its password and is root's alone, and this
|
||||||
|
// process is not root.
|
||||||
|
func floatingAddress(path string) string {
|
||||||
|
b, err := os.ReadFile(path) //nolint:gosec // a fixed path of our own
|
||||||
|
if err != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
addr, _, _ := strings.Cut(strings.TrimSpace(string(b)), "/")
|
||||||
|
if net.ParseIP(addr) == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return addr
|
||||||
|
}
|
||||||
@@ -0,0 +1,142 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestStatusSaysWhichControlPlaneLeadsAndWhetherTheOthersStandBy(t *testing.T) {
|
||||||
|
render := func(s adminapi.Summary) string {
|
||||||
|
cmd := &cobra.Command{}
|
||||||
|
var b bytes.Buffer
|
||||||
|
cmd.SetOut(&b)
|
||||||
|
printSummary(cmd, s)
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
out := render(adminapi.Summary{ControlPlanes: adminapi.ControlPlanes{
|
||||||
|
Members: []adminapi.ControlPlane{
|
||||||
|
{Name: "llmgw-ctl", State: "leading", This: true},
|
||||||
|
{Name: "llmgw-ctl02", State: "standing by"},
|
||||||
|
},
|
||||||
|
Floating: "10.0.0.161", FloatingUp: true,
|
||||||
|
}})
|
||||||
|
if !strings.Contains(out, "CONTROL llmgw-ctl leads (this one), llmgw-ctl02 standing by · 10.0.0.161 on the leader") {
|
||||||
|
t.Errorf("status:\n%s", out)
|
||||||
|
}
|
||||||
|
out = render(adminapi.Summary{ControlPlanes: adminapi.ControlPlanes{
|
||||||
|
Members: []adminapi.ControlPlane{
|
||||||
|
{Name: "llmgw-ctl", State: "not answering"},
|
||||||
|
{Name: "llmgw-ctl02", State: "standing by", This: true},
|
||||||
|
},
|
||||||
|
Floating: "10.0.0.161",
|
||||||
|
}})
|
||||||
|
for _, want := range []string{"NO LEADER", "llmgw-ctl not answering", "10.0.0.161 NOT ANSWERING"} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Errorf("%q missing:\n%s", want, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if out := render(adminapi.Summary{}); strings.Contains(out, "CONTROL") {
|
||||||
|
t.Error("a single control plane got a control line")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheProbeAsksEachControlPlaneWhetherItLeads(t *testing.T) {
|
||||||
|
leading := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
|
||||||
|
defer leading.Close()
|
||||||
|
standby := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusServiceUnavailable)
|
||||||
|
}))
|
||||||
|
defer standby.Close()
|
||||||
|
gone := httptest.NewServer(http.NotFoundHandler())
|
||||||
|
gone.Close()
|
||||||
|
|
||||||
|
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
for _, cp := range []store.ControlPlane{
|
||||||
|
{URL: "https://10.0.0.1:7443", Name: "one"},
|
||||||
|
{URL: "https://10.0.0.2:7443", Name: "two"},
|
||||||
|
{URL: "https://10.0.0.3:7443", Name: "three"},
|
||||||
|
} {
|
||||||
|
if err := st.PutControlPlane(t.Context(), cp); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
at := map[string]string{"10.0.0.1": leading.URL, "10.0.0.2": standby.URL, "10.0.0.3": gone.URL, "10.0.0.161": leading.URL}
|
||||||
|
p := newControlPlaneProbe(st, ":8080")
|
||||||
|
floating := filepath.Join(t.TempDir(), "floating")
|
||||||
|
if err := os.WriteFile(floating, []byte("10.0.0.161/24\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
p.self, p.floating = "two", floating
|
||||||
|
p.leaderURL = func(host string) string { return at[host] + "/leader" }
|
||||||
|
|
||||||
|
s := p.Status()
|
||||||
|
got := map[string]adminapi.ControlPlane{}
|
||||||
|
for _, m := range s.Members {
|
||||||
|
got[m.Name] = m
|
||||||
|
}
|
||||||
|
if got["one"].State != "leading" || got["two"].State != "standing by" || got["three"].State != "not answering" {
|
||||||
|
t.Errorf("states: %+v", got)
|
||||||
|
}
|
||||||
|
if !got["two"].This || got["one"].This || got["one"].Address != "10.0.0.1" {
|
||||||
|
t.Errorf("which and where: %+v", got)
|
||||||
|
}
|
||||||
|
if !s.FloatingUp {
|
||||||
|
t.Error("a floating address on the leader read as down")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOneControlPlaneIsNotASetOfThem(t *testing.T) {
|
||||||
|
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
if err := st.PutControlPlane(t.Context(), store.ControlPlane{URL: "https://10.0.0.1:7443", Name: "one"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if s := newControlPlaneProbe(st, ":8080").Status(); len(s.Members) != 0 {
|
||||||
|
t.Errorf("one control plane reported as a set: %+v", s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheFloatingAddressIsReadFromWhatServerAddWrote(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
file := filepath.Join(dir, "floating")
|
||||||
|
if err := os.WriteFile(file, []byte("10.0.0.161/24\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := floatingAddress(file); got != "10.0.0.161" {
|
||||||
|
t.Errorf("read %q", got)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(file, []byte("not an address\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := floatingAddress(file); got != "" {
|
||||||
|
t.Errorf("nonsense read as %q", got)
|
||||||
|
}
|
||||||
|
if got := floatingAddress(filepath.Join(dir, "none")); got != "" {
|
||||||
|
t.Errorf("no file read as %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheProbeReadsTheFloatingAddressFromItsFile(t *testing.T) {
|
||||||
|
// The file, not what it held when the process started: server add
|
||||||
|
// writes it after restarting the control plane it adds.
|
||||||
|
if p := newControlPlaneProbe(nil, ":8080"); p.floating != floatingPath {
|
||||||
|
t.Errorf("the probe reads %q, not %s", p.floating, floatingPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
+187
-58
@@ -16,6 +16,7 @@ import (
|
|||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
|
|
||||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||||
)
|
)
|
||||||
@@ -27,19 +28,25 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
|
|||||||
releases string
|
releases string
|
||||||
perNode time.Duration
|
perNode time.Duration
|
||||||
serverSvc string
|
serverSvc string
|
||||||
|
parallel int
|
||||||
|
sshUser string
|
||||||
|
othersCP bool
|
||||||
)
|
)
|
||||||
c := &cobra.Command{
|
c := &cobra.Command{
|
||||||
Use: "upgrade VERSION",
|
Use: "upgrade VERSION",
|
||||||
Short: "Upgrade this control plane, then every node, one at a time",
|
Short: "Upgrade the control planes, then every node, in waves",
|
||||||
Long: "Moves the cluster to a release: v0.10.4, or a candidate like v0.10.4-rc.2.\n\n" +
|
Long: "Moves the cluster to a release: v0.10.4, or a candidate like v0.10.4-rc.2.\n\n" +
|
||||||
"This machine first — run it with sudo on the control plane, since it\n" +
|
"The other control planes first, over ssh as --ssh-user (the one sudo was\n" +
|
||||||
"replaces /usr/local/bin/llmgw and restarts llmgw-server. Then each node, in\n" +
|
"run by), then this machine — run it with sudo on a control plane, since it\n" +
|
||||||
"turn: its agent fetches the release for its own platform, checks it\n" +
|
"replaces /usr/local/bin/llmgw and restarts llmgw-server. A leader restarted\n" +
|
||||||
"against the release's checksums, swaps it in and restarts. Engines keep\n" +
|
"last hands over to a standby already on the release.\n\n" +
|
||||||
"serving through all of it and are adopted again; no ssh into the nodes is\n" +
|
"Then the nodes, in waves: one first, then twice as many each time a wave\n" +
|
||||||
"needed.\n\n" +
|
"comes back whole, up to --parallel at once. Each agent fetches the release\n" +
|
||||||
|
"for its own platform, checks it against the release's checksums, swaps it\n" +
|
||||||
|
"in and restarts; engines keep serving through all of it and are adopted\n" +
|
||||||
|
"again, and no ssh into the nodes is needed.\n\n" +
|
||||||
"A node counts as done when it is back on the new version, ready, with as\n" +
|
"A node counts as done when it is back on the new version, ready, with as\n" +
|
||||||
"many ready replicas as before. One that is not within --timeout stops the\n" +
|
"many ready replicas as before. A wave not back within --timeout stops the\n" +
|
||||||
"upgrade there, so a bad release reaches one node rather than all of them.\n" +
|
"upgrade there, so a bad release reaches one node rather than all of them.\n" +
|
||||||
"Nodes already on the version are skipped, so running it again resumes.",
|
"Nodes already on the version are skipped, so running it again resumes.",
|
||||||
Args: cobra.ExactArgs(1),
|
Args: cobra.ExactArgs(1),
|
||||||
@@ -61,11 +68,28 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !skipCP {
|
if !skipCP {
|
||||||
if err := upgradeControlPlane(ctx, out, client, v, releases, serverSvc); err != nil {
|
// The leader last, wherever this runs: every other control
|
||||||
return err
|
// plane is a standby, upgraded with nothing moving, and the
|
||||||
|
// leader's restart hands over once, to a machine already on
|
||||||
|
// the release.
|
||||||
|
others := func() error {
|
||||||
|
if !othersCP {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return upgradeOtherControlPlanes(ctx, out, client, v, releases, sshUser)
|
||||||
|
}
|
||||||
|
self := func() error { return upgradeControlPlane(ctx, out, client, v, releases, serverSvc) }
|
||||||
|
order := []func() error{others, self}
|
||||||
|
if !thisLeads(ctx, client) {
|
||||||
|
order = []func() error{self, others}
|
||||||
|
}
|
||||||
|
for _, step := range order {
|
||||||
|
if err := step(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return upgradeNodes(ctx, out, client, v, nodes, perNode)
|
return upgradeNodes(ctx, out, client, v, nodes, perNode, parallel)
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
c.Flags().StringSliceVar(&nodes, "nodes", nil, "only these nodes (default: all)")
|
c.Flags().StringSliceVar(&nodes, "nodes", nil, "only these nodes (default: all)")
|
||||||
@@ -74,9 +98,60 @@ func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
|
|||||||
"(default: as server.yaml says, the mirror's copy on an offline cluster, else the project's)")
|
"(default: as server.yaml says, the mirror's copy on an offline cluster, else the project's)")
|
||||||
c.Flags().DurationVar(&perNode, "timeout", 3*time.Minute, "how long a node may take to come back")
|
c.Flags().DurationVar(&perNode, "timeout", 3*time.Minute, "how long a node may take to come back")
|
||||||
c.Flags().StringVar(&serverSvc, "service", "llmgw-server", "this machine's systemd unit")
|
c.Flags().StringVar(&serverSvc, "service", "llmgw-server", "this machine's systemd unit")
|
||||||
|
c.Flags().IntVar(¶llel, "parallel", 20, "the most nodes upgraded at once, once the first waves are back")
|
||||||
|
c.Flags().StringVar(&sshUser, "ssh-user", os.Getenv("SUDO_USER"), "who to log in to the other control planes as")
|
||||||
|
c.Flags().BoolVar(&othersCP, "other-control-planes", true, "upgrade the cluster's other control planes too, over ssh")
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// thisLeads says whether this machine's control plane leads; one that does
|
||||||
|
// not say, alone or older, is taken to.
|
||||||
|
func thisLeads(ctx context.Context, client func() *ctlClient) bool {
|
||||||
|
var sum adminapi.Summary
|
||||||
|
if err := client().get(ctx, "/api/v1/summary", &sum); err != nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, cp := range sum.ControlPlanes.Members {
|
||||||
|
if cp.This {
|
||||||
|
return cp.State == "leading"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// upgradeOtherControlPlanes moves every other control plane of the cluster to
|
||||||
|
// v, over ssh with `llmgw upgrade`. Nothing is changed here if one cannot be
|
||||||
|
// reached.
|
||||||
|
func upgradeOtherControlPlanes(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, user string) error {
|
||||||
|
var sum adminapi.Summary
|
||||||
|
if err := client().get(ctx, "/api/v1/summary", &sum); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, cp := range sum.ControlPlanes.Members {
|
||||||
|
if cp.This {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
r := &onboard.SSH{Addr: cp.Address, User: user}
|
||||||
|
have, err := r.Run(ctx, "llmgw --version", nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("%s (%s) cannot be reached over ssh as %q: %v\n"+
|
||||||
|
"run `sudo llmgw upgrade %s --service llmgw-server` there, then this again with "+
|
||||||
|
"--other-control-planes=false", cp.Name, cp.Address, user, err, v)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(have), " "+v+"\n") || strings.HasSuffix(strings.TrimSpace(string(have)), " "+v) {
|
||||||
|
fmt.Fprintf(out, "== %s already runs %s\n", cp.Name, v)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "== %s (%s): %s -> %s\n", cp.Name, cp.State, strings.TrimSpace(string(have)), v)
|
||||||
|
res, err := r.Run(ctx, "sudo llmgw upgrade "+shellQuote(v)+" --service llmgw-server --releases "+shellQuote(releases), nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("upgrading %s: %v: %s", cp.Name, err, strings.TrimSpace(string(res)))
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, " upgraded and restarted\n")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
|
func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
|
||||||
if version.Version == v {
|
if version.Version == v {
|
||||||
fmt.Fprintf(out, "this machine already runs %s\n", v)
|
fmt.Fprintf(out, "this machine already runs %s\n", v)
|
||||||
@@ -133,13 +208,23 @@ func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlC
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient, v string, only []string, perNode time.Duration) error {
|
// upgradeNodes moves nodes to v in waves: one node first, then twice as
|
||||||
|
// many each time one comes back whole, up to parallel at once.
|
||||||
|
//
|
||||||
|
// Waves rather than one at a time, because one at a time is hours on a
|
||||||
|
// fleet of hundreds, and an agent's restart costs nothing served — the
|
||||||
|
// engines keep running and are adopted again. One first, because a release
|
||||||
|
// that does not come back should reach one node rather than a wave of them.
|
||||||
|
func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient, v string, only []string,
|
||||||
|
perNode time.Duration, parallel int,
|
||||||
|
) error {
|
||||||
var nodes []adminapi.NodeView
|
var nodes []adminapi.NodeView
|
||||||
if err := client().get(ctx, "/api/v1/nodes", &nodes); err != nil {
|
if err := client().get(ctx, "/api/v1/nodes", &nodes); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
|
slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
|
||||||
done, skipped := 0, 0
|
var todo []adminapi.NodeView
|
||||||
|
skipped := 0
|
||||||
for _, n := range nodes {
|
for _, n := range nodes {
|
||||||
if len(only) > 0 && !slices.Contains(only, n.ID) {
|
if len(only) > 0 && !slices.Contains(only, n.ID) {
|
||||||
continue
|
continue
|
||||||
@@ -148,41 +233,21 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
|||||||
skipped++
|
skipped++
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
start := time.Now()
|
todo = append(todo, n)
|
||||||
before := readyReplicasOn(ctx, client, n.ID)
|
|
||||||
fmt.Fprintf(out, "== %s: %s -> %s\n", n.ID, n.AgentVersion, v)
|
|
||||||
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
|
|
||||||
if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
|
|
||||||
return fmt.Errorf("%s: %w\nstopped here; `llmgw ctl logs %s` says more", n.ID, err, n.ID)
|
|
||||||
}
|
|
||||||
fmt.Fprintf(out, " new binary in place; waiting for it to come back with its %d replica(s)\n", before)
|
|
||||||
deadline := time.Now().Add(perNode)
|
|
||||||
for {
|
|
||||||
var now []adminapi.NodeView
|
|
||||||
_ = client().get(ctx, "/api/v1/nodes", &now)
|
|
||||||
var cur adminapi.NodeView
|
|
||||||
for _, x := range now {
|
|
||||||
if x.ID == n.ID {
|
|
||||||
cur = x
|
|
||||||
}
|
|
||||||
}
|
|
||||||
after := readyReplicasOn(ctx, client, n.ID)
|
|
||||||
if cur.AgentVersion == v && cur.State == "ready" && after >= before {
|
|
||||||
fmt.Fprintf(out, " done in %s: ready, replicas %d/%d\n",
|
|
||||||
time.Since(start).Round(time.Second), after, before)
|
|
||||||
done++
|
|
||||||
break
|
|
||||||
}
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
return fmt.Errorf("%s did not come back within %s (version %q, state %q, replicas %d/%d)\n"+
|
|
||||||
"stopped here, before the next node; `llmgw ctl logs %s` says why, and the node "+
|
|
||||||
"keeps its previous binary beside the new one", n.ID, perNode, cur.AgentVersion,
|
|
||||||
cur.State, after, before, n.ID)
|
|
||||||
}
|
|
||||||
time.Sleep(2 * time.Second)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
|
parallel = max(parallel, 1)
|
||||||
|
start, size, done := time.Now(), 1, 0
|
||||||
|
for len(todo) > 0 {
|
||||||
|
wave := todo[:min(size, len(todo))]
|
||||||
|
todo = todo[len(wave):]
|
||||||
|
if err := upgradeWave(ctx, out, client, v, wave, perNode); err != nil {
|
||||||
|
return fmt.Errorf("%w\nstopped after this wave: %d upgraded, %d not tried", err, done, len(todo))
|
||||||
|
}
|
||||||
|
done += len(wave)
|
||||||
|
size = min(size*2, parallel)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "%d node(s) upgraded to %s in %s, %d already on it\n",
|
||||||
|
done, v, time.Since(start).Round(time.Second), skipped)
|
||||||
// The one machine this does not reach. Said every time, because it is
|
// The one machine this does not reach. Said every time, because it is
|
||||||
// exactly the kind of thing that stays three versions behind unnoticed.
|
// exactly the kind of thing that stays three versions behind unnoticed.
|
||||||
fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
|
fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
|
||||||
@@ -190,23 +255,87 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// readyReplicasOn counts the ready replicas with a shard on a node.
|
// upgradeWave tells every node in the wave at once, and waits until each is
|
||||||
func readyReplicasOn(ctx context.Context, client func() *ctlClient, node string) int {
|
// back on v, ready, with as many ready replicas as it had.
|
||||||
var reps []adminapi.ReplicaView
|
func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v string,
|
||||||
if err := client().get(ctx, "/api/v1/replicas", &reps); err != nil {
|
wave []adminapi.NodeView, perNode time.Duration,
|
||||||
return 0
|
) error {
|
||||||
|
start := time.Now()
|
||||||
|
before := readyReplicasByNode(ctx, client)
|
||||||
|
ids := make([]string, len(wave))
|
||||||
|
for i, n := range wave {
|
||||||
|
ids[i] = n.ID
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "== %d node(s): %s\n", len(wave), abbreviate(ids, 8))
|
||||||
|
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
|
||||||
|
var failed []string
|
||||||
|
for _, n := range wave {
|
||||||
|
if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
|
||||||
|
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(failed) > 0 {
|
||||||
|
return fmt.Errorf("not told to upgrade:\n %s\n`llmgw ctl logs NODE` says more", strings.Join(failed, "\n "))
|
||||||
|
}
|
||||||
|
waiting := slices.Clone(ids)
|
||||||
|
deadline := time.Now().Add(perNode)
|
||||||
|
for {
|
||||||
|
var now []adminapi.NodeView
|
||||||
|
_ = client().get(ctx, "/api/v1/nodes", &now)
|
||||||
|
after := readyReplicasByNode(ctx, client)
|
||||||
|
state := map[string]adminapi.NodeView{}
|
||||||
|
for _, x := range now {
|
||||||
|
state[x.ID] = x
|
||||||
|
}
|
||||||
|
waiting = slices.DeleteFunc(waiting, func(id string) bool {
|
||||||
|
cur := state[id]
|
||||||
|
return cur.AgentVersion == v && cur.State == "ready" && after[id] >= before[id]
|
||||||
|
})
|
||||||
|
if len(waiting) == 0 {
|
||||||
|
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
var why []string
|
||||||
|
for _, id := range waiting {
|
||||||
|
cur := state[id]
|
||||||
|
why = append(why, fmt.Sprintf("%s (version %q, state %q, replicas %d/%d)",
|
||||||
|
id, cur.AgentVersion, cur.State, after[id], before[id]))
|
||||||
|
}
|
||||||
|
return fmt.Errorf("not back within %s:\n %s\n`llmgw ctl logs NODE` says why, and each keeps its "+
|
||||||
|
"previous binary beside the new one", perNode, strings.Join(why, "\n "))
|
||||||
|
}
|
||||||
|
time.Sleep(2 * time.Second)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// abbreviate lists up to n names, and says how many more.
|
||||||
|
func abbreviate(names []string, n int) string {
|
||||||
|
if len(names) <= n {
|
||||||
|
return strings.Join(names, ", ")
|
||||||
|
}
|
||||||
|
return strings.Join(names[:n], ", ") + fmt.Sprintf(" and %d more", len(names)-n)
|
||||||
|
}
|
||||||
|
|
||||||
|
// readyReplicasByNode counts, for each node, the ready replicas with a
|
||||||
|
// shard on it.
|
||||||
|
func readyReplicasByNode(ctx context.Context, client func() *ctlClient) map[string]int {
|
||||||
|
var reps []adminapi.ReplicaView
|
||||||
|
out := map[string]int{}
|
||||||
|
if err := client().get(ctx, "/api/v1/replicas", &reps); err != nil {
|
||||||
|
return out
|
||||||
}
|
}
|
||||||
n := 0
|
|
||||||
for _, r := range reps {
|
for _, r := range reps {
|
||||||
if r.State != "ready" {
|
if r.State != "ready" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
for _, s := range r.Shards {
|
seen := map[string]bool{}
|
||||||
if s.Node == node {
|
for _, sh := range r.Shards {
|
||||||
n++
|
if !seen[sh.Node] {
|
||||||
break
|
seen[sh.Node] = true
|
||||||
|
out[sh.Node]++
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return n
|
return out
|
||||||
}
|
}
|
||||||
@@ -3,6 +3,7 @@ package cli
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -57,32 +58,76 @@ func (f *fakeFleet) server(t *testing.T) *httptest.Server {
|
|||||||
return srv
|
return srv
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestAnUpgradeGoesNodeByNodeAndSkipsTheDone(t *testing.T) {
|
func TestAnUpgradeSkipsTheNodesAlreadyDone(t *testing.T) {
|
||||||
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.1", "c": "v1.0.0"}}
|
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.1", "c": "v1.0.0"}}
|
||||||
srv := f.server(t)
|
srv := f.server(t)
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second)
|
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 1)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if strings.Join(f.told, ",") != "a,c" {
|
if strings.Join(f.told, ",") != "a,c" {
|
||||||
t.Errorf("told %v; b was already on the version", f.told)
|
t.Errorf("told %v; b was already on the version", f.told)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "2 node(s) upgraded to v1.0.1, 1 already on it") {
|
if !strings.Contains(out.String(), "2 node(s) upgraded to v1.0.1 in") || !strings.Contains(out.String(), "1 already on it") {
|
||||||
t.Errorf("output:\n%s", out.String())
|
t.Errorf("output:\n%s", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestAnUpgradeStopsAtTheFirstNodeThatDoesNotComeBack(t *testing.T) {
|
func TestAnUpgradeStopsAtTheFirstWaveThatDoesNotComeBack(t *testing.T) {
|
||||||
// A bad release reaches one node, not all of them.
|
// A bad release reaches one node, not all of them.
|
||||||
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0"}, stuck: map[string]bool{"a": true}}
|
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0"}, stuck: map[string]bool{"a": true}}
|
||||||
srv := f.server(t)
|
srv := f.server(t)
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 3*time.Second)
|
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 3*time.Second, 1)
|
||||||
if err == nil || !strings.Contains(err.Error(), "a did not come back") {
|
if err == nil || !strings.Contains(err.Error(), "not back within") || !strings.Contains(err.Error(), "a (version") {
|
||||||
t.Fatalf("err %v", err)
|
t.Fatalf("err %v", err)
|
||||||
}
|
}
|
||||||
if len(f.told) != 1 {
|
if len(f.told) != 1 {
|
||||||
t.Errorf("went on to %v after a node failed to come back", f.told)
|
t.Errorf("went on to %v after a node failed to come back", f.told)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAnUpgradeGoesInWavesThatGrowUpToTheLimit(t *testing.T) {
|
||||||
|
// One first, then twice as many each time, up to --parallel: hundreds of
|
||||||
|
// nodes in minutes rather than hours, and a bad release still on one.
|
||||||
|
versions := map[string]string{}
|
||||||
|
for i := range 12 {
|
||||||
|
versions[fmt.Sprintf("n%02d", i)] = "v1.0.0"
|
||||||
|
}
|
||||||
|
f := &fakeFleet{versions: versions}
|
||||||
|
srv := f.server(t)
|
||||||
|
var out bytes.Buffer
|
||||||
|
if err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 4); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var sizes []string
|
||||||
|
for _, line := range strings.Split(out.String(), "\n") {
|
||||||
|
if n, ok := strings.CutPrefix(line, "== "); ok {
|
||||||
|
sizes = append(sizes, strings.Fields(n)[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got := strings.Join(sizes, ","); got != "1,2,4,4,1" {
|
||||||
|
t.Errorf("waves %s:\n%s", got, out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAnUpgradeKnowsWhetherItRunsOnTheLeader(t *testing.T) {
|
||||||
|
sum := adminapi.Summary{}
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
_ = json.NewEncoder(w).Encode(sum)
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
client := func() *ctlClient { return newCtlClient(srv.URL) }
|
||||||
|
if !thisLeads(t.Context(), client) {
|
||||||
|
t.Error("a control plane with no rival does not lead")
|
||||||
|
}
|
||||||
|
sum.ControlPlanes.Members = []adminapi.ControlPlane{{Name: "a", State: "leading"}, {Name: "b", State: "standing by", This: true}}
|
||||||
|
if thisLeads(t.Context(), client) {
|
||||||
|
t.Error("a standby took itself for the leader: it would upgrade the leader before itself")
|
||||||
|
}
|
||||||
|
sum.ControlPlanes.Members[0].This, sum.ControlPlanes.Members[1].This = true, false
|
||||||
|
if !thisLeads(t.Context(), client) {
|
||||||
|
t.Error("the leader took itself for a standby")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,257 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/x509"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httputil"
|
||||||
|
"net/url"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/net/http2"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||||
|
)
|
||||||
|
|
||||||
|
// forwardPrefix is where the leader answers a standby's admin API, on the
|
||||||
|
// agent port: the one port control planes already reach each other on, with
|
||||||
|
// the cluster's certificates both ways. The admin port itself listens on
|
||||||
|
// loopback, and should — it serves pprof.
|
||||||
|
const forwardPrefix = "/_admin"
|
||||||
|
|
||||||
|
// Headers a forwarded request and its answer carry: which standby asked, and
|
||||||
|
// which control plane answered.
|
||||||
|
const (
|
||||||
|
hdrForwardedBy = "X-Llmgw-Forwarded-By"
|
||||||
|
hdrAnsweredBy = "X-Llmgw-Answered-By"
|
||||||
|
)
|
||||||
|
|
||||||
|
// leaderForward sends a standby's admin API to the leader.
|
||||||
|
//
|
||||||
|
// Everything the console and ctl show is the leader's: nodes, replicas and
|
||||||
|
// jobs live in its memory, put there by the agent streams only it holds. A
|
||||||
|
// standby answering from its own would show an empty cluster, and accept
|
||||||
|
// changes it cannot carry out. So a standby answers nothing under /api/
|
||||||
|
// itself; the leader does, and the console looks the same on either machine.
|
||||||
|
type leaderForward struct {
|
||||||
|
st *store.DB
|
||||||
|
lead interface{ IsLeader() bool }
|
||||||
|
self string
|
||||||
|
tls tlsconf.Config
|
||||||
|
log *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
// standby wraps the admin handler: /api/ goes to the leader while this
|
||||||
|
// process is not it.
|
||||||
|
func (f *leaderForward) standby(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if adminStaysHere(r.URL.Path) || f.lead.IsLeader() {
|
||||||
|
next.ServeHTTP(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
target, name, err := f.leader(r.Context())
|
||||||
|
if err != nil {
|
||||||
|
writeForwardError(w, http.StatusServiceUnavailable,
|
||||||
|
"this control plane is standing by, and cannot reach the leader: "+err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
tc, err := f.tls.ClientConfig(target.Hostname())
|
||||||
|
if err != nil {
|
||||||
|
writeForwardError(w, http.StatusInternalServerError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p := &httputil.ReverseProxy{
|
||||||
|
Rewrite: func(pr *httputil.ProxyRequest) {
|
||||||
|
pr.Out.URL.Scheme, pr.Out.URL.Host = "https", target.Host
|
||||||
|
pr.Out.URL.Path = forwardPrefix + pr.In.URL.Path
|
||||||
|
pr.Out.URL.RawPath = ""
|
||||||
|
pr.Out.Host = target.Host
|
||||||
|
// The leader's own token is not this one's, and the
|
||||||
|
// certificate is what it believes.
|
||||||
|
pr.Out.Header.Del("Authorization")
|
||||||
|
pr.Out.Header.Set(hdrForwardedBy, f.self)
|
||||||
|
},
|
||||||
|
Transport: &http2.Transport{TLSClientConfig: tc},
|
||||||
|
// Streams, such as a job's progress, pass through as they come.
|
||||||
|
FlushInterval: -1,
|
||||||
|
ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) {
|
||||||
|
writeForwardError(w, http.StatusBadGateway, fmt.Sprintf("the leader, %s, did not answer: %v", name, err))
|
||||||
|
},
|
||||||
|
}
|
||||||
|
p.ServeHTTP(w, r)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// adminStaysHere are the admin paths about this process rather than the
|
||||||
|
// cluster: its profile, its own metrics, its own health. Never forwarded, and
|
||||||
|
// never answered for another — pprof dumps the heap, keys and all.
|
||||||
|
func adminStaysHere(path string) bool {
|
||||||
|
return strings.HasPrefix(path, "/debug/") || path == "/metrics" || path == "/health"
|
||||||
|
}
|
||||||
|
|
||||||
|
// publicStaysHere are the client port's paths that must describe this
|
||||||
|
// process: a load balancer and keepalived ask them which machine to send
|
||||||
|
// traffic to, and a standby saying the leader is ready would draw it.
|
||||||
|
func publicStaysHere(path string) bool {
|
||||||
|
return path == "/health" || path == "/ready" || path == "/leader" || path == "/metrics"
|
||||||
|
}
|
||||||
|
|
||||||
|
// public wraps the client-facing handler: on a standby, everything but its
|
||||||
|
// own health goes to the leader's client port — the portal, its admin pages
|
||||||
|
// and the API, which on a standby would show an empty cluster and serve
|
||||||
|
// nothing. Plain HTTP, as the port is; the leader trusts a control plane's
|
||||||
|
// X-Forwarded-For, so a client is still known by its own address.
|
||||||
|
func (f *leaderForward) public(next http.Handler, port string) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
// Once only: a request another control plane forwarded is answered
|
||||||
|
// here, leading or not, rather than sent on while the lease moves.
|
||||||
|
if publicStaysHere(r.URL.Path) || f.lead.IsLeader() || r.Header.Get(hdrForwardedBy) != "" {
|
||||||
|
next.ServeHTTP(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
target, _, err := f.leader(r.Context())
|
||||||
|
if err != nil || target.Hostname() == "" {
|
||||||
|
next.ServeHTTP(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
host := net.JoinHostPort(target.Hostname(), port)
|
||||||
|
p := &httputil.ReverseProxy{
|
||||||
|
Rewrite: func(pr *httputil.ProxyRequest) {
|
||||||
|
pr.Out.URL.Scheme, pr.Out.URL.Host = "http", host
|
||||||
|
// The name the client used, for the portal's links and cookies.
|
||||||
|
pr.Out.Host = pr.In.Host
|
||||||
|
peer, _, _ := net.SplitHostPort(pr.In.RemoteAddr)
|
||||||
|
xff := pr.In.Header.Get("X-Forwarded-For")
|
||||||
|
if xff != "" {
|
||||||
|
xff += ", "
|
||||||
|
}
|
||||||
|
pr.Out.Header.Set("X-Forwarded-For", xff+peer)
|
||||||
|
for _, h := range []string{"X-Forwarded-Proto", "X-Forwarded-Host"} {
|
||||||
|
if v := pr.In.Header.Get(h); v != "" {
|
||||||
|
pr.Out.Header.Set(h, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
pr.Out.Header.Set(hdrForwardedBy, f.self)
|
||||||
|
},
|
||||||
|
FlushInterval: -1,
|
||||||
|
ErrorHandler: func(w http.ResponseWriter, _ *http.Request, err error) {
|
||||||
|
writeForwardError(w, http.StatusBadGateway, "this control plane is standing by, and the leader did not answer: "+err.Error())
|
||||||
|
},
|
||||||
|
}
|
||||||
|
p.ServeHTTP(w, r)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// trustControlPlanes keeps the client port trusting the other control
|
||||||
|
// planes' X-Forwarded-For, besides router.trusted_proxies: a standby passes
|
||||||
|
// requests on, and the client is the address it was given, not the standby.
|
||||||
|
// Read again every half minute, for one added since.
|
||||||
|
func (f *leaderForward) trustControlPlanes(ctx context.Context, configured []string) {
|
||||||
|
for {
|
||||||
|
list := slices.Clone(configured)
|
||||||
|
if cps, err := f.st.ControlPlanes(ctx); err == nil {
|
||||||
|
for _, cp := range cps {
|
||||||
|
if u, err := url.Parse(cp.URL); err == nil && u.Hostname() != "" {
|
||||||
|
list = append(list, u.Hostname())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
clientip.SetTrusted(list)
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-time.After(30 * time.Second):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// leader is where the lease's holder is reached, and its name.
|
||||||
|
func (f *leaderForward) leader(ctx context.Context) (*url.URL, string, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
lease, err := f.st.Leadership(ctx, control.ControlRole)
|
||||||
|
if err != nil || lease.Owner == "" || time.Now().After(lease.ExpiresAt) {
|
||||||
|
return nil, "", errors.New("no control plane holds the lease right now")
|
||||||
|
}
|
||||||
|
name := control.HolderName(lease.Owner)
|
||||||
|
if name == f.self {
|
||||||
|
// This machine's previous process, whose lease has not run out.
|
||||||
|
return nil, name, errors.New("the lease is still held by this machine's previous run; it runs out within seconds")
|
||||||
|
}
|
||||||
|
cps, err := f.st.ControlPlanes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return nil, name, err
|
||||||
|
}
|
||||||
|
for _, cp := range cps {
|
||||||
|
if cp.Name == name {
|
||||||
|
u, err := url.Parse(cp.URL)
|
||||||
|
return u, name, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil, name, fmt.Errorf("%s leads, and is not among the control planes the store lists", name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// leaderSide answers forwarded requests, on the agent port: only from another
|
||||||
|
// control plane of this cluster, only while this one leads, and only the API.
|
||||||
|
func (f *leaderForward) leaderSide(admin http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
path := strings.TrimPrefix(r.URL.Path, forwardPrefix)
|
||||||
|
if adminStaysHere(path) {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := f.fromControlPlane(r); err != nil {
|
||||||
|
writeForwardError(w, http.StatusForbidden, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !f.lead.IsLeader() {
|
||||||
|
// Not passed on again: two standbys must not bounce a request
|
||||||
|
// between them while the lease moves.
|
||||||
|
writeForwardError(w, http.StatusServiceUnavailable, "this control plane no longer leads; ask again")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r2 := r.Clone(r.Context())
|
||||||
|
r2.URL.Path, r2.URL.RawPath = path, ""
|
||||||
|
w.Header().Set(hdrAnsweredBy, f.self)
|
||||||
|
admin.ServeHTTP(w, r2)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// fromControlPlane checks the client certificate names a control plane the
|
||||||
|
// store lists. A node's certificate is from the same authority, and a node
|
||||||
|
// must not reach the admin API by presenting it.
|
||||||
|
func (f *leaderForward) fromControlPlane(r *http.Request) error {
|
||||||
|
if r.TLS == nil || len(r.TLS.VerifiedChains) == 0 {
|
||||||
|
return errors.New("a control plane's certificate is required")
|
||||||
|
}
|
||||||
|
cert := r.TLS.VerifiedChains[0][0]
|
||||||
|
if !slices.Contains(cert.ExtKeyUsage, x509.ExtKeyUsageServerAuth) {
|
||||||
|
return errors.New("not a control plane's certificate")
|
||||||
|
}
|
||||||
|
cps, err := f.st.ControlPlanes(r.Context())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, cp := range cps {
|
||||||
|
if cp.Name == cert.Subject.CommonName {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fmt.Errorf("%s is not a control plane of this cluster", cert.Subject.CommonName)
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeForwardError(w http.ResponseWriter, code int, msg string) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(code)
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]string{"error": msg})
|
||||||
|
}
|
||||||
@@ -0,0 +1,244 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||||
|
)
|
||||||
|
|
||||||
|
type leading struct{ v atomic.Bool }
|
||||||
|
|
||||||
|
func (l *leading) IsLeader() bool { return l.v.Load() }
|
||||||
|
|
||||||
|
// forwardCluster is two control planes of one cluster on one store: a
|
||||||
|
// leader whose agent port answers forwarded admin requests, and a standby.
|
||||||
|
type forwardCluster struct {
|
||||||
|
st *store.DB
|
||||||
|
dir string
|
||||||
|
leaderLeads *leading
|
||||||
|
standby, leader *leaderForward
|
||||||
|
leaderURL string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newForwardCluster(t *testing.T) *forwardCluster {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
ca, err := tlsconf.NewCA("test CA")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := ca.Write(dir); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, name := range []string{"ctl-a", "ctl-b"} {
|
||||||
|
leaf, err := ca.IssuePeer(name, []string{"127.0.0.1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := leaf.Write(dir, name); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
node, err := ca.IssueClient("node1")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := node.Write(dir, "node1"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cfgFor := func(name string) tlsconf.Config {
|
||||||
|
return tlsconf.Config{CA: filepath.Join(dir, "ca.pem"), Cert: filepath.Join(dir, name+".pem"), Key: filepath.Join(dir, name+"-key.pem")}
|
||||||
|
}
|
||||||
|
|
||||||
|
st, err := store.Open(t.Context(), filepath.Join(dir, "s.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { st.Close() })
|
||||||
|
|
||||||
|
c := &forwardCluster{st: st, dir: dir, leaderLeads: &leading{}}
|
||||||
|
c.leaderLeads.v.Store(true)
|
||||||
|
log := slog.New(slog.DiscardHandler)
|
||||||
|
c.leader = &leaderForward{st: st, lead: c.leaderLeads, self: "ctl-a", tls: cfgFor("ctl-a"), log: log}
|
||||||
|
standbyLeads := &leading{}
|
||||||
|
c.standby = &leaderForward{st: st, lead: standbyLeads, self: "ctl-b", tls: cfgFor("ctl-b"), log: log}
|
||||||
|
|
||||||
|
// The leader's admin API: says whose it is, and who it was asked by.
|
||||||
|
admin := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]string{"path": r.URL.Path, "by": r.Header.Get(hdrForwardedBy),
|
||||||
|
"auth": r.Header.Get("Authorization"), "actor": r.Header.Get("X-Llmgw-Actor")})
|
||||||
|
})
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.Handle(forwardPrefix+"/", c.leader.leaderSide(admin))
|
||||||
|
srv := httptest.NewUnstartedServer(mux)
|
||||||
|
tc, err := cfgFor("ctl-a").ServerConfig()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
srv.TLS = tc
|
||||||
|
srv.EnableHTTP2 = true
|
||||||
|
srv.StartTLS()
|
||||||
|
t.Cleanup(srv.Close)
|
||||||
|
c.leaderURL = srv.URL
|
||||||
|
|
||||||
|
for _, cp := range []store.ControlPlane{{URL: srv.URL, Name: "ctl-a"}, {URL: "https://127.0.0.1:1", Name: "ctl-b"}} {
|
||||||
|
if err := st.PutControlPlane(t.Context(), cp); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, err := st.AcquireLeadership(t.Context(), control.ControlRole, "ctl-a-0123456789abcdef", time.Minute); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
// ask sends one request through the standby's admin API.
|
||||||
|
func (c *forwardCluster) ask(t *testing.T, path string) (*http.Response, map[string]string) {
|
||||||
|
t.Helper()
|
||||||
|
local := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]string{"path": "local"})
|
||||||
|
})
|
||||||
|
srv := httptest.NewServer(c.standby.standby(local))
|
||||||
|
defer srv.Close()
|
||||||
|
req, _ := http.NewRequestWithContext(t.Context(), http.MethodGet, srv.URL+path, nil)
|
||||||
|
req.Header.Set("Authorization", "Bearer the-standbys-own-token")
|
||||||
|
req.Header.Set("X-Llmgw-Actor", "root@ctl-b")
|
||||||
|
resp, err := http.DefaultClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
b, _ := io.ReadAll(resp.Body)
|
||||||
|
var got map[string]string
|
||||||
|
_ = json.Unmarshal(b, &got)
|
||||||
|
return resp, got
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAStandbysAdminAPIIsTheLeaders(t *testing.T) {
|
||||||
|
c := newForwardCluster(t)
|
||||||
|
resp, got := c.ask(t, "/api/v1/summary")
|
||||||
|
if resp.StatusCode != http.StatusOK || got["path"] != "/api/v1/summary" {
|
||||||
|
t.Fatalf("%d %v", resp.StatusCode, got)
|
||||||
|
}
|
||||||
|
if resp.Header.Get(hdrAnsweredBy) != "ctl-a" || got["by"] != "ctl-b" {
|
||||||
|
t.Errorf("answered by %q, asked by %q", resp.Header.Get(hdrAnsweredBy), got["by"])
|
||||||
|
}
|
||||||
|
if got["auth"] != "" {
|
||||||
|
t.Error("the standby's token was passed on to the leader")
|
||||||
|
}
|
||||||
|
if got["actor"] != "root@ctl-b" {
|
||||||
|
t.Errorf("who asked was lost: %q", got["actor"])
|
||||||
|
}
|
||||||
|
// The dashboard's pages are the leader's too.
|
||||||
|
if _, got := c.ask(t, "/nodes/cuda01"); got["path"] != "/nodes/cuda01" {
|
||||||
|
t.Errorf("a dashboard page was answered here: %v", got)
|
||||||
|
}
|
||||||
|
// What is about this process stays on it: pprof, its metrics.
|
||||||
|
for _, p := range []string{"/debug/pprof/", "/metrics"} {
|
||||||
|
if _, got := c.ask(t, p); got["path"] != "local" {
|
||||||
|
t.Errorf("%s was forwarded: %v", p, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheLeaderAnswersOnlyControlPlanesAndOnlyWhileItLeads(t *testing.T) {
|
||||||
|
c := newForwardCluster(t)
|
||||||
|
call := func(cert string) int {
|
||||||
|
cfg := tlsconf.Config{CA: filepath.Join(c.dir, "ca.pem"), Cert: filepath.Join(c.dir, cert+".pem"), Key: filepath.Join(c.dir, cert+"-key.pem")}
|
||||||
|
tc, err := cfg.ClientConfig("127.0.0.1")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
tc.NextProtos = nil
|
||||||
|
client := &http.Client{Transport: &http.Transport{TLSClientConfig: tc}}
|
||||||
|
resp, err := client.Get(c.leaderURL + forwardPrefix + "/api/v1/summary")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
return resp.StatusCode
|
||||||
|
}
|
||||||
|
if code := call("ctl-b"); code != http.StatusOK {
|
||||||
|
t.Errorf("a control plane was refused: %d", code)
|
||||||
|
}
|
||||||
|
// A node's certificate is from the same authority, and is no way in.
|
||||||
|
if code := call("node1"); code != http.StatusForbidden {
|
||||||
|
t.Errorf("a node reached the admin API: %d", code)
|
||||||
|
}
|
||||||
|
c.leaderLeads.v.Store(false)
|
||||||
|
if code := call("ctl-b"); code != http.StatusServiceUnavailable {
|
||||||
|
t.Errorf("a control plane that no longer leads answered: %d", code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAStandbyWithNoLeaderSaysSo(t *testing.T) {
|
||||||
|
c := newForwardCluster(t)
|
||||||
|
if err := c.st.ReleaseLeadership(t.Context(), control.ControlRole, "ctl-a-0123456789abcdef"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
resp, got := c.ask(t, "/api/v1/summary")
|
||||||
|
if resp.StatusCode != http.StatusServiceUnavailable || !strings.Contains(got["error"], "standing by") {
|
||||||
|
t.Errorf("%d %v", resp.StatusCode, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAStandbysPortalIsTheLeaders(t *testing.T) {
|
||||||
|
c := newForwardCluster(t)
|
||||||
|
// The leader's client port, on the address the store lists for it.
|
||||||
|
var got http.Header
|
||||||
|
var gotHost string
|
||||||
|
leader := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
got, gotHost = r.Header.Clone(), r.Host
|
||||||
|
_, _ = w.Write([]byte("leader " + r.URL.Path))
|
||||||
|
}))
|
||||||
|
defer leader.Close()
|
||||||
|
_, port, _ := net.SplitHostPort(strings.TrimPrefix(leader.URL, "http://"))
|
||||||
|
|
||||||
|
local := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("standby " + r.URL.Path)) })
|
||||||
|
srv := httptest.NewServer(c.standby.public(local, port))
|
||||||
|
defer srv.Close()
|
||||||
|
get := func(path string, hdr map[string]string) string {
|
||||||
|
req, _ := http.NewRequestWithContext(t.Context(), http.MethodGet, srv.URL+path, nil)
|
||||||
|
req.Host = "llmgw.example"
|
||||||
|
for k, v := range hdr {
|
||||||
|
req.Header.Set(k, v)
|
||||||
|
}
|
||||||
|
resp, err := http.DefaultClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
b, _ := io.ReadAll(resp.Body)
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
if body := get("/admin/replicas", map[string]string{"X-Forwarded-For": "203.0.113.7", "X-Forwarded-Proto": "https"}); body != "leader /admin/replicas" {
|
||||||
|
t.Fatalf("the portal was answered by %q", body)
|
||||||
|
}
|
||||||
|
if gotHost != "llmgw.example" || got.Get("X-Forwarded-Proto") != "https" ||
|
||||||
|
!strings.HasPrefix(got.Get("X-Forwarded-For"), "203.0.113.7, ") || got.Get(hdrForwardedBy) != "ctl-b" {
|
||||||
|
t.Errorf("host %q, headers %v", gotHost, got)
|
||||||
|
}
|
||||||
|
// What a load balancer asks is about this machine.
|
||||||
|
for _, p := range []string{"/ready", "/health", "/leader"} {
|
||||||
|
if body := get(p, nil); body != "standby "+p {
|
||||||
|
t.Errorf("%s answered by %q", p, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Forwarded once only.
|
||||||
|
if body := get("/v1/models", map[string]string{hdrForwardedBy: "ctl-c"}); body != "standby /v1/models" {
|
||||||
|
t.Errorf("a forwarded request was forwarded again: %q", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -73,6 +73,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
|||||||
f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
|
f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
|
||||||
f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
|
f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
|
||||||
f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
|
f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
|
||||||
|
f.BoolVar(&req.Tools, "tools", false,
|
||||||
|
"say it makes tool calls (for agents and editors); check a running one with llmgw ctl tool-check")
|
||||||
f.StringVar(&req.GuardFormat, "guard-format", "",
|
f.StringVar(&req.GuardFormat, "guard-format", "",
|
||||||
"serve it as the guardrails' judge (role: guard), read as llamaguard3, shieldgemma or json; hidden from clients")
|
"serve it as the guardrails' judge (role: guard), read as llamaguard3, shieldgemma or json; hidden from clients")
|
||||||
f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
|
f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
|
||||||
|
|||||||
+34
-2
@@ -330,6 +330,9 @@ func newServerCmd() *cobra.Command {
|
|||||||
return out, err
|
return out, err
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
if st := rt.Store(); st != nil {
|
||||||
|
api.SetControlPlanes(newControlPlaneProbe(st, cfg.Listen).Status)
|
||||||
|
}
|
||||||
adminAPI = api
|
adminAPI = api
|
||||||
api.SetReleases(cfg.Releases)
|
api.SetReleases(cfg.Releases)
|
||||||
api.SetNotifier(notifier)
|
api.SetNotifier(notifier)
|
||||||
@@ -499,8 +502,23 @@ func newServerCmd() *cobra.Command {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
adminHandler := auditAPI(rt, adminMux, log)
|
||||||
|
// With a rival, a standby's admin API is the leader's: what the
|
||||||
|
// console and ctl show lives in the leader's memory.
|
||||||
|
var forward *leaderForward
|
||||||
|
if admin != nil && admin.Leadership() != nil && rt.Store() != nil && cfg.AgentTLS.Enabled() {
|
||||||
|
self, _ := os.Hostname()
|
||||||
|
forward = &leaderForward{st: rt.Store(), lead: admin.Leadership(), self: self, tls: cfg.AgentTLS, log: log}
|
||||||
|
}
|
||||||
|
served := adminHandler
|
||||||
|
if forward != nil {
|
||||||
|
served = forward.standby(adminHandler)
|
||||||
|
_, port, _ := net.SplitHostPort(cfg.Listen)
|
||||||
|
apiSrv.Handler = forward.public(apiSrv.Handler, port)
|
||||||
|
go forward.trustControlPlanes(ctx, cfg.Router.TrustedProxies)
|
||||||
|
}
|
||||||
adminSrv := withLimits(&http.Server{
|
adminSrv := withLimits(&http.Server{
|
||||||
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, auditAPI(rt, adminMux, log))),
|
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, served)),
|
||||||
ReadTimeout: time.Minute,
|
ReadTimeout: time.Minute,
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -566,6 +584,9 @@ func newServerCmd() *cobra.Command {
|
|||||||
if agentHandler != nil {
|
if agentHandler != nil {
|
||||||
mux := http.NewServeMux()
|
mux := http.NewServeMux()
|
||||||
mux.Handle(agentPath, agentHandler)
|
mux.Handle(agentPath, agentHandler)
|
||||||
|
if forward != nil {
|
||||||
|
mux.Handle(forwardPrefix+"/", router.AdminGuard(forward.leaderSide(adminHandler)))
|
||||||
|
}
|
||||||
// No read timeout: an agent's stream is one request that
|
// No read timeout: an agent's stream is one request that
|
||||||
// lasts as long as the agent does.
|
// lasts as long as the agent does.
|
||||||
agentSrv = withLimits(&http.Server{Handler: mux})
|
agentSrv = withLimits(&http.Server{Handler: mux})
|
||||||
@@ -617,8 +638,19 @@ func newServerCmd() *cobra.Command {
|
|||||||
log.Info("shutting down")
|
log.Info("shutting down")
|
||||||
sctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
sctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
// The lease first: a standby polls for it, and every agent
|
||||||
|
// about to be let go will be refused by a standby that has
|
||||||
|
// not taken it yet.
|
||||||
|
if admin != nil && admin.Leadership() != nil {
|
||||||
|
admin.Leadership().StepDown()
|
||||||
|
}
|
||||||
if agentSrv != nil {
|
if agentSrv != nil {
|
||||||
_ = agentSrv.Shutdown(sctx)
|
// Closed, not shut down gracefully: an agent's stream
|
||||||
|
// never goes idle, so Shutdown would wait out its whole
|
||||||
|
// timeout while every node stayed on a control plane
|
||||||
|
// that is leaving, and a standby could not be found
|
||||||
|
// until it gave up. Closed, they move at once.
|
||||||
|
_ = agentSrv.Close()
|
||||||
}
|
}
|
||||||
if metricsSrv != nil {
|
if metricsSrv != nil {
|
||||||
_ = metricsSrv.Shutdown(sctx)
|
_ = metricsSrv.Shutdown(sctx)
|
||||||
|
|||||||
@@ -121,6 +121,7 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
|||||||
{"connect", sa.connect},
|
{"connect", sa.connect},
|
||||||
{"binary", sa.binary},
|
{"binary", sa.binary},
|
||||||
{"directories", sa.dirs},
|
{"directories", sa.dirs},
|
||||||
|
{"this certificate", sa.ownCertificate},
|
||||||
{"certificate", sa.certificate},
|
{"certificate", sa.certificate},
|
||||||
{"configuration", sa.configuration},
|
{"configuration", sa.configuration},
|
||||||
{"store", sa.storeAccess},
|
{"store", sa.storeAccess},
|
||||||
@@ -132,6 +133,10 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
|||||||
fn func(context.Context) error
|
fn func(context.Context) error
|
||||||
}{"firewall", sa.firewall})
|
}{"firewall", sa.firewall})
|
||||||
}
|
}
|
||||||
|
steps = append(steps, struct {
|
||||||
|
name string
|
||||||
|
fn func(context.Context) error
|
||||||
|
}{"mirror", sa.mirror})
|
||||||
if sa.vip != "" {
|
if sa.vip != "" {
|
||||||
steps = append(steps, struct {
|
steps = append(steps, struct {
|
||||||
name string
|
name string
|
||||||
@@ -227,6 +232,40 @@ sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw`, nil)
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ownCertificate makes this control plane's certificate one it can also
|
||||||
|
// present as a client — a standby asks the leader for the console's data
|
||||||
|
// with it — reissuing one from before that with the same names. The running
|
||||||
|
// server reads it as it starts.
|
||||||
|
func (sa *serverAdd) ownCertificate(context.Context) error {
|
||||||
|
path := filepath.Join(sa.etcDir, "server.pem")
|
||||||
|
cert, err := readCert(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if clientCapable(cert) {
|
||||||
|
sa.kept("this control plane's certificate")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
ca, err := loadCA(sa.pkiDir)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("the authority's key is not in %s: run this where it is (%v)", sa.pkiDir, err)
|
||||||
|
}
|
||||||
|
hosts := strings.Split(sanNames(cert), ", ")
|
||||||
|
leaf, err := ca.IssuePeer(cert.Subject.CommonName, hosts)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, leaf.CertPEM, 0o644); err != nil { //nolint:gosec // a certificate
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(sa.etcDir, "server-key.pem"), leaf.KeyPEM, 0o640); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
sa.done("this control plane's certificate reissued for %s, to ask the leader as well as answer;\n"+
|
||||||
|
" restart llmgw-server here for it to take effect — %s takes over meanwhile", strings.Join(hosts, ", "), sa.name)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// certificate issues it a server certificate from this cluster's authority,
|
// certificate issues it a server certificate from this cluster's authority,
|
||||||
// for its name and addresses, unless it holds one that already names them.
|
// for its name and addresses, unless it holds one that already names them.
|
||||||
func (sa *serverAdd) certificate(ctx context.Context) error {
|
func (sa *serverAdd) certificate(ctx context.Context) error {
|
||||||
@@ -251,14 +290,14 @@ func (sa *serverAdd) certificate(ctx context.Context) error {
|
|||||||
for _, w := range want {
|
for _, w := range want {
|
||||||
missing = missing || !slices.Contains(names, w)
|
missing = missing || !slices.Contains(names, w)
|
||||||
}
|
}
|
||||||
if !missing {
|
if !missing && clientCapable(cert) {
|
||||||
sa.kept("certificate for %s", strings.Join(names, ", "))
|
sa.kept("certificate for %s", strings.Join(names, ", "))
|
||||||
return sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM)
|
return sa.put(ctx, "/etc/llmgw/ca.pem", "0644", "root:root", ca.PEM)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
leaf, err := ca.IssueServer(sa.name, want)
|
leaf, err := ca.IssuePeer(sa.name, want)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -419,6 +458,38 @@ func (sa *serverAdd) service(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// mirror checks that the mirror's management API answers the new control
|
||||||
|
// plane. It answers only the addresses and certificates it was told to, and
|
||||||
|
// one told only of the first control plane refuses the second the moment it
|
||||||
|
// leads: models cannot be fetched, updated or served from the registry.
|
||||||
|
//
|
||||||
|
// Said, not fixed: the mirror is its own machine with its own unit, and
|
||||||
|
// llmgw does not log in to it.
|
||||||
|
func (sa *serverAdd) mirror(ctx context.Context) error {
|
||||||
|
cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
|
||||||
|
if err != nil || cfg.Mirror.API == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if cfg.Mirror.TLS.Enabled() {
|
||||||
|
fmt.Fprintf(sa.out, " note the mirror at %s answers only the certificates it names: add\n"+
|
||||||
|
" --api-client %s to its llmgw-mirror unit, or it refuses %s when it leads\n",
|
||||||
|
cfg.Mirror.API, sa.name, sa.name)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
code, _ := sa.sh(ctx, "curl -s -o /dev/null -m 3 -w '%{http_code}' "+shellQuote(strings.TrimRight(cfg.Mirror.API, "/")+"/")+" || true", nil)
|
||||||
|
switch code = strings.TrimSpace(code); code {
|
||||||
|
case "403":
|
||||||
|
fmt.Fprintf(sa.out, " !! the mirror at %s refuses %s: add --api-allow %s to its\n"+
|
||||||
|
" llmgw-mirror unit and restart it, or models cannot be fetched while %s leads\n",
|
||||||
|
cfg.Mirror.API, sa.name, sa.ip, sa.name)
|
||||||
|
case "", "000":
|
||||||
|
fmt.Fprintf(sa.out, " !! the mirror at %s does not answer %s\n", cfg.Mirror.API, sa.name)
|
||||||
|
default:
|
||||||
|
sa.done("the mirror at %s answers it", cfg.Mirror.API)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// floating keeps an address on whichever of the two leads, with keepalived
|
// floating keeps an address on whichever of the two leads, with keepalived
|
||||||
// on both, following /leader.
|
// on both, following /leader.
|
||||||
func (sa *serverAdd) floating(ctx context.Context) error {
|
func (sa *serverAdd) floating(ctx context.Context) error {
|
||||||
@@ -453,6 +524,12 @@ sudo systemctl restart keepalived`, []byte(conf)); err != nil {
|
|||||||
return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
|
return fmt.Errorf("on %s: %v: %s", m.name, err, strings.TrimSpace(string(out)))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Where the summary reads it, on both: readable, unlike keepalived's.
|
||||||
|
for _, r := range []onboard.Remote{here, sa.remote} {
|
||||||
|
if out, err := r.Run(ctx, "sudo sh -c 'cat > "+floatingPath+" && chmod 0644 "+floatingPath+"'", []byte(sa.vip+"\n")); err != nil {
|
||||||
|
return fmt.Errorf("recording it: %v: %s", err, strings.TrimSpace(string(out)))
|
||||||
|
}
|
||||||
|
}
|
||||||
sa.done("%s follows whichever control plane leads (keepalived on both)", sa.vip)
|
sa.done("%s follows whichever control plane leads (keepalived on both)", sa.vip)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -503,3 +580,9 @@ vrrp_instance llmgw {
|
|||||||
}
|
}
|
||||||
`, iface, priority, pass, vip, prefix)
|
`, iface, priority, pass, vip, prefix)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// clientCapable says a control plane's certificate can also present itself
|
||||||
|
// as a client, as a standby does to the leader.
|
||||||
|
func clientCapable(cert *x509.Certificate) bool {
|
||||||
|
return slices.Contains(cert.ExtKeyUsage, x509.ExtKeyUsageClientAuth)
|
||||||
|
}
|
||||||
@@ -462,7 +462,10 @@ func (in *serverInit) ensureCerts(context.Context) error {
|
|||||||
// A certificate from this authority naming every address is left
|
// A certificate from this authority naming every address is left
|
||||||
// alone. One naming fewer is reissued with the union, so an address
|
// alone. One naming fewer is reissued with the union, so an address
|
||||||
// that worked yesterday is never dropped by adding one today.
|
// that worked yesterday is never dropped by adding one today.
|
||||||
if len(missing) == 0 && cert.CheckSignatureFrom(ca.Cert) == nil {
|
// And one that can also present itself as a client, which a
|
||||||
|
// standby's admin API needs to reach the leader's; certificates
|
||||||
|
// from before that are reissued.
|
||||||
|
if len(missing) == 0 && cert.CheckSignatureFrom(ca.Cert) == nil && clientCapable(cert) {
|
||||||
in.kept("server certificate for %s", strings.Join(have, ", "))
|
in.kept("server certificate for %s", strings.Join(have, ", "))
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -477,7 +480,7 @@ func (in *serverInit) ensureCerts(context.Context) error {
|
|||||||
if name == "" {
|
if name == "" {
|
||||||
name = "llmgw-control"
|
name = "llmgw-control"
|
||||||
}
|
}
|
||||||
leaf, err := ca.IssueServer(name, want)
|
leaf, err := ca.IssuePeer(name, want)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -95,6 +95,39 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
|
|||||||
fmt.Fprintf(out, "\nMODELS %d of %d have the replicas they asked for\n",
|
fmt.Fprintf(out, "\nMODELS %d of %d have the replicas they asked for\n",
|
||||||
s.Models.Satisfied, s.Models.Total)
|
s.Models.Satisfied, s.Models.Total)
|
||||||
|
|
||||||
|
// The control planes, when there are several: who leads, whether the
|
||||||
|
// others would be there to take over, and the address clients use.
|
||||||
|
if cps := s.ControlPlanes; len(cps.Members) > 0 {
|
||||||
|
var parts []string
|
||||||
|
leader := false
|
||||||
|
for _, cp := range cps.Members {
|
||||||
|
p := cp.Name
|
||||||
|
switch cp.State {
|
||||||
|
case "leading":
|
||||||
|
leader = true
|
||||||
|
p += " leads"
|
||||||
|
default:
|
||||||
|
p += " " + cp.State
|
||||||
|
}
|
||||||
|
if cp.This {
|
||||||
|
p += " (this one)"
|
||||||
|
}
|
||||||
|
parts = append(parts, p)
|
||||||
|
}
|
||||||
|
if !leader {
|
||||||
|
parts = append([]string{"NO LEADER — nothing is placed"}, parts...)
|
||||||
|
}
|
||||||
|
line := strings.Join(parts, ", ")
|
||||||
|
if cps.Floating != "" {
|
||||||
|
if cps.FloatingUp {
|
||||||
|
line += " · " + cps.Floating + " on the leader"
|
||||||
|
} else {
|
||||||
|
line += " · " + cps.Floating + " NOT ANSWERING"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "CONTROL %s\n", line)
|
||||||
|
}
|
||||||
|
|
||||||
// The store, when it is a replicated cluster: one line, saying who leads
|
// The store, when it is a replicated cluster: one line, saying who leads
|
||||||
// and whether every other member keeps up.
|
// and whether every other member keeps up.
|
||||||
switch {
|
switch {
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"net"
|
||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
@@ -42,6 +43,7 @@ const (
|
|||||||
cReplicas
|
cReplicas
|
||||||
cUsage
|
cUsage
|
||||||
cQuotas
|
cQuotas
|
||||||
|
cGuard
|
||||||
cFederation
|
cFederation
|
||||||
cMirror
|
cMirror
|
||||||
cJobs
|
cJobs
|
||||||
@@ -53,6 +55,7 @@ const (
|
|||||||
cManual
|
cManual
|
||||||
cSettings
|
cSettings
|
||||||
cAddNode
|
cAddNode
|
||||||
|
cAddControl
|
||||||
cReplicate
|
cReplicate
|
||||||
cServe
|
cServe
|
||||||
cUpgrade
|
cUpgrade
|
||||||
@@ -62,8 +65,8 @@ const (
|
|||||||
cCount
|
cCount
|
||||||
)
|
)
|
||||||
|
|
||||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
|
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
|
||||||
"server.yaml", "Add a node", "Replicate the store", "Serve a model", "Upgrade the cluster",
|
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Upgrade the cluster",
|
||||||
"Restart all replicas", "Back up now", "Leave"}
|
"Restart all replicas", "Back up now", "Leave"}
|
||||||
|
|
||||||
// consoleAction is what is done once the screen closes: each runs its own
|
// consoleAction is what is done once the screen closes: each runs its own
|
||||||
@@ -167,6 +170,7 @@ type consoleModel struct {
|
|||||||
|
|
||||||
quotas []adminapi.QuotaView
|
quotas []adminapi.QuotaView
|
||||||
quotasErr error
|
quotasErr error
|
||||||
|
guard guardState
|
||||||
|
|
||||||
st *storeHolder
|
st *storeHolder
|
||||||
people peopleData
|
people peopleData
|
||||||
@@ -290,6 +294,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
|||||||
if m.menu == cQuotas {
|
if m.menu == cQuotas {
|
||||||
cmds = append(cmds, m.fetchQuotas())
|
cmds = append(cmds, m.fetchQuotas())
|
||||||
}
|
}
|
||||||
|
if m.menu == cGuard {
|
||||||
|
cmds = append(cmds, m.fetchGuard())
|
||||||
|
}
|
||||||
if m.menu == cMirror {
|
if m.menu == cMirror {
|
||||||
cmds = append(cmds, m.fetchMirror())
|
cmds = append(cmds, m.fetchMirror())
|
||||||
}
|
}
|
||||||
@@ -348,6 +355,9 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
|||||||
m.people = peopleData(msg)
|
m.people = peopleData(msg)
|
||||||
case consoleQuotasMsg:
|
case consoleQuotasMsg:
|
||||||
m.quotas, m.quotasErr = msg.rows, msg.err
|
m.quotas, m.quotasErr = msg.rows, msg.err
|
||||||
|
case consoleGuardMsg:
|
||||||
|
window := m.guard.window
|
||||||
|
m.guard = guardState{view: msg.view, events: msg.events, err: msg.err, window: window}
|
||||||
case evalSuitesMsg:
|
case evalSuitesMsg:
|
||||||
return m.onEvalSuites(msg)
|
return m.onEvalSuites(msg)
|
||||||
case evalPollMsg:
|
case evalPollMsg:
|
||||||
@@ -447,6 +457,14 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
|||||||
switch m.menu {
|
switch m.menu {
|
||||||
case cAddNode:
|
case cAddNode:
|
||||||
return m.ask("Add a node", "user@address of the machine", 1)
|
return m.ask("Add a node", "user@address of the machine", 1)
|
||||||
|
case cAddControl:
|
||||||
|
if !m.storeReplicated() {
|
||||||
|
m.dialog, m.dialogTitle = "error", "Add a control plane"
|
||||||
|
m.dialogText = "A second control plane needs a store both can reach, and this one's\n" +
|
||||||
|
"is on one machine. Replicate the store first (Do → Replicate the store)."
|
||||||
|
return m, nil
|
||||||
|
}
|
||||||
|
return m.ask("Add a control plane", "user@address of the machine", 2)
|
||||||
case cReplicate:
|
case cReplicate:
|
||||||
return m.ask("Replicate the store", "user@address of database machine", 2)
|
return m.ask("Replicate the store", "user@address of database machine", 2)
|
||||||
case cLeave:
|
case cLeave:
|
||||||
@@ -491,6 +509,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
|||||||
m.menuD = m.replicaMenu(m.data.replicas[m.row])
|
m.menuD = m.replicaMenu(m.data.replicas[m.row])
|
||||||
case cQuotas:
|
case cQuotas:
|
||||||
m.menuD = m.quotaMenu(m.quotas[m.row])
|
m.menuD = m.quotaMenu(m.quotas[m.row])
|
||||||
|
case cGuard:
|
||||||
|
m.menuD = m.guardAttachmentMenu(m.guard.view.Attachments[m.row])
|
||||||
case cMirror:
|
case cMirror:
|
||||||
m.menuD = m.mirrorMenu(m.mirrorGroups()[m.row])
|
m.menuD = m.mirrorMenu(m.mirrorGroups()[m.row])
|
||||||
case cJobs:
|
case cJobs:
|
||||||
@@ -546,6 +566,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
|||||||
switch m.menu {
|
switch m.menu {
|
||||||
case cQuotas:
|
case cQuotas:
|
||||||
m.quotaForm(adminapi.QuotaView{})
|
m.quotaForm(adminapi.QuotaView{})
|
||||||
|
case cGuard:
|
||||||
|
m.guardAttachForm()
|
||||||
case cModels:
|
case cModels:
|
||||||
m.addModelForm()
|
m.addModelForm()
|
||||||
case cMirror:
|
case cMirror:
|
||||||
@@ -557,6 +579,11 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
|||||||
case cOrgs:
|
case cOrgs:
|
||||||
m.createOrgForm()
|
m.createOrgForm()
|
||||||
}
|
}
|
||||||
|
case "w":
|
||||||
|
if m.menu == cGuard {
|
||||||
|
m.guard.window = (m.guard.window + 1) % len(guardWindows)
|
||||||
|
return m, m.fetchGuard()
|
||||||
|
}
|
||||||
case "i":
|
case "i":
|
||||||
if m.menu == cMirror {
|
if m.menu == cMirror {
|
||||||
m.mirrorImportOllamaForm()
|
m.mirrorImportOllamaForm()
|
||||||
@@ -594,6 +621,8 @@ func (m consoleModel) onEnter() tea.Cmd {
|
|||||||
return m.fetchUsage()
|
return m.fetchUsage()
|
||||||
case cQuotas:
|
case cQuotas:
|
||||||
return m.fetchQuotas()
|
return m.fetchQuotas()
|
||||||
|
case cGuard:
|
||||||
|
return m.fetchGuard()
|
||||||
case cMirror:
|
case cMirror:
|
||||||
return m.fetchMirror()
|
return m.fetchMirror()
|
||||||
case cMonitor:
|
case cMonitor:
|
||||||
@@ -659,19 +688,38 @@ func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
|||||||
v := strings.TrimSpace(m.prompt.Value())
|
v := strings.TrimSpace(m.prompt.Value())
|
||||||
switch m.dialog {
|
switch m.dialog {
|
||||||
case "target":
|
case "target":
|
||||||
if v == "" {
|
// The floating address is the one answer that may be left empty.
|
||||||
|
floating := m.menu == cAddControl && len(m.asked) == 1
|
||||||
|
if v == "" && !floating {
|
||||||
return m, nil
|
return m, nil
|
||||||
}
|
}
|
||||||
|
if floating && v != "" {
|
||||||
|
if _, _, err := net.ParseCIDR(v); err != nil {
|
||||||
|
m.dialogText = floatingQuestion + "\n\n" + sBad.Render("not an address with its prefix, like 10.0.0.161/24")
|
||||||
|
return m, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
m.asked = append(m.asked, v)
|
m.asked = append(m.asked, v)
|
||||||
m.prompt.SetValue("")
|
m.prompt.SetValue("")
|
||||||
if len(m.asked) == int(m.editKind) {
|
if len(m.asked) == int(m.editKind) {
|
||||||
kind := "node"
|
switch m.menu {
|
||||||
if m.menu == cReplicate {
|
case cAddControl:
|
||||||
kind = "patroni"
|
args := []string{"server", "add", m.asked[0]}
|
||||||
|
if m.asked[1] != "" {
|
||||||
|
args = append(args, "--vip", m.asked[1])
|
||||||
|
}
|
||||||
|
m.action = consoleAction{kind: "run", targets: args}
|
||||||
|
case cReplicate:
|
||||||
|
m.action = consoleAction{kind: "patroni", targets: m.asked}
|
||||||
|
default:
|
||||||
|
m.action = consoleAction{kind: "node", targets: m.asked}
|
||||||
}
|
}
|
||||||
m.action = consoleAction{kind: kind, targets: m.asked}
|
|
||||||
return m, tea.Quit
|
return m, tea.Quit
|
||||||
}
|
}
|
||||||
|
if m.menu == cAddControl {
|
||||||
|
m.dialogText = floatingQuestion
|
||||||
|
m.prompt.Placeholder = "10.0.0.161/24"
|
||||||
|
}
|
||||||
case "edit":
|
case "edit":
|
||||||
if err := m.doc.Set(m.editPath, m.editKind, m.prompt.Value()); err != nil {
|
if err := m.doc.Set(m.editPath, m.editKind, m.prompt.Value()); err != nil {
|
||||||
m.dialogText = m.editQuestion() + "\n\n" + sBad.Render(err.Error())
|
m.dialogText = m.editQuestion() + "\n\n" + sBad.Render(err.Error())
|
||||||
@@ -910,6 +958,8 @@ func (m consoleModel) rowCount() int {
|
|||||||
return len(m.usage)
|
return len(m.usage)
|
||||||
case cQuotas:
|
case cQuotas:
|
||||||
return len(m.quotas)
|
return len(m.quotas)
|
||||||
|
case cGuard:
|
||||||
|
return len(m.guard.view.Attachments)
|
||||||
case cMirror:
|
case cMirror:
|
||||||
return len(m.mirrorGroups())
|
return len(m.mirrorGroups())
|
||||||
case cJobs:
|
case cJobs:
|
||||||
@@ -1036,7 +1086,7 @@ func (m consoleModel) panel() (title, body, keys string) {
|
|||||||
c := &cobra.Command{}
|
c := &cobra.Command{}
|
||||||
c.SetOut(&b)
|
c.SetOut(&b)
|
||||||
printSummary(c, m.data.sum)
|
printSummary(c, m.data.sum)
|
||||||
return "Overview", "\n" + indent(b.String(), " "), keys
|
return "Overview", m.standbyNote() + "\n" + indent(b.String(), " "), keys
|
||||||
case cNodes, cLogs:
|
case cNodes, cLogs:
|
||||||
if m.detail && m.row < len(m.data.nodes) {
|
if m.detail && m.row < len(m.data.nodes) {
|
||||||
n := m.data.nodes[m.row]
|
n := m.data.nodes[m.row]
|
||||||
@@ -1137,6 +1187,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
|||||||
return "Usage", head + table([]string{"WHO", "REQUESTS", "TOKENS", "ERRORS", "SPENT"}, rows, m.row, pw, m.inPanel), keys
|
return "Usage", head + table([]string{"WHO", "REQUESTS", "TOKENS", "ERRORS", "SPENT"}, rows, m.row, pw, m.inPanel), keys
|
||||||
case cQuotas:
|
case cQuotas:
|
||||||
return m.quotasPanel(pw)
|
return m.quotasPanel(pw)
|
||||||
|
case cGuard:
|
||||||
|
return m.guardPanel(pw)
|
||||||
case cAccounts, cKeys, cOrgs:
|
case cAccounts, cKeys, cOrgs:
|
||||||
return m.peoplePanel(pw)
|
return m.peoplePanel(pw)
|
||||||
case cManual:
|
case cManual:
|
||||||
@@ -1173,6 +1225,10 @@ func (m consoleModel) panel() (title, body, keys string) {
|
|||||||
case cAddNode:
|
case cAddNode:
|
||||||
item = "\n Adds a machine as a node: installs llmgw on it over ssh, checks it,\n gives it a certificate and starts its agent.\n\n " +
|
item = "\n Adds a machine as a node: installs llmgw on it over ssh, checks it,\n gives it a certificate and starts its agent.\n\n " +
|
||||||
sDim.Render("On the command line:") + "\n sudo llmgw node add <user>@<address>"
|
sDim.Render("On the command line:") + "\n sudo llmgw node add <user>@<address>"
|
||||||
|
case cAddControl:
|
||||||
|
item = "\n Makes another machine a control plane of this cluster: a standby\n that takes over within seconds when this one is lost. Every node is\n told of it; none is touched. With a floating address, keepalived on\n both keeps it on whichever leads — the address to give clients.\n\n " +
|
||||||
|
sDim.Render("Needs the store replicated first, so both reach it.") + "\n\n " +
|
||||||
|
sDim.Render("On the command line:") + "\n sudo --preserve-env=SSH_AUTH_SOCK llmgw server add <user>@<address> --vip <address>/<prefix>"
|
||||||
case cReplicate:
|
case cReplicate:
|
||||||
item = "\n Moves the store onto two machines under Patroni, replicated,\n so losing one loses nothing.\n\n " +
|
item = "\n Moves the store onto two machines under Patroni, replicated,\n so losing one loses nothing.\n\n " +
|
||||||
sDim.Render("On the command line:") +
|
sDim.Render("On the command line:") +
|
||||||
@@ -1531,3 +1587,26 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
|
|||||||
_, _ = bufio.NewReader(os.Stdin).ReadString('\n')
|
_, _ = bufio.NewReader(os.Stdin).ReadString('\n')
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// floatingQuestion is the second question when adding a control plane.
|
||||||
|
const floatingQuestion = "A floating address for clients, kept on whichever control plane leads,\n" +
|
||||||
|
"as 10.0.0.161/24 — a free one on this network. Empty for none, when a load\n" +
|
||||||
|
"balancer in front checks /ready instead."
|
||||||
|
|
||||||
|
// standbyNote says, on a control plane standing by, whose cluster the
|
||||||
|
// console shows: the leader's, which this one's admin API passes it on to.
|
||||||
|
func (m consoleModel) standbyNote() string {
|
||||||
|
var here, leader string
|
||||||
|
for _, cp := range m.data.sum.ControlPlanes.Members {
|
||||||
|
if cp.This {
|
||||||
|
here = cp.State
|
||||||
|
}
|
||||||
|
if cp.State == "leading" {
|
||||||
|
leader = cp.Name
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if here != "standing by" || leader == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return "\n " + sDim.Render("Standing by. Shown, and changed, on the leader: "+leader) + "\n"
|
||||||
|
}
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
tea "github.com/charmbracelet/bubbletea"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The guardrails, as the console shows them: the policies server.yaml
|
||||||
|
// defines and what each checks, where they are attached, whether the judge
|
||||||
|
// models they lean on are serving, and what they did in a window.
|
||||||
|
|
||||||
|
// guardWindows are the windows W steps through.
|
||||||
|
var guardWindows = []string{"24h", "7d", "1h"}
|
||||||
|
|
||||||
|
type guardState struct {
|
||||||
|
view adminapi.GuardView
|
||||||
|
events adminapi.GuardEventsView
|
||||||
|
err error
|
||||||
|
// window is an index into guardWindows.
|
||||||
|
window int
|
||||||
|
}
|
||||||
|
|
||||||
|
type consoleGuardMsg struct {
|
||||||
|
view adminapi.GuardView
|
||||||
|
events adminapi.GuardEventsView
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m consoleModel) fetchGuard() tea.Cmd {
|
||||||
|
c, since := m.client, guardWindows[m.guard.window]
|
||||||
|
return func() tea.Msg {
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
var msg consoleGuardMsg
|
||||||
|
if msg.err = c.get(ctx, "/api/v1/guard", &msg.view); msg.err != nil {
|
||||||
|
return msg
|
||||||
|
}
|
||||||
|
// What they did is a second question; a store without the record
|
||||||
|
// still shows the policies.
|
||||||
|
_ = c.get(ctx, "/api/v1/guard/events?since="+since, &msg.events)
|
||||||
|
return msg
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m consoleModel) guardPanel(pw int) (string, string, string) {
|
||||||
|
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||||
|
if m.inPanel {
|
||||||
|
keys = "↑↓ attachment · Enter actions · A attach · W window · ← back · Q leave"
|
||||||
|
}
|
||||||
|
g := m.guard
|
||||||
|
var b strings.Builder
|
||||||
|
b.WriteString("\n " + sDim.Render("What requests and answers are checked for, where, and what that did. A to attach a policy.") + "\n\n")
|
||||||
|
if g.err != nil {
|
||||||
|
if strings.Contains(g.err.Error(), "501") || strings.Contains(g.err.Error(), "no guard policies") {
|
||||||
|
b.WriteString(" " + sDim.Render("No policies: define them under guard: in server.yaml (Settings), then restart."))
|
||||||
|
} else {
|
||||||
|
b.WriteString(" " + sBad.Render(g.err.Error()))
|
||||||
|
}
|
||||||
|
return "Guardrails", b.String(), keys
|
||||||
|
}
|
||||||
|
if len(g.view.Details) == 0 && len(g.view.Policies) == 0 {
|
||||||
|
b.WriteString(" " + sDim.Render("asking the control plane…"))
|
||||||
|
return "Guardrails", b.String(), keys
|
||||||
|
}
|
||||||
|
|
||||||
|
def := "none"
|
||||||
|
if len(g.view.Default) > 0 {
|
||||||
|
def = strings.Join(g.view.Default, ", ")
|
||||||
|
}
|
||||||
|
b.WriteString(" " + sTitle.Render("POLICIES") + " " + sDim.Render("on every request: "+def) + "\n")
|
||||||
|
for _, p := range g.view.Details {
|
||||||
|
line := fmt.Sprintf(" %-14s %s · fail %s", p.Name, p.Mode, p.Fail)
|
||||||
|
if p.NoStore {
|
||||||
|
line += " · keeps nothing"
|
||||||
|
}
|
||||||
|
if p.Judge != "" {
|
||||||
|
line += " · judge " + p.Judge + " " + m.judgeState(p.Judge)
|
||||||
|
}
|
||||||
|
b.WriteString(line + "\n")
|
||||||
|
for _, c := range p.Checks {
|
||||||
|
b.WriteString(" " + strings.Repeat(" ", 15) + sDim.Render(c) + "\n")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
b.WriteString("\n " + sTitle.Render("ATTACHED") + " " + sDim.Render("besides the default, to part of the tenancy tree, a person or a key") + "\n")
|
||||||
|
if len(g.view.Attachments) == 0 {
|
||||||
|
b.WriteString(" " + sDim.Render("Nowhere. A to attach one.") + "\n")
|
||||||
|
} else {
|
||||||
|
var rows [][]string
|
||||||
|
for _, a := range g.view.Attachments {
|
||||||
|
who := a.Name
|
||||||
|
if who == "" {
|
||||||
|
who = a.ScopeID
|
||||||
|
}
|
||||||
|
rows = append(rows, []string{a.Scope, who, a.Policy})
|
||||||
|
}
|
||||||
|
b.WriteString(table([]string{"SCOPE", "WHO", "POLICY"}, rows, m.row, pw, m.inPanel) + "\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
window := guardWindows[g.window]
|
||||||
|
b.WriteString("\n " + sTitle.Render("IN THE LAST "+strings.ToUpper(window)) + " " + sDim.Render("W for another window") + "\n")
|
||||||
|
if len(g.events.Counts) == 0 {
|
||||||
|
b.WriteString(" " + sDim.Render("Nothing flagged, redacted or blocked.") + "\n")
|
||||||
|
} else {
|
||||||
|
var rows [][]string
|
||||||
|
for i, c := range g.events.Counts {
|
||||||
|
if i == 10 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
rows = append(rows, []string{c.Action, c.Policy, dash(c.Category), dash(c.Project), fmt.Sprint(c.Count)})
|
||||||
|
}
|
||||||
|
b.WriteString(table([]string{"ACTION", "POLICY", "CATEGORY", "PROJECT", "REQUESTS"}, rows, -1, pw, false) + "\n")
|
||||||
|
}
|
||||||
|
if len(g.events.Events) > 0 {
|
||||||
|
b.WriteString("\n " + sTitle.Render("NEWEST") + "\n")
|
||||||
|
var rows [][]string
|
||||||
|
for i, e := range g.events.Events {
|
||||||
|
if i == 10 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
rows = append(rows, []string{e.At.Local().Format("01-02 15:04:05"), e.Action, e.Policy, dash(e.Category),
|
||||||
|
e.Model, fmt.Sprint(e.Status), fmt.Sprintf("%dms", e.GuardMs), e.RequestID})
|
||||||
|
}
|
||||||
|
b.WriteString(table([]string{"AT", "ACTION", "POLICY", "CATEGORY", "MODEL", "STATUS", "GUARD", "REQUEST"}, rows, -1, pw, false))
|
||||||
|
}
|
||||||
|
return "Guardrails", b.String(), keys
|
||||||
|
}
|
||||||
|
|
||||||
|
// judgeState is how many of a judge model's replicas are ready: a policy
|
||||||
|
// that fails closed refuses everything while it has none.
|
||||||
|
func (m consoleModel) judgeState(model string) string {
|
||||||
|
for _, md := range m.data.models {
|
||||||
|
if md.Name == model {
|
||||||
|
s := fmt.Sprintf("(%d/%d ready)", md.Ready, md.Desired)
|
||||||
|
if md.Ready == 0 {
|
||||||
|
return sBad.Render(s)
|
||||||
|
}
|
||||||
|
return sOK.Render(s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return sBad.Render("(not in the catalog)")
|
||||||
|
}
|
||||||
|
|
||||||
|
// guardAttachForm attaches a policy.
|
||||||
|
func (m *consoleModel) guardAttachForm() {
|
||||||
|
if len(m.guard.view.Policies) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.openForm(&formDialog{title: "Attach a guardrail policy",
|
||||||
|
note: "Requests from there are checked by it as well as by the default.",
|
||||||
|
fields: []formField{
|
||||||
|
choiceF("to", []string{"project", "team", "org", "user", "key"}, "project"),
|
||||||
|
textF("which", "", "ORG/TEAM/PROJECT, ORG/TEAM, ORG, or the ID"),
|
||||||
|
choiceF("policy", m.guard.view.Policies, m.guard.view.Policies[0]),
|
||||||
|
},
|
||||||
|
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||||
|
if strings.TrimSpace(v[1]) == "" {
|
||||||
|
return nil, fmt.Errorf("which %s?", v[0])
|
||||||
|
}
|
||||||
|
req := adminapi.GuardAttachmentRequest{Target: v[0] + ":" + strings.TrimSpace(v[1]), Policy: v[2]}
|
||||||
|
return m.call("Attach "+v[2], http.MethodPost, "/api/v1/guard/attachments", req, nil), nil
|
||||||
|
}})
|
||||||
|
}
|
||||||
|
|
||||||
|
// guardAttachmentMenu is what can be done to one attachment.
|
||||||
|
func (m consoleModel) guardAttachmentMenu(a adminapi.GuardAttachmentView) *menuDialog {
|
||||||
|
who := a.Name
|
||||||
|
if who == "" {
|
||||||
|
who = a.ScopeID
|
||||||
|
}
|
||||||
|
return &menuDialog{title: a.Policy + " on " + a.Scope + " " + who, items: []menuItem{
|
||||||
|
{"Detach…", func(m *consoleModel) tea.Cmd {
|
||||||
|
m.confirmD = &confirmDialog{title: "Detach " + a.Policy,
|
||||||
|
text: "Requests from " + who + " are then checked by the default policies alone.",
|
||||||
|
yes: func(m *consoleModel) tea.Cmd {
|
||||||
|
req := adminapi.GuardAttachmentRequest{Target: a.Scope + ":" + a.ScopeID, Policy: a.Policy}
|
||||||
|
return m.call("Detach "+a.Policy, http.MethodDelete, "/api/v1/guard/attachments", req, nil)
|
||||||
|
}}
|
||||||
|
return nil
|
||||||
|
}},
|
||||||
|
}}
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package cli
|
|||||||
import (
|
import (
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -171,3 +172,75 @@ func TestTheConsoleWatchesTheClusterAndEditsItsSettings(t *testing.T) {
|
|||||||
t.Errorf("server.yaml:\n%s", b)
|
t.Errorf("server.yaml:\n%s", b)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestTheConsoleAddsAControlPlaneWithAFloatingAddress(t *testing.T) {
|
||||||
|
replicated := consoleModel{menu: cAddControl, loaded: true, width: 100, height: 30}
|
||||||
|
replicated.data.sum.Store = []adminapi.StoreMember{{Name: "a", Role: "leader"}}
|
||||||
|
|
||||||
|
// The address, then the floating address; the console closes to run it.
|
||||||
|
var mm tea.Model = replicated
|
||||||
|
mm = press(mm, "enter", "ubuntu@10.0.0.150", "enter", "10.0.0.161/24", "enter")
|
||||||
|
want := []string{"server", "add", "ubuntu@10.0.0.150", "--vip", "10.0.0.161/24"}
|
||||||
|
if a := mm.(consoleModel).action; a.kind != "run" || !slices.Equal(a.targets, want) {
|
||||||
|
t.Errorf("%+v", a)
|
||||||
|
}
|
||||||
|
|
||||||
|
// No floating address is an answer too.
|
||||||
|
mm = press(replicated, "enter", "ubuntu@10.0.0.150", "enter", "enter")
|
||||||
|
if a := mm.(consoleModel).action; !slices.Equal(a.targets, want[:3]) {
|
||||||
|
t.Errorf("without one: %+v", a)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A mistyped one is caught here rather than half way through.
|
||||||
|
mm = press(replicated, "enter", "ubuntu@10.0.0.150", "enter", "10.0.0.161", "enter")
|
||||||
|
if c := mm.(consoleModel); c.action.kind != "" || !strings.Contains(c.dialogText, "prefix") {
|
||||||
|
t.Errorf("an address without its prefix went through: %+v", c.action)
|
||||||
|
}
|
||||||
|
|
||||||
|
// On a store on one machine it says what to do first.
|
||||||
|
mm = press(consoleModel{menu: cAddControl, loaded: true, width: 100, height: 30}, "enter")
|
||||||
|
if c := mm.(consoleModel); c.dialog != "error" || !strings.Contains(c.dialogText, "Replicate the store") {
|
||||||
|
t.Errorf("dialog %q: %s", c.dialog, c.dialogText)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheConsoleOnAStandbySaysWhoseClusterItShows(t *testing.T) {
|
||||||
|
m := consoleModel{menu: cOverview, loaded: true, width: 120, height: 40}
|
||||||
|
m.data.sum.ControlPlanes = adminapi.ControlPlanes{Members: []adminapi.ControlPlane{
|
||||||
|
{Name: "llmgw-ctl", State: "leading"},
|
||||||
|
{Name: "llmgw-ctl02", State: "standing by", This: true},
|
||||||
|
}}
|
||||||
|
if v := m.View(); !strings.Contains(v, "Standing by. Shown, and changed, on the leader: llmgw-ctl") {
|
||||||
|
t.Errorf("the overview on a standby:\n%s", v)
|
||||||
|
}
|
||||||
|
m.data.sum.ControlPlanes.Members[0].This, m.data.sum.ControlPlanes.Members[1].This = true, false
|
||||||
|
if v := m.View(); strings.Contains(v, "Standing by.") {
|
||||||
|
t.Error("the leader's console says it is standing by")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheConsoleShowsTheGuardrails(t *testing.T) {
|
||||||
|
m := consoleModel{menu: cGuard, loaded: true, width: 140, height: 60}
|
||||||
|
m.data.models = []adminapi.ModelView{{Name: "llama-guard-3-1b", Ready: 2, Desired: 2}}
|
||||||
|
m.guard.view = adminapi.GuardView{
|
||||||
|
Policies: []string{"baseline"}, Default: []string{"baseline"},
|
||||||
|
Details: []adminapi.GuardPolicyView{{Name: "baseline", Mode: "enforce", Fail: "closed", Judge: "llama-guard-3-1b",
|
||||||
|
Checks: []string{"input: pii email → redact"}}},
|
||||||
|
Attachments: []adminapi.GuardAttachmentView{{Name: "acme/ml"}},
|
||||||
|
}
|
||||||
|
m.guard.view.Attachments[0].Scope, m.guard.view.Attachments[0].Policy = "team", "baseline"
|
||||||
|
m.guard.events = adminapi.GuardEventsView{Counts: []adminapi.GuardCountView{{Action: "block", Policy: "baseline", Category: "S1", Count: 7}}}
|
||||||
|
v := m.View()
|
||||||
|
for _, want := range []string{"Guardrails", "baseline", "fail closed", "judge llama-guard-3-1b (2/2 ready)",
|
||||||
|
"input: pii email → redact", "acme/ml", "IN THE LAST 24H", "S1"} {
|
||||||
|
if !strings.Contains(v, want) {
|
||||||
|
t.Errorf("%q missing:\n%s", want, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// W moves the window, and asks again.
|
||||||
|
m.inPanel = true
|
||||||
|
mm, cmd := m.Update(tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune("w")})
|
||||||
|
if c := mm.(consoleModel); guardWindows[c.guard.window] != "7d" || cmd == nil {
|
||||||
|
t.Errorf("window %q", guardWindows[c.guard.window])
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -54,6 +54,9 @@ type API struct {
|
|||||||
releases string
|
releases string
|
||||||
// storeStatus reads a replicated store's members; nil for one database.
|
// storeStatus reads a replicated store's members; nil for one database.
|
||||||
storeStatus func() ([]StoreMember, error)
|
storeStatus func() ([]StoreMember, error)
|
||||||
|
// controlPlanes says what each control plane answers; nil for a cluster
|
||||||
|
// of one.
|
||||||
|
controlPlanes func() ControlPlanes
|
||||||
// serve runs registry-to-replica jobs; nil without a mirror and a store.
|
// serve runs registry-to-replica jobs; nil without a mirror and a store.
|
||||||
serve *modelserve.Server
|
serve *modelserve.Server
|
||||||
// history is the Prometheus the console's charts read from.
|
// history is the Prometheus the console's charts read from.
|
||||||
@@ -870,6 +873,10 @@ func (a *API) SetUsage(u Usage) { a.usage = u }
|
|||||||
// SetStoreStatus supplies the state of a replicated store, for the summary.
|
// SetStoreStatus supplies the state of a replicated store, for the summary.
|
||||||
func (a *API) SetStoreStatus(f func() ([]StoreMember, error)) { a.storeStatus = f }
|
func (a *API) SetStoreStatus(f func() ([]StoreMember, error)) { a.storeStatus = f }
|
||||||
|
|
||||||
|
// SetControlPlanes sets where the summary learns which control plane leads
|
||||||
|
// and which stand by.
|
||||||
|
func (a *API) SetControlPlanes(f func() ControlPlanes) { a.controlPlanes = f }
|
||||||
|
|
||||||
// ScopeUsageView is one group's spend, with its quota when it has one.
|
// ScopeUsageView is one group's spend, with its quota when it has one.
|
||||||
type ScopeUsageView struct {
|
type ScopeUsageView struct {
|
||||||
Scope string `json:"scope"`
|
Scope string `json:"scope"`
|
||||||
@@ -1155,8 +1162,18 @@ func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeJSON(w, http.StatusOK, Removed{Removed: name})
|
writeJSON(w, http.StatusOK, Removed{Removed: name})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *API) summary(w http.ResponseWriter, _ *http.Request) {
|
func (a *API) summary(w http.ResponseWriter, r *http.Request) {
|
||||||
writeJSON(w, http.StatusOK, a.Summary())
|
s := a.Summary()
|
||||||
|
// Asked through a standby, which passes on the console's questions to
|
||||||
|
// the leader: "this one" is the standby the console is on.
|
||||||
|
if by := r.Header.Get("X-Llmgw-Forwarded-By"); by != "" && len(s.ControlPlanes.Members) > 0 {
|
||||||
|
members := slices.Clone(s.ControlPlanes.Members)
|
||||||
|
for i := range members {
|
||||||
|
members[i].This = members[i].Name == by
|
||||||
|
}
|
||||||
|
s.ControlPlanes.Members = members
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, s)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
|
func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
|
||||||
|
|||||||
@@ -4,11 +4,14 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"maps"
|
||||||
"net/http"
|
"net/http"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
// GuardStore is where policy attachments are kept.
|
// GuardStore is where policy attachments are kept.
|
||||||
@@ -34,6 +37,114 @@ type GuardView struct {
|
|||||||
Policies []string `json:"policies"`
|
Policies []string `json:"policies"`
|
||||||
Default []string `json:"default"`
|
Default []string `json:"default"`
|
||||||
Attachments []GuardAttachmentView `json:"attachments"`
|
Attachments []GuardAttachmentView `json:"attachments"`
|
||||||
|
// Details says what each policy does, in the order of Policies.
|
||||||
|
Details []GuardPolicyView `json:"details"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// GuardPolicyView is what one policy does, in brief.
|
||||||
|
type GuardPolicyView struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
// Mode is enforce, shadow or off; Fail is open or closed.
|
||||||
|
Mode string `json:"mode"`
|
||||||
|
Fail string `json:"fail"`
|
||||||
|
NoStore bool `json:"no_store,omitempty"`
|
||||||
|
// Judge is the judge model, when one reads the text.
|
||||||
|
Judge string `json:"judge,omitempty"`
|
||||||
|
// Checks are what it looks for, one line each: "input: pii email → redact".
|
||||||
|
Checks []string `json:"checks"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// GuardHistory is what the guardrails did, from the usage record; the store
|
||||||
|
// has it, and a store that does not leaves the events endpoint unanswered.
|
||||||
|
type GuardHistory interface {
|
||||||
|
GuardEvents(ctx context.Context, since time.Time, limit int) ([]store.GuardEvent, error)
|
||||||
|
GuardCounts(ctx context.Context, since time.Time) ([]store.GuardCount, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// GuardEventsView is what the guardrails did since a moment: totals, and the
|
||||||
|
// newest requests.
|
||||||
|
type GuardEventsView struct {
|
||||||
|
Since time.Time `json:"since"`
|
||||||
|
Counts []GuardCountView `json:"counts"`
|
||||||
|
Events []GuardEventView `json:"events"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// GuardCountView is how many requests one policy did one thing to.
|
||||||
|
type GuardCountView struct {
|
||||||
|
Project string `json:"project,omitempty"`
|
||||||
|
Action string `json:"action"`
|
||||||
|
Policy string `json:"policy"`
|
||||||
|
Category string `json:"category,omitempty"`
|
||||||
|
Count int64 `json:"count"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// GuardEventView is one request the guardrails did something to.
|
||||||
|
type GuardEventView struct {
|
||||||
|
At time.Time `json:"at"`
|
||||||
|
RequestID string `json:"request_id"`
|
||||||
|
Project string `json:"project,omitempty"`
|
||||||
|
Model string `json:"model"`
|
||||||
|
Status int `json:"status"`
|
||||||
|
Action string `json:"action"`
|
||||||
|
Policy string `json:"policy"`
|
||||||
|
Category string `json:"category,omitempty"`
|
||||||
|
GuardMs int64 `json:"guard_ms"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// policyView sums one policy up.
|
||||||
|
func policyView(name string, p guard.Policy) GuardPolicyView {
|
||||||
|
v := GuardPolicyView{Name: name, Mode: or(p.Mode, "enforce"), Fail: or(p.Fail, "open"), NoStore: p.NoStore, Checks: []string{}}
|
||||||
|
rules := func(dir string, r guard.Rules) {
|
||||||
|
for _, kind := range slices.Sorted(maps.Keys(r.PII)) {
|
||||||
|
v.Checks = append(v.Checks, fmt.Sprintf("%s: pii %s → %s", dir, kind, r.PII[kind]))
|
||||||
|
}
|
||||||
|
if r.Injection != "" {
|
||||||
|
v.Checks = append(v.Checks, fmt.Sprintf("%s: prompt injection → %s", dir, r.Injection))
|
||||||
|
}
|
||||||
|
for _, d := range r.Deny {
|
||||||
|
v.Checks = append(v.Checks, fmt.Sprintf("%s: pattern %s → %s", dir, d.Name, or(d.Action, "block")))
|
||||||
|
}
|
||||||
|
if r.MaxChars > 0 {
|
||||||
|
v.Checks = append(v.Checks, fmt.Sprintf("%s: at most %d characters", dir, r.MaxChars))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rules("input", p.Input)
|
||||||
|
rules("output", p.Output)
|
||||||
|
if j := p.Judge; j != nil {
|
||||||
|
v.Judge = j.Model
|
||||||
|
var dirs []string
|
||||||
|
if j.Input != "" {
|
||||||
|
dirs = append(dirs, "input → "+j.Input)
|
||||||
|
}
|
||||||
|
if j.Output != "" {
|
||||||
|
dirs = append(dirs, "output → "+j.Output)
|
||||||
|
}
|
||||||
|
line := "judge " + j.Model + ": " + strings.Join(dirs, ", ")
|
||||||
|
if len(j.Categories) > 0 {
|
||||||
|
line += " (" + strings.Join(j.Categories, " ") + ")"
|
||||||
|
}
|
||||||
|
v.Checks = append(v.Checks, line)
|
||||||
|
}
|
||||||
|
if len(p.Tools.Allow) > 0 {
|
||||||
|
v.Checks = append(v.Checks, "tools: only "+strings.Join(p.Tools.Allow, ", "))
|
||||||
|
}
|
||||||
|
if len(p.Tools.Deny) > 0 {
|
||||||
|
v.Checks = append(v.Checks, "tools: never "+strings.Join(p.Tools.Deny, ", "))
|
||||||
|
}
|
||||||
|
if p.MaxOutputTokens > 0 {
|
||||||
|
v.Checks = append(v.Checks, fmt.Sprintf("answers of at most %d tokens", p.MaxOutputTokens))
|
||||||
|
}
|
||||||
|
if p.SystemPrompt != "" {
|
||||||
|
v.Checks = append(v.Checks, "a system prompt of its own")
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
func or(s, dflt string) string {
|
||||||
|
if s == "" {
|
||||||
|
return dflt
|
||||||
|
}
|
||||||
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
// GuardAttachmentView is one attachment, with the name people know its
|
// GuardAttachmentView is one attachment, with the name people know its
|
||||||
@@ -43,6 +154,15 @@ type GuardAttachmentView struct {
|
|||||||
Name string `json:"name,omitempty"`
|
Name string `json:"name,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// guardHistory is where what the guardrails did is read, when there is one.
|
||||||
|
func (a *API) guardHistory() (GuardHistory, bool) {
|
||||||
|
if a.guard == nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
h, ok := a.guard.Store.(GuardHistory)
|
||||||
|
return h, ok
|
||||||
|
}
|
||||||
|
|
||||||
// guardTarget reads where a policy is to be attached.
|
// guardTarget reads where a policy is to be attached.
|
||||||
func (a *API) guardTarget(ctx context.Context, target string) (guard.Attachment, error) {
|
func (a *API) guardTarget(ctx context.Context, target string) (guard.Attachment, error) {
|
||||||
kind, what, ok := strings.Cut(strings.TrimSpace(target), ":")
|
kind, what, ok := strings.Cut(strings.TrimSpace(target), ":")
|
||||||
@@ -99,6 +219,57 @@ func (a *API) registerGuard(mux Mux) {
|
|||||||
v.Policies = append(v.Policies, name)
|
v.Policies = append(v.Policies, name)
|
||||||
}
|
}
|
||||||
slices.Sort(v.Policies)
|
slices.Sort(v.Policies)
|
||||||
|
for _, name := range v.Policies {
|
||||||
|
v.Details = append(v.Details, policyView(name, a.guard.Library.Policies[name]))
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, v)
|
||||||
|
})
|
||||||
|
mux.HandleFunc("GET /api/v1/guard/events", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
h, ok := a.guardHistory()
|
||||||
|
if !ok {
|
||||||
|
writeError(w, http.StatusNotImplemented, "no guard policies, or no store to record what they did")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
since := 24 * time.Hour
|
||||||
|
if q := r.URL.Query().Get("since"); q != "" {
|
||||||
|
d, err := parseSince(q)
|
||||||
|
if err != nil {
|
||||||
|
writeError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
since = d
|
||||||
|
}
|
||||||
|
from := time.Now().Add(-since)
|
||||||
|
counts, err := h.GuardCounts(r.Context(), from)
|
||||||
|
if err != nil {
|
||||||
|
writeError(w, http.StatusInternalServerError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
events, err := h.GuardEvents(r.Context(), from, 100)
|
||||||
|
if err != nil {
|
||||||
|
writeError(w, http.StatusInternalServerError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var names map[string]string
|
||||||
|
if a.quotas != nil {
|
||||||
|
names, _ = a.scopeNames(r.Context())
|
||||||
|
}
|
||||||
|
project := func(id string) string {
|
||||||
|
if n := names["project:"+id]; n != "" {
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
v := GuardEventsView{Since: from, Counts: []GuardCountView{}, Events: []GuardEventView{}}
|
||||||
|
for _, c := range counts {
|
||||||
|
v.Counts = append(v.Counts, GuardCountView{Project: project(c.ProjectID), Action: c.Action,
|
||||||
|
Policy: c.Policy, Category: c.Category, Count: c.Count})
|
||||||
|
}
|
||||||
|
slices.SortStableFunc(v.Counts, func(a, b GuardCountView) int { return int(b.Count - a.Count) })
|
||||||
|
for _, e := range events {
|
||||||
|
v.Events = append(v.Events, GuardEventView{At: e.At, RequestID: e.RequestID, Project: project(e.ProjectID),
|
||||||
|
Model: e.Model, Status: e.Status, Action: e.Action, Policy: e.Policy, Category: e.Category, GuardMs: e.GuardMs})
|
||||||
|
}
|
||||||
writeJSON(w, http.StatusOK, v)
|
writeJSON(w, http.StatusOK, v)
|
||||||
})
|
})
|
||||||
change := func(attach bool) http.HandlerFunc {
|
change := func(attach bool) http.HandlerFunc {
|
||||||
|
|||||||
@@ -116,3 +116,54 @@ func TestUsageIsExportedAsCSVThatASpreadsheetCannotRun(t *testing.T) {
|
|||||||
t.Errorf("%d %s %+v", rec.Code, body, m)
|
t.Errorf("%d %s %+v", rec.Code, body, m)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// memGuardHistory is a guard store that also remembers what was done.
|
||||||
|
type memGuardHistory struct {
|
||||||
|
memGuard
|
||||||
|
since time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memGuardHistory) GuardEvents(_ context.Context, since time.Time, _ int) ([]store.GuardEvent, error) {
|
||||||
|
m.since = since
|
||||||
|
return []store.GuardEvent{{RequestID: "r1", Model: "m", Status: 400, Action: "block", Policy: "base", Category: "S1", GuardMs: 410}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memGuardHistory) GuardCounts(context.Context, time.Time) ([]store.GuardCount, error) {
|
||||||
|
return []store.GuardCount{
|
||||||
|
{Action: "flag", Policy: "base", Category: "email", Count: 2},
|
||||||
|
{Action: "block", Policy: "base", Category: "S1", Count: 7},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheGuardrailsSayWhatEachPolicyDoesAndWhatTheyDid(t *testing.T) {
|
||||||
|
a, _ := setup(t, testModels())
|
||||||
|
h := &memGuardHistory{}
|
||||||
|
a.SetGuard(&GuardAdmin{Store: h, Library: guard.Config{Policies: map[string]guard.Policy{
|
||||||
|
"base": {Fail: "closed", Input: guard.Rules{PII: map[string]string{"email": "redact"}, Injection: "block"},
|
||||||
|
Judge: &guard.JudgePolicy{Model: "llama-guard-3-1b", Input: "block", Categories: []string{"S1", "S9"}}},
|
||||||
|
}}})
|
||||||
|
|
||||||
|
var v GuardView
|
||||||
|
if code := call(t, a, "GET", "/api/v1/guard", "", &v); code != 200 || len(v.Details) != 1 {
|
||||||
|
t.Fatalf("%d %+v", code, v)
|
||||||
|
}
|
||||||
|
d := v.Details[0]
|
||||||
|
want := []string{"input: pii email → redact", "input: prompt injection → block", "judge llama-guard-3-1b: input → block (S1 S9)"}
|
||||||
|
if d.Mode != "enforce" || d.Fail != "closed" || d.Judge != "llama-guard-3-1b" || !slices.Equal(d.Checks, want) {
|
||||||
|
t.Errorf("%+v", d)
|
||||||
|
}
|
||||||
|
|
||||||
|
var ev GuardEventsView
|
||||||
|
if code := call(t, a, "GET", "/api/v1/guard/events?since=1h", "", &ev); code != 200 {
|
||||||
|
t.Fatalf("events: %d", code)
|
||||||
|
}
|
||||||
|
if time.Since(h.since) > time.Hour+time.Minute || time.Since(h.since) < time.Hour-time.Minute {
|
||||||
|
t.Errorf("asked since %s", h.since)
|
||||||
|
}
|
||||||
|
if len(ev.Counts) != 2 || ev.Counts[0].Count != 7 || len(ev.Events) != 1 || ev.Events[0].GuardMs != 410 {
|
||||||
|
t.Errorf("most first, and the events: %+v", ev)
|
||||||
|
}
|
||||||
|
if code := call(t, a, "GET", "/api/v1/guard/events?since=soon", "", nil); code != 400 {
|
||||||
|
t.Errorf("a bad window: %d", code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -249,6 +249,11 @@ var Routes = []Route{
|
|||||||
Response: Removed{}},
|
Response: Removed{}},
|
||||||
{Method: "GET", Path: "/api/v1/guard", Summary: "The guardrail policies and where they are attached",
|
{Method: "GET", Path: "/api/v1/guard", Summary: "The guardrail policies and where they are attached",
|
||||||
Response: GuardView{}},
|
Response: GuardView{}},
|
||||||
|
{Method: "GET", Path: "/api/v1/guard/events", Summary: "What the guardrails did: totals and the newest requests",
|
||||||
|
Query: []Param{
|
||||||
|
{Name: "since", Description: "How far back: 90m, 24h, 7d; 24h when absent."},
|
||||||
|
},
|
||||||
|
Response: GuardEventsView{}},
|
||||||
{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
|
{Method: "POST", Path: "/api/v1/guard/attachments", Summary: "Attach a guardrail policy",
|
||||||
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
|
Request: GuardAttachmentRequest{}, Response: guard.Attachment{}},
|
||||||
{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
|
{Method: "DELETE", Path: "/api/v1/guard/attachments", Summary: "Detach a guardrail policy",
|
||||||
|
|||||||
@@ -35,6 +35,9 @@ type Summary struct {
|
|||||||
// StoreError why it could not be read; both empty for a single database.
|
// StoreError why it could not be read; both empty for a single database.
|
||||||
Store []StoreMember `json:"store,omitempty"`
|
Store []StoreMember `json:"store,omitempty"`
|
||||||
StoreError string `json:"store_error,omitempty"`
|
StoreError string `json:"store_error,omitempty"`
|
||||||
|
// ControlPlanes is the cluster's control planes when there is more than
|
||||||
|
// one: which leads, which stand by, and the floating address in front.
|
||||||
|
ControlPlanes ControlPlanes `json:"control_planes,omitzero"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// StoreMember is one member of a replicated store, as its Patroni reports it.
|
// StoreMember is one member of a replicated store, as its Patroni reports it.
|
||||||
@@ -47,6 +50,27 @@ type StoreMember struct {
|
|||||||
Lag int64 `json:"lag"`
|
Lag int64 `json:"lag"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ControlPlanes is every control plane of a cluster that has several, as
|
||||||
|
// each answered when asked.
|
||||||
|
type ControlPlanes struct {
|
||||||
|
Members []ControlPlane `json:"members,omitempty"`
|
||||||
|
// Floating is the address keepalived keeps on the leader (10.0.0.161),
|
||||||
|
// and FloatingUp whether it answered as a leader when asked; empty when
|
||||||
|
// nothing floats.
|
||||||
|
Floating string `json:"floating,omitempty"`
|
||||||
|
FloatingUp bool `json:"floating_up,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ControlPlane is one control plane.
|
||||||
|
type ControlPlane struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Address string `json:"address"`
|
||||||
|
// State is "leading", "standing by" or "not answering".
|
||||||
|
State string `json:"state"`
|
||||||
|
// This is the control plane that answered the summary.
|
||||||
|
This bool `json:"this,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
// NodeTotals counts nodes by the states that need different responses.
|
// NodeTotals counts nodes by the states that need different responses.
|
||||||
type NodeTotals struct {
|
type NodeTotals struct {
|
||||||
Total int `json:"total"`
|
Total int `json:"total"`
|
||||||
@@ -128,6 +152,9 @@ func (a *API) Summary() Summary {
|
|||||||
s.StoreError = err.Error()
|
s.StoreError = err.Error()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if a.controlPlanes != nil {
|
||||||
|
s.ControlPlanes = a.controlPlanes()
|
||||||
|
}
|
||||||
|
|
||||||
nodes := a.Nodes()
|
nodes := a.Nodes()
|
||||||
byClass := map[string]*ClassUse{}
|
byClass := map[string]*ClassUse{}
|
||||||
|
|||||||
@@ -302,16 +302,58 @@ func (s *Service) Connect(
|
|||||||
}()
|
}()
|
||||||
defer wg.Wait()
|
defer wg.Wait()
|
||||||
|
|
||||||
|
// Read in a goroutine of its own, so the stream can also be ended from
|
||||||
|
// this side: Receive blocks until the agent sends, and an agent that is
|
||||||
|
// only heartbeating would keep a stream this process should no longer
|
||||||
|
// hold for as long as it likes. Returning ends it; the reader's Receive
|
||||||
|
// then fails and it follows.
|
||||||
|
read := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
defer close(read)
|
||||||
|
s.readLoop(ctx, stream, c, nodeID)
|
||||||
|
}()
|
||||||
|
tick := time.NewTicker(leaderCheck)
|
||||||
|
defer tick.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-read:
|
||||||
|
return nil
|
||||||
|
case <-c.closed:
|
||||||
|
// Replaced by a newer stream from the same node, or the writer
|
||||||
|
// failed.
|
||||||
|
return nil
|
||||||
|
case <-tick.C:
|
||||||
|
// Leadership lost while the process lives on — the store was
|
||||||
|
// slow, or another took the lease. A standby holding agents is
|
||||||
|
// a cluster whose leader cannot reach its nodes, so let them go
|
||||||
|
// and they find the leader.
|
||||||
|
if s.leader != nil && !s.leader.IsLeader() {
|
||||||
|
s.log.Info("no longer leading; letting the agent go", "node", nodeID)
|
||||||
|
c.close()
|
||||||
|
return connect.NewError(connect.CodeUnavailable,
|
||||||
|
errors.New("this control plane is a standby; connect to the leader"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// leaderCheck is how often an open stream checks that this process still
|
||||||
|
// leads.
|
||||||
|
var leaderCheck = time.Second
|
||||||
|
|
||||||
|
func (s *Service) readLoop(ctx context.Context,
|
||||||
|
stream *connect.BidiStream[llmgwv1.AgentMessage, llmgwv1.ControlMessage], c *conn, nodeID string,
|
||||||
|
) {
|
||||||
for {
|
for {
|
||||||
msg, err := stream.Receive()
|
msg, err := stream.Receive()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
c.close()
|
c.close()
|
||||||
if ctx.Err() != nil || errors.Is(err, io.EOF) {
|
if ctx.Err() != nil || errors.Is(err, io.EOF) {
|
||||||
s.log.Info("agent disconnected", "node", nodeID)
|
s.log.Info("agent disconnected", "node", nodeID)
|
||||||
return nil
|
return
|
||||||
}
|
}
|
||||||
s.log.Warn("agent stream failed", "node", nodeID, "err", err)
|
s.log.Warn("agent stream failed", "node", nodeID, "err", err)
|
||||||
return nil
|
return
|
||||||
}
|
}
|
||||||
if kv := msg.GetKvEvents(); kv != nil {
|
if kv := msg.GetKvEvents(); kv != nil {
|
||||||
s.kvEvents(c, kv)
|
s.kvEvents(c, kv)
|
||||||
|
|||||||
@@ -231,6 +231,10 @@ func (c *Controller) SetLeadership(l *Leadership) {
|
|||||||
c.rec.SetLeadership(l)
|
c.rec.SetLeadership(l)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Leadership is what SetLeadership set; nil when there is nobody to compete
|
||||||
|
// with.
|
||||||
|
func (c *Controller) Leadership() *Leadership { return c.lead }
|
||||||
|
|
||||||
// IsLeader reports whether this process is placing work. True when leadership
|
// IsLeader reports whether this process is placing work. True when leadership
|
||||||
// was never configured, because then there is nobody to compete with.
|
// was never configured, because then there is nobody to compete with.
|
||||||
func (c *Controller) IsLeader() bool { return c.lead == nil || c.lead.IsLeader() }
|
func (c *Controller) IsLeader() bool { return c.lead == nil || c.lead.IsLeader() }
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -51,6 +52,8 @@ type Leadership struct {
|
|||||||
renew time.Duration
|
renew time.Duration
|
||||||
|
|
||||||
leader atomic.Bool
|
leader atomic.Bool
|
||||||
|
// stopped is set by StepDown: nothing takes the lease again after it.
|
||||||
|
stopped atomic.Bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewLeadership prepares leadership over a store.
|
// NewLeadership prepares leadership over a store.
|
||||||
@@ -117,8 +120,16 @@ func (l *Leadership) Run(ctx context.Context) {
|
|||||||
// process: a line per attempt would bury everything else, and the thing an
|
// process: a line per attempt would bury everything else, and the thing an
|
||||||
// operator needs to see is the moment leadership moved.
|
// operator needs to see is the moment leadership moved.
|
||||||
func (l *Leadership) attempt(ctx context.Context) {
|
func (l *Leadership) attempt(ctx context.Context) {
|
||||||
|
if l.stopped.Load() {
|
||||||
|
return
|
||||||
|
}
|
||||||
lease, err := l.store.AcquireLeadership(ctx, controlRole, l.owner, l.ttl)
|
lease, err := l.store.AcquireLeadership(ctx, controlRole, l.owner, l.ttl)
|
||||||
switch {
|
switch {
|
||||||
|
case err == nil && l.stopped.Load():
|
||||||
|
// Stepped down while this was on its way to the store: give back
|
||||||
|
// what it just took.
|
||||||
|
l.leader.Store(true)
|
||||||
|
l.step(ctx)
|
||||||
case err == nil:
|
case err == nil:
|
||||||
if l.leader.CompareAndSwap(false, true) {
|
if l.leader.CompareAndSwap(false, true) {
|
||||||
l.log.Info("this control plane is now placing work", "owner", l.owner)
|
l.log.Info("this control plane is now placing work", "owner", l.owner)
|
||||||
@@ -145,10 +156,24 @@ func (l *Leadership) attempt(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// StepDown gives the lease up now, for a process on its way out: called
|
||||||
|
// before the agents are let go, so the standby they move to is already the
|
||||||
|
// leader, and before the process exits, which would otherwise race the
|
||||||
|
// release in Run and leave the standby to wait out the lease. Run's own
|
||||||
|
// release after it is then nothing.
|
||||||
|
func (l *Leadership) StepDown() {
|
||||||
|
if l.store == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
l.stopped.Store(true)
|
||||||
|
l.step(context.Background())
|
||||||
|
}
|
||||||
|
|
||||||
// step gives the lease up on the way out, so a standby takes over in seconds
|
// step gives the lease up on the way out, so a standby takes over in seconds
|
||||||
// rather than waiting out the clock.
|
// rather than waiting out the clock.
|
||||||
func (l *Leadership) step(ctx context.Context) {
|
func (l *Leadership) step(ctx context.Context) {
|
||||||
if !l.leader.Load() {
|
// Once: StepDown and Run's own way out both come here.
|
||||||
|
if !l.leader.CompareAndSwap(true, false) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
// A fresh context: the one that just ended is the reason this is running.
|
// A fresh context: the one that just ended is the reason this is running.
|
||||||
@@ -180,3 +205,16 @@ func newOwnerID() string {
|
|||||||
}
|
}
|
||||||
return host + "-" + hex.EncodeToString(b[:])
|
return host + "-" + hex.EncodeToString(b[:])
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ControlRole is the lease's name in the store, for readers that ask who
|
||||||
|
// holds it without competing.
|
||||||
|
const ControlRole = controlRole
|
||||||
|
|
||||||
|
// HolderName is the machine an owner id names: the hostname it was made
|
||||||
|
// from, without the random part.
|
||||||
|
func HolderName(owner string) string {
|
||||||
|
if i := strings.LastIndexByte(owner, '-'); i > 0 {
|
||||||
|
return owner[:i]
|
||||||
|
}
|
||||||
|
return owner
|
||||||
|
}
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
package control
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAStoppingLeaderHandsOverAtOnceAndNeverTakesItBack(t *testing.T) {
|
||||||
|
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer st.Close()
|
||||||
|
log := slog.New(slog.DiscardHandler)
|
||||||
|
a := NewLeadership(st, time.Minute, log)
|
||||||
|
b := NewLeadership(st, time.Minute, log)
|
||||||
|
|
||||||
|
a.attempt(t.Context())
|
||||||
|
b.attempt(t.Context())
|
||||||
|
if !a.IsLeader() || b.IsLeader() {
|
||||||
|
t.Fatalf("a %v, b %v", a.IsLeader(), b.IsLeader())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stepping down gives the lease back, rather than leaving b to wait out
|
||||||
|
// the minute it was taken for.
|
||||||
|
a.StepDown()
|
||||||
|
if a.IsLeader() {
|
||||||
|
t.Error("still leading after stepping down")
|
||||||
|
}
|
||||||
|
// A renewal already on its way when the process began to stop must not
|
||||||
|
// take it back.
|
||||||
|
a.attempt(t.Context())
|
||||||
|
if a.IsLeader() {
|
||||||
|
t.Error("took the lease again after stepping down")
|
||||||
|
}
|
||||||
|
b.attempt(t.Context())
|
||||||
|
if !b.IsLeader() {
|
||||||
|
t.Error("the standby could not take the lease that was given up")
|
||||||
|
}
|
||||||
|
lease, err := st.Leadership(t.Context(), ControlRole)
|
||||||
|
if err != nil || lease.Owner != b.Owner() {
|
||||||
|
t.Errorf("the lease is %q (%v), not b's", lease.Owner, err)
|
||||||
|
}
|
||||||
|
if got := HolderName(b.Owner()); got == "" || got == b.Owner() {
|
||||||
|
t.Errorf("holder name %q from %q", got, b.Owner())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -140,9 +140,14 @@ type Request struct {
|
|||||||
// llamaguard3, shieldgemma or json. Empty serves clients.
|
// llamaguard3, shieldgemma or json. Empty serves clients.
|
||||||
GuardFormat string `json:"guard_format,omitempty"`
|
GuardFormat string `json:"guard_format,omitempty"`
|
||||||
// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
|
// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
|
||||||
// use. They default to 16384 and 4096, what four parallel slots divide.
|
// use. They default to 16384 and 4096, what four parallel slots divide;
|
||||||
|
// llama.cpp is given as many slots as the pool holds requests.
|
||||||
CtxPool int `json:"ctx_pool,omitempty"`
|
CtxPool int `json:"ctx_pool,omitempty"`
|
||||||
ContextLength int `json:"context_length,omitempty"`
|
ContextLength int `json:"context_length,omitempty"`
|
||||||
|
// Tools says the model makes tool calls the engine returns as calls: it
|
||||||
|
// is written into the catalog, where clients and editors read it. Check
|
||||||
|
// a running replica with llmgw ctl tool-check before saying so.
|
||||||
|
Tools bool `json:"tools,omitempty"`
|
||||||
// CPUCores is what a replica computes on; zero takes half the smallest
|
// CPUCores is what a replica computes on; zero takes half the smallest
|
||||||
// node it fits on.
|
// node it fits on.
|
||||||
CPUCores int `json:"cpu_cores,omitempty"`
|
CPUCores int `json:"cpu_cores,omitempty"`
|
||||||
@@ -261,6 +266,10 @@ func (s *Server) Start(req Request) (Job, error) {
|
|||||||
if req.ContextLength <= 0 {
|
if req.ContextLength <= 0 {
|
||||||
req.ContextLength = 4096
|
req.ContextLength = 4096
|
||||||
}
|
}
|
||||||
|
if req.ContextLength > req.CtxPool {
|
||||||
|
return Job{}, fmt.Errorf("a request of %d tokens cannot fit a pool of %d: give --ctx-pool at least --context-length",
|
||||||
|
req.ContextLength, req.CtxPool)
|
||||||
|
}
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
s.seq++
|
s.seq++
|
||||||
j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
|
j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
|
||||||
@@ -545,12 +554,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
|||||||
switch {
|
switch {
|
||||||
case o.CPU:
|
case o.CPU:
|
||||||
p.Name = "cpu"
|
p.Name = "cpu"
|
||||||
p.Args = []string{"--ctx-size", strconv.Itoa(req.CtxPool)}
|
p.Args = llamaArgs(req)
|
||||||
p.MinMemoryGB, p.CPUCores = memGB, cores
|
p.MinMemoryGB, p.CPUCores = memGB, cores
|
||||||
case o.Engine == "llamacpp":
|
case o.Engine == "llamacpp":
|
||||||
p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUsPerNode, o.Class)
|
p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUsPerNode, o.Class)
|
||||||
p.GPUClass, p.GPUsPerNode = o.Class, o.GPUsPerNode
|
p.GPUClass, p.GPUsPerNode = o.Class, o.GPUsPerNode
|
||||||
p.Args = []string{"--ctx-size", strconv.Itoa(req.CtxPool)}
|
p.Args = llamaArgs(req)
|
||||||
default:
|
default:
|
||||||
p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUs(), o.Class)
|
p.Name = fmt.Sprintf("gpu-%dx%s", o.GPUs(), o.Class)
|
||||||
p.GPUClass, p.TP = o.Class, o.GPUsPerNode
|
p.GPUClass, p.TP = o.Class, o.GPUsPerNode
|
||||||
@@ -564,6 +573,23 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
|
||||||
|
// by the slots, so slots of the context the catalog promises are as many as
|
||||||
|
// the pool holds: the engine's default of four turned a 32768 pool into
|
||||||
|
// slots of 8192, and the gateway, told 32768, sent requests the engine then
|
||||||
|
// refused.
|
||||||
|
func llamaArgs(req Request) []string {
|
||||||
|
return []string{"--ctx-size", strconv.Itoa(req.CtxPool),
|
||||||
|
"--parallel", strconv.Itoa(max(1, req.CtxPool/req.ContextLength))}
|
||||||
|
}
|
||||||
|
|
||||||
|
// maxOutput is the longest answer the catalog allows: a quarter of the
|
||||||
|
// context, from 1024 to 8192. 1024 everywhere cut an editor's agent off in
|
||||||
|
// the middle of the file it was writing.
|
||||||
|
func maxOutput(contextLength int) int {
|
||||||
|
return min(8192, max(1024, contextLength/4))
|
||||||
|
}
|
||||||
|
|
||||||
// addAndStart puts the model in the catalog with its profile, waits for a
|
// addAndStart puts the model in the catalog with its profile, waits for a
|
||||||
// replica and asks it to answer.
|
// replica and asks it to answer.
|
||||||
// hardwareExists is whether the cluster has what the ways chosen need, busy
|
// hardwareExists is whether the cluster has what the ways chosen need, busy
|
||||||
@@ -583,10 +609,10 @@ func (s *Server) addAndStart(ctx context.Context, j *Job, repo hf.Repo, set hf.F
|
|||||||
// 4. The catalog.
|
// 4. The catalog.
|
||||||
m := catalog.Model{
|
m := catalog.Model{
|
||||||
Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
|
Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
|
||||||
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
|
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: maxOutput(req.ContextLength),
|
||||||
Profiles: profs,
|
Profiles: profs,
|
||||||
Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
|
Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
|
||||||
Capabilities: catalog.Capabilities{Vision: req.Projector != ""},
|
Capabilities: catalog.Capabilities{Vision: req.Projector != "", Tools: req.Tools},
|
||||||
Origin: &catalog.Origin{Repo: repo.ID, Revision: repo.Revision, Files: originFiles(set)},
|
Origin: &catalog.Origin{Repo: repo.ID, Revision: repo.Revision, Files: originFiles(set)},
|
||||||
}
|
}
|
||||||
if req.GuardFormat != "" {
|
if req.GuardFormat != "" {
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
package modelserve
|
||||||
|
|
||||||
|
import (
|
||||||
|
"slices"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLlamaCppIsGivenTheSlotsTheCatalogPromises(t *testing.T) {
|
||||||
|
// A 32768 pool for 32768-token requests is one slot: the engine's own
|
||||||
|
// four made slots of 8192, and requests the gateway admitted were refused.
|
||||||
|
for _, c := range []struct {
|
||||||
|
pool, ctx int
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{32768, 32768, []string{"--ctx-size", "32768", "--parallel", "1"}},
|
||||||
|
{16384, 4096, []string{"--ctx-size", "16384", "--parallel", "4"}},
|
||||||
|
{65536, 16384, []string{"--ctx-size", "65536", "--parallel", "4"}},
|
||||||
|
} {
|
||||||
|
if got := llamaArgs(Request{CtxPool: c.pool, ContextLength: c.ctx}); !slices.Equal(got, c.want) {
|
||||||
|
t.Errorf("pool %d, context %d: %v", c.pool, c.ctx, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTheLongestAnswerGrowsWithTheContext(t *testing.T) {
|
||||||
|
for ctx, want := range map[int]int{2048: 1024, 4096: 1024, 16384: 4096, 32768: 8192, 131072: 8192} {
|
||||||
|
if got := maxOutput(ctx); got != want {
|
||||||
|
t.Errorf("context %d: max output %d, want %d", ctx, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -208,6 +208,10 @@ func (r *Router) Handler() http.Handler {
|
|||||||
// So the example on the profile page names a model that is actually
|
// So the example on the profile page names a model that is actually
|
||||||
// running, rather than a placeholder that gets sent as-is.
|
// running, rather than a placeholder that gets sent as-is.
|
||||||
pages.ServedModels(r.exampleModels)
|
pages.ServedModels(r.exampleModels)
|
||||||
|
pages.SetModelFacts(func(name string) web.ModelFacts {
|
||||||
|
_, m := r.resolve(name)
|
||||||
|
return web.ModelFacts{ContextLength: m.ContextLength, MaxOutput: m.MaxOutput, Tools: m.Capabilities.Tools}
|
||||||
|
})
|
||||||
pages.SetPersonalOff(r.personalOff)
|
pages.SetPersonalOff(r.personalOff)
|
||||||
pages.SetLlamaCppVersion(r.llamaVersion)
|
pages.SetLlamaCppVersion(r.llamaVersion)
|
||||||
pages.SetPersonalDefaults(r.personalDefaults)
|
pages.SetPersonalDefaults(r.personalDefaults)
|
||||||
|
|||||||
@@ -77,6 +77,8 @@ type AccountPages struct {
|
|||||||
// field in that snippet is real, so the one placeholder left in it was
|
// field in that snippet is real, so the one placeholder left in it was
|
||||||
// copied verbatim and sent as the literal string MODEL.
|
// copied verbatim and sent as the literal string MODEL.
|
||||||
served func() []string
|
served func() []string
|
||||||
|
// modelFacts is what the catalog says of a model, for an editor.
|
||||||
|
modelFacts func(model string) ModelFacts
|
||||||
// hasMirror is set when /admin/mirror is registered, so the other admin
|
// hasMirror is set when /admin/mirror is registered, so the other admin
|
||||||
// pages can link to it and only then.
|
// pages can link to it and only then.
|
||||||
hasMirror bool
|
hasMirror bool
|
||||||
@@ -132,6 +134,9 @@ type AccountPages struct {
|
|||||||
// example falls back to a placeholder, which is what it always was.
|
// example falls back to a placeholder, which is what it always was.
|
||||||
func (p *AccountPages) ServedModels(f func() []string) { p.served = f }
|
func (p *AccountPages) ServedModels(f func() []string) { p.served = f }
|
||||||
|
|
||||||
|
// SetModelFacts supplies what an editor's configuration says of a model.
|
||||||
|
func (p *AccountPages) SetModelFacts(f func(model string) ModelFacts) { p.modelFacts = f }
|
||||||
|
|
||||||
// exampleModel picks the model the copy-pasteable example should name.
|
// exampleModel picks the model the copy-pasteable example should name.
|
||||||
//
|
//
|
||||||
// A real model over a simulated one is not expressible here — this package does
|
// A real model over a simulated one is not expressible here — this package does
|
||||||
|
|||||||
+60
-12
@@ -45,10 +45,25 @@ type connectField struct{ Label, Value string }
|
|||||||
// unquoted: the names this gateway serves, and nothing a shell reads twice.
|
// unquoted: the names this gateway serves, and nothing a shell reads twice.
|
||||||
var safeModel = regexp.MustCompile(`^[A-Za-z0-9._:/@+-]{1,200}$`)
|
var safeModel = regexp.MustCompile(`^[A-Za-z0-9._:/@+-]{1,200}$`)
|
||||||
|
|
||||||
|
// ModelFacts is what an editor's configuration needs to know of a model: how
|
||||||
|
// much one request may hold, and whether its tool calls come back as calls.
|
||||||
|
type ModelFacts struct {
|
||||||
|
ContextLength int
|
||||||
|
// MaxOutput is the longest answer the gateway allows; 0 is no limit.
|
||||||
|
MaxOutput int
|
||||||
|
Tools bool
|
||||||
|
}
|
||||||
|
|
||||||
// continueYAML is a Continue assistant pointing at the gateway: a file of its
|
// continueYAML is a Continue assistant pointing at the gateway: a file of its
|
||||||
// own under ~/.continue/assistants, so the person's config.yaml is not touched.
|
// own under ~/.continue/assistants, so the person's config.yaml is not touched.
|
||||||
func continueYAML(base, model, key string) string {
|
//
|
||||||
return fmt.Sprintf(`# Written by llmgw for Continue: the gateway at %[1]s.
|
// What it says of the model is what the catalog says. Continue assumed a
|
||||||
|
// context far larger than a CPU replica's, and every request over it was
|
||||||
|
// refused; and told a model uses tools when its calls come back as text,
|
||||||
|
// Continue's agent stalls on them rather than acting.
|
||||||
|
func continueYAML(base, model, key string, f ModelFacts) string {
|
||||||
|
var b strings.Builder
|
||||||
|
fmt.Fprintf(&b, `# Written by llmgw for Continue: the gateway at %[1]s.
|
||||||
name: llmgw
|
name: llmgw
|
||||||
version: 1.0.0
|
version: 1.0.0
|
||||||
schema: v1
|
schema: v1
|
||||||
@@ -62,11 +77,25 @@ models:
|
|||||||
- chat
|
- chat
|
||||||
- edit
|
- edit
|
||||||
- apply
|
- apply
|
||||||
capabilities:
|
|
||||||
- tool_use
|
|
||||||
# For completion as you type, add a small, fast model here with the role
|
|
||||||
# autocomplete; a large one makes every keystroke wait for it.
|
|
||||||
`, base, model, key)
|
`, base, model, key)
|
||||||
|
if f.Tools {
|
||||||
|
b.WriteString(" capabilities:\n - tool_use\n")
|
||||||
|
}
|
||||||
|
if f.ContextLength > 0 {
|
||||||
|
fmt.Fprintf(&b, ` # What the gateway lets one request hold; Continue trims to fit.
|
||||||
|
defaultCompletionOptions:
|
||||||
|
contextLength: %d
|
||||||
|
maxTokens: %d
|
||||||
|
`, f.ContextLength, maxTokensFor(f))
|
||||||
|
}
|
||||||
|
if !f.Tools {
|
||||||
|
b.WriteString("# This model's catalog entry does not say it makes tool calls, so Continue\n" +
|
||||||
|
"# chats and edits with it but its agent cannot act: choose one that does.\n")
|
||||||
|
}
|
||||||
|
b.WriteString(`# For completion as you type, add a small, fast model here with the role
|
||||||
|
# autocomplete; a large one makes every keystroke wait for it.
|
||||||
|
`)
|
||||||
|
return b.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// codexTOML is the provider and profile Codex needs, and nothing else of its
|
// codexTOML is the provider and profile Codex needs, and nothing else of its
|
||||||
@@ -87,13 +116,13 @@ model = "%[2]s"
|
|||||||
}
|
}
|
||||||
|
|
||||||
// connectFor builds what the page shows for a tool, in English.
|
// connectFor builds what the page shows for a tool, in English.
|
||||||
func connectFor(tool, base, model, key string) connectView {
|
func connectFor(tool, base, model, key string, f ModelFacts) connectView {
|
||||||
return connectIn(english, tool, base, model, key)
|
return connectIn(english, tool, base, model, key, f)
|
||||||
}
|
}
|
||||||
|
|
||||||
// connectIn builds what the page shows for a tool, its sentences in l. The
|
// connectIn builds what the page shows for a tool, its sentences in l. The
|
||||||
// files, commands and paths are the same in every language.
|
// files, commands and paths are the same in every language.
|
||||||
func connectIn(l *Lang, tool, base, model, key string) connectView {
|
func connectIn(l *Lang, tool, base, model, key string, f ModelFacts) connectView {
|
||||||
v := connectView{Tool: tool, Model: model, Key: key}
|
v := connectView{Tool: tool, Model: model, Key: key}
|
||||||
for _, t := range editorTools {
|
for _, t := range editorTools {
|
||||||
if t.Key == tool {
|
if t.Key == tool {
|
||||||
@@ -104,7 +133,7 @@ func connectIn(l *Lang, tool, base, model, key string) connectView {
|
|||||||
switch tool {
|
switch tool {
|
||||||
case "continue":
|
case "continue":
|
||||||
v.Files = []connectFile{{Path: "~/.continue/assistants/llmgw.yaml", Name: "llmgw.yaml",
|
v.Files = []connectFile{{Path: "~/.continue/assistants/llmgw.yaml", Name: "llmgw.yaml",
|
||||||
Body: continueYAML(base, model, key)}}
|
Body: continueYAML(base, model, key, f)}}
|
||||||
v.Command = fmt.Sprintf("curl -fsSL '%s' | LLMGW_KEY='%s' sh", script, key)
|
v.Command = fmt.Sprintf("curl -fsSL '%s' | LLMGW_KEY='%s' sh", script, key)
|
||||||
v.Run = l.T("connect.run.continue")
|
v.Run = l.T("connect.run.continue")
|
||||||
case "codex":
|
case "codex":
|
||||||
@@ -173,7 +202,7 @@ func (p *AccountPages) connect(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
p.log.Info("a user connected an editor", "user", user.ID, "tool", tool, "key", k.ID)
|
p.log.Info("a user connected an editor", "user", user.ID, "tool", tool, "key", k.ID)
|
||||||
c := connectIn(l, tool, baseURL(r), model, secret)
|
c := connectIn(l, tool, baseURL(r), model, secret, p.facts(model))
|
||||||
p.render(w, r, sess, user, profileData{Connect: &c})
|
p.render(w, r, sess, user, profileData{Connect: &c})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -210,7 +239,7 @@ cat > "$file" <<'LLMGW'
|
|||||||
tmp="$file.tmp"
|
tmp="$file.tmp"
|
||||||
sed "s|__LLMGW_KEY__|$LLMGW_KEY|" "$file" > "$tmp" && mv "$tmp" "$file"
|
sed "s|__LLMGW_KEY__|$LLMGW_KEY|" "$file" > "$tmp" && mv "$tmp" "$file"
|
||||||
echo "Wrote $file. In Continue, choose the llmgw assistant."
|
echo "Wrote $file. In Continue, choose the llmgw assistant."
|
||||||
`, base, continueYAML(base, model, "__LLMGW_KEY__"))
|
`, base, continueYAML(base, model, "__LLMGW_KEY__", p.facts(model)))
|
||||||
case "codex":
|
case "codex":
|
||||||
body = fmt.Sprintf(`#!/bin/sh
|
body = fmt.Sprintf(`#!/bin/sh
|
||||||
# Connects Codex to the llmgw gateway at %[1]s.
|
# Connects Codex to the llmgw gateway at %[1]s.
|
||||||
@@ -242,3 +271,22 @@ echo "Run: LLMGW_API_KEY=<your key> codex --profile llmgw"
|
|||||||
w.Header().Set("Cache-Control", "no-store")
|
w.Header().Set("Cache-Control", "no-store")
|
||||||
_, _ = w.Write([]byte(body))
|
_, _ = w.Write([]byte(body))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// facts is what the catalog says of a model, when there is one to ask.
|
||||||
|
func (p *AccountPages) facts(model string) ModelFacts {
|
||||||
|
if p.modelFacts == nil {
|
||||||
|
return ModelFacts{}
|
||||||
|
}
|
||||||
|
return p.modelFacts(model)
|
||||||
|
}
|
||||||
|
|
||||||
|
// maxTokensFor is the answer length Continue asks for: a quarter of the
|
||||||
|
// context, at most 4096, and never over what the gateway allows — asked for
|
||||||
|
// more, it refuses the request.
|
||||||
|
func maxTokensFor(f ModelFacts) int {
|
||||||
|
n := min(4096, max(256, f.ContextLength/4))
|
||||||
|
if f.MaxOutput > 0 {
|
||||||
|
n = min(n, f.MaxOutput)
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
@@ -11,7 +11,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestEachToolGetsItsConfigurationWithTheKey(t *testing.T) {
|
func TestEachToolGetsItsConfigurationWithTheKey(t *testing.T) {
|
||||||
c := connectFor("continue", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc")
|
c := connectFor("continue", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc", ModelFacts{})
|
||||||
if !strings.Contains(c.Files[0].Body, "apiBase: https://gw.example/v1") || !strings.Contains(c.Files[0].Body, "apiKey: sk-llmgw-abc") ||
|
if !strings.Contains(c.Files[0].Body, "apiBase: https://gw.example/v1") || !strings.Contains(c.Files[0].Body, "apiKey: sk-llmgw-abc") ||
|
||||||
c.Files[0].Path != "~/.continue/assistants/llmgw.yaml" {
|
c.Files[0].Path != "~/.continue/assistants/llmgw.yaml" {
|
||||||
t.Errorf("continue = %+v", c.Files)
|
t.Errorf("continue = %+v", c.Files)
|
||||||
@@ -21,11 +21,11 @@ func TestEachToolGetsItsConfigurationWithTheKey(t *testing.T) {
|
|||||||
if strings.Contains(strings.Split(c.Command, "|")[0], "sk-llmgw-abc") {
|
if strings.Contains(strings.Split(c.Command, "|")[0], "sk-llmgw-abc") {
|
||||||
t.Errorf("the key is in the URL: %s", c.Command)
|
t.Errorf("the key is in the URL: %s", c.Command)
|
||||||
}
|
}
|
||||||
x := connectFor("codex", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc")
|
x := connectFor("codex", "https://gw.example", "qwen2.5-coder-7b", "sk-llmgw-abc", ModelFacts{})
|
||||||
if strings.Contains(x.Files[0].Body, "sk-llmgw-abc") || !strings.Contains(x.Files[0].Body, `env_key = "LLMGW_API_KEY"`) {
|
if strings.Contains(x.Files[0].Body, "sk-llmgw-abc") || !strings.Contains(x.Files[0].Body, `env_key = "LLMGW_API_KEY"`) {
|
||||||
t.Errorf("codex writes the key into its file: %s", x.Files[0].Body)
|
t.Errorf("codex writes the key into its file: %s", x.Files[0].Body)
|
||||||
}
|
}
|
||||||
cl := connectFor("cline", "https://gw.example", "m", "sk-llmgw-abc")
|
cl := connectFor("cline", "https://gw.example", "m", "sk-llmgw-abc", ModelFacts{})
|
||||||
if len(cl.Fields) != 4 || cl.Fields[1].Value != "https://gw.example/v1" || cl.Command != "" {
|
if len(cl.Fields) != 4 || cl.Fields[1].Value != "https://gw.example/v1" || cl.Command != "" {
|
||||||
t.Errorf("cline = %+v", cl)
|
t.Errorf("cline = %+v", cl)
|
||||||
}
|
}
|
||||||
@@ -115,3 +115,31 @@ func TestASetupScriptRefusesANameAShellWouldRead(t *testing.T) {
|
|||||||
t.Errorf("%d", w.Code)
|
t.Errorf("%d", w.Code)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestContinueIsToldTheModelsContextAndWhetherItCanUseTools(t *testing.T) {
|
||||||
|
// Told a context far larger than the gateway allows, Continue sends
|
||||||
|
// prompts that are refused; told a model uses tools when its calls come
|
||||||
|
// back as text, its agent stalls.
|
||||||
|
agent := continueYAML("https://gw.example", "coder-30b", "k", ModelFacts{ContextLength: 32768, Tools: true})
|
||||||
|
for _, want := range []string{"tool_use", "contextLength: 32768", "maxTokens: 4096"} {
|
||||||
|
if !strings.Contains(agent, want) {
|
||||||
|
t.Errorf("%q missing:\n%s", want, agent)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
chat := continueYAML("https://gw.example", "coder", "k", ModelFacts{ContextLength: 4096})
|
||||||
|
if strings.Contains(chat, "tool_use") || !strings.Contains(chat, "contextLength: 4096") ||
|
||||||
|
!strings.Contains(chat, "maxTokens: 1024") || !strings.Contains(chat, "its agent cannot act") {
|
||||||
|
t.Errorf("a model without tools:\n%s", chat)
|
||||||
|
}
|
||||||
|
if unknown := continueYAML("https://gw.example", "m", "k", ModelFacts{}); strings.Contains(unknown, "contextLength") {
|
||||||
|
t.Errorf("a context nobody knows was written down:\n%s", unknown)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContinueNeverAsksForMoreThanTheGatewayAllows(t *testing.T) {
|
||||||
|
// Asked for more than max_output, the gateway refuses the request.
|
||||||
|
y := continueYAML("https://gw.example", "m", "k", ModelFacts{ContextLength: 32768, MaxOutput: 2048, Tools: true})
|
||||||
|
if !strings.Contains(y, "maxTokens: 2048") {
|
||||||
|
t.Errorf("not capped at max_output:\n%s", y)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -171,18 +171,18 @@ func TestTheProfilePageShowsTurkishNumbersAndDates(t *testing.T) {
|
|||||||
|
|
||||||
func TestTheEditorSetupExplainsItselfInTurkish(t *testing.T) {
|
func TestTheEditorSetupExplainsItselfInTurkish(t *testing.T) {
|
||||||
tr := langByCode("tr")
|
tr := langByCode("tr")
|
||||||
c := connectIn(tr, "cline", "https://gw.example", "m", "sk-llmgw-abc")
|
c := connectIn(tr, "cline", "https://gw.example", "m", "sk-llmgw-abc", ModelFacts{})
|
||||||
if c.Fields[2].Label != "API anahtarı (API Key)" || c.Fields[0].Value != "OpenAI Compatible" ||
|
if c.Fields[2].Label != "API anahtarı (API Key)" || c.Fields[0].Value != "OpenAI Compatible" ||
|
||||||
!strings.Contains(c.Run, "dişli simgesine") {
|
!strings.Contains(c.Run, "dişli simgesine") {
|
||||||
t.Errorf("cline in Turkish = %+v", c)
|
t.Errorf("cline in Turkish = %+v", c)
|
||||||
}
|
}
|
||||||
x := connectIn(tr, "codex", "https://gw.example", "m", "sk-llmgw-abc")
|
x := connectIn(tr, "codex", "https://gw.example", "m", "sk-llmgw-abc", ModelFacts{})
|
||||||
// What is typed or written to a file stays what the tool reads.
|
// What is typed or written to a file stays what the tool reads.
|
||||||
if x.Run != "LLMGW_API_KEY='sk-llmgw-abc' codex --profile llmgw" || !strings.Contains(x.Files[0].Body, "[profiles.llmgw]") ||
|
if x.Run != "LLMGW_API_KEY='sk-llmgw-abc' codex --profile llmgw" || !strings.Contains(x.Files[0].Body, "[profiles.llmgw]") ||
|
||||||
!strings.Contains(x.Notes[0], "~/.codex/config.toml") {
|
!strings.Contains(x.Notes[0], "~/.codex/config.toml") {
|
||||||
t.Errorf("codex in Turkish = %+v", x)
|
t.Errorf("codex in Turkish = %+v", x)
|
||||||
}
|
}
|
||||||
if en := connectFor("continue", "https://gw.example", "m", "k"); en.Run != `In Continue, choose the "llmgw" assistant at the top of its panel.` {
|
if en := connectFor("continue", "https://gw.example", "m", "k", ModelFacts{}); en.Run != `In Continue, choose the "llmgw" assistant at the top of its panel.` {
|
||||||
t.Errorf("English Continue: %q", en.Run)
|
t.Errorf("English Continue: %q", en.Run)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Binary file not shown.
@@ -261,7 +261,7 @@ func TestControlPlaneFeedsRouter(t *testing.T) {
|
|||||||
}()
|
}()
|
||||||
defer close(stop)
|
defer close(stop)
|
||||||
|
|
||||||
h.waitForEndpoints(t, 2, 15*time.Second)
|
h.waitForEndpoints(t, 2, 30*time.Second)
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
@@ -304,7 +304,7 @@ func TestRouterKeepsServingWhileNodeIsLost(t *testing.T) {
|
|||||||
}()
|
}()
|
||||||
defer close(stop)
|
defer close(stop)
|
||||||
|
|
||||||
h.waitForEndpoints(t, 2, 15*time.Second)
|
h.waitForEndpoints(t, 2, 30*time.Second)
|
||||||
|
|
||||||
// Continuous traffic for the whole experiment.
|
// Continuous traffic for the whole experiment.
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
|||||||
Reference in new issue
Block a user