Federation: another llmgw cluster as a provider, its models as PEER/MODEL
ctl federation add/list/remove/sync keeps peers in the store (migration 0029); their /v1/models are offered here as external models named PEER/MODEL and kept in step, a model still named by a fallback kept withdrawn. Peers are checked at /ready, timed, and taken out of routing while down. Requests to a peer carry x-llmgw-hop and are refused (508) when they come back to a cluster they passed through; residency and no_store keep requests off peers; X-Request-Id and, with --send-tags, attribution tags go along. ctl model fallbacks sets overflow to a peer. Console view, metrics, admin API routes and docs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
40e9fd2ab1
33 files changed
+2944
-23
No files matched your search
@@ -1062,6 +1062,63 @@ prices and budgets, rate limits, usage records, and key allowlists.
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### Federating clusters
|
||||
|
||||
Another llmgw cluster — another site, another team's GPUs — can be a provider
|
||||
of this one, and llmgw knows it is one. Ask the other cluster's operator for a
|
||||
key in one of their projects (`llmgw keys create`), put it in a file on this
|
||||
control plane (on every control plane, with HA), and add the peer:
|
||||
|
||||
```
|
||||
llmgw ctl federation add east --url https://llm.east.example --key-file /etc/llmgw/secrets/east.key
|
||||
llmgw ctl federation list
|
||||
llmgw ctl model fallbacks qwen-7b east/qwen-7b # overflow goes to east
|
||||
```
|
||||
|
||||
- **Its models, named PEER/MODEL.** What the peer's `/v1/models` offers that
|
||||
key is offered here as `east/qwen-7b` and so on, always with the prefix —
|
||||
never under the peer's own name, even when nothing here uses it, so
|
||||
whether a request leaves the cluster never depends on what else is in the
|
||||
catalog, and every usage record and `x-llmgw-served-model` header says which
|
||||
site answered. A model here called `east/...` wins over the peer's of the
|
||||
same name. The peer's aliases, and models the peer has from peers of its
|
||||
own, are not taken: federation is one hop.
|
||||
- **Kept in step.** Each peer is checked every `federation.interval` (10s):
|
||||
its `/ready`, timed — the latency `ctl federation list` shows — and its
|
||||
`/v1/models`. A model it adds appears here; one it drops goes, unless a
|
||||
model here still falls back to it, when it is kept *withdrawn* (in the
|
||||
catalog, sent nothing) until the fallback is changed. `ctl federation sync
|
||||
east` checks now.
|
||||
- **Taken out when down.** A peer whose `/ready` fails, which refuses the key,
|
||||
or which does not answer is sent nothing until it answers again; its models
|
||||
stay in the catalog. A peer's `/ready` counts only what it serves itself, so
|
||||
two clusters federating each other are not each ready because of the other.
|
||||
- **Overflow.** Named in a model's fallbacks (`ctl model fallbacks`, for
|
||||
models added at runtime), a peer's model is used only when the model has
|
||||
no capacity here — as for any external model.
|
||||
- **What stays here, stays here.** A peer's model is an external model: a
|
||||
project with `keys residency --in-cluster`, or under a guardrails policy with
|
||||
`no_store`, is refused it by name and never sent to it as a fallback.
|
||||
- **One key, and who is asking.** The peer sees this cluster's key, so its
|
||||
usage, limits and budgets are one tenant's. The request's `X-Request-Id`
|
||||
goes with it, so one id finds the request on both sides. With
|
||||
`--send-tags` the request also carries `x-llmgw-tags:
|
||||
federated_from=<federation.name>,org=…,team=…,project=…` (ids), for the
|
||||
peer's usage records to attribute it; off unless given, because it tells
|
||||
the other site who here is asking.
|
||||
- **No loops.** Each cluster has an id (`ctl federation list` shows it), and a
|
||||
request sent to a peer carries `x-llmgw-hop` with the ids it has passed
|
||||
through. A cluster refuses a request that has already passed through it
|
||||
(508, `federation_loop`), and does not send one to a peer it came from —
|
||||
so two clusters falling back to each other try each once.
|
||||
- `llmgw_federation_peer_up`, `_latency_seconds` and `_models` are on
|
||||
`/metrics`; the console has a Federation view. `ctl federation remove east`
|
||||
takes the peer and its models away, refused while a fallback names one.
|
||||
|
||||
Not yet: models declared in server.yaml cannot name a peer's model as a
|
||||
fallback (the catalog in the file is checked before the peers are read), and
|
||||
a peer behind a private CA needs that CA in the system's trust store.
|
||||
|
||||
### Trying new weights on some of the traffic first
|
||||
|
||||
When a model's repository publishes new weights, `llmgw ctl model update`
|
||||
|
||||
Reference in new issue
Block a user