Federation: another llmgw cluster as a provider, its models as PEER/MODEL

ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:21:30 +02:00
1 parent d41104a452
commit 40e9fd2ab1
33 files changed
+2944 -23

No files matched your search

+57
View File
@@ -1062,6 +1062,63 @@ prices and budgets, rate limits, usage records, and key allowlists.
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.
### Federating clusters
Another llmgw cluster — another site, another team's GPUs — can be a provider
of this one, and llmgw knows it is one. Ask the other cluster's operator for a
key in one of their projects (`llmgw keys create`), put it in a file on this
control plane (on every control plane, with HA), and add the peer:
```
llmgw ctl federation add east --url https://llm.east.example --key-file /etc/llmgw/secrets/east.key
llmgw ctl federation list
llmgw ctl model fallbacks qwen-7b east/qwen-7b # overflow goes to east
```
- **Its models, named PEER/MODEL.** What the peer's `/v1/models` offers that
key is offered here as `east/qwen-7b` and so on, always with the prefix —
never under the peer's own name, even when nothing here uses it, so
whether a request leaves the cluster never depends on what else is in the
catalog, and every usage record and `x-llmgw-served-model` header says which
site answered. A model here called `east/...` wins over the peer's of the
same name. The peer's aliases, and models the peer has from peers of its
own, are not taken: federation is one hop.
- **Kept in step.** Each peer is checked every `federation.interval` (10s):
its `/ready`, timed — the latency `ctl federation list` shows — and its
`/v1/models`. A model it adds appears here; one it drops goes, unless a
model here still falls back to it, when it is kept *withdrawn* (in the
catalog, sent nothing) until the fallback is changed. `ctl federation sync
east` checks now.
- **Taken out when down.** A peer whose `/ready` fails, which refuses the key,
or which does not answer is sent nothing until it answers again; its models
stay in the catalog. A peer's `/ready` counts only what it serves itself, so
two clusters federating each other are not each ready because of the other.
- **Overflow.** Named in a model's fallbacks (`ctl model fallbacks`, for
models added at runtime), a peer's model is used only when the model has
no capacity here — as for any external model.
- **What stays here, stays here.** A peer's model is an external model: a
project with `keys residency --in-cluster`, or under a guardrails policy with
`no_store`, is refused it by name and never sent to it as a fallback.
- **One key, and who is asking.** The peer sees this cluster's key, so its
usage, limits and budgets are one tenant's. The request's `X-Request-Id`
goes with it, so one id finds the request on both sides. With
`--send-tags` the request also carries `x-llmgw-tags:
federated_from=<federation.name>,org=…,team=…,project=…` (ids), for the
peer's usage records to attribute it; off unless given, because it tells
the other site who here is asking.
- **No loops.** Each cluster has an id (`ctl federation list` shows it), and a
request sent to a peer carries `x-llmgw-hop` with the ids it has passed
through. A cluster refuses a request that has already passed through it
(508, `federation_loop`), and does not send one to a peer it came from —
so two clusters falling back to each other try each once.
- `llmgw_federation_peer_up`, `_latency_seconds` and `_models` are on
`/metrics`; the console has a Federation view. `ctl federation remove east`
takes the peer and its models away, refused while a fallback names one.
Not yet: models declared in server.yaml cannot name a peer's model as a
fallback (the catalog in the file is checked before the peers are read), and
a peer behind a private CA needs that CA in the system's trust store.
### Trying new weights on some of the traffic first
When a model's repository publishes new weights, `llmgw ctl model update`