Federation: another llmgw cluster as a provider, its models as PEER/MODEL
ctl federation add/list/remove/sync keeps peers in the store (migration 0029); their /v1/models are offered here as external models named PEER/MODEL and kept in step, a model still named by a fallback kept withdrawn. Peers are checked at /ready, timed, and taken out of routing while down. Requests to a peer carry x-llmgw-hop and are refused (508) when they come back to a cluster they passed through; residency and no_store keep requests off peers; X-Request-Id and, with --send-tags, attribution tags go along. ctl model fallbacks sets overflow to a peer. Console view, metrics, admin API routes and docs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
40e9fd2ab1
33 files changed
+2944
-23
No files matched your search
@@ -0,0 +1,40 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Federating clusters: another llmgw cluster as a provider
|
||||
|
||||
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
|
||||
makes another llmgw gateway a peer. It is checked first (its `/ready`,
|
||||
timed, and its `/v1/models` with the key) and kept in the store (migration
|
||||
0029, `federation_peers`) with the models it offers.
|
||||
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
|
||||
follow the peer's own list every `federation.interval` (10s). A model the
|
||||
peer drops goes, unless a model here still falls back to it: then it is
|
||||
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
|
||||
The peer's aliases and its own peers' models are not taken.
|
||||
- A peer whose `/ready` fails, which refuses the key, or which does not answer
|
||||
is taken out of routing until it answers again. `/ready` now counts only a
|
||||
gateway's own endpoints, not its peers'.
|
||||
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
|
||||
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
|
||||
overflow can go to a peer's model.
|
||||
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
|
||||
have passed through (each cluster's id is kept in the store). A request that
|
||||
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
|
||||
request came from is not sent it.
|
||||
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
|
||||
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
|
||||
org=,team=,project=` (ids); off by default.
|
||||
- In-cluster residency and `no_store` keep requests off peers, by name and as
|
||||
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
|
||||
as the replica.
|
||||
- `ctl federation list|remove|sync`, a Federation view in the console,
|
||||
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
|
||||
API `/api/v1/federation`, and a "Federating clusters" section in
|
||||
docs/install.md. New `federation:` block in server.yaml (`name`,
|
||||
`interval`).
|
||||
- A response header the gateway already set (X-Request-Id,
|
||||
x-llmgw-served-model) is no longer repeated from the upstream's.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [x] 24. Multi-cluster federation — built: peers via `ctl federation` kept in the store, their models as PEER/MODEL synced from /v1/models (withdrawn-while-referenced), /ready health with latency and routing withdrawal, `ctl model fallbacks` for overflow, hop header loop refusal, residency/no_store hold, optional attribution tags, console view, metrics, docs. Left: server.yaml-declared models cannot name a peer model as a fallback (config catalog is validated before peers are read); per-peer CA / mTLS for peers behind a private CA; transitive federation (A→B→C models) deliberately not offered; latency is /ready RTT only (no per-request peer TTFT used in routing or weighting); the peer's adapters are not imported (listed as aliases); no console form to add/remove peers (CLI only); key file must exist on every control plane.
|
||||
Reference in new issue
Block a user