Files
llmgw/notes/c24-federation.md
T
Ilker ManapandClaude Opus 5.5 40e9fd2ab1 Federation: another llmgw cluster as a provider, its models as PEER/MODEL
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:30 +02:00

2.9 KiB

CHANGELOG

Federating clusters: another llmgw cluster as a provider

  • llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags] makes another llmgw gateway a peer. It is checked first (its /ready, timed, and its /v1/models with the key) and kept in the store (migration 0029, federation_peers) with the models it offers.
  • A peer's models join the catalog as NAME/MODEL, always prefixed, and follow the peer's own list every federation.interval (10s). A model the peer drops goes, unless a model here still falls back to it: then it is kept withdrawn (in the catalog, sent nothing) until the fallback changes. The peer's aliases and its own peers' models are not taken.
  • A peer whose /ready fails, which refuses the key, or which does not answer is taken out of routing until it answers again. /ready now counts only a gateway's own endpoints, not its peers'.
  • llmgw ctl model fallbacks MODEL [FALLBACK...] (and PUT /api/v1/models/{name}/fallbacks) sets a runtime model's fallbacks, so overflow can go to a peer's model.
  • Requests to a peer carry x-llmgw-hop with the ids of the clusters they have passed through (each cluster's id is kept in the store). A request that comes back to a cluster is refused with 508 federation_loop, and a peer a request came from is not sent it.
  • The peer sees one key, this cluster's, and the request's X-Request-Id. With --send-tags, also x-llmgw-tags: federated_from=<federation.name>, org=,team=,project= (ids); off by default.
  • In-cluster residency and no_store keep requests off peers, by name and as fallbacks. Usage records the peer's model as served, and peer:NAME/MODEL as the replica.
  • ctl federation list|remove|sync, a Federation view in the console, llmgw_federation_peer_up, _latency_seconds and _models metrics, admin API /api/v1/federation, and a "Federating clusters" section in docs/install.md. New federation: block in server.yaml (name, interval).
  • A response header the gateway already set (X-Request-Id, x-llmgw-served-model) is no longer repeated from the upstream's.

BACKLOG

  • 24. Multi-cluster federation — built: peers via ctl federation kept in the store, their models as PEER/MODEL synced from /v1/models (withdrawn-while-referenced), /ready health with latency and routing withdrawal, ctl model fallbacks for overflow, hop header loop refusal, residency/no_store hold, optional attribution tags, console view, metrics, docs. Left: server.yaml-declared models cannot name a peer model as a fallback (config catalog is validated before peers are read); per-peer CA / mTLS for peers behind a private CA; transitive federation (A→B→C models) deliberately not offered; latency is /ready RTT only (no per-request peer TTFT used in routing or weighting); the peer's adapters are not imported (listed as aliases); no console form to add/remove peers (CLI only); key file must exist on every control plane.