Federation: another llmgw cluster as a provider, its models as PEER/MODEL

ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:21:30 +02:00
1 parent d41104a452
commit 40e9fd2ab1
33 files changed
+2944 -23

No files matched your search

+41
View File
@@ -86,6 +86,9 @@ rather than repeated.
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
| `peer` | `string` | [External](#external) |
| `name` | `string` | [Federation](#federation) |
| `interval` | `time.Duration` | [Federation](#federation) |
| `format` | `string` | [GuardModel](#guardmodel) |
| `categories` | `[]string` | [GuardModel](#guardmodel) |
| `model` | `string` | [JudgePolicy](#judgepolicy) |
@@ -271,6 +274,7 @@ rather than repeated.
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
| `models` | `[]catalog.Model` | [Server](#server) |
| `federation` | `Federation` | [Server](#server) |
| `endpoints` | `[]Endpoint` | [Server](#server) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
@@ -839,6 +843,33 @@ APIKeyFile holds the provider's key, on the control plane. A file
rather than the key itself, so the key is in neither the catalog, the
store nor the audit log; empty for a provider that asks for none.
### `peer`
`string`
Peer names the federated llmgw cluster this model is offered by, for
a model that federation added (`llmgw ctl federation add`) rather
than one written by hand. Such a model is named PEER/MODEL, comes and
goes with the peer's own list, and is sent only while the peer is up.
## Federation
### `name`
`string`
Name is this cluster as a peer reads it, in the federated_from tag
sent with a request to a peer set to receive tags (`ctl federation add
--send-tags`). Empty is this machine's host name.
### `interval`
`time.Duration`
Interval is how often each peer is asked whether it is ready, timed,
and what it offers. A peer that does not answer is taken out of
routing until it does. Zero is 10s.
## GuardModel
### `format`
@@ -2249,6 +2280,16 @@ Takes a [Control](#control) block.
`[]catalog.Model`
### `federation`
`Federation`
Federation is how this cluster takes part in federation with other
llmgw clusters. The peers themselves are added at runtime, with
`llmgw ctl federation add`, and kept in the store.
Takes a [Federation](#federation) block.
### `endpoints`
`[]Endpoint`