An external model takes api_key_files, each a file and an optional weight, in place of api_key_file. Each key is an endpoint of its own (external:MODEL#1, #2), so the router's scoring, retries and breakers work per key. A key answering 429 is passed over while its breaker is open. A key answering 401 is passed over too, and external_key_refused is sent, naming the endpoint and never the key; with a single key the 401 is still the client's. A key whose file cannot be read is left out, and usage records which key answered by its endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
688b205604
commit
fe310576e7
14 files changed
+283
-15
No files matched your search
@@ -125,11 +125,16 @@ var Reference = []Block{
|
||||
{Key: "url", Type: "string", Doc: "URL is the provider's base address, without /v1: https://api.openai.com.", Nested: ""},
|
||||
{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
|
||||
{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
|
||||
{Key: "api_key_files", Type: "[]ExternalKey", Doc: "APIKeyFiles are several keys for the provider, in place of\nAPIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that\nanswers 429 or 401 is passed over while its breaker is open and the\nrequest goes to another. A provider's limits are per key.", Nested: ""},
|
||||
{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
|
||||
{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
|
||||
{Key: "output", Type: "string", Doc: "Output is what it makes: pictures, or meshes (see ComfyUI.Output).", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "ExternalKey", Title: "ExternalKey", Doc: "", Fields: []Field{
|
||||
{Key: "file", Type: "string", Doc: "File holds the key, on the control plane, as APIKeyFile does.", Nested: ""},
|
||||
{Key: "weight", Type: "float64", Doc: "Weight is its share of the requests against the others'; zero is 1.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
|
||||
{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
|
||||
{Key: "margin", Type: "float64", Doc: "Margin is the share of every cap kept free, 0.1 for ten percent. Zero\nkeeps none, for caps already given derated.", Nested: ""},
|
||||
|
||||
Reference in new issue
Block a user