A profile may name the card it needs rather than its class, and fit any site
CI / test (push) Failing after 6m19s

A catalog entry meant for many sites cannot know their GPU class names:
the PoC has a Quadro RTX 6000, a customer an A100, another an L40S. A
profile may now say gpu: {min_compute_capability, min_memory_gb} in
place of gpu_class, and stands for every class in the cluster that meets
it, named <profile>@<class>, at its own priority; placement still sees
exact classes. ProfileByName and EngineProfile resolve such a name to
the profile on that class, so every caller that maps a replica back to
its profile works unchanged. A ComfyUI profile may carry its own
workflow, checkpoint and nodes, and with its own source an fp8 workflow
and weights go to cards with fp8 and fp16 ones to Turing.

A profile by need is left out of placement until the cluster's cards are
known: with no class it would have read as a CPU profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-11 16:02:32 +02:00
1 parent b6bfc33ef7
commit 777a9223a7
11 files changed
+419 -14

No files matched your search

+40
View File
@@ -8869,6 +8869,20 @@
}
}
},
"catalog.GPUNeed": {
"type": "object",
"description": "GPUNeed is what card a profile needs.",
"properties": {
"min_compute_capability": {
"type": "number",
"description": "MinComputeCapability is the oldest architecture that runs it: 7.5 is Turing, 8.0 Ampere (bf16, flash-attention 2), 8.9 Ada (fp8), 9.0 Hopper."
},
"min_memory_gb": {
"type": "integer",
"description": "MinMemoryGB is the card's own memory it needs."
}
}
},
"catalog.GuardModel": {
"type": "object",
"description": "GuardModel is how a judge model's answers are read.",
@@ -9227,6 +9241,32 @@
"kv_events": {
"type": "boolean",
"description": "KVEvents has vLLM publish what its prefix cache holds, and the router route this model's requests by it: to the replica holding the most of a prompt's leading blocks (plan §7.5 tier 2). vLLM only.\n\nOff by default, because it costs something on every request: the router asks a replica to tokenize the prompt before placing it, a round trip and the chat template run twice. It pays where prompts share long prefixes that the approximate tree guesses wrong — many replicas, conversations that move between them, evictions. The engine is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED, which the router's hashing assumes; arguments overriding either leave the index empty, and the agent says so in its log."
},
"gpu": {
"$ref": "#/components/schemas/catalog.GPUNeed",
"description": "GPU says what card the profile needs, in place of naming its class: it stands for every class in the cluster that meets it. A catalog entry written once then places sensibly on any site's cards."
},
"comfyui": {
"$ref": "#/components/schemas/catalog.ProfileComfyUI",
"description": "ComfyUI is this profile's own workflow, checkpoint and nodes, over the model's: an fp8 workflow where the card has fp8, fp16 where it has not."
}
}
},
"catalog.ProfileComfyUI": {
"type": "object",
"description": "ProfileComfyUI is what a ComfyUI profile runs differently from the model: each field set replaces the model's.",
"properties": {
"workflow": {
"type": "string"
},
"checkpoint": {
"type": "string"
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/catalog.CustomNode"
}
}
}
},