A profile may name the card it needs rather than its class, and fit any site
CI / test (push) Failing after 6m19s
CI / test (push) Failing after 6m19s
A catalog entry meant for many sites cannot know their GPU class names:
the PoC has a Quadro RTX 6000, a customer an A100, another an L40S. A
profile may now say gpu: {min_compute_capability, min_memory_gb} in
place of gpu_class, and stands for every class in the cluster that meets
it, named <profile>@<class>, at its own priority; placement still sees
exact classes. ProfileByName and EngineProfile resolve such a name to
the profile on that class, so every caller that maps a replica back to
its profile works unchanged. A ComfyUI profile may carry its own
workflow, checkpoint and nodes, and with its own source an fp8 workflow
and weights go to cards with fp8 and fp16 ones to Turing.
A profile by need is left out of placement until the cluster's cards are
known: with no class it would have read as a CPU profile.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
b6bfc33ef7
commit
777a9223a7
11 files changed
+419
-14
No files matched your search
@@ -8869,6 +8869,20 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.GPUNeed": {
|
||||
"type": "object",
|
||||
"description": "GPUNeed is what card a profile needs.",
|
||||
"properties": {
|
||||
"min_compute_capability": {
|
||||
"type": "number",
|
||||
"description": "MinComputeCapability is the oldest architecture that runs it: 7.5 is Turing, 8.0 Ampere (bf16, flash-attention 2), 8.9 Ada (fp8), 9.0 Hopper."
|
||||
},
|
||||
"min_memory_gb": {
|
||||
"type": "integer",
|
||||
"description": "MinMemoryGB is the card's own memory it needs."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.GuardModel": {
|
||||
"type": "object",
|
||||
"description": "GuardModel is how a judge model's answers are read.",
|
||||
@@ -9227,6 +9241,32 @@
|
||||
"kv_events": {
|
||||
"type": "boolean",
|
||||
"description": "KVEvents has vLLM publish what its prefix cache holds, and the router route this model's requests by it: to the replica holding the most of a prompt's leading blocks (plan §7.5 tier 2). vLLM only.\n\nOff by default, because it costs something on every request: the router asks a replica to tokenize the prompt before placing it, a round trip and the chat template run twice. It pays where prompts share long prefixes that the approximate tree guesses wrong — many replicas, conversations that move between them, evictions. The engine is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED, which the router's hashing assumes; arguments overriding either leave the index empty, and the agent says so in its log."
|
||||
},
|
||||
"gpu": {
|
||||
"$ref": "#/components/schemas/catalog.GPUNeed",
|
||||
"description": "GPU says what card the profile needs, in place of naming its class: it stands for every class in the cluster that meets it. A catalog entry written once then places sensibly on any site's cards."
|
||||
},
|
||||
"comfyui": {
|
||||
"$ref": "#/components/schemas/catalog.ProfileComfyUI",
|
||||
"description": "ComfyUI is this profile's own workflow, checkpoint and nodes, over the model's: an fp8 workflow where the card has fp8, fp16 where it has not."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.ProfileComfyUI": {
|
||||
"type": "object",
|
||||
"description": "ProfileComfyUI is what a ComfyUI profile runs differently from the model: each field set replaces the model's.",
|
||||
"properties": {
|
||||
"workflow": {
|
||||
"type": "string"
|
||||
},
|
||||
"checkpoint": {
|
||||
"type": "string"
|
||||
},
|
||||
"nodes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/catalog.CustomNode"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
Reference in new issue
Block a user