Facility-aware placement: racks, PDUs, holds and power caps
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels, from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's). Placement spreads a model's replicas across racks and PDUs, keeps a multi-node replica within one domain when its profile says keep_within, places nothing in a domain held with `llmgw ctl facility hold` (optionally draining it), and refuses a shard that would take a domain past its power cap from facility.budgets, counting declared, derived and measured draw. Explain-placement names the domain, and keeps the rejections of a placed replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the domains; the console's Nodes view shows and sets location and holds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
0d3eb82b9b
42 files changed
+3510
-63
No files matched your search
@@ -99,6 +99,19 @@ var Reference = []Block{
|
||||
{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
|
||||
{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
|
||||
{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
|
||||
{Key: "margin", Type: "float64", Doc: "Margin is the share of every cap kept free, 0.1 for ten percent. Zero\nkeeps none, for caps already given derated.", Nested: ""},
|
||||
{Key: "gpu_watts", Type: "float64", Doc: "GPUWatts is a card's draw when it does not report its power limit:\nan agent without nvidia-smi's power fields, or a node that has not\nsent a heartbeat yet. Zero counts such a card as drawing nothing.", Nested: ""},
|
||||
{Key: "cpu_core_watts", Type: "float64", Doc: "CPUCoreWatts is the draw of a core reserved by a replica with no GPU\n(cpu_cores). Zero counts such replicas as drawing nothing.", Nested: ""},
|
||||
{Key: "node_watts", Type: "float64", Doc: "NodeWatts is what a node draws besides its cards and reserved cores —\nidle CPUs, memory, fans, disks — counted for every node in a capped\ndomain, busy or not.", Nested: ""},
|
||||
{Key: "measured_fresh", Type: "time.Duration", Doc: "MeasuredFresh is how old a node's power reading may be and still be\ncounted against its domains. Zero takes two minutes.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "FacilityBudget", Title: "FacilityBudget", Doc: "", Fields: []Field{
|
||||
{Key: "domain", Type: "string", Doc: "Domain is level:name, as the nodes' labels give it: rack:r12-07,\npdu:r12-07-a, row:12.", Nested: ""},
|
||||
{Key: "cap_watts", Type: "float64", Doc: "CapWatts is what the domain may draw. For a dual-fed rack it is what\none feed carries alone, derated as the facility derates it (usually\n80%), since that is what the rack has to survive.", Nested: ""},
|
||||
{Key: "other_watts", Type: "float64", Doc: "OtherWatts is drawn in the domain by what llmgw does not place on:\nswitches, storage, machines outside the cluster.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "GuardModel", Title: "GuardModel", Doc: "", Fields: []Field{
|
||||
{Key: "format", Type: "string", Doc: "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and\nthe categories), shieldgemma (Google's ShieldGemma: Yes or No), or\njson ({\"safe\": false, \"categories\": [\"...\"]}).", Nested: ""},
|
||||
{Key: "categories", Type: "[]string", Doc: "Categories the model is asked about, for shieldgemma — each a policy\nsentence — and the names reported for llamaguard3's S1, S2, ….", Nested: ""},
|
||||
@@ -240,6 +253,8 @@ var Reference = []Block{
|
||||
{Key: "cpu_cores", Type: "int", Doc: "CPUCores is the cores a replica of a profile with no GPU is given:\nreserved out of its node's CPUs by placement, as memory is, and what\nits engine computes on (llama.cpp's --threads).\n\nReserved because an engine takes every core it is told about, and two\non one node that each think they have all of them spin against each\nother: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second\nto under one each. Zero reserves nothing and leaves the thread count to\nthe node, which is every profile from before this.", Nested: ""},
|
||||
{Key: "min_disk_gb", Type: "int", Doc: "MinDiskGB is the free space a node needs before it can hold these\nweights.\n\nOnly checked when the node does not already have them: a node holding\nthe file needs no room for it, and requiring the space twice would keep\na model off the one node best placed to run it.\n\nDeclared rather than derived, because the size of a model that has never\nbeen fetched anywhere is not knowable from here — and a wrong guess\nfails late, halfway through a download, which is the failure this exists\nto prevent. Zero means unconstrained, and a node that reports no storage\nis not excluded: unknown is not full.", Nested: ""},
|
||||
{Key: "node_selector", Type: "map[string]string", Doc: "NodeSelector restricts this profile to nodes carrying every one of these\nlabels, with these values.\n\nA CPU profile otherwise fits everywhere, because there is no card to be\nfree — which is right for a 0.5B model and wrong for a 14B one. Nothing\nin the inventory says how much memory a node has, so the operator says\nwhich nodes are meant for a model instead of the scheduler inferring it.\nAgents have reported labels since the beginning; this is what reads them.", Nested: ""},
|
||||
{Key: "power_watts", Type: "float64", Doc: "PowerWatts is what one replica of this profile draws, in watts, split\nevenly over its nodes, counted against the power cap of every rack,\nPDU or other domain its nodes are in (facility.budgets in server.yaml).\nZero derives it: each card at the power limit it reports, each\nreserved core at facility.cpu_core_watts. Only matters where a domain\nhas a cap.", Nested: ""},
|
||||
{Key: "keep_within", Type: "string", Doc: "KeepWithin keeps a multi-node replica's nodes inside one rack, row,\nroom or site, read from their llmgw.io/rack (and so on) labels: for a\nfabric whose leaves do not reach across racks, or cables that do not.\nNodes without the label are passed over for such a profile. Empty is\nno constraint.", Nested: ""},
|
||||
{Key: "weight", Type: "float64", Doc: "Weight is the router's capacity weight. It should come from ctl bench\nrather than a guess (plan §6.2.1).", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "Priority orders profiles during placement; highest that fits wins.", Nested: ""},
|
||||
{Key: "startup_timeout", Type: "time.Duration", Doc: "StartupTimeout bounds model loading. A 70B model across two nodes takes\ntens of minutes, so this is per profile rather than global.", Nested: ""},
|
||||
@@ -337,6 +352,7 @@ var Reference = []Block{
|
||||
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
|
||||
{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
|
||||
{Key: "models", Type: "[]catalog.Model", Doc: "", Nested: ""},
|
||||
{Key: "facility", Type: "Facility", Doc: "Facility is the power caps of racks, PDUs and other domains, and how\na replica's draw is estimated against them. Where a node is comes from\nits llmgw.io/site, room, row, rack and pdu labels; holding a domain for\nmaintenance is `llmgw ctl facility hold`, not configuration.", Nested: "config.Facility"},
|
||||
{Key: "endpoints", Type: "[]Endpoint", Doc: "Endpoints is the phase-1 static replica list. It is only consulted when\nno model declares a profile, which is what makes a router-only smoke test\nagainst sim-engines possible without a control plane.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
|
||||
|
||||
Reference in new issue
Block a user