Release v0.10.5

Collects rc.1 to rc.14: the mirror page in tabs with a card per model, one line per record in every admin table, reserved CPU cores and required memory, capacity classes and preemption, tenant capacity quotas, sizing by measured throughput, and node sensor readings. Adds `llmgw upgrade` for the mirror host, which `ctl upgrade` does not reach, and says so when it ends. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Mark facility steps 1-5 deferred; node sensors collected
2026-09-30 22:26:19 +02:00 · 2026-09-30 20:25:50 +02:00 · 2026-09-30 19:49:08 +02:00 · 2026-09-30 19:25:31 +02:00 · 2026-09-30 18:41:43 +02:00 · 2026-09-30 18:23:08 +02:00
34 changed files with 2019 additions and 194 deletions
@@ -20,10 +20,21 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.

-## Unreleased
+## v0.10.5 — 2026-09-30

-Changes since v0.10.4, published as release candidates (v0.10.5-rc.N) until
-there are enough for v0.10.5.
+Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
+format check and has no binaries.
+
+After upgrading:
+
+- **Declare what CPU models need.** A CPU model added from now on must say its
+  memory; one stored before this still loads and is flagged in `ctl models`
+  until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
+  are then reserved per replica and become the engine's thread count, for
+  replicas started after the change: `llmgw ctl restart --model MODEL`.
+- **Upgrade the mirror on its own host**, now with one command:
+  `sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
+  mirror older than this). `llmgw ctl upgrade` says so at the end.

 ### CPU cores are reserved, and memory has to be declared

@@ -70,9 +81,12 @@ these rules:
  loaded, nor one spanning several nodes.
 - Idle on-demand replicas go first, then what lower classes grew into, then
  the same within the class, then lower classes' minimums; the longest unused
-  first within each.
+  first within each, and of equally idle ones the smallest.
 - The set is tried on a copy of the inventory first, at most three replicas,
-  and nothing is stopped unless it makes the room.
+  and nothing is stopped unless it makes the room. Of a set that does, each
+  replica whose stopping turns out not to be needed is left running, so an
+  idle replica tried first on the wrong node is not stopped with the one that
+  made the room.

 A replica stopped this way is drained and given 30 seconds to finish what it
 is serving. Its model is held back for up to five minutes, or until the model
@@ -106,6 +120,57 @@ two is refused. Changes take effect within half a minute. Metrics:
 `llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
 by scope and resource; alert: CapacityAtLimit.

+### Sized by what a replica was measured to serve
+
+What `llmgw ctl bench MODEL --save` measured — the most requests at once one
+replica served while meeting its SLO, and its throughput there — now drives
+decisions rather than only being printed:
+
+- **Scaling up** when requests per replica pass what was measured, not only
+  when the engines' queues fill: an engine takes requests into its slots long
+  after latency has gone.
+- **Scaling down** only when the replicas left would stay under 70% of what
+  they were measured to hold, so a replica taken back does not put the rest
+  straight over.
+- **Choosing a profile.** Among profiles of equal priority, the one measured
+  to serve the most per GPU — or per core — is tried first. Priority set by an
+  operator still wins.
+- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
+  replicas that takes, how many run, and whether the rest fit, with the reason
+  when they do not; `--apply` sets it if they do. Headroom reports what one
+  more replica would serve.
+
+A model never measured behaves exactly as before. Measurements are read from
+the store every half minute.
+
+### Upgrading a machine the control plane does not reach
+
+`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
+the mirror's host above all, whose service runs unprivileged and cannot replace
+its own binary as an agent does. The release's checksums are checked, the old
+binary kept, the unit restarted and watched until it stays up.
+
+### Nodes report their own heat and power
+
+The first step of facility-aware placement (docs/facility-aware-placement.md),
+observe-only: nothing is decided on these yet. Every agent reads, at most every
+ten seconds, and sends with its heartbeat:
+
+- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
+  it, power draw and the enforced limit, and what is holding the clocks down —
+  power cap, thermal slowdown, power brake. A GPU's temperature was read once,
+  at registration; it is now current.
+- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
+  the package temperature, and the board's inlet air sensor where it has one.
+- **CPU power** from RAPL's energy counters.
+
+A sensor a machine does not have is not reported, never shown as zero. New
+metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
+`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
+`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
+`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
+table of temperature, power against its limit, and what is throttling it.
+
 ### The mirror page in tabs

 /admin/mirror was one long page: what the mirror holds, the Hugging Face
@@ -90,7 +90,7 @@ func (x ShardStatus_State) Number() protoreflect.EnumNumber {

 // Deprecated: Use ShardStatus_State.Descriptor instead.
 func (ShardStatus_State) EnumDescriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{19, 0}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{20, 0}
 }

 // AgentMessage is anything a node tells the control plane.
@@ -1415,8 +1415,15 @@ type NodeLoad struct {
 	MemoryUsedBytes      int64        `protobuf:"varint,6,opt,name=memory_used_bytes,json=memoryUsedBytes,proto3" json:"memory_used_bytes,omitempty"`
 	MemoryAvailableBytes int64        `protobuf:"varint,7,opt,name=memory_available_bytes,json=memoryAvailableBytes,proto3" json:"memory_available_bytes,omitempty"`
 	Shards               []*ShardLoad `protobuf:"bytes,8,rep,name=shards,proto3" json:"shards,omitempty"`
-	unknownFields        protoimpl.UnknownFields
-	sizeCache            protoimpl.SizeCache
+	// What the machine's own sensors say; zero is not measured. cpu_temp_c is
+	// the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
+	// inlet_temp_c the board's inlet or ambient sensor where it has one.
+	CpuTempC      float64      `protobuf:"fixed64,9,opt,name=cpu_temp_c,json=cpuTempC,proto3" json:"cpu_temp_c,omitempty"`
+	CpuPowerW     float64      `protobuf:"fixed64,10,opt,name=cpu_power_w,json=cpuPowerW,proto3" json:"cpu_power_w,omitempty"`
+	InletTempC    float64      `protobuf:"fixed64,11,opt,name=inlet_temp_c,json=inletTempC,proto3" json:"inlet_temp_c,omitempty"`
+	Gpus          []*GPUSensor `protobuf:"bytes,12,rep,name=gpus,proto3" json:"gpus,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
 }

 func (x *NodeLoad) Reset() {
@@ -1505,6 +1512,121 @@ func (x *NodeLoad) GetShards() []*ShardLoad {
 	return nil
 }

+func (x *NodeLoad) GetCpuTempC() float64 {
+	if x != nil {
+		return x.CpuTempC
+	}
+	return 0
+}
+
+func (x *NodeLoad) GetCpuPowerW() float64 {
+	if x != nil {
+		return x.CpuPowerW
+	}
+	return 0
+}
+
+func (x *NodeLoad) GetInletTempC() float64 {
+	if x != nil {
+		return x.InletTempC
+	}
+	return 0
+}
+
+func (x *NodeLoad) GetGpus() []*GPUSensor {
+	if x != nil {
+		return x.Gpus
+	}
+	return nil
+}
+
+// GPUSensor is one card's heat, power, and what is holding it back.
+type GPUSensor struct {
+	state       protoimpl.MessageState `protogen:"open.v1"`
+	Index       int32                  `protobuf:"varint,1,opt,name=index,proto3" json:"index,omitempty"`
+	TempC       float64                `protobuf:"fixed64,2,opt,name=temp_c,json=tempC,proto3" json:"temp_c,omitempty"`
+	MemoryTempC float64                `protobuf:"fixed64,3,opt,name=memory_temp_c,json=memoryTempC,proto3" json:"memory_temp_c,omitempty"`
+	PowerW      float64                `protobuf:"fixed64,4,opt,name=power_w,json=powerW,proto3" json:"power_w,omitempty"`
+	PowerLimitW float64                `protobuf:"fixed64,5,opt,name=power_limit_w,json=powerLimitW,proto3" json:"power_limit_w,omitempty"`
+	// throttle names what is holding the clocks down now: sw_power_cap,
+	// hw_slowdown, sw_thermal, hw_thermal, power_brake.
+	Throttle      []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
+	unknownFields protoimpl.UnknownFields
+	sizeCache     protoimpl.SizeCache
+}
+
+func (x *GPUSensor) Reset() {
+	*x = GPUSensor{}
+	mi := &file_llmgw_v1_agent_proto_msgTypes[10]
+	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+	ms.StoreMessageInfo(mi)
+}
+
+func (x *GPUSensor) String() string {
+	return protoimpl.X.MessageStringOf(x)
+}
+
+func (*GPUSensor) ProtoMessage() {}
+
+func (x *GPUSensor) ProtoReflect() protoreflect.Message {
+	mi := &file_llmgw_v1_agent_proto_msgTypes[10]
+	if x != nil {
+		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
+		if ms.LoadMessageInfo() == nil {
+			ms.StoreMessageInfo(mi)
+		}
+		return ms
+	}
+	return mi.MessageOf(x)
+}
+
+// Deprecated: Use GPUSensor.ProtoReflect.Descriptor instead.
+func (*GPUSensor) Descriptor() ([]byte, []int) {
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{10}
+}
+
+func (x *GPUSensor) GetIndex() int32 {
+	if x != nil {
+		return x.Index
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetTempC() float64 {
+	if x != nil {
+		return x.TempC
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetMemoryTempC() float64 {
+	if x != nil {
+		return x.MemoryTempC
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetPowerW() float64 {
+	if x != nil {
+		return x.PowerW
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetPowerLimitW() float64 {
+	if x != nil {
+		return x.PowerLimitW
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetThrottle() []string {
+	if x != nil {
+		return x.Throttle
+	}
+	return nil
+}
+
 // ShardLoad is one engine process group's use.
 type ShardLoad struct {
 	state   protoimpl.MessageState `protogen:"open.v1"`
@@ -1520,7 +1642,7 @@ type ShardLoad struct {

 func (x *ShardLoad) Reset() {
 	*x = ShardLoad{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[10]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[11]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1532,7 +1654,7 @@ func (x *ShardLoad) String() string {
 func (*ShardLoad) ProtoMessage() {}

 func (x *ShardLoad) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[10]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[11]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1545,7 +1667,7 @@ func (x *ShardLoad) ProtoReflect() protoreflect.Message {

 // Deprecated: Use ShardLoad.ProtoReflect.Descriptor instead.
 func (*ShardLoad) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{10}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{11}
 }

 func (x *ShardLoad) GetShardId() string {
@@ -1613,7 +1735,7 @@ type IBPort struct {

 func (x *IBPort) Reset() {
 	*x = IBPort{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[11]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[12]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1625,7 +1747,7 @@ func (x *IBPort) String() string {
 func (*IBPort) ProtoMessage() {}

 func (x *IBPort) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[11]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[12]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1638,7 +1760,7 @@ func (x *IBPort) ProtoReflect() protoreflect.Message {

 // Deprecated: Use IBPort.ProtoReflect.Descriptor instead.
 func (*IBPort) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{11}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{12}
 }

 func (x *IBPort) GetHca() string {
@@ -1781,7 +1903,7 @@ type SubnetManager struct {

 func (x *SubnetManager) Reset() {
 	*x = SubnetManager{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[12]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[13]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1793,7 +1915,7 @@ func (x *SubnetManager) String() string {
 func (*SubnetManager) ProtoMessage() {}

 func (x *SubnetManager) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[12]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[13]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1806,7 +1928,7 @@ func (x *SubnetManager) ProtoReflect() protoreflect.Message {

 // Deprecated: Use SubnetManager.ProtoReflect.Descriptor instead.
 func (*SubnetManager) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{12}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{13}
 }

 func (x *SubnetManager) GetGuid() string {
@@ -1849,7 +1971,7 @@ type IMEX struct {

 func (x *IMEX) Reset() {
 	*x = IMEX{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[13]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[14]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1861,7 +1983,7 @@ func (x *IMEX) String() string {
 func (*IMEX) ProtoMessage() {}

 func (x *IMEX) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[13]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[14]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1874,7 +1996,7 @@ func (x *IMEX) ProtoReflect() protoreflect.Message {

 // Deprecated: Use IMEX.ProtoReflect.Descriptor instead.
 func (*IMEX) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{13}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{14}
 }

 func (x *IMEX) GetState() string {
@@ -1906,7 +2028,7 @@ type GPUFabricState struct {

 func (x *GPUFabricState) Reset() {
 	*x = GPUFabricState{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[14]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[15]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -1918,7 +2040,7 @@ func (x *GPUFabricState) String() string {
 func (*GPUFabricState) ProtoMessage() {}

 func (x *GPUFabricState) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[14]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[15]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -1931,7 +2053,7 @@ func (x *GPUFabricState) ProtoReflect() protoreflect.Message {

 // Deprecated: Use GPUFabricState.ProtoReflect.Descriptor instead.
 func (*GPUFabricState) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{14}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{15}
 }

 func (x *GPUFabricState) GetIndex() int32 {
@@ -2038,7 +2160,7 @@ type Heartbeat struct {

 func (x *Heartbeat) Reset() {
 	*x = Heartbeat{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[15]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[16]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2050,7 +2172,7 @@ func (x *Heartbeat) String() string {
 func (*Heartbeat) ProtoMessage() {}

 func (x *Heartbeat) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[15]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[16]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2063,7 +2185,7 @@ func (x *Heartbeat) ProtoReflect() protoreflect.Message {

 // Deprecated: Use Heartbeat.ProtoReflect.Descriptor instead.
 func (*Heartbeat) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{15}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{16}
 }

 func (x *Heartbeat) GetNodeId() string {
@@ -2211,7 +2333,7 @@ type Provision struct {

 func (x *Provision) Reset() {
 	*x = Provision{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[16]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[17]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2223,7 +2345,7 @@ func (x *Provision) String() string {
 func (*Provision) ProtoMessage() {}

 func (x *Provision) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[16]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[17]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2236,7 +2358,7 @@ func (x *Provision) ProtoReflect() protoreflect.Message {

 // Deprecated: Use Provision.ProtoReflect.Descriptor instead.
 func (*Provision) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{16}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{17}
 }

 func (x *Provision) GetEngines() []*EngineWant {
@@ -2257,7 +2379,7 @@ type EngineWant struct {

 func (x *EngineWant) Reset() {
 	*x = EngineWant{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[17]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[18]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2269,7 +2391,7 @@ func (x *EngineWant) String() string {
 func (*EngineWant) ProtoMessage() {}

 func (x *EngineWant) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[17]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[18]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2282,7 +2404,7 @@ func (x *EngineWant) ProtoReflect() protoreflect.Message {

 // Deprecated: Use EngineWant.ProtoReflect.Descriptor instead.
 func (*EngineWant) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{17}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{18}
 }

 func (x *EngineWant) GetEngine() string {
@@ -2315,7 +2437,7 @@ type ProvisionStatus struct {

 func (x *ProvisionStatus) Reset() {
 	*x = ProvisionStatus{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[18]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[19]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2327,7 +2449,7 @@ func (x *ProvisionStatus) String() string {
 func (*ProvisionStatus) ProtoMessage() {}

 func (x *ProvisionStatus) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[18]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[19]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2340,7 +2462,7 @@ func (x *ProvisionStatus) ProtoReflect() protoreflect.Message {

 // Deprecated: Use ProvisionStatus.ProtoReflect.Descriptor instead.
 func (*ProvisionStatus) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{18}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{19}
 }

 func (x *ProvisionStatus) GetEngine() string {
@@ -2404,7 +2526,7 @@ type ShardStatus struct {

 func (x *ShardStatus) Reset() {
 	*x = ShardStatus{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[19]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[20]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2416,7 +2538,7 @@ func (x *ShardStatus) String() string {
 func (*ShardStatus) ProtoMessage() {}

 func (x *ShardStatus) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[19]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[20]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2429,7 +2551,7 @@ func (x *ShardStatus) ProtoReflect() protoreflect.Message {

 // Deprecated: Use ShardStatus.ProtoReflect.Descriptor instead.
 func (*ShardStatus) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{19}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{20}
 }

 func (x *ShardStatus) GetNodeId() string {
@@ -2503,7 +2625,7 @@ type FetchProgress struct {

 func (x *FetchProgress) Reset() {
 	*x = FetchProgress{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[20]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[21]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2515,7 +2637,7 @@ func (x *FetchProgress) String() string {
 func (*FetchProgress) ProtoMessage() {}

 func (x *FetchProgress) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[20]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[21]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2528,7 +2650,7 @@ func (x *FetchProgress) ProtoReflect() protoreflect.Message {

 // Deprecated: Use FetchProgress.ProtoReflect.Descriptor instead.
 func (*FetchProgress) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{20}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{21}
 }

 func (x *FetchProgress) GetSource() string {
@@ -2591,7 +2713,7 @@ type StartShard struct {

 func (x *StartShard) Reset() {
 	*x = StartShard{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[21]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[22]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2603,7 +2725,7 @@ func (x *StartShard) String() string {
 func (*StartShard) ProtoMessage() {}

 func (x *StartShard) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[21]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[22]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2616,7 +2738,7 @@ func (x *StartShard) ProtoReflect() protoreflect.Message {

 // Deprecated: Use StartShard.ProtoReflect.Descriptor instead.
 func (*StartShard) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{21}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{22}
 }

 func (x *StartShard) GetReplicaId() string {
@@ -2757,7 +2879,7 @@ type EngineProfile struct {

 func (x *EngineProfile) Reset() {
 	*x = EngineProfile{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[22]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[23]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2769,7 +2891,7 @@ func (x *EngineProfile) String() string {
 func (*EngineProfile) ProtoMessage() {}

 func (x *EngineProfile) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[22]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[23]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2782,7 +2904,7 @@ func (x *EngineProfile) ProtoReflect() protoreflect.Message {

 // Deprecated: Use EngineProfile.ProtoReflect.Descriptor instead.
 func (*EngineProfile) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{22}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{23}
 }

 func (x *EngineProfile) GetName() string {
@@ -2878,7 +3000,7 @@ type RemoveWeights struct {

 func (x *RemoveWeights) Reset() {
 	*x = RemoveWeights{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[23]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[24]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2890,7 +3012,7 @@ func (x *RemoveWeights) String() string {
 func (*RemoveWeights) ProtoMessage() {}

 func (x *RemoveWeights) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[23]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[24]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2903,7 +3025,7 @@ func (x *RemoveWeights) ProtoReflect() protoreflect.Message {

 // Deprecated: Use RemoveWeights.ProtoReflect.Descriptor instead.
 func (*RemoveWeights) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{23}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{24}
 }

 func (x *RemoveWeights) GetPaths() []string {
@@ -2923,7 +3045,7 @@ type StopShard struct {

 func (x *StopShard) Reset() {
 	*x = StopShard{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[24]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[25]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2935,7 +3057,7 @@ func (x *StopShard) String() string {
 func (*StopShard) ProtoMessage() {}

 func (x *StopShard) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[24]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[25]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2948,7 +3070,7 @@ func (x *StopShard) ProtoReflect() protoreflect.Message {

 // Deprecated: Use StopShard.ProtoReflect.Descriptor instead.
 func (*StopShard) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{24}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{25}
 }

 func (x *StopShard) GetShardId() string {
@@ -2968,7 +3090,7 @@ type DrainShard struct {

 func (x *DrainShard) Reset() {
 	*x = DrainShard{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[25]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[26]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -2980,7 +3102,7 @@ func (x *DrainShard) String() string {
 func (*DrainShard) ProtoMessage() {}

 func (x *DrainShard) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[25]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[26]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -2993,7 +3115,7 @@ func (x *DrainShard) ProtoReflect() protoreflect.Message {

 // Deprecated: Use DrainShard.ProtoReflect.Descriptor instead.
 func (*DrainShard) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{25}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{26}
 }

 func (x *DrainShard) GetShardId() string {
@@ -3015,7 +3137,7 @@ type Ack struct {

 func (x *Ack) Reset() {
 	*x = Ack{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[26]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[27]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3027,7 +3149,7 @@ func (x *Ack) String() string {
 func (*Ack) ProtoMessage() {}

 func (x *Ack) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[26]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[27]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3040,7 +3162,7 @@ func (x *Ack) ProtoReflect() protoreflect.Message {

 // Deprecated: Use Ack.ProtoReflect.Descriptor instead.
 func (*Ack) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{26}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{27}
 }

 func (x *Ack) GetMessageId() string {
@@ -3084,7 +3206,7 @@ type LogsRequest struct {

 func (x *LogsRequest) Reset() {
 	*x = LogsRequest{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[27]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[28]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3096,7 +3218,7 @@ func (x *LogsRequest) String() string {
 func (*LogsRequest) ProtoMessage() {}

 func (x *LogsRequest) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[27]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[28]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3109,7 +3231,7 @@ func (x *LogsRequest) ProtoReflect() protoreflect.Message {

 // Deprecated: Use LogsRequest.ProtoReflect.Descriptor instead.
 func (*LogsRequest) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{27}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{28}
 }

 func (x *LogsRequest) GetRequestId() string {
@@ -3149,7 +3271,7 @@ type LogsReply struct {

 func (x *LogsReply) Reset() {
 	*x = LogsReply{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[28]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[29]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3161,7 +3283,7 @@ func (x *LogsReply) String() string {
 func (*LogsReply) ProtoMessage() {}

 func (x *LogsReply) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[28]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[29]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3174,7 +3296,7 @@ func (x *LogsReply) ProtoReflect() protoreflect.Message {

 // Deprecated: Use LogsReply.ProtoReflect.Descriptor instead.
 func (*LogsReply) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{28}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{29}
 }

 func (x *LogsReply) GetRequestId() string {
@@ -3220,7 +3342,7 @@ type FabricScanRequest struct {

 func (x *FabricScanRequest) Reset() {
 	*x = FabricScanRequest{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[29]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[30]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3232,7 +3354,7 @@ func (x *FabricScanRequest) String() string {
 func (*FabricScanRequest) ProtoMessage() {}

 func (x *FabricScanRequest) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[29]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[30]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3245,7 +3367,7 @@ func (x *FabricScanRequest) ProtoReflect() protoreflect.Message {

 // Deprecated: Use FabricScanRequest.ProtoReflect.Descriptor instead.
 func (*FabricScanRequest) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{29}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{30}
 }

 func (x *FabricScanRequest) GetRequestId() string {
@@ -3284,7 +3406,7 @@ type FabricScanReply struct {

 func (x *FabricScanReply) Reset() {
 	*x = FabricScanReply{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[30]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[31]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3296,7 +3418,7 @@ func (x *FabricScanReply) String() string {
 func (*FabricScanReply) ProtoMessage() {}

 func (x *FabricScanReply) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[30]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[31]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3309,7 +3431,7 @@ func (x *FabricScanReply) ProtoReflect() protoreflect.Message {

 // Deprecated: Use FabricScanReply.ProtoReflect.Descriptor instead.
 func (*FabricScanReply) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{30}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{31}
 }

 func (x *FabricScanReply) GetRequestId() string {
@@ -3365,7 +3487,7 @@ type FabricTestRequest struct {

 func (x *FabricTestRequest) Reset() {
 	*x = FabricTestRequest{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[31]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[32]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3377,7 +3499,7 @@ func (x *FabricTestRequest) String() string {
 func (*FabricTestRequest) ProtoMessage() {}

 func (x *FabricTestRequest) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[31]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[32]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3390,7 +3512,7 @@ func (x *FabricTestRequest) ProtoReflect() protoreflect.Message {

 // Deprecated: Use FabricTestRequest.ProtoReflect.Descriptor instead.
 func (*FabricTestRequest) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{31}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{32}
 }

 func (x *FabricTestRequest) GetRequestId() string {
@@ -3459,7 +3581,7 @@ type FabricTestReply struct {

 func (x *FabricTestReply) Reset() {
 	*x = FabricTestReply{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[32]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3471,7 +3593,7 @@ func (x *FabricTestReply) String() string {
 func (*FabricTestReply) ProtoMessage() {}

 func (x *FabricTestReply) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[32]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3484,7 +3606,7 @@ func (x *FabricTestReply) ProtoReflect() protoreflect.Message {

 // Deprecated: Use FabricTestReply.ProtoReflect.Descriptor instead.
 func (*FabricTestReply) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{32}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{33}
 }

 func (x *FabricTestReply) GetRequestId() string {
@@ -3544,7 +3666,7 @@ type UpgradeAgent struct {

 func (x *UpgradeAgent) Reset() {
 	*x = UpgradeAgent{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3556,7 +3678,7 @@ func (x *UpgradeAgent) String() string {
 func (*UpgradeAgent) ProtoMessage() {}

 func (x *UpgradeAgent) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[33]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3569,7 +3691,7 @@ func (x *UpgradeAgent) ProtoReflect() protoreflect.Message {

 // Deprecated: Use UpgradeAgent.ProtoReflect.Descriptor instead.
 func (*UpgradeAgent) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{33}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{34}
 }

 func (x *UpgradeAgent) GetRequestId() string {
@@ -3606,7 +3728,7 @@ type UpgradeReply struct {

 func (x *UpgradeReply) Reset() {
 	*x = UpgradeReply{}
-	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[35]
 	ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 	ms.StoreMessageInfo(mi)
 }
@@ -3618,7 +3740,7 @@ func (x *UpgradeReply) String() string {
 func (*UpgradeReply) ProtoMessage() {}

 func (x *UpgradeReply) ProtoReflect() protoreflect.Message {
-	mi := &file_llmgw_v1_agent_proto_msgTypes[34]
+	mi := &file_llmgw_v1_agent_proto_msgTypes[35]
 	if x != nil {
 		ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
 		if ms.LoadMessageInfo() == nil {
@@ -3631,7 +3753,7 @@ func (x *UpgradeReply) ProtoReflect() protoreflect.Message {

 // Deprecated: Use UpgradeReply.ProtoReflect.Descriptor instead.
 func (*UpgradeReply) Descriptor() ([]byte, []int) {
-	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{34}
+	return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{35}
 }

 func (x *UpgradeReply) GetRequestId() string {
@@ -3786,7 +3908,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
 	"\n" +
 	"size_bytes\x18\x02 \x01(\x03R\tsizeBytes\x12\x16\n" +
-	"\x06shared\x18\x03 \x01(\bR\x06shared\"\x9b\x02\n" +
+	"\x06shared\x18\x03 \x01(\bR\x06shared\"\xa4\x03\n" +
 	"\bNodeLoad\x12\x1f\n" +
 	"\vcpu_percent\x18\x01 \x01(\x01R\n" +
 	"cpuPercent\x12\x1b\n" +
@@ -3796,7 +3918,21 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\x06load15\x18\x05 \x01(\x01R\x06load15\x12*\n" +
 	"\x11memory_used_bytes\x18\x06 \x01(\x03R\x0fmemoryUsedBytes\x124\n" +
 	"\x16memory_available_bytes\x18\a \x01(\x03R\x14memoryAvailableBytes\x12+\n" +
-	"\x06shards\x18\b \x03(\v2\x13.llmgw.v1.ShardLoadR\x06shards\"m\n" +
+	"\x06shards\x18\b \x03(\v2\x13.llmgw.v1.ShardLoadR\x06shards\x12\x1c\n" +
+	"\n" +
+	"cpu_temp_c\x18\t \x01(\x01R\bcpuTempC\x12\x1e\n" +
+	"\vcpu_power_w\x18\n" +
+	" \x01(\x01R\tcpuPowerW\x12 \n" +
+	"\finlet_temp_c\x18\v \x01(\x01R\n" +
+	"inletTempC\x12'\n" +
+	"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xb5\x01\n" +
+	"\tGPUSensor\x12\x14\n" +
+	"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
+	"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
+	"\rmemory_temp_c\x18\x03 \x01(\x01R\vmemoryTempC\x12\x17\n" +
+	"\apower_w\x18\x04 \x01(\x01R\x06powerW\x12\"\n" +
+	"\rpower_limit_w\x18\x05 \x01(\x01R\vpowerLimitW\x12\x1a\n" +
+	"\bthrottle\x18\x06 \x03(\tR\bthrottle\"m\n" +
 	"\tShardLoad\x12\x19\n" +
 	"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
 	"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
@@ -4026,7 +4162,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
 }

 var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
-var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 41)
+var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 42)
 var file_llmgw_v1_agent_proto_goTypes = []any{
 	(ShardStatus_State)(0),    // 0: llmgw.v1.ShardStatus.State
 	(*AgentMessage)(nil),      // 1: llmgw.v1.AgentMessage
@@ -4039,91 +4175,93 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
 	(*FabricAttachment)(nil),  // 8: llmgw.v1.FabricAttachment
 	(*LocalWeights)(nil),      // 9: llmgw.v1.LocalWeights
 	(*NodeLoad)(nil),          // 10: llmgw.v1.NodeLoad
-	(*ShardLoad)(nil),         // 11: llmgw.v1.ShardLoad
-	(*IBPort)(nil),            // 12: llmgw.v1.IBPort
-	(*SubnetManager)(nil),     // 13: llmgw.v1.SubnetManager
-	(*IMEX)(nil),              // 14: llmgw.v1.IMEX
-	(*GPUFabricState)(nil),    // 15: llmgw.v1.GPUFabricState
-	(*Heartbeat)(nil),         // 16: llmgw.v1.Heartbeat
-	(*Provision)(nil),         // 17: llmgw.v1.Provision
-	(*EngineWant)(nil),        // 18: llmgw.v1.EngineWant
-	(*ProvisionStatus)(nil),   // 19: llmgw.v1.ProvisionStatus
-	(*ShardStatus)(nil),       // 20: llmgw.v1.ShardStatus
-	(*FetchProgress)(nil),     // 21: llmgw.v1.FetchProgress
-	(*StartShard)(nil),        // 22: llmgw.v1.StartShard
-	(*EngineProfile)(nil),     // 23: llmgw.v1.EngineProfile
-	(*RemoveWeights)(nil),     // 24: llmgw.v1.RemoveWeights
-	(*StopShard)(nil),         // 25: llmgw.v1.StopShard
-	(*DrainShard)(nil),        // 26: llmgw.v1.DrainShard
-	(*Ack)(nil),               // 27: llmgw.v1.Ack
-	(*LogsRequest)(nil),       // 28: llmgw.v1.LogsRequest
-	(*LogsReply)(nil),         // 29: llmgw.v1.LogsReply
-	(*FabricScanRequest)(nil), // 30: llmgw.v1.FabricScanRequest
-	(*FabricScanReply)(nil),   // 31: llmgw.v1.FabricScanReply
-	(*FabricTestRequest)(nil), // 32: llmgw.v1.FabricTestRequest
-	(*FabricTestReply)(nil),   // 33: llmgw.v1.FabricTestReply
-	(*UpgradeAgent)(nil),      // 34: llmgw.v1.UpgradeAgent
-	(*UpgradeReply)(nil),      // 35: llmgw.v1.UpgradeReply
-	nil,                       // 36: llmgw.v1.Register.LabelsEntry
-	nil,                       // 37: llmgw.v1.Register.EngineVersionsEntry
-	nil,                       // 38: llmgw.v1.GPU.NicPathsEntry
-	nil,                       // 39: llmgw.v1.IBPort.CountersEntry
-	nil,                       // 40: llmgw.v1.Heartbeat.EngineVersionsEntry
-	nil,                       // 41: llmgw.v1.EngineProfile.EnvEntry
+	(*GPUSensor)(nil),         // 11: llmgw.v1.GPUSensor
+	(*ShardLoad)(nil),         // 12: llmgw.v1.ShardLoad
+	(*IBPort)(nil),            // 13: llmgw.v1.IBPort
+	(*SubnetManager)(nil),     // 14: llmgw.v1.SubnetManager
+	(*IMEX)(nil),              // 15: llmgw.v1.IMEX
+	(*GPUFabricState)(nil),    // 16: llmgw.v1.GPUFabricState
+	(*Heartbeat)(nil),         // 17: llmgw.v1.Heartbeat
+	(*Provision)(nil),         // 18: llmgw.v1.Provision
+	(*EngineWant)(nil),        // 19: llmgw.v1.EngineWant
+	(*ProvisionStatus)(nil),   // 20: llmgw.v1.ProvisionStatus
+	(*ShardStatus)(nil),       // 21: llmgw.v1.ShardStatus
+	(*FetchProgress)(nil),     // 22: llmgw.v1.FetchProgress
+	(*StartShard)(nil),        // 23: llmgw.v1.StartShard
+	(*EngineProfile)(nil),     // 24: llmgw.v1.EngineProfile
+	(*RemoveWeights)(nil),     // 25: llmgw.v1.RemoveWeights
+	(*StopShard)(nil),         // 26: llmgw.v1.StopShard
+	(*DrainShard)(nil),        // 27: llmgw.v1.DrainShard
+	(*Ack)(nil),               // 28: llmgw.v1.Ack
+	(*LogsRequest)(nil),       // 29: llmgw.v1.LogsRequest
+	(*LogsReply)(nil),         // 30: llmgw.v1.LogsReply
+	(*FabricScanRequest)(nil), // 31: llmgw.v1.FabricScanRequest
+	(*FabricScanReply)(nil),   // 32: llmgw.v1.FabricScanReply
+	(*FabricTestRequest)(nil), // 33: llmgw.v1.FabricTestRequest
+	(*FabricTestReply)(nil),   // 34: llmgw.v1.FabricTestReply
+	(*UpgradeAgent)(nil),      // 35: llmgw.v1.UpgradeAgent
+	(*UpgradeReply)(nil),      // 36: llmgw.v1.UpgradeReply
+	nil,                       // 37: llmgw.v1.Register.LabelsEntry
+	nil,                       // 38: llmgw.v1.Register.EngineVersionsEntry
+	nil,                       // 39: llmgw.v1.GPU.NicPathsEntry
+	nil,                       // 40: llmgw.v1.IBPort.CountersEntry
+	nil,                       // 41: llmgw.v1.Heartbeat.EngineVersionsEntry
+	nil,                       // 42: llmgw.v1.EngineProfile.EnvEntry
 }
 var file_llmgw_v1_agent_proto_depIdxs = []int32{
 	3,  // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
-	16, // 1: llmgw.v1.AgentMessage.heartbeat:type_name -> llmgw.v1.Heartbeat
-	20, // 2: llmgw.v1.AgentMessage.shard:type_name -> llmgw.v1.ShardStatus
-	29, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
-	31, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
-	33, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
-	35, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
-	22, // 7: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
-	25, // 8: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
-	26, // 9: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
-	27, // 10: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
-	24, // 11: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
-	17, // 12: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
-	28, // 13: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
-	30, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
-	32, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
-	34, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
-	36, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
+	17, // 1: llmgw.v1.AgentMessage.heartbeat:type_name -> llmgw.v1.Heartbeat
+	21, // 2: llmgw.v1.AgentMessage.shard:type_name -> llmgw.v1.ShardStatus
+	30, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
+	32, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
+	34, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
+	36, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
+	23, // 7: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
+	26, // 8: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
+	27, // 9: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
+	28, // 10: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
+	25, // 11: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
+	18, // 12: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
+	29, // 13: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
+	31, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
+	33, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
+	35, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
+	37, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
 	7,  // 18: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
 	8,  // 19: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
-	37, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
+	38, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
 	6,  // 21: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
 	9,  // 22: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
 	5,  // 23: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
 	4,  // 24: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
-	14, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
+	15, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
 	0,  // 26: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
-	38, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
-	11, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
-	39, // 29: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
-	13, // 30: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
-	5,  // 31: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
-	9,  // 32: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
-	40, // 33: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
-	19, // 34: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
-	10, // 35: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
-	12, // 36: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
-	8,  // 37: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
-	15, // 38: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
-	14, // 39: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
-	18, // 40: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
-	0,  // 41: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
-	21, // 42: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
-	23, // 43: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
-	41, // 44: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
-	1,  // 45: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
-	2,  // 46: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
-	46, // [46:47] is the sub-list for method output_type
-	45, // [45:46] is the sub-list for method input_type
-	45, // [45:45] is the sub-list for extension type_name
-	45, // [45:45] is the sub-list for extension extendee
-	0,  // [0:45] is the sub-list for field type_name
+	39, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
+	12, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
+	11, // 29: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
+	40, // 30: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
+	14, // 31: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
+	5,  // 32: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
+	9,  // 33: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
+	41, // 34: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
+	20, // 35: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
+	10, // 36: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
+	13, // 37: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
+	8,  // 38: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
+	16, // 39: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
+	15, // 40: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
+	19, // 41: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
+	0,  // 42: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
+	22, // 43: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
+	24, // 44: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
+	42, // 45: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
+	1,  // 46: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
+	2,  // 47: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
+	47, // [47:48] is the sub-list for method output_type
+	46, // [46:47] is the sub-list for method input_type
+	46, // [46:46] is the sub-list for extension type_name
+	46, // [46:46] is the sub-list for extension extendee
+	0,  // [0:46] is the sub-list for field type_name
 }

 func init() { file_llmgw_v1_agent_proto_init() }
@@ -4158,7 +4296,7 @@ func file_llmgw_v1_agent_proto_init() {
 			GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
 			RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
 			NumEnums:      1,
-			NumMessages:   41,
+			NumMessages:   42,
 			NumExtensions: 0,
 			NumServices:   1,
 		},
@@ -285,6 +285,25 @@ message NodeLoad {
  int64 memory_used_bytes = 6;
  int64 memory_available_bytes = 7;
  repeated ShardLoad shards = 8;
+  // What the machine's own sensors say; zero is not measured. cpu_temp_c is
+  // the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
+  // inlet_temp_c the board's inlet or ambient sensor where it has one.
+  double cpu_temp_c = 9;
+  double cpu_power_w = 10;
+  double inlet_temp_c = 11;
+  repeated GPUSensor gpus = 12;
+}
+
+// GPUSensor is one card's heat, power, and what is holding it back.
+message GPUSensor {
+  int32 index = 1;
+  double temp_c = 2;
+  double memory_temp_c = 3;
+  double power_w = 4;
+  double power_limit_w = 5;
+  // throttle names what is holding the clocks down now: sw_power_cap,
+  // hw_slowdown, sw_thermal, hw_thermal, power_brake.
+  repeated string throttle = 6;
 }

 // ShardLoad is one engine process group's use.
@@ -0,0 +1,303 @@
+# Facility-aware placement: rack power, temperature and cooling
+
+**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
+deferred** until the rack, PDU and power information of a real site is at
+hand — without it they would be designed against guesses.
+This is the design for step 5 of resource allocation, after reserved CPU cores
+and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
+and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
+what follows, only the node sensors (rollout step 0) exist in the code: they
+are read, published and shown, and change no decision.
+
+## Why
+
+In a large installation the scarce thing is often not a GPU but the room it
+sits in. Three failures are invisible to placement today:
+
+- **A rack goes over its power budget.** Every node in it reports healthy;
+  placement sees free cards and fills them; the breaker, or the facility's
+  power cap, decides the rest. A rack with A and B feeds is worse: it runs
+  fine until one feed fails and the other cannot carry the load.
+- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
+  replica placed in a rack whose inlet air is too warm serves fewer tokens per
+  second than its measured capacity says, and nothing connects the two.
+- **Correlated failure.** Two replicas of a model on two nodes in the same rack
+  or on the same PDU are one failure away from zero. Placement spreads replicas
+  across nodes, not across racks.
+
+The goal is for placement to treat facility limits the way it already treats
+memory and cores: reserve against a budget, refuse what does not fit, say why,
+and never let missing data stop the cluster.
+
+## What it builds on
+
+| Already in place | Used for |
+|---|---|
+| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
+| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
+| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
+| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
+| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
+| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
+| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
+| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
+
+## Concepts
+
+**Location.** Each node has a place in a tree: site, room, row, rack. Power has
+its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
+feed. Both are read from node labels with well-known keys:
+
+```yaml
+# agent.yaml
+labels:
+  llmgw.io/site: sto1
+  llmgw.io/room: hall-b
+  llmgw.io/row: "12"
+  llmgw.io/rack: r12-07
+  llmgw.io/pdu: r12-07-a,r12-07-b   # both feeds, for a dual-fed rack
+```
+
+A node without these labels is its own rack of one: placement behaves as it
+does today.
+
+**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
+condition. Domains are declared in a facility file (below) or discovered from
+node labels; a label naming a domain the file does not declare creates it with
+no budget and no condition.
+
+**Budget.** A domain may have a power cap in watts. Placement reserves each
+replica's estimated draw against every domain its node belongs to and refuses a
+placement that would take any of them past its cap less a safety margin. For a
+dual-fed rack the cap that matters is what one feed can carry alone, since that
+is what the rack must survive.
+
+**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
+or `unknown`, derived from readings against thresholds with hysteresis.
+Conditions change what placement prefers and allows; they are not amounts to
+fill.
+
+**Reading.** A measured value with a time: inlet temperature, power draw, a
+cooling unit's alarm state. Readings older than their staleness limit are
+treated as absent.
+
+## Estimating a replica's draw
+
+What placement reserves has to be known before the replica runs, so it is an
+estimate first and a measurement later:
+
+1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
+2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
+   (the agent already reads the cards; the power limit is one more field) plus
+   a per-node share for CPU, memory and fans; CPU profiles take their reserved
+   cores times a per-core figure for the node's CPU model.
+3. **Measured.** Once replicas run, the agent reports their cards' actual draw
+   (NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
+   domain's headroom is computed from the larger of what is reserved and what
+   is measured, so an estimate that was too low is corrected by the room it
+   stops offering, not by an outage.
+
+## Decisions
+
+### Power
+
+- **Hard limit.** A placement that would take any domain past `cap × (1 −
+  margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
+  16.0 kW reserved, replica needs ~2.8, margin 10%*.
+- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
+  times its derating (typically 80%), not the sum.
+- **Already over.** A domain found over its cap, because the estimate was low
+  or the cap was lowered, gets no new placements, and an alert. It is not
+  emptied automatically; shedding power is the facility's decision, and
+  stopping inference to do it is ours only if asked (see shedding below).
+
+### Temperature and cooling
+
+Thresholds are per domain with defaults, and each condition needs to hold for a
+while before it changes, in both directions, so a single reading neither
+starts nor ends anything:
+
+| Condition | Enters when | Placement |
+|---|---|---|
+| `ok` | below the warm threshold for 5 min | normal |
+| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
+| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
+| `unknown` | no fresh reading | as `ok`, and reported as unknown |
+
+**Shedding** (optional, off by default): a domain `hot` for longer than its
+shed delay has load moved off it using step 2's machinery. The lowest class goes
+first, and replicas above their model's minimum go before any at it. A critical
+minimum is never touched, and moves are rate limited to one replica per domain
+per interval. The replica is placed elsewhere first where there is room, so
+shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
+-pl`) is a gentler alternative worth measuring against shedding: less
+throughput, no restart.
+
+### Spreading
+
+Replicas of one model are spread across racks, and across PDUs, the way they
+are spread across nodes today. This is a preference, dropped when it is the only
+thing standing in the way, as node spreading already is.
+
+### Scoring
+
+Placement today filters, then takes the first candidate in a fixed order
+(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
+the weights first). Facility signals are better as a score than as more
+filters. Candidates that pass the hard limits are ordered by:
+
+1. condition: `ok` before `warm`;
+2. power headroom left in the rack after this replica, more first, so load
+   levels across racks;
+3. the existing order.
+
+Once measurements include power, step 4's profile preference can use tokens per
+watt instead of tokens per GPU where both are known.
+
+## Where the data comes from
+
+A `FacilitySource` interface, implemented several ways, chosen in
+`server.yaml`:
+
+```go
+type FacilitySource interface {
+    // Domains is what the source knows: budgets, thresholds, members.
+    Domains(ctx) ([]Domain, error)
+    // Readings is the latest value of each signal, per domain.
+    Readings(ctx) ([]Reading, error)
+}
+```
+
+- **Static file.** Domains, caps and thresholds, written by hand or generated
+  from a DCIM export. Enough for power budgets with no live data at all.
+- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
+  readings usually end up in Prometheus. The source runs configured instant
+  queries and maps their labels onto domains:
+
+  ```yaml
+  facility:
+    source: prometheus
+    prometheus: http://monitoring:9090
+    signals:
+      inlet_temp_c:
+        query: max by (rack) (dcim_rack_inlet_temperature_celsius)
+        domain_label: rack
+        stale_after: 2m
+      power_watts:
+        query: sum by (pdu) (pdu_outlet_power_watts)
+        domain_label: pdu
+      cooling_alarm:
+        query: max by (row) (crac_alarm)
+        domain_label: row
+  ```
+- **Agents (built, step 0).** What each node reads about itself, every ten
+  seconds at most, sent with its heartbeat:
+  - **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
+    driver reports it, power draw, the enforced power limit, and what is
+    holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
+    `hw_thermal`, `power_brake`.
+  - **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
+    package temperature, or the hottest core; and the board's inlet or ambient
+    sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
+  - **CPU power, from RAPL:** the packages' energy counters over time,
+    subdomains not counted twice, counter wrap handled.
+
+  A machine without a sensor reports nothing for it, never a zero; a VM has
+  none of them. Published as `llmgw_gpu_temperature_celsius`,
+  `llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
+  `llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
+  `llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
+  `llmgw_node_inlet_temperature_celsius`, and on each node's page.
+
+  What they can and cannot say:
+  - **Throttling is the strongest signal there is.** A temperature is a proxy;
+    a card holding its clocks down is the fact that invalidates its measured
+    capacity, and it needs no threshold to be chosen.
+  - **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
+    power-supply losses are not in it, so a node's draw is that plus an
+    estimated remainder. A rack's is its llmgw nodes' plus a declared figure
+    for everything else in it: switches and storage report nothing here.
+  - **CPU and GPU temperatures follow the load, not the room.** A hot rack is
+    inferred from many nodes running hot at comparable load, or from
+    throttling, not from one node's CPU. Only a board inlet sensor reads the
+    air, and most boards have none.
+  - **No budget comes from a sensor.** A reading is a draw; the cap it is
+    measured against still comes from the facility.
+- **Simulated.** Readings from a script, for tests and for rehearsing a
+  thermal event on a cluster without facility data.
+
+## Safety
+
+- **Missing data never blocks.** A source that is down, a reading that is stale,
+  a node without location labels: placement behaves as it does today, and says
+  the data is missing (a metric, a line on the cluster page). Facility
+  awareness can make placement more careful, never impossible.
+- **Hysteresis and hold times** on every condition, so a noisy sensor does not
+  flap a rack between allowed and forbidden.
+- **Observe-only first.** A mode where every decision is computed, logged and
+  shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
+  for long enough to trust the thresholds before turning enforcement on.
+- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
+  facility team to mark a domain unavailable, or to clear a condition they know
+  to be a sensor fault.
+- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
+
+## What people see
+
+- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
+  31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
+- `llmgw ctl facility` lists domains with budget, reserved, measured and
+  condition. The cluster page gets a rack summary beside the nodes table.
+- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
+  `_measured_watts`, `llmgw_domain_condition{condition}`,
+  `llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
+  domain hot, a domain over its cap, and facility data gone stale.
+
+## Rollout
+
+Each step ships on its own and is useful without the next:
+
+0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
+   CPU heat and power, inlet air where the board has a sensor; published and
+   shown, no decision changed.
+1. **Location and spreading.** Well-known labels, domains from labels, replicas
+   spread across racks and PDUs, locations on the node views. No facility data
+   needed.
+2. **Power budgets from a static file.** Caps and margins, estimated draw per
+   replica (declared or derived), hard limit in placement and headroom.
+3. **Readings, observe-only.** The Prometheus source beside the agents',
+   conditions with hysteresis (throttling first among them), measured power
+   folded into headroom; decisions shown, not enforced.
+4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
+   release.
+5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
+   preemption; measured power in profile preference.
+
+## Testing
+
+- The simulated source drives unit tests of every condition transition, the
+  hysteresis, staleness, and the placement outcomes, with mutation checks as
+  for steps 1–4.
+- Property tests: whatever the readings, placement never exceeds a declared
+  cap with fresh data, and never refuses everything because data is missing.
+- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
+  with a static file of made-up caps, steps 3–4 with the simulated source
+  replaying a thermal event.
+- Before enforcement anywhere real: a period in observe-only mode against the
+  site's actual data.
+
+## Open questions
+
+- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
+  of it already reaches a Prometheus. The source design assumes Prometheus is
+  the common ground; a site without it needs another source.
+- **Who owns the numbers.** Caps, deratings and thresholds belong to the
+  facility team. Whether they edit llmgw's file, or llmgw reads theirs.
+- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
+  should see phases or only the PDU total.
+- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
+  the domain tree allows either, but the defaults should match what sites
+  actually measure.
+- **Power capping instead of shedding.** Whether to lower GPU power limits in a
+  warm rack rather than move replicas, and how that interacts with measured
+  capacity from step 4, which would no longer hold at the lower limit.
@@ -940,8 +940,18 @@ restart either — it rebuilds the fleet from what the agents report.
 fields, and both sides treat an absent one as unknown rather than as a default.
 Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
 installing the release binary and restarting `llmgw-agent`, and every upgrade
-after that is the command above. The mirror, which is not an agent, is
-upgraded the same way by hand.
+after that is the command above.
+
+**The mirror is upgraded on its own host**, since it is not an agent and its
+service cannot replace its own binary:
+
+```
+sudo llmgw upgrade v0.10.5 --service llmgw-mirror
+```
+
+The same checks — the release's checksums, the old binary kept beside the new —
+then a restart of the unit, waiting for it to stay up. A mirror older than
+v0.10.5 does not have the command: install the release binary by hand once.

 Then confirm what you have, rather than assuming:

@@ -101,6 +101,9 @@ type Agent struct {
 	smsErr map[string]string
 	// load measures the machine between heartbeats.
 	load *discovery.LoadSampler
+	// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
+	// from nvidia-smi.
+	sensors *discovery.SensorSampler

 	// fetchWatchers are the shards waiting on each source, so progress on one
 	// download reaches every shard that asked for it. A fetch is single-flight
@@ -146,6 +149,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		logRing:      ring,
 		provKick:     make(chan struct{}, 1),
 		load:         discovery.NewLoadSampler(),
+		sensors:      discovery.NewSensorSampler(),
 	}
 	a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
 	return a, nil
@@ -638,6 +642,16 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
 			ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
 		})
 	}
+	if a.sensors != nil {
+		se := a.sensors.Sample(context.Background())
+		out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
+		for _, g := range se.GPUs {
+			out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
+				Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
+				PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
+			})
+		}
+	}
 	return out
 }

@@ -0,0 +1,242 @@
+package discovery
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Sensors is what a node's own hardware says about heat and power: the CPUs,
+// the board where it has an inlet sensor, and each GPU. A zero is not
+// measured, never a reading of zero — a VM has none of these, and must not
+// look like a machine running at 0 °C on no power.
+type Sensors struct {
+	// CPUTempC is the hottest CPU sensor: the package, or the hottest core
+	// where no package reading exists.
+	CPUTempC float64
+	// CPUPowerW is the CPU packages' draw, from RAPL: the change in their
+	// energy counters over the time between two samples.
+	CPUPowerW float64
+	// InletTempC is the air coming in, where the board has a sensor it calls
+	// that. Most do not, and a CPU's temperature is not a substitute: it
+	// follows the load, not the room.
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat, power and whether it is slowing itself down.
+type GPUSensor struct {
+	Index       int
+	TempC       float64
+	MemoryTempC float64
+	PowerW      float64
+	PowerLimitW float64
+	// Throttle names what is holding the card's clocks down now:
+	// sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake. Empty
+	// is running free. Idle and application clock settings are left out;
+	// they are not the card being held back.
+	Throttle []string
+}
+
+// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
+// than temperatures change, and each run costs a few hundred milliseconds.
+const sensorEvery = 10 * time.Second
+
+// SensorSampler reads Sensors. The first sample has no CPU power, which needs
+// two readings of the energy counters.
+type SensorSampler struct {
+	sys string // /sys, or a test's copy of it
+	smi func(ctx context.Context, fields string) ([]byte, error)
+	now func() time.Time
+
+	mu         sync.Mutex
+	last       Sensors
+	lastAt     time.Time
+	prevEnergy map[string]uint64
+	prevAt     time.Time
+}
+
+// NewSensorSampler reads this machine.
+func NewSensorSampler() *SensorSampler {
+	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
+}
+
+func runSMI(ctx context.Context, fields string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	//nolint:gosec // fixed binary, fields from this file
+	return exec.CommandContext(ctx, "nvidia-smi", "--query-gpu="+fields,
+		"--format=csv,noheader,nounits").Output()
+}
+
+// Sample reads the sensors, or returns the last reading when that is recent.
+func (s *SensorSampler) Sample(ctx context.Context) Sensors {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	now := s.now()
+	if !s.lastAt.IsZero() && now.Sub(s.lastAt) < sensorEvery {
+		return s.last
+	}
+	out := Sensors{}
+	out.CPUTempC, out.InletTempC = s.hwmon()
+	out.CPUPowerW = s.rapl(now)
+	out.GPUs = s.gpus(ctx)
+	s.last, s.lastAt = out, now
+	return out
+}
+
+// hwmon reads the hottest CPU sensor and an inlet sensor, from every hwmon
+// device the kernel exposes — what lm_sensors reads, without it installed.
+func (s *SensorSampler) hwmon() (cpu, inlet float64) {
+	devs, _ := filepath.Glob(filepath.Join(s.sys, "class/hwmon/hwmon*"))
+	var pkg, core float64
+	for _, d := range devs {
+		name := readTrim(filepath.Join(d, "name"))
+		inputs, _ := filepath.Glob(filepath.Join(d, "temp*_input"))
+		for _, in := range inputs {
+			milli, err := strconv.ParseFloat(readTrim(in), 64)
+			if err != nil || milli <= 0 {
+				continue
+			}
+			c := milli / 1000
+			label := strings.ToLower(readTrim(strings.TrimSuffix(in, "_input") + "_label"))
+			switch {
+			case isCPUSensor(name) && (strings.HasPrefix(label, "package") || label == "tctl" || label == "tdie"):
+				pkg = max(pkg, c)
+			case isCPUSensor(name):
+				core = max(core, c)
+			case label == "cpu":
+				// A board's own reading of the CPU, as Dell's SMM gives.
+				core = max(core, c)
+			case strings.Contains(label, "inlet") || strings.Contains(label, "ambient") ||
+				strings.Contains(label, "intake") || label == "systin":
+				inlet = max(inlet, c)
+			}
+		}
+	}
+	if pkg > 0 {
+		return pkg, inlet
+	}
+	return core, inlet
+}
+
+func isCPUSensor(name string) bool {
+	switch name {
+	case "coretemp", "k10temp", "zenpower", "cpu_thermal":
+		return true
+	}
+	return false
+}
+
+// rapl is the CPU packages' power from their energy counters. Package
+// domains only: the dram and core subdomains are parts of a package, and
+// adding them would count the same energy twice.
+func (s *SensorSampler) rapl(now time.Time) float64 {
+	doms, _ := filepath.Glob(filepath.Join(s.sys, "class/powercap/intel-rapl:*"))
+	energy := map[string]uint64{}
+	wrap := map[string]uint64{}
+	for _, d := range doms {
+		if strings.Count(filepath.Base(d), ":") != 1 {
+			continue // intel-rapl:0:0 is a subdomain of intel-rapl:0
+		}
+		e, err := strconv.ParseUint(readTrim(filepath.Join(d, "energy_uj")), 10, 64)
+		if err != nil {
+			continue // unreadable without root on newer kernels
+		}
+		energy[d] = e
+		wrap[d], _ = strconv.ParseUint(readTrim(filepath.Join(d, "max_energy_range_uj")), 10, 64)
+	}
+	prev, prevAt := s.prevEnergy, s.prevAt
+	s.prevEnergy, s.prevAt = energy, now
+	dt := now.Sub(prevAt).Seconds()
+	if prev == nil || dt <= 0 || len(energy) == 0 {
+		return 0
+	}
+	var joules float64
+	for d, e := range energy {
+		p, ok := prev[d]
+		if !ok {
+			return 0 // a domain appeared: no fair difference this time
+		}
+		if e < p { // the counter wrapped
+			e += wrap[d]
+		}
+		joules += float64(e-p) / 1e6
+	}
+	return joules / dt
+}
+
+// The throttle reasons nvidia-smi reports, as bits. Only those that mean the
+// card is being held back are named.
+var throttleBits = []struct {
+	bit  uint64
+	name string
+}{
+	{0x4, "sw_power_cap"},
+	{0x8, "hw_slowdown"},
+	{0x20, "sw_thermal"},
+	{0x40, "hw_thermal"},
+	{0x80, "power_brake"},
+}
+
+// gpus asks nvidia-smi for each card's heat and power. The memory temperature
+// is not known to every driver, and a field nvidia-smi does not know fails the
+// whole query, so it is asked for first and dropped on failure.
+func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
+	withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
+	out, err := s.smi(ctx, withMem)
+	if err != nil {
+		if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
+			return nil
+		}
+	}
+	var gs []GPUSensor
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Split(line, ",")
+		if len(f) < 5 {
+			continue
+		}
+		for i := range f {
+			f[i] = strings.TrimSpace(f[i])
+		}
+		idx, err := strconv.Atoi(f[0])
+		if err != nil {
+			continue
+		}
+		g := GPUSensor{Index: idx, TempC: num(f[1]), PowerW: num(f[2]), PowerLimitW: num(f[3])}
+		if bits, err := strconv.ParseUint(strings.TrimPrefix(f[4], "0x"), 16, 64); err == nil {
+			for _, b := range throttleBits {
+				if bits&b.bit != 0 {
+					g.Throttle = append(g.Throttle, b.name)
+				}
+			}
+		}
+		if len(f) > 5 {
+			g.MemoryTempC = num(f[5])
+		}
+		gs = append(gs, g)
+	}
+	return gs
+}
+
+// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
+func num(s string) float64 {
+	v, err := strconv.ParseFloat(s, 64)
+	if err != nil || v < 0 {
+		return 0
+	}
+	return v
+}
+
+func readTrim(path string) string {
+	b, err := os.ReadFile(path) //nolint:gosec // sysfs
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}
@@ -0,0 +1,169 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func put(t *testing.T, root, rel, body string) {
+	t.Helper()
+	p := filepath.Join(root, rel)
+	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(p, []byte(body+"\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+}
+
+// dellHost is the layout of the Proxmox host these tests were written
+// against: Dell's SMM with a CPU and SODIMM sensors, coretemp with a package
+// and cores, and RAPL with a package and its dram subdomain.
+func dellHost(t *testing.T) string {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon0/name", "dell_smm")
+	put(t, root, "class/hwmon/hwmon0/temp1_input", "55000")
+	put(t, root, "class/hwmon/hwmon0/temp1_label", "CPU")
+	put(t, root, "class/hwmon/hwmon0/temp2_input", "47000")
+	put(t, root, "class/hwmon/hwmon0/temp2_label", "SODIMM")
+	put(t, root, "class/hwmon/hwmon1/name", "coretemp")
+	put(t, root, "class/hwmon/hwmon1/temp1_input", "58000")
+	put(t, root, "class/hwmon/hwmon1/temp1_label", "Package id 0")
+	put(t, root, "class/hwmon/hwmon1/temp10_input", "61000")
+	put(t, root, "class/hwmon/hwmon1/temp10_label", "Core 8")
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1000000000")
+	put(t, root, "class/powercap/intel-rapl:0/max_energy_range_uj", "262143328850")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "5000000")
+	return root
+}
+
+func sampler(sys string, smi func(context.Context, string) ([]byte, error), clock *time.Time) *SensorSampler {
+	return &SensorSampler{sys: sys, smi: smi, now: func() time.Time { return *clock }}
+}
+
+func noSMI(context.Context, string) ([]byte, error) { return nil, errors.New("not found") }
+
+func TestAHostsCPUHeatAndPowerAreReadAsLmSensorsWould(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	s := sampler(root, noSMI, &now)
+
+	first := s.Sample(t.Context())
+	// The package reading, not the hotter core: the package is what the
+	// chip itself reports as its temperature.
+	if first.CPUTempC != 58 {
+		t.Errorf("CPU %.1f °C, expected the package's 58", first.CPUTempC)
+	}
+	if first.CPUPowerW != 0 {
+		t.Error("power from one reading of a counter is invented")
+	}
+	if first.InletTempC != 0 || first.GPUs != nil {
+		t.Errorf("sensors this host does not have were reported: %+v", first)
+	}
+
+	// 10 s later the package has used 650 J, and its dram subdomain some of
+	// that — counted once.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1650000000")
+	put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "99000000")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 64.9 || got > 65.1 {
+		t.Errorf("CPU power %.2f W, expected 65", got)
+	}
+}
+
+func TestACounterThatWrapsIsNotANegativeDraw(t *testing.T) {
+	root := dellHost(t)
+	now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "262143000000")
+	s := sampler(root, noSMI, &now)
+	s.Sample(t.Context())
+	// 328,850 µJ to the wrap, then 499,671,150 past it: 500 J in 10 s.
+	put(t, root, "class/powercap/intel-rapl:0/energy_uj", "499671150")
+	now = now.Add(10 * time.Second)
+	if got := s.Sample(t.Context()).CPUPowerW; got < 49.9 || got > 50.1 {
+		t.Errorf("across a wrap: %.2f W, expected 50", got)
+	}
+}
+
+func TestAVMHasNoSensorsAndSaysNothing(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/thermal/cooling_device0/type", "Processor")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.CPUTempC != 0 || got.CPUPowerW != 0 || got.InletTempC != 0 || got.GPUs != nil {
+		t.Errorf("a machine with no sensors reported %+v", got)
+	}
+}
+
+func TestABoardsInletSensorIsTheRoomNotTheCPU(t *testing.T) {
+	root := t.TempDir()
+	put(t, root, "class/hwmon/hwmon2/name", "nct6798")
+	put(t, root, "class/hwmon/hwmon2/temp1_input", "24500")
+	put(t, root, "class/hwmon/hwmon2/temp1_label", "SYSTIN")
+	put(t, root, "class/hwmon/hwmon3/name", "k10temp")
+	put(t, root, "class/hwmon/hwmon3/temp1_input", "71250")
+	put(t, root, "class/hwmon/hwmon3/temp1_label", "Tctl")
+	now := time.Now()
+	got := sampler(root, noSMI, &now).Sample(t.Context())
+	if got.InletTempC != 24.5 || got.CPUTempC != 71.25 {
+		t.Errorf("inlet %.2f, CPU %.2f", got.InletTempC, got.CPUTempC)
+	}
+}
+
+func TestEachGPUSaysHowHotHowHungryAndWhatHoldsItBack(t *testing.T) {
+	var asked []string
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		asked = append(asked, fields)
+		return []byte(
+			"0, 71, 612.45, 700.00, 0x0000000000000000, 84\n" +
+				"1, 88, 698.10, 700.00, 0x0000000000000024, 95\n" + // sw_power_cap + sw_thermal
+				"2, 45, [N/A], 700.00, 0x0000000000000001, [N/A]\n"), nil // idle only
+	}
+	now := time.Now()
+	ss := sampler(t.TempDir(), smi, &now)
+	got := ss.Sample(t.Context())
+	if len(got.GPUs) != 3 {
+		t.Fatalf("%+v", got.GPUs)
+	}
+	g0, g1, g2 := got.GPUs[0], got.GPUs[1], got.GPUs[2]
+	if g0.TempC != 71 || g0.PowerW != 612.45 || g0.PowerLimitW != 700 || g0.MemoryTempC != 84 || g0.Throttle != nil {
+		t.Errorf("card 0 %+v", g0)
+	}
+	if strings.Join(g1.Throttle, ",") != "sw_power_cap,sw_thermal" {
+		t.Errorf("card 1 throttled by %v", g1.Throttle)
+	}
+	if g2.PowerW != 0 || g2.MemoryTempC != 0 || g2.Throttle != nil {
+		t.Errorf("card 2: N/A is not a reading, and idle is not being held back: %+v", g2)
+	}
+	// Within ten seconds the last reading is reused rather than running
+	// nvidia-smi on every heartbeat; after, it is asked again.
+	now = now.Add(5 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 1 {
+		t.Errorf("nvidia-smi was asked %d times within ten seconds", len(asked))
+	}
+	now = now.Add(6 * time.Second)
+	ss.Sample(t.Context())
+	if len(asked) != 2 {
+		t.Errorf("nvidia-smi was asked %d times after eleven seconds, expected 2", len(asked))
+	}
+}
+
+func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T) {
+	smi := func(_ context.Context, fields string) ([]byte, error) {
+		if strings.Contains(fields, "temperature.memory") {
+			return nil, errors.New(`Field "temperature.memory" is not a valid field to query.`)
+		}
+		return []byte("0, 66, 250.0, 300.0, 0x0000000000000040\n"), nil
+	}
+	now := time.Now()
+	got := sampler(t.TempDir(), smi, &now).Sample(t.Context())
+	if len(got.GPUs) != 1 || got.GPUs[0].TempC != 66 || strings.Join(got.GPUs[0].Throttle, ",") != "hw_thermal" {
+		t.Errorf("%+v", got.GPUs)
+	}
+}
@@ -137,7 +137,7 @@ func newCtlCmd() *cobra.Command {
 		// apply to it; it lives here because this is where operators look.
 		newBenchCmd(client, &asJSON),
 		newStatusCmd(client, &asJSON),
-		newModelCmd(client),
+		newModelCmd(client, &asJSON),
 		newScaleCmd(client),
 		newRestartCmd(client),
 		newRetryCmd(client),
@@ -172,6 +172,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
 		}
 	}
 	fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
+	// The one machine this does not reach. Said every time, because it is
+	// exactly the kind of thing that stays three versions behind unnoticed.
+	fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
+		"  sudo llmgw upgrade %s --service llmgw-mirror\n", v)
 	return nil
 }

@@ -24,13 +24,14 @@ import (
 // minutes of outage. For a fixed production set that is a reasonable trade; for
 // a platform whose promise is that people can try things, it is the difference
 // between trying one and not bothering.
-func newModelCmd(client func() *ctlClient) *cobra.Command {
+func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 	c := &cobra.Command{
 		Use:   "model",
 		Short: "Add or remove models while the cluster is running",
 	}
 	c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
 		newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
+		newTargetCmd(client, asJSON),
 		newModelServeCmd(client))
 	return c
 }
@@ -162,3 +162,69 @@ func newOwnerCmd(client func() *ctlClient) *cobra.Command {
 		},
 	}
 }
+
+func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	var tps float64
+	var apply bool
+	c := &cobra.Command{
+		Use:   "target <model> --tokens-per-sec N [--apply]",
+		Short: "How many replicas a model needs for a throughput, and whether they fit",
+		Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
+			"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
+			"--apply makes that the number it always runs, if the cluster has room.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if tps <= 0 {
+				return fmt.Errorf("--tokens-per-sec is required")
+			}
+			var v adminapi.TargetView
+			path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
+				strconv.FormatFloat(tps, 'f', -1, 64)
+			if err := client().get(cmd.Context(), path, &v); err != nil {
+				return err
+			}
+			if *asJSON {
+				return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
+			}
+			out := cmd.OutOrStdout()
+			fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
+				v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
+			switch {
+			case v.More == 0:
+				fmt.Fprintln(out, "It already has them.")
+			case v.Fits:
+				fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
+			default:
+				fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
+				for _, b := range v.Blocked {
+					fmt.Fprintln(out, "  "+b)
+				}
+			}
+			if !apply || v.More == 0 {
+				return nil
+			}
+			if !v.Fits {
+				return fmt.Errorf("not applied: the cluster has no room for them")
+			}
+			var models []adminapi.ModelView
+			if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
+				return err
+			}
+			top := v.Needed
+			for _, m := range models {
+				if m.Name == v.Model {
+					top = max(top, m.Max)
+				}
+			}
+			body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
+			if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
+			return nil
+		},
+	}
+	c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
+	c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
+	return c
+}
@@ -47,6 +47,7 @@ func Root() *cobra.Command {
 		newSimEngineCmd(),
 		newDoctorCmd(),
 		newNodeCmd(),
+		newSelfUpgradeCmd(),
 	)
 	return root
 }
@@ -0,0 +1,87 @@
+package cli
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"runtime"
+	"strings"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/release"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
+)
+
+// newSelfUpgradeCmd upgrades this machine's llmgw on its own, for the machines
+// `llmgw ctl upgrade` does not reach: a mirror host, above all, whose service
+// runs unprivileged with the filesystem read-only and so cannot replace its own
+// binary the way an agent does.
+func newSelfUpgradeCmd() *cobra.Command {
+	var releases, service string
+	c := &cobra.Command{
+		Use:   "upgrade VERSION [--service UNIT]",
+		Short: "Upgrade this machine's llmgw, and restart the service that runs it",
+		Long: "For a machine the control plane does not upgrade — the model mirror's host.\n" +
+			"The control plane and the nodes are upgraded with `llmgw ctl upgrade`.\n\n" +
+			"Fetches the release for this platform, checks it against the release's\n" +
+			"checksums, keeps the old binary beside the new one, and restarts --service,\n" +
+			"waiting for it to be running again. Run it with sudo.",
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			v := args[0]
+			out := cmd.OutOrStdout()
+			if !release.Tag.MatchString(v) {
+				return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
+			}
+			if version.Version == v {
+				fmt.Fprintf(out, "this machine already runs %s\n", v)
+				return nil
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("replacing this machine's binary needs root: run it with sudo")
+			}
+			self, err := os.Executable()
+			if err != nil {
+				return err
+			}
+			backup, err := release.Install(cmd.Context(), nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintf(out, "%s -> %s: installed, checksum verified; the old binary is %s\n",
+				version.Version, v, backup)
+			if service == "" {
+				return nil
+			}
+			//nolint:gosec // the unit name is the operator's own flag
+			if b, err := exec.CommandContext(cmd.Context(), "systemctl", "restart", service).CombinedOutput(); err != nil {
+				return fmt.Errorf("restarting %s: %v: %s", service, err, strings.TrimSpace(string(b)))
+			}
+			// Running, and still running a few seconds on: a unit that starts
+			// and dies on a bad flag is "active" for a moment first.
+			deadline := time.Now().Add(30 * time.Second)
+			steady := 0
+			for time.Now().Before(deadline) {
+				//nolint:gosec // as above
+				b, _ := exec.CommandContext(cmd.Context(), "systemctl", "is-active", service).Output()
+				if strings.TrimSpace(string(b)) == "active" {
+					if steady++; steady >= 3 {
+						fmt.Fprintf(out, "%s restarted and running\n", service)
+						return nil
+					}
+				} else {
+					steady = 0
+				}
+				time.Sleep(2 * time.Second)
+			}
+			return fmt.Errorf("%s is not running after the upgrade; see journalctl -u %s, and %s holds the previous binary",
+				service, service, backup)
+		},
+	}
+	c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from")
+	c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
+	return c
+}
@@ -488,6 +488,9 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
 			// The tenancy tree's capacity quotas: what each part is
 			// guaranteed and may hold.
 			ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
+			// And what each model was measured to serve, for scaling and for
+			// choosing between profiles.
+			ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
 			log.Info("competing to be the control plane that places work",
 				"owner", lead.Owner())
 		}
@@ -60,6 +60,7 @@ func (a *API) Register(mux *http.ServeMux) {
 	mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
 	mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
 	a.registerQuotas(mux)
+	mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
 	mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
 		if a.models == nil {
 			writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -248,6 +249,11 @@ type LoadView struct {
 	// MemPercent is used over used plus available.
 	MemPercent float64 `json:"memory_percent"`
 	AgeSec     int64   `json:"age_seconds"`
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64               `json:"cpu_temp_c,omitempty"`
+	CPUPowerW  float64               `json:"cpu_power_w,omitempty"`
+	InletTempC float64               `json:"inlet_temp_c,omitempty"`
+	GPUs       []inventory.GPUSensor `json:"gpu_sensors,omitempty"`
 }

 // loadFresh is how old a measurement may be and still be shown as current.
@@ -261,7 +267,8 @@ func loadView(l *inventory.Load, now time.Time) *LoadView {
 		CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
 		Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
 		MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
-		AgeSec: int64(now.Sub(l.At).Seconds()),
+		AgeSec:   int64(now.Sub(l.At).Seconds()),
+		CPUTempC: l.CPUTempC, CPUPowerW: l.CPUPowerW, InletTempC: l.InletTempC, GPUs: l.GPUs,
 	}
 	if v.MemTotal > 0 {
 		v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
@@ -399,6 +406,8 @@ type HeadroomView struct {
 	Limited bool `json:"limited,omitempty"`
 	// Blocked says why one more does not fit, a line per node considered.
 	Blocked []string `json:"blocked,omitempty"`
+	// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
+	TokPerSec float64 `json:"tok_per_sec,omitempty"`
 }

 // RejectionView is one rejected candidate.
@@ -958,7 +967,7 @@ func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
 		return HeadroomView{}, false
 	}
 	return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
-		Limited: h.Limited, Blocked: h.Blocked}, true
+		Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec}, true
 }

 func (a *API) Explain(model string) ExplainView {
@@ -0,0 +1,63 @@
+package adminapi
+
+import (
+	"fmt"
+	"math"
+	"net/http"
+	"strconv"
+)
+
+// TargetView is how many replicas a model needs to serve a throughput, from
+// what one replica was measured to serve, and whether the cluster has room.
+type TargetView struct {
+	Model string `json:"model"`
+	// TokPerSec is what was asked for; PerReplica what one replica was
+	// measured to serve at its SLO, on Profile.
+	TokPerSec  float64 `json:"tok_per_sec"`
+	PerReplica float64 `json:"per_replica_tok_per_sec"`
+	Profile    string  `json:"profile"`
+	// Needed replicas in all; Current hold capacity now; More is how many of
+	// Needed are not running, and Room how many more fit.
+	Needed  int      `json:"needed"`
+	Current int      `json:"current"`
+	More    int      `json:"more"`
+	Room    int      `json:"room"`
+	Fits    bool     `json:"fits"`
+	Blocked []string `json:"blocked,omitempty"`
+}
+
+// Target sizes a model for a throughput.
+func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
+	meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
+	if !ok || meas.TokPerSec <= 0 {
+		return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
+	}
+	v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
+		Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
+	h, ok := a.Headroom(model, v.Needed)
+	if !ok {
+		return TargetView{}, fmt.Errorf("no model %q", model)
+	}
+	v.Current = h.Current
+	v.More = max(v.Needed-h.Current, 0)
+	v.Room = h.More
+	v.Fits = v.More <= h.More || h.Limited
+	if !v.Fits {
+		v.Blocked = h.Blocked
+	}
+	return v, nil
+}
+
+func (a *API) target(w http.ResponseWriter, r *http.Request) {
+	tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
+	if err != nil || tps <= 0 {
+		writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
+		return
+	}
+	v, err := a.Target(r.PathValue("name"), tps)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, v)
+}
@@ -0,0 +1,30 @@
+package adminapi
+
+import (
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+)
+
+func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
+	// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
+	a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
+	if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
+		t.Errorf("an unmeasured model should say how to measure it: %v", err)
+	}
+	ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
+		return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
+	})
+	v, err := a.Target("llama", 1200)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
+		t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
+	}
+	v, _ = a.Target("llama", 2000)
+	if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
+		t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
+	}
+}
@@ -545,6 +545,13 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
 			CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
 		}
 	}
+	out.CPUTempC, out.CPUPowerW, out.InletTempC = l.GetCpuTempC(), l.GetCpuPowerW(), l.GetInletTempC()
+	for _, g := range l.GetGpus() {
+		out.GPUs = append(out.GPUs, inventory.GPUSensor{
+			Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
+			PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
+		})
+	}
 	return out
 }

@@ -74,6 +74,13 @@ type Collector struct {
 	gpuMemoryUsed  *prometheus.Desc
 	gpuUtil        *prometheus.Desc
 	gpuTemp        *prometheus.Desc
+	gpuMemTemp     *prometheus.Desc
+	gpuPower       *prometheus.Desc
+	gpuPowerLimit  *prometheus.Desc
+	gpuThrottled   *prometheus.Desc
+	nodeCPUTemp    *prometheus.Desc
+	nodeCPUPower   *prometheus.Desc
+	nodeInletTemp  *prometheus.Desc
 	gpuHealthy     *prometheus.Desc
 	gpuNVLink      *prometheus.Desc
 	gpuECC         *prometheus.Desc
@@ -142,6 +149,21 @@ func New(src Source) *Collector {
 			"How busy a GPU was over the last sample period. Absent when the driver does not "+
 				"report it, which is not the same as idle. Memory used says a model is loaded; "+
 				"this says whether it is doing anything.", gpu, nil),
+		gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
+			"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
+		gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
+			"A GPU's power draw now.", gpu, nil),
+		gpuPowerLimit: prometheus.NewDesc("llmgw_gpu_power_limit_watts",
+			"The power limit a GPU is held to.", gpu, nil),
+		gpuThrottled: prometheus.NewDesc("llmgw_gpu_throttled",
+			"1 while something holds a GPU's clocks down: sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.",
+			append(gpu, "reason"), nil),
+		nodeCPUTemp: prometheus.NewDesc("llmgw_node_cpu_temperature_celsius",
+			"A node's hottest CPU sensor.", node, nil),
+		nodeCPUPower: prometheus.NewDesc("llmgw_node_cpu_power_watts",
+			"A node's CPU packages' power draw, from RAPL.", node, nil),
+		nodeInletTemp: prometheus.NewDesc("llmgw_node_inlet_temperature_celsius",
+			"A node's inlet or ambient air sensor, where its board has one.", node, nil),
 		gpuTemp: prometheus.NewDesc("llmgw_gpu_temperature_celsius",
 			"A GPU's reported temperature.", gpu, nil),
 		gpuHealthy: prometheus.NewDesc("llmgw_gpu_healthy",
@@ -250,6 +272,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
 		c.gpuNVLJoined, c.imexUp,
 		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
+		c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
 		c.gpuNVLink, c.gpuECC,
 		c.replicaState, c.replicaReady, c.replicaIdle,
 		c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
@@ -297,6 +320,16 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 		}

 		if l := n.Load; l != nil && now.Sub(l.At) < loadFresh {
+			// Each only when measured: a VM has none of these, and a zero
+			// would read as a machine at 0 °C drawing nothing.
+			for _, v := range []struct {
+				d   *prometheus.Desc
+				val float64
+			}{{c.nodeCPUTemp, l.CPUTempC}, {c.nodeCPUPower, l.CPUPowerW}, {c.nodeInletTemp, l.InletTempC}} {
+				if v.val > 0 {
+					ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, n.ID)
+				}
+			}
 			ch <- prometheus.MustNewConstMetric(c.nodeCPU, prometheus.GaugeValue, l.CPUPercent, n.ID)
 			if l.CPUCount > 0 {
 				ch <- prometheus.MustNewConstMetric(c.nodeCPUs, prometheus.GaugeValue, float64(l.CPUCount), n.ID)
@@ -366,7 +399,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
 				ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
 					float64(g.UtilizationPercent), labels...)
 			}
-			if g.Health.TemperatureC > 0 {
+			// The sensors' fresh reading when there is one; the temperature
+			// read at registration otherwise, which is all an older agent sends.
+			if s, ok := sensorFor(n.Load, g.Index, now); ok {
+				c.gpuSensor(ch, s, labels)
+			} else if g.Health.TemperatureC > 0 {
 				ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
 					float64(g.Health.TemperatureC), labels...)
 			}
@@ -518,3 +555,30 @@ func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
 		}
 	}
 }
+
+// sensorFor is a GPU's fresh sensor reading, if the node sent one.
+func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor, bool) {
+	if l == nil || now.Sub(l.At) >= loadFresh {
+		return inventory.GPUSensor{}, false
+	}
+	for _, g := range l.GPUs {
+		if g.Index == index {
+			return g, true
+		}
+	}
+	return inventory.GPUSensor{}, false
+}
+
+func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
+	for _, v := range []struct {
+		d   *prometheus.Desc
+		val float64
+	}{{c.gpuTemp, s.TempC}, {c.gpuMemTemp, s.MemoryTempC}, {c.gpuPower, s.PowerW}, {c.gpuPowerLimit, s.PowerLimitW}} {
+		if v.val > 0 {
+			ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, labels...)
+		}
+	}
+	for _, reason := range s.Throttle {
+		ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
+	}
+}
@@ -0,0 +1,41 @@
+package fleetmetrics
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
+	gpuNode := inventory.Node{ID: "g1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559, Health: discovery.GPUHealth{TemperatureC: 40}}},
+		Load: &inventory.Load{At: time.Now(), CPUTempC: 58, CPUPowerW: 65,
+			GPUs: []inventory.GPUSensor{{Index: 0, TempC: 88, PowerW: 698, PowerLimitW: 700,
+				Throttle: []string{"sw_power_cap", "sw_thermal"}}}}}
+	vm := inventory.Node{ID: "vm1", State: inventory.Ready, Load: &inventory.Load{At: time.Now(), CPUPercent: 3}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{gpuNode, vm}})
+
+	for _, want := range []string{
+		"llmgw_node_cpu_temperature_celsius node=g1 -> 58",
+		"llmgw_node_cpu_power_watts node=g1 -> 65",
+		"llmgw_gpu_power_watts class=h100-80g gpu=0 node=g1 -> 698",
+		"llmgw_gpu_power_limit_watts class=h100-80g gpu=0 node=g1 -> 700",
+		"llmgw_gpu_throttled class=h100-80g gpu=0 node=g1 reason=sw_thermal -> 1",
+		// The fresh reading, not the 40 °C from registration.
+		"llmgw_gpu_temperature_celsius class=h100-80g gpu=0 node=g1 -> 88",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %q", want)
+		}
+	}
+	for _, absent := range []string{"node=vm1 -> 0", "llmgw_node_inlet_temperature_celsius", "llmgw_gpu_memory_temperature_celsius"} {
+		for _, line := range strings.Split(out, "\n") {
+			if strings.Contains(line, absent) && (strings.Contains(line, "temperature") || strings.Contains(line, "power")) {
+				t.Errorf("published what was not measured: %s", line)
+			}
+		}
+	}
+}
@@ -304,6 +304,21 @@ type Load struct {
 	MemAvailBytes int64
 	// Shards is each engine's use, by shard ID.
 	Shards map[string]ShardLoad
+	// What the node's sensors say; zero is not measured.
+	CPUTempC   float64
+	CPUPowerW  float64
+	InletTempC float64
+	GPUs       []GPUSensor
+}
+
+// GPUSensor is one card's heat and power, and what is throttling it.
+type GPUSensor struct {
+	Index       int      `json:"index"`
+	TempC       float64  `json:"temp_c,omitempty"`
+	MemoryTempC float64  `json:"memory_temp_c,omitempty"`
+	PowerW      float64  `json:"power_w,omitempty"`
+	PowerLimitW float64  `json:"power_limit_w,omitempty"`
+	Throttle    []string `json:"throttle,omitempty"`
 }

 // ShardLoad is one engine's use: cores busy, and resident memory.
@@ -2,11 +2,14 @@ package control

 import (
 	"context"
+	"encoding/json"
 	"log/slog"
 	"sync"
 	"time"

+	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
 )

@@ -50,3 +53,43 @@ func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
 		return cur
 	}
 }
+
+// MeasurementStore is where benchmark results are kept.
+type MeasurementStore interface {
+	Capacities(ctx context.Context) ([]store.StoredCapacity, error)
+}
+
+// Measurements returns what the reconciler asks for benchmark results, keyed
+// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
+// keeps the last good answer when the store fails.
+func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
+	var mu sync.Mutex
+	var cur map[string]reconciler.Measured
+	var at time.Time
+	return func() map[string]reconciler.Measured {
+		mu.Lock()
+		defer mu.Unlock()
+		if cur != nil && time.Since(at) < quotaRefresh {
+			return cur
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		at = time.Now()
+		rows, err := st.Capacities(ctx)
+		if err != nil {
+			log.Warn("could not read benchmark results; keeping the last ones", "err", err)
+			return cur
+		}
+		next := map[string]reconciler.Measured{}
+		for _, row := range rows {
+			var c bench.Capacity
+			if json.Unmarshal(row.Measurement, &c) != nil {
+				continue
+			}
+			next[row.Model+"/"+row.Profile] = reconciler.Measured{
+				MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
+		}
+		cur = next
+		return cur
+	}
+}
@@ -157,8 +157,31 @@ type Load struct {
 // the last one is still starting: on a small CPU fleet each replica is minutes
 // of weights and gigabytes of memory, and a spike that is over by the time it
 // loads should not leave three behind.
+//
+// With a benchmark stored for the profile a model runs on, its measured
+// capacity is used too: more requests per replica than were measured to meet
+// the SLO is busy, even with the engines' queues empty — an engine will take
+// requests into its slots long after latency has gone — and a replica is only
+// taken back when the rest would still be under scaleDownHeadroom of what they
+// were measured to hold, so scaling down does not land on the edge.
 func (r *Reconciler) RecordLoad(loads map[string]Load) {
 	now := r.now()
+	// The profile each model's replicas run on, for its measurement; read
+	// before the loop, which holds the lock.
+	running := map[string]string{}
+	r.mu.Lock()
+	for _, rep := range r.replicas {
+		if rep.State.Counts() {
+			running[rep.Model] = rep.Profile
+		}
+	}
+	r.mu.Unlock()
+	measured := map[string]Measured{}
+	for model, profile := range running {
+		if m, ok := r.measured(model, profile); ok {
+			measured[model] = m
+		}
+	}
 	for _, m := range r.cat.Load().Models() {
 		base := r.baseFor(m)
 		r.mu.Lock()
@@ -178,8 +201,13 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 		l := loads[m.Name]
 		perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
 		extra := r.extra[m.Name]
+		meas, hasMeas := measured[m.Name]
+		overMeasured := hasMeas &&
+			float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
+		fitsInFewer := !hasMeas ||
+			float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
 		switch {
-		case perReplica >= r.cfg.ScaleUpWaiting:
+		case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
 			delete(r.quietSince, m.Name)
 			since, ok := r.busySince[m.Name]
 			if !ok {
@@ -190,9 +218,10 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 				r.extra[m.Name] = extra + 1
 				r.busySince[m.Name] = now
 				r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
-					"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
+					"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
+					"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
 			}
-		case l.Waiting == 0 && extra > 0:
+		case l.Waiting == 0 && extra > 0 && fitsInFewer:
 			delete(r.busySince, m.Name)
 			since, ok := r.quietSince[m.Name]
 			if !ok {
@@ -212,3 +241,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
 		r.mu.Unlock()
 	}
 }
+
+// scaleDownHeadroom is how full, of their measured capacity, the replicas left
+// after a scale-down may be. Below one so that the load that was there a
+// minute ago does not put them straight back over.
+const scaleDownHeadroom = 0.7
@@ -25,6 +25,9 @@ type Headroom struct {
 	// Blocked is why one more than More does not fit, a line per node that
 	// was considered. Empty when Limited.
 	Blocked []string
+	// TokPerSec is what one more replica was measured to serve, on the
+	// profile placement would give it; zero when it was never measured.
+	TokPerSec float64
 }

 // Headroom tries placing up to limit more replicas of a model on a copy of
@@ -59,10 +62,10 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
 		usage = pol.Usage(r.Holdings())
 	}
 	for h.More < limit {
-		res := sched.Place(m.PlacementProfiles(), avoid)
+		res := sched.Place(r.orderedProfiles(m), avoid)
 		if res.Placement == nil && len(avoid) > 0 {
 			// As scaleUp does: spreading is a preference, not a rule.
-			res = sched.Place(m.PlacementProfiles(), nil)
+			res = sched.Place(r.orderedProfiles(m), nil)
 		}
 		if res.Placement == nil {
 			for _, rej := range res.Rejections {
@@ -87,6 +90,11 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
 		if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
 			mem, cores = prof.MinMemoryGB, cpuCores(prof)
 		}
+		if h.More == 0 {
+			if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
+				h.TokPerSec = meas.TokPerSec
+			}
+		}
 		var nodes []string
 		for _, a := range res.Placement.Nodes {
 			inv.Allocate(a.NodeID, inventory.Allocation{
@@ -0,0 +1,108 @@
+package reconciler
+
+import (
+	"sort"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
+)
+
+// Measured is what one replica of a model on a profile was measured to serve
+// while meeting its SLO, by `llmgw ctl bench --save`.
+type Measured struct {
+	// MaxConcurrency is the most requests at once that still met the SLO.
+	MaxConcurrency int
+	// TokPerSec is the throughput at that concurrency.
+	TokPerSec float64
+}
+
+// SetMeasurements supplies benchmark results, keyed model/profile, read
+// afresh each time it is called. Without them scaling goes by queue alone and
+// profiles are chosen by priority alone.
+func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.measurements = f
+}
+
+// measured is a model's measurement on one profile.
+func (r *Reconciler) measured(model, profile string) (Measured, bool) {
+	r.mu.Lock()
+	f := r.measurements
+	r.mu.Unlock()
+	if f == nil {
+		return Measured{}, false
+	}
+	m, ok := f()[model+"/"+profile]
+	return m, ok && m.MaxConcurrency > 0
+}
+
+// orderedProfiles is a model's profiles as placement should try them: by
+// priority, and among equal priorities the one measured to serve the most per
+// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
+// their catalog order behind measured ones of the same priority.
+//
+// Placement sorts by priority itself, so the order found here is written back
+// as priorities, descending, for it to keep.
+func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
+	ps := m.PlacementProfiles()
+	eff := make([]float64, len(ps))
+	for i, p := range ps {
+		meas, ok := r.measured(m.Name, p.Name)
+		if !ok {
+			continue
+		}
+		units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
+		if units == 0 {
+			units = float64(max(p.CPUCores, 1))
+		}
+		eff[i] = meas.TokPerSec / units
+	}
+	idx := make([]int, len(ps))
+	for i := range idx {
+		idx[i] = i
+	}
+	sort.SliceStable(idx, func(a, b int) bool {
+		pa, pb := ps[idx[a]], ps[idx[b]]
+		if pa.Priority != pb.Priority {
+			return pa.Priority > pb.Priority
+		}
+		return eff[idx[a]] > eff[idx[b]]
+	})
+	out := make([]placement.Profile, len(ps))
+	for rank, i := range idx {
+		out[rank] = ps[i]
+		out[rank].Priority = len(ps) - rank
+	}
+	return out
+}
+
+// MeasuredFor is the measurement that sizes a model: that of the profile its
+// replicas run on, or, with none running, of the first profile placement
+// would try that was measured.
+func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
+	r.mu.Lock()
+	profile := ""
+	for _, rep := range r.replicas {
+		if rep.Model == model && rep.State.Counts() {
+			profile = rep.Profile
+			break
+		}
+	}
+	r.mu.Unlock()
+	if profile != "" {
+		if m, ok := r.measured(model, profile); ok {
+			return m, profile, true
+		}
+	}
+	cm, ok := r.cat.Load().Resolve(model)
+	if !ok {
+		return Measured{}, "", false
+	}
+	for _, p := range r.orderedProfiles(cm) {
+		if m, ok := r.measured(cm.Name, p.Name); ok {
+			return m, p.Name, true
+		}
+	}
+	return Measured{}, "", false
+}
@@ -0,0 +1,96 @@
+package reconciler
+
+import (
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func measure(h *harness, byKey map[string]Measured) {
+	h.r.SetMeasurements(func() map[string]Measured { return byKey })
+}
+
+func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	// Six in flight, nothing waiting at the engine: by the queue it is fine.
+	load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
+	hold := func() {
+		h.r.RecordLoad(load)
+		h.advance(61 * time.Second)
+		h.r.RecordLoad(load)
+		h.r.Reconcile(t.Context())
+	}
+	hold()
+	if n := liveCount(h); n != 1 {
+		t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
+	}
+	// Measured to meet its SLO at four at once: six is past it.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	hold()
+	if n := liveCount(h); n != 2 {
+		t.Errorf("%d replicas with 6 in flight against a measured 4", n)
+	}
+}
+
+func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
+	m := testModel(1)
+	m.Scaling.Max = 2
+	h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	h.r.mu.Lock()
+	h.r.extra["llama"] = 1
+	h.r.mu.Unlock()
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	if liveCount(h) != 2 {
+		t.Fatal("setup: not at two replicas")
+	}
+	quietFor := func(l Load) {
+		for range 3 {
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.advance(DefaultConfig().ScaleDownAfter + time.Second)
+			h.r.RecordLoad(map[string]Load{"llama": l})
+			h.r.Reconcile(t.Context())
+		}
+	}
+	// Nothing waiting, but three running: one replica left would be at 3 of
+	// its 4, past 70%.
+	quietFor(Load{Replicas: 2, Running: 3})
+	if n := liveCount(h); n != 2 {
+		t.Errorf("scaled down to the edge of what one replica holds: %d", n)
+	}
+	quietFor(Load{Replicas: 2, Running: 2})
+	if n := liveCount(h); n != 1 {
+		t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
+	}
+}
+
+func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
+	m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{
+			{Name: "tp4", GPUClass: "h100-80g", TP: 4},
+			{Name: "tp2", GPUClass: "h100-80g", TP: 2},
+			{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
+		}}
+	h := newHarness(t, m, gpuNode("1", 8))
+	// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
+	// the higher priority, which an operator set and a benchmark does not
+	// overrule.
+	measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
+		"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
+	cm, _ := h.r.cat.Load().Resolve("llama")
+	var order []string
+	for _, p := range h.r.orderedProfiles(cm) {
+		order = append(order, p.Name)
+	}
+	if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
+		t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
+	}
+}
@@ -29,6 +29,9 @@ type victim struct {
 	AboveMin bool
 	Tier     int
 	Idle     time.Duration
+	// Size is what stopping it frees, for choosing the smaller of two
+	// otherwise equal victims.
+	Size quota.Amount
 }

 // preemptFor stops replicas of lower classes when a model cannot otherwise be
@@ -54,28 +57,46 @@ func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []st
 	if len(cands) == 0 {
 		return false
 	}
-	trial := r.inv.Clone()
-	sched := placement.New(trial)
+	// Candidates in order until the model fits; then every one whose
+	// stopping turns out not to be needed is kept after all. Without that
+	// second step an idle replica on the wrong node, tried first, would be
+	// stopped along with the one that actually made the room.
 	var chosen []victim
 	for _, v := range cands {
 		if len(chosen) == maxVictims {
 			break
 		}
-		trial.ReleaseReplica(v.ID)
 		chosen = append(chosen, v)
-		res := sched.Place(m.PlacementProfiles(), avoid)
-		if res.Placement == nil && len(avoid) > 0 {
-			res = sched.Place(m.PlacementProfiles(), nil)
-		}
-		if res.Placement == nil {
+		if !r.fitsWithout(m, avoid, chosen) {
 			continue
 		}
+		for i := len(chosen) - 1; i >= 0 && len(chosen) > 1; i-- {
+			fewer := append(append([]victim(nil), chosen[:i]...), chosen[i+1:]...)
+			if r.fitsWithout(m, avoid, fewer) {
+				chosen = fewer
+			}
+		}
 		r.preempt(ctx, m, chosen)
 		return true
 	}
 	return false
 }

+// fitsWithout reports whether m could be placed with these replicas gone,
+// tried on a copy of the inventory.
+func (r *Reconciler) fitsWithout(m catalog.Model, avoid []string, gone []victim) bool {
+	trial := r.inv.Clone()
+	for _, v := range gone {
+		trial.ReleaseReplica(v.ID)
+	}
+	sched := placement.New(trial)
+	res := sched.Place(r.orderedProfiles(m), avoid)
+	if res.Placement == nil && len(avoid) > 0 {
+		res = sched.Place(r.orderedProfiles(m), nil)
+	}
+	return res.Placement != nil
+}
+
 // preempt drains the chosen replicas and holds their models back.
 func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
 	now := r.now()
@@ -245,15 +266,32 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
 			if len(nodesOf(rep)) > 1 {
 				continue // see makeRoomFor: a multi-node replica is its own decision
 			}
+			prof, _ := other.ProfileByName(rep.Profile)
+			var size quota.Amount
+			for _, sh := range rep.Shards {
+				size = size.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
+			}
 			out = append(out, victim{ID: rep.ID, Model: model, Class: class,
-				AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now)})
+				AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now), Size: size})
 		}
 	}
 	sort.SliceStable(out, func(i, j int) bool {
-		if out[i].Tier != out[j].Tier {
-			return out[i].Tier < out[j].Tier
+		a, b := out[i], out[j]
+		if a.Tier != b.Tier {
+			return a.Tier < b.Tier
 		}
-		return out[i].Idle > out[j].Idle
+		if a.Idle != b.Idle {
+			return a.Idle > b.Idle
+		}
+		// The smaller first: the least served is lost if it is enough, and
+		// the pruning after keeps a larger one only when it is needed.
+		if a.Size.GPUs != b.Size.GPUs {
+			return a.Size.GPUs < b.Size.GPUs
+		}
+		if a.Size.MemoryGB != b.Size.MemoryGB {
+			return a.Size.MemoryGB < b.Size.MemoryGB
+		}
+		return a.ID < b.ID
 	})
 	return out
 }
@@ -235,3 +235,80 @@ func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
 		t.Error("a hold outlived its time")
 	}
 }
+
+func TestOnlyTheReplicasThatMakeTheRoomAreStopped(t *testing.T) {
+	// Two half-used nodes. "idle" on node 1 is on demand and long unused, so
+	// it is first in line; but the critical model needs node 2's other half,
+	// which only stopping "chat" frees.
+	idle := catalog.Model{Name: "idle", Source: "/m", Scaling: catalog.Scaling{Min: 0, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
+	chat := catalog.Model{Name: "chat", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp6", Engine: "vllm", GPUClass: "h100-80g", TP: 6}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
+	h := newFleetHarness(t, []catalog.Model{idle, chat}, gpuNode("1", 6), gpuNode("2", 8))
+	h.r.Want("idle")
+	h.settle(t, 4)
+	var idleNode, chatNode string
+	for _, rep := range h.r.Replicas() {
+		switch rep.Model {
+		case "idle":
+			idleNode = rep.Shards[0].NodeID
+		case "chat":
+			chatNode = rep.Shards[0].NodeID
+		}
+	}
+	if idleNode == "" || chatNode != "2" {
+		t.Fatalf("setup: idle on %q, chat on %q; chat must hold node 2: %+v", idleNode, chatNode, h.r.Replicas())
+	}
+	h.r.mu.Lock()
+	for _, rep := range h.r.replicas {
+		rep.LastUsedAt = h.r.now().Add(-time.Hour)
+	}
+	h.r.mu.Unlock()
+	h.advance(h.r.cfg.MinLifetime + time.Minute)
+	cat, _ := catalog.New([]catalog.Model{idle, chat, vital})
+	h.r.cat.Store(cat)
+
+	h.r.Reconcile(t.Context())
+	var stopped []string
+	for _, rep := range h.r.Replicas() {
+		if rep.PreemptedFor == "vital" {
+			stopped = append(stopped, rep.Model)
+		}
+	}
+	if len(stopped) != 1 || stopped[0] != "chat" {
+		t.Errorf("stopped %v; only chat's replica makes room for a whole node", stopped)
+	}
+}
+
+func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
+	cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 31 * 1024, CPUCount: 8}
+	big := catalog.Model{Name: "big", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12, CPUCores: 4}}}
+	small := catalog.Model{Name: "small", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
+	vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, CPUCores: 4}}}
+	// 8 cores: big 4 + small 1 leaves 3, and vital needs 4. Stopping either
+	// is enough; stopping big loses far more.
+	for range 5 { // the choice must not depend on map order
+		h := newFleetHarness(t, []catalog.Model{big, small}, cpu)
+		h.settle(t, 3)
+		h.advance(h.r.cfg.MinLifetime + time.Minute)
+		cat, _ := catalog.New([]catalog.Model{big, small, vital})
+		h.r.cat.Store(cat)
+		h.r.Reconcile(t.Context())
+		var stopped []string
+		for _, rep := range h.r.Replicas() {
+			if rep.PreemptedFor == "vital" {
+				stopped = append(stopped, rep.Model)
+			}
+		}
+		if len(stopped) != 1 || stopped[0] != "small" {
+			t.Fatalf("stopped %v; the small replica alone makes room", stopped)
+		}
+	}
+}
@@ -273,6 +273,8 @@ type Reconciler struct {
 	preempted  map[string]int
 	// quotas is the tenancy tree's capacity quotas as last read; nil is none.
 	quotas func() *quota.Policy
+	// measurements are benchmark results by model/profile; nil is none.
+	measurements func() map[string]Measured

 	wake chan struct{}
 	stop chan struct{}
@@ -753,10 +755,10 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		r.mu.Lock()
 		rep, ok := r.replicas[id]
 		var expired bool
+		grace, preemptedFor := r.cfg.DrainGrace, ""
 		if ok {
-			grace := r.cfg.DrainGrace
 			if rep.PreemptedFor != "" {
-				grace = r.cfg.PreemptGrace
+				grace, preemptedFor = r.cfg.PreemptGrace, rep.PreemptedFor
 			}
 			expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
 		}
@@ -766,11 +768,15 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		case !ok:
 		case replaced:
 			r.stopReplica(ctx, id)
+		case expired && preemptedFor != "":
+			r.log.Info("stopping a preempted replica", "replica", id,
+				"for", preemptedFor, "after", grace)
+			r.stopReplica(ctx, id)
 		case expired:
 			// The replacement never arrived — often because nowhere else can
 			// host it. Draining forever would leave the node undrainable.
 			r.log.Warn("stopping a drained replica without a replacement",
-				"replica", id, "after", r.cfg.DrainGrace)
+				"replica", id, "after", grace)
 			r.stopReplica(ctx, id)
 		}
 	}
@@ -784,12 +790,13 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 // allocated, and starting several at once from the same inventory snapshot
 // would hand them the same GPUs.
 func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
-	res := r.sched.Place(m.PlacementProfiles(), avoid)
+	profiles := r.orderedProfiles(m)
+	res := r.sched.Place(profiles, avoid)
 	if res.Placement == nil && len(avoid) > 0 {
 		// Spreading across failure domains is a preference, not a rule. If it
 		// is the only thing standing in the way, take the placement anyway:
 		// a second replica on one node still survives an engine crash.
-		res = r.sched.Place(m.PlacementProfiles(), nil)
+		res = r.sched.Place(profiles, nil)
 	}

 	r.mu.Lock()
@@ -7,6 +7,7 @@ import (
 	"time"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
@@ -174,3 +175,33 @@ func TestTheNVLinkPageDrawsEachDomainAsARack(t *testing.T) {
 		t.Error("the cluster page does not lead to the racks")
 	}
 }
+
+// hotSource is a GPU node whose sensors report, one card held back.
+type hotSource struct{ fakeSource }
+
+func (hotSource) Nodes() []adminapi.NodeView {
+	n := fakeSource{}.Nodes()[0]
+	n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
+		CPUTempC: 58, CPUPowerW: 312, InletTempC: 24.5,
+		GPUs: []inventory.GPUSensor{
+			{Index: 0, TempC: 71, MemoryTempC: 84, PowerW: 612, PowerLimitW: 700},
+			{Index: 1, TempC: 88, PowerW: 698, PowerLimitW: 700, Throttle: []string{"sw_power_cap", "sw_thermal"}},
+		}}
+	return []adminapi.NodeView{n}
+}
+
+func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
+	_, body := get(t, nodeMux(hotSource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
+	for _, want := range []string{"58 °C", "312 W", "24.5 °C", "inlet air", "84 °C", "612 W", "of 700",
+		`<span class="bad">sw_power_cap, sw_thermal</span>`} {
+		if !strings.Contains(body, want) {
+			t.Errorf("the page lacks %q", want)
+		}
+	}
+	_, vm := get(t, nodeMux(busySource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
+	for _, absent := range []string{"°C", "<h2>GPUs</h2>", "inlet air"} {
+		if strings.Contains(vm, absent) {
+			t.Errorf("a node with no sensors shows %q", absent)
+		}
+	}
+}
@@ -21,6 +21,10 @@
  <div class="tile"><div class="v">{{printf "%.0f" .CPUPercent}}%</div><div class="k">CPU{{if .CPUs}} · {{.CPUs}} cores{{end}}</div>{{template "minibar" pctbar .CPUPercent}}</div>
  <div class="tile"><div class="v">{{bytes .MemUsed}}</div><div class="k">RAM in use · of {{bytes .MemTotal}}</div>{{template "minibar" pctbar .MemPercent}}</div>
  <div class="tile"><div class="v">{{printf "%.2f" .Load1}}</div><div class="k">load · {{printf "%.2f" .Load5}} 5m · {{printf "%.2f" .Load15}} 15m</div></div>
+  {{/* Only what this machine has sensors for: a VM has none, and a tile
+       saying 0 °C would be a reading nobody took. */}}
+  {{if or .CPUTempC .CPUPowerW}}<div class="tile"><div class="v">{{if .CPUTempC}}{{printf "%.0f" .CPUTempC}} °C{{else}}—{{end}}</div><div class="k">CPU{{if .CPUPowerW}} · {{printf "%.0f" .CPUPowerW}} W{{end}}</div></div>{{end}}
+  {{if .InletTempC}}<div class="tile"><div class="v">{{printf "%.1f" .InletTempC}} °C</div><div class="k">inlet air</div></div>{{end}}
  {{else}}
  <div class="tile"><div class="v">—</div><div class="k">CPU and RAM not measured lately</div></div>
  {{end}}
@@ -29,6 +33,17 @@
 </div>
 {{if .Warnings}}{{range .Warnings}}<p class="note bad">{{.}}</p>{{end}}{{end}}
 {{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
+{{with .Load}}{{if .GPUs}}
+<h2>GPUs</h2>
+<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
+{{range .GPUs}}<tr>
+  <td class="num">{{.Index}}</td>
+  <td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
+  <td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
+  <td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
+  <td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
+</tr>{{end}}</table>
+{{end}}{{end}}
 {{end}}

 {{if or .Node.IBPorts .Node.Fabrics}}
@@ -109,6 +109,13 @@ var funcs = template.FuncMap{
 		return p
 	},
 	"ago": ago,
+	// percent is part of whole, 0 to 100, for a bar.
+	"percent": func(part, whole float64) float64 {
+		if whole <= 0 {
+			return 0
+		}
+		return part * 100 / whole
+	},
 	// shortNode is a node's name without what every node's name starts with.
 	"shortNode": strings.TrimPrefix,
 	"age":       age,
@@ -1463,8 +1463,8 @@ The launch path is now built too:
 - [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
      level meeting the SLO, plus the queue depth and KV usage observed there,
      so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
-      rather than printed
+- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
+      and the autoscaler and profile choice use what `ctl bench --save` measured
 - [x] OTel tracing: an inference span with admission, schedule and each
      upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
      set, and the traceparent carried on to the engine
@@ -1480,7 +1480,17 @@ The launch path is now built too:
 - [ ] Admission displacement (dropping requests from lower bands)
 - [x] Autoscaling on load: a replica added while requests wait, up to a
      model's max, and taken back after ten quiet minutes
- [ ] Priority-based preemption
+- [x] Priority-based preemption: capacity classes (critical, standard,
+      opportunistic), preemption tried on a copy of the inventory first
+- [x] Capacity quotas per organization, team and project: guarantees kept,
+      borrowing taken back, limits enforced
+- [x] CPU cores reserved like memory, and memory required of CPU models
+- [x] Node sensors collected: GPU temperature, power and throttling, CPU
+      temperature and RAPL power, board inlet air; published and shown
+- [ ] Rack power, temperature and cooling as placement signals: planned in
+      [docs/facility-aware-placement.md](docs/facility-aware-placement.md),
+      deferred — it needs each site's rack, PDU and power data first
+- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
 - [ ] Control plane HA (Raft)
 - [x] Weight downloading and cache management: the mirror, agent fetches
      checked against their digest, and reclaiming what nothing uses
Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	5cf0003e74	Release v0.10.5 Release / notes (push) Successful in 3s Details Release / build (push) Successful in 1m12s Details CI / test (push) Successful in 3m53s Details Collects rc.1 to rc.14: the mirror page in tabs with a card per model, one line per record in every admin table, reserved CPU cores and required memory, capacity classes and preemption, tenant capacity quotas, sizing by measured throughput, and node sensor readings. Adds `llmgw upgrade` for the mirror host, which `ctl upgrade` does not reach, and says so when it ends. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 22:26:19 +02:00
Ilker ManapandClaude Opus 5.5	d10fad064e	Mark facility steps 1-5 deferred; node sensors collected CI / test (push) Successful in 3m45s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 20:25:50 +02:00
Ilker ManapandClaude Opus 5.5	cd10e086c8	Read nodes' heat and power: GPUs, CPUs, board inlet, RAPL Release / build (push) Successful in 1m10s Details CI / test (push) Successful in 3m51s Details Release / notes (push) Successful in 3s Details Every agent samples, at most every ten seconds: nvidia-smi's temperature, HBM temperature, power draw, enforced limit and throttle reasons per GPU; hwmon's CPU package and board inlet temperatures; and CPU package power from RAPL's energy counters. Sent with heartbeats, published as metrics and shown on the node page; unmeasured sensors are absent, never zero. Observe-only, step 0 of docs/facility-aware-placement.md, which now describes the agent source and what these readings can and cannot say. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 19:49:08 +02:00
Ilker ManapandClaude Opus 5.5	8724dc027a	Plan facility-aware placement: rack power, temperature and cooling CI / test (push) Successful in 3m41s Details A design document for step 5 of resource allocation, deferred: location labels and rack spreading, power budgets with estimated then measured draw, thermal and cooling conditions with hysteresis, opt-in shedding through preemption, pluggable facility sources, and a rollout that is useful at each step and safe when data is missing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 19:25:31 +02:00
Ilker ManapandClaude Opus 5.5	e7a8105aa4	Log a preempted replica's stop with the grace it had Release / build (push) Successful in 1m7s Details Release / notes (push) Successful in 2s Details CI / test (push) Successful in 3m47s Details It said "stopping a drained replica without a replacement after=5m0s" when it had been given the 30-second preemption grace. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 18:41:43 +02:00
Ilker ManapandClaude Opus 5.5	d0e9813ce9	Of equally idle victims, try the smallest first Release / build (push) Successful in 1m11s Details Release / notes (push) Successful in 3s Details CI / test (push) Successful in 3m55s Details Idle times are unknown after a control plane restart, which left the choice between a 1-core and a 4-core replica to map order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 18:23:08 +02:00
Ilker ManapandClaude Opus 5.5	0e73f37128	Stop only the replicas that make the room Release / build (push) Successful in 1m4s Details CI / test (push) Successful in 4m32s Details Release / notes (push) Successful in 3s Details Candidates were added in order until the model fitted, and every one added was stopped, including those tried first that freed room on the wrong node. A set that works is now pruned to the replicas it needs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 18:21:58 +02:00
Ilker ManapandClaude Opus 5.5	78e41460f9	Scale and place by what a replica was measured to serve Release / build (push) Successful in 1m8s Details CI / test (push) Successful in 4m24s Details Release / notes (push) Successful in 3s Details Stored ctl bench results reach the reconciler: requests per replica past the measured SLO concurrency scale a model up, a replica is taken back only if the rest stay under 70% of theirs, and among profiles of equal priority the most measured tokens per GPU or core is tried first. ctl model target sizes a model for a throughput against the room there is. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 18:21:08 +02:00