Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3d3c96e422 | ||
|
|
623cdb5ba3 | ||
|
|
561667c9be | ||
|
|
cf0178bf05 | ||
|
|
f288f3c423 | ||
|
|
5cf0003e74 | ||
|
|
d10fad064e | ||
|
|
cd10e086c8 | ||
|
|
8724dc027a | ||
|
|
e7a8105aa4 | ||
|
|
d0e9813ce9 | ||
|
|
0e73f37128 | ||
|
|
78e41460f9 |
No files matched your search
@@ -93,6 +93,24 @@ jobs:
|
||||
./dist/llmgw-${version}-linux-amd64 --version
|
||||
cat dist/checksums.txt
|
||||
|
||||
# Signed when the repository has a signing key, so a machine that trusts
|
||||
# it installs nothing the server alone could have put there. Without the
|
||||
# secret nothing is signed and releases are checked on their checksums,
|
||||
# as before; a build that trusts a key refuses those, so the secret has
|
||||
# to be in place before trusted.pub is filled in.
|
||||
- name: Sign the checksums
|
||||
env:
|
||||
RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
|
||||
if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
|
||||
echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
|
||||
exit 0
|
||||
fi
|
||||
./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
|
||||
echo "signed: dist/checksums.txt.sig"
|
||||
|
||||
- name: Attach to the release
|
||||
env:
|
||||
TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
|
||||
+120
-4
@@ -22,8 +22,70 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
Changes since v0.10.4, published as release candidates (v0.10.5-rc.N) until
|
||||
there are enough for v0.10.5.
|
||||
Changes since v0.10.5, published as release candidates (v0.10.6-rc.N) until
|
||||
there are enough for v0.10.6.
|
||||
|
||||
### Capacity from the admin pages
|
||||
|
||||
What v0.10.5 made settable from the command line is now on the pages too:
|
||||
|
||||
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
|
||||
cores and memory beside what it holds, with a form to set, change or remove
|
||||
one. It goes through the same checks as `llmgw ctl quota`.
|
||||
- **A model's card** on the mirror page sets its class, its owner (when there
|
||||
are quotas to count against), and the memory and cores a replica reserves,
|
||||
beside its replica counts. Only what changed is applied, and each change is
|
||||
said.
|
||||
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
|
||||
the headroom API count the replicas a model could get by stopping those its
|
||||
class outranks, separately from free room, and name whose they would be. A
|
||||
critical model's card no longer refuses a replica that preemption would make
|
||||
room for.
|
||||
|
||||
### Displacement: a higher band is not refused while a lower one waits
|
||||
|
||||
Each band's queue was bounded, and nothing bounded them together, so a full
|
||||
gateway turned an interactive request away while batch requests sat queued
|
||||
behind it. `admission.queue_capacity_total` now bounds everything queued —
|
||||
by default half of every band's capacity together. Reached, a request is not
|
||||
refused while a lower band has any queued: the newest request of the lowest
|
||||
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
|
||||
and returns. The newest, because the oldest have waited longest and are
|
||||
nearest their TTL. Counted as `displaced` in the admission rejection metric.
|
||||
|
||||
### Releases can be signed
|
||||
|
||||
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
|
||||
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
|
||||
public key in `internal/release/trusted.pub`, every binary built after refuses
|
||||
a release not signed with it, wherever it installs one. Off until then, and
|
||||
docs/install.md says in which order to turn it on.
|
||||
|
||||
### Restarts borrow a lower class's room
|
||||
|
||||
A restart places the replacement beside the replica it replaces, and where
|
||||
there is no free room it now makes some the way any placement does: by
|
||||
stopping a replica its class outranks, which gets its room back once the old
|
||||
replica has gone. The in-place restart — the model unserved while its
|
||||
replacement loads — is left for when nothing of a lower class is running. On a
|
||||
cluster where every model is standard, that is every full one: mark the models
|
||||
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
|
||||
|
||||
## v0.10.5 — 2026-09-30
|
||||
|
||||
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
|
||||
format check and has no binaries.
|
||||
|
||||
After upgrading:
|
||||
|
||||
- **Declare what CPU models need.** A CPU model added from now on must say its
|
||||
memory; one stored before this still loads and is flagged in `ctl models`
|
||||
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
|
||||
are then reserved per replica and become the engine's thread count, for
|
||||
replicas started after the change: `llmgw ctl restart --model MODEL`.
|
||||
- **Upgrade the mirror on its own host**, now with one command:
|
||||
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
|
||||
mirror older than this). `llmgw ctl upgrade` says so at the end.
|
||||
|
||||
### CPU cores are reserved, and memory has to be declared
|
||||
|
||||
@@ -70,9 +132,12 @@ these rules:
|
||||
loaded, nor one spanning several nodes.
|
||||
- Idle on-demand replicas go first, then what lower classes grew into, then
|
||||
the same within the class, then lower classes' minimums; the longest unused
|
||||
first within each.
|
||||
first within each, and of equally idle ones the smallest.
|
||||
- The set is tried on a copy of the inventory first, at most three replicas,
|
||||
and nothing is stopped unless it makes the room.
|
||||
and nothing is stopped unless it makes the room. Of a set that does, each
|
||||
replica whose stopping turns out not to be needed is left running, so an
|
||||
idle replica tried first on the wrong node is not stopped with the one that
|
||||
made the room.
|
||||
|
||||
A replica stopped this way is drained and given 30 seconds to finish what it
|
||||
is serving. Its model is held back for up to five minutes, or until the model
|
||||
@@ -106,6 +171,57 @@ two is refused. Changes take effect within half a minute. Metrics:
|
||||
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
|
||||
by scope and resource; alert: CapacityAtLimit.
|
||||
|
||||
### Sized by what a replica was measured to serve
|
||||
|
||||
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
|
||||
replica served while meeting its SLO, and its throughput there — now drives
|
||||
decisions rather than only being printed:
|
||||
|
||||
- **Scaling up** when requests per replica pass what was measured, not only
|
||||
when the engines' queues fill: an engine takes requests into its slots long
|
||||
after latency has gone.
|
||||
- **Scaling down** only when the replicas left would stay under 70% of what
|
||||
they were measured to hold, so a replica taken back does not put the rest
|
||||
straight over.
|
||||
- **Choosing a profile.** Among profiles of equal priority, the one measured
|
||||
to serve the most per GPU — or per core — is tried first. Priority set by an
|
||||
operator still wins.
|
||||
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
|
||||
replicas that takes, how many run, and whether the rest fit, with the reason
|
||||
when they do not; `--apply` sets it if they do. Headroom reports what one
|
||||
more replica would serve.
|
||||
|
||||
A model never measured behaves exactly as before. Measurements are read from
|
||||
the store every half minute.
|
||||
|
||||
### Upgrading a machine the control plane does not reach
|
||||
|
||||
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
|
||||
the mirror's host above all, whose service runs unprivileged and cannot replace
|
||||
its own binary as an agent does. The release's checksums are checked, the old
|
||||
binary kept, the unit restarted and watched until it stays up.
|
||||
|
||||
### Nodes report their own heat and power
|
||||
|
||||
The first step of facility-aware placement (docs/facility-aware-placement.md),
|
||||
observe-only: nothing is decided on these yet. Every agent reads, at most every
|
||||
ten seconds, and sends with its heartbeat:
|
||||
|
||||
- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
|
||||
it, power draw and the enforced limit, and what is holding the clocks down —
|
||||
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
|
||||
at registration; it is now current.
|
||||
- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
|
||||
the package temperature, and the board's inlet air sensor where it has one.
|
||||
- **CPU power** from RAPL's energy counters.
|
||||
|
||||
A sensor a machine does not have is not reported, never shown as zero. New
|
||||
metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
|
||||
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
|
||||
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
|
||||
`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
|
||||
table of temperature, power against its limit, and what is throttling it.
|
||||
|
||||
### The mirror page in tabs
|
||||
|
||||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||||
|
||||
+295
-157
@@ -90,7 +90,7 @@ func (x ShardStatus_State) Number() protoreflect.EnumNumber {
|
||||
|
||||
// Deprecated: Use ShardStatus_State.Descriptor instead.
|
||||
func (ShardStatus_State) EnumDescriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{19, 0}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{20, 0}
|
||||
}
|
||||
|
||||
// AgentMessage is anything a node tells the control plane.
|
||||
@@ -1415,8 +1415,15 @@ type NodeLoad struct {
|
||||
MemoryUsedBytes int64 `protobuf:"varint,6,opt,name=memory_used_bytes,json=memoryUsedBytes,proto3" json:"memory_used_bytes,omitempty"`
|
||||
MemoryAvailableBytes int64 `protobuf:"varint,7,opt,name=memory_available_bytes,json=memoryAvailableBytes,proto3" json:"memory_available_bytes,omitempty"`
|
||||
Shards []*ShardLoad `protobuf:"bytes,8,rep,name=shards,proto3" json:"shards,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
// What the machine's own sensors say; zero is not measured. cpu_temp_c is
|
||||
// the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
|
||||
// inlet_temp_c the board's inlet or ambient sensor where it has one.
|
||||
CpuTempC float64 `protobuf:"fixed64,9,opt,name=cpu_temp_c,json=cpuTempC,proto3" json:"cpu_temp_c,omitempty"`
|
||||
CpuPowerW float64 `protobuf:"fixed64,10,opt,name=cpu_power_w,json=cpuPowerW,proto3" json:"cpu_power_w,omitempty"`
|
||||
InletTempC float64 `protobuf:"fixed64,11,opt,name=inlet_temp_c,json=inletTempC,proto3" json:"inlet_temp_c,omitempty"`
|
||||
Gpus []*GPUSensor `protobuf:"bytes,12,rep,name=gpus,proto3" json:"gpus,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *NodeLoad) Reset() {
|
||||
@@ -1505,6 +1512,121 @@ func (x *NodeLoad) GetShards() []*ShardLoad {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *NodeLoad) GetCpuTempC() float64 {
|
||||
if x != nil {
|
||||
return x.CpuTempC
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *NodeLoad) GetCpuPowerW() float64 {
|
||||
if x != nil {
|
||||
return x.CpuPowerW
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *NodeLoad) GetInletTempC() float64 {
|
||||
if x != nil {
|
||||
return x.InletTempC
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *NodeLoad) GetGpus() []*GPUSensor {
|
||||
if x != nil {
|
||||
return x.Gpus
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GPUSensor is one card's heat, power, and what is holding it back.
|
||||
type GPUSensor struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Index int32 `protobuf:"varint,1,opt,name=index,proto3" json:"index,omitempty"`
|
||||
TempC float64 `protobuf:"fixed64,2,opt,name=temp_c,json=tempC,proto3" json:"temp_c,omitempty"`
|
||||
MemoryTempC float64 `protobuf:"fixed64,3,opt,name=memory_temp_c,json=memoryTempC,proto3" json:"memory_temp_c,omitempty"`
|
||||
PowerW float64 `protobuf:"fixed64,4,opt,name=power_w,json=powerW,proto3" json:"power_w,omitempty"`
|
||||
PowerLimitW float64 `protobuf:"fixed64,5,opt,name=power_limit_w,json=powerLimitW,proto3" json:"power_limit_w,omitempty"`
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *GPUSensor) Reset() {
|
||||
*x = GPUSensor{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[10]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *GPUSensor) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*GPUSensor) ProtoMessage() {}
|
||||
|
||||
func (x *GPUSensor) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[10]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use GPUSensor.ProtoReflect.Descriptor instead.
|
||||
func (*GPUSensor) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{10}
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetIndex() int32 {
|
||||
if x != nil {
|
||||
return x.Index
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetTempC() float64 {
|
||||
if x != nil {
|
||||
return x.TempC
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetMemoryTempC() float64 {
|
||||
if x != nil {
|
||||
return x.MemoryTempC
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetPowerW() float64 {
|
||||
if x != nil {
|
||||
return x.PowerW
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetPowerLimitW() float64 {
|
||||
if x != nil {
|
||||
return x.PowerLimitW
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *GPUSensor) GetThrottle() []string {
|
||||
if x != nil {
|
||||
return x.Throttle
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
type ShardLoad struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -1520,7 +1642,7 @@ type ShardLoad struct {
|
||||
|
||||
func (x *ShardLoad) Reset() {
|
||||
*x = ShardLoad{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[10]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[11]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -1532,7 +1654,7 @@ func (x *ShardLoad) String() string {
|
||||
func (*ShardLoad) ProtoMessage() {}
|
||||
|
||||
func (x *ShardLoad) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[10]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[11]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -1545,7 +1667,7 @@ func (x *ShardLoad) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use ShardLoad.ProtoReflect.Descriptor instead.
|
||||
func (*ShardLoad) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{10}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{11}
|
||||
}
|
||||
|
||||
func (x *ShardLoad) GetShardId() string {
|
||||
@@ -1613,7 +1735,7 @@ type IBPort struct {
|
||||
|
||||
func (x *IBPort) Reset() {
|
||||
*x = IBPort{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[11]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[12]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -1625,7 +1747,7 @@ func (x *IBPort) String() string {
|
||||
func (*IBPort) ProtoMessage() {}
|
||||
|
||||
func (x *IBPort) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[11]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[12]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -1638,7 +1760,7 @@ func (x *IBPort) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use IBPort.ProtoReflect.Descriptor instead.
|
||||
func (*IBPort) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{11}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{12}
|
||||
}
|
||||
|
||||
func (x *IBPort) GetHca() string {
|
||||
@@ -1781,7 +1903,7 @@ type SubnetManager struct {
|
||||
|
||||
func (x *SubnetManager) Reset() {
|
||||
*x = SubnetManager{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[12]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[13]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -1793,7 +1915,7 @@ func (x *SubnetManager) String() string {
|
||||
func (*SubnetManager) ProtoMessage() {}
|
||||
|
||||
func (x *SubnetManager) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[12]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[13]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -1806,7 +1928,7 @@ func (x *SubnetManager) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use SubnetManager.ProtoReflect.Descriptor instead.
|
||||
func (*SubnetManager) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{12}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{13}
|
||||
}
|
||||
|
||||
func (x *SubnetManager) GetGuid() string {
|
||||
@@ -1849,7 +1971,7 @@ type IMEX struct {
|
||||
|
||||
func (x *IMEX) Reset() {
|
||||
*x = IMEX{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[13]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[14]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -1861,7 +1983,7 @@ func (x *IMEX) String() string {
|
||||
func (*IMEX) ProtoMessage() {}
|
||||
|
||||
func (x *IMEX) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[13]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[14]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -1874,7 +1996,7 @@ func (x *IMEX) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use IMEX.ProtoReflect.Descriptor instead.
|
||||
func (*IMEX) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{13}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{14}
|
||||
}
|
||||
|
||||
func (x *IMEX) GetState() string {
|
||||
@@ -1906,7 +2028,7 @@ type GPUFabricState struct {
|
||||
|
||||
func (x *GPUFabricState) Reset() {
|
||||
*x = GPUFabricState{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[14]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[15]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -1918,7 +2040,7 @@ func (x *GPUFabricState) String() string {
|
||||
func (*GPUFabricState) ProtoMessage() {}
|
||||
|
||||
func (x *GPUFabricState) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[14]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[15]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -1931,7 +2053,7 @@ func (x *GPUFabricState) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use GPUFabricState.ProtoReflect.Descriptor instead.
|
||||
func (*GPUFabricState) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{14}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{15}
|
||||
}
|
||||
|
||||
func (x *GPUFabricState) GetIndex() int32 {
|
||||
@@ -2038,7 +2160,7 @@ type Heartbeat struct {
|
||||
|
||||
func (x *Heartbeat) Reset() {
|
||||
*x = Heartbeat{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[15]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[16]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2050,7 +2172,7 @@ func (x *Heartbeat) String() string {
|
||||
func (*Heartbeat) ProtoMessage() {}
|
||||
|
||||
func (x *Heartbeat) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[15]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[16]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2063,7 +2185,7 @@ func (x *Heartbeat) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use Heartbeat.ProtoReflect.Descriptor instead.
|
||||
func (*Heartbeat) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{15}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{16}
|
||||
}
|
||||
|
||||
func (x *Heartbeat) GetNodeId() string {
|
||||
@@ -2211,7 +2333,7 @@ type Provision struct {
|
||||
|
||||
func (x *Provision) Reset() {
|
||||
*x = Provision{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[16]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[17]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2223,7 +2345,7 @@ func (x *Provision) String() string {
|
||||
func (*Provision) ProtoMessage() {}
|
||||
|
||||
func (x *Provision) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[16]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[17]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2236,7 +2358,7 @@ func (x *Provision) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use Provision.ProtoReflect.Descriptor instead.
|
||||
func (*Provision) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{16}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{17}
|
||||
}
|
||||
|
||||
func (x *Provision) GetEngines() []*EngineWant {
|
||||
@@ -2257,7 +2379,7 @@ type EngineWant struct {
|
||||
|
||||
func (x *EngineWant) Reset() {
|
||||
*x = EngineWant{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[17]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[18]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2269,7 +2391,7 @@ func (x *EngineWant) String() string {
|
||||
func (*EngineWant) ProtoMessage() {}
|
||||
|
||||
func (x *EngineWant) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[17]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[18]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2282,7 +2404,7 @@ func (x *EngineWant) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use EngineWant.ProtoReflect.Descriptor instead.
|
||||
func (*EngineWant) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{17}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{18}
|
||||
}
|
||||
|
||||
func (x *EngineWant) GetEngine() string {
|
||||
@@ -2315,7 +2437,7 @@ type ProvisionStatus struct {
|
||||
|
||||
func (x *ProvisionStatus) Reset() {
|
||||
*x = ProvisionStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[18]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[19]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2327,7 +2449,7 @@ func (x *ProvisionStatus) String() string {
|
||||
func (*ProvisionStatus) ProtoMessage() {}
|
||||
|
||||
func (x *ProvisionStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[18]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[19]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2340,7 +2462,7 @@ func (x *ProvisionStatus) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use ProvisionStatus.ProtoReflect.Descriptor instead.
|
||||
func (*ProvisionStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{18}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{19}
|
||||
}
|
||||
|
||||
func (x *ProvisionStatus) GetEngine() string {
|
||||
@@ -2404,7 +2526,7 @@ type ShardStatus struct {
|
||||
|
||||
func (x *ShardStatus) Reset() {
|
||||
*x = ShardStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[19]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[20]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2416,7 +2538,7 @@ func (x *ShardStatus) String() string {
|
||||
func (*ShardStatus) ProtoMessage() {}
|
||||
|
||||
func (x *ShardStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[19]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[20]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2429,7 +2551,7 @@ func (x *ShardStatus) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use ShardStatus.ProtoReflect.Descriptor instead.
|
||||
func (*ShardStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{19}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{20}
|
||||
}
|
||||
|
||||
func (x *ShardStatus) GetNodeId() string {
|
||||
@@ -2503,7 +2625,7 @@ type FetchProgress struct {
|
||||
|
||||
func (x *FetchProgress) Reset() {
|
||||
*x = FetchProgress{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[20]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[21]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2515,7 +2637,7 @@ func (x *FetchProgress) String() string {
|
||||
func (*FetchProgress) ProtoMessage() {}
|
||||
|
||||
func (x *FetchProgress) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[20]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[21]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2528,7 +2650,7 @@ func (x *FetchProgress) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use FetchProgress.ProtoReflect.Descriptor instead.
|
||||
func (*FetchProgress) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{20}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{21}
|
||||
}
|
||||
|
||||
func (x *FetchProgress) GetSource() string {
|
||||
@@ -2591,7 +2713,7 @@ type StartShard struct {
|
||||
|
||||
func (x *StartShard) Reset() {
|
||||
*x = StartShard{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[21]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[22]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2603,7 +2725,7 @@ func (x *StartShard) String() string {
|
||||
func (*StartShard) ProtoMessage() {}
|
||||
|
||||
func (x *StartShard) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[21]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[22]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2616,7 +2738,7 @@ func (x *StartShard) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use StartShard.ProtoReflect.Descriptor instead.
|
||||
func (*StartShard) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{21}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{22}
|
||||
}
|
||||
|
||||
func (x *StartShard) GetReplicaId() string {
|
||||
@@ -2757,7 +2879,7 @@ type EngineProfile struct {
|
||||
|
||||
func (x *EngineProfile) Reset() {
|
||||
*x = EngineProfile{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[22]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[23]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2769,7 +2891,7 @@ func (x *EngineProfile) String() string {
|
||||
func (*EngineProfile) ProtoMessage() {}
|
||||
|
||||
func (x *EngineProfile) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[22]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[23]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2782,7 +2904,7 @@ func (x *EngineProfile) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use EngineProfile.ProtoReflect.Descriptor instead.
|
||||
func (*EngineProfile) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{22}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{23}
|
||||
}
|
||||
|
||||
func (x *EngineProfile) GetName() string {
|
||||
@@ -2878,7 +3000,7 @@ type RemoveWeights struct {
|
||||
|
||||
func (x *RemoveWeights) Reset() {
|
||||
*x = RemoveWeights{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[23]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[24]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2890,7 +3012,7 @@ func (x *RemoveWeights) String() string {
|
||||
func (*RemoveWeights) ProtoMessage() {}
|
||||
|
||||
func (x *RemoveWeights) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[23]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[24]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2903,7 +3025,7 @@ func (x *RemoveWeights) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use RemoveWeights.ProtoReflect.Descriptor instead.
|
||||
func (*RemoveWeights) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{23}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{24}
|
||||
}
|
||||
|
||||
func (x *RemoveWeights) GetPaths() []string {
|
||||
@@ -2923,7 +3045,7 @@ type StopShard struct {
|
||||
|
||||
func (x *StopShard) Reset() {
|
||||
*x = StopShard{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[24]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[25]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2935,7 +3057,7 @@ func (x *StopShard) String() string {
|
||||
func (*StopShard) ProtoMessage() {}
|
||||
|
||||
func (x *StopShard) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[24]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[25]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2948,7 +3070,7 @@ func (x *StopShard) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use StopShard.ProtoReflect.Descriptor instead.
|
||||
func (*StopShard) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{24}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{25}
|
||||
}
|
||||
|
||||
func (x *StopShard) GetShardId() string {
|
||||
@@ -2968,7 +3090,7 @@ type DrainShard struct {
|
||||
|
||||
func (x *DrainShard) Reset() {
|
||||
*x = DrainShard{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[25]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[26]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -2980,7 +3102,7 @@ func (x *DrainShard) String() string {
|
||||
func (*DrainShard) ProtoMessage() {}
|
||||
|
||||
func (x *DrainShard) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[25]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[26]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -2993,7 +3115,7 @@ func (x *DrainShard) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use DrainShard.ProtoReflect.Descriptor instead.
|
||||
func (*DrainShard) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{25}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{26}
|
||||
}
|
||||
|
||||
func (x *DrainShard) GetShardId() string {
|
||||
@@ -3015,7 +3137,7 @@ type Ack struct {
|
||||
|
||||
func (x *Ack) Reset() {
|
||||
*x = Ack{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[26]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[27]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3027,7 +3149,7 @@ func (x *Ack) String() string {
|
||||
func (*Ack) ProtoMessage() {}
|
||||
|
||||
func (x *Ack) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[26]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[27]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3040,7 +3162,7 @@ func (x *Ack) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use Ack.ProtoReflect.Descriptor instead.
|
||||
func (*Ack) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{26}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{27}
|
||||
}
|
||||
|
||||
func (x *Ack) GetMessageId() string {
|
||||
@@ -3084,7 +3206,7 @@ type LogsRequest struct {
|
||||
|
||||
func (x *LogsRequest) Reset() {
|
||||
*x = LogsRequest{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[27]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[28]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3096,7 +3218,7 @@ func (x *LogsRequest) String() string {
|
||||
func (*LogsRequest) ProtoMessage() {}
|
||||
|
||||
func (x *LogsRequest) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[27]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[28]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3109,7 +3231,7 @@ func (x *LogsRequest) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use LogsRequest.ProtoReflect.Descriptor instead.
|
||||
func (*LogsRequest) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{27}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{28}
|
||||
}
|
||||
|
||||
func (x *LogsRequest) GetRequestId() string {
|
||||
@@ -3149,7 +3271,7 @@ type LogsReply struct {
|
||||
|
||||
func (x *LogsReply) Reset() {
|
||||
*x = LogsReply{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[28]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[29]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3161,7 +3283,7 @@ func (x *LogsReply) String() string {
|
||||
func (*LogsReply) ProtoMessage() {}
|
||||
|
||||
func (x *LogsReply) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[28]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[29]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3174,7 +3296,7 @@ func (x *LogsReply) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use LogsReply.ProtoReflect.Descriptor instead.
|
||||
func (*LogsReply) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{28}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{29}
|
||||
}
|
||||
|
||||
func (x *LogsReply) GetRequestId() string {
|
||||
@@ -3220,7 +3342,7 @@ type FabricScanRequest struct {
|
||||
|
||||
func (x *FabricScanRequest) Reset() {
|
||||
*x = FabricScanRequest{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[29]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[30]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3232,7 +3354,7 @@ func (x *FabricScanRequest) String() string {
|
||||
func (*FabricScanRequest) ProtoMessage() {}
|
||||
|
||||
func (x *FabricScanRequest) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[29]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[30]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3245,7 +3367,7 @@ func (x *FabricScanRequest) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use FabricScanRequest.ProtoReflect.Descriptor instead.
|
||||
func (*FabricScanRequest) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{29}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{30}
|
||||
}
|
||||
|
||||
func (x *FabricScanRequest) GetRequestId() string {
|
||||
@@ -3284,7 +3406,7 @@ type FabricScanReply struct {
|
||||
|
||||
func (x *FabricScanReply) Reset() {
|
||||
*x = FabricScanReply{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[30]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[31]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3296,7 +3418,7 @@ func (x *FabricScanReply) String() string {
|
||||
func (*FabricScanReply) ProtoMessage() {}
|
||||
|
||||
func (x *FabricScanReply) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[30]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[31]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3309,7 +3431,7 @@ func (x *FabricScanReply) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use FabricScanReply.ProtoReflect.Descriptor instead.
|
||||
func (*FabricScanReply) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{30}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{31}
|
||||
}
|
||||
|
||||
func (x *FabricScanReply) GetRequestId() string {
|
||||
@@ -3365,7 +3487,7 @@ type FabricTestRequest struct {
|
||||
|
||||
func (x *FabricTestRequest) Reset() {
|
||||
*x = FabricTestRequest{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[31]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[32]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3377,7 +3499,7 @@ func (x *FabricTestRequest) String() string {
|
||||
func (*FabricTestRequest) ProtoMessage() {}
|
||||
|
||||
func (x *FabricTestRequest) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[31]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[32]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3390,7 +3512,7 @@ func (x *FabricTestRequest) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use FabricTestRequest.ProtoReflect.Descriptor instead.
|
||||
func (*FabricTestRequest) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{31}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{32}
|
||||
}
|
||||
|
||||
func (x *FabricTestRequest) GetRequestId() string {
|
||||
@@ -3459,7 +3581,7 @@ type FabricTestReply struct {
|
||||
|
||||
func (x *FabricTestReply) Reset() {
|
||||
*x = FabricTestReply{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[32]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[33]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3471,7 +3593,7 @@ func (x *FabricTestReply) String() string {
|
||||
func (*FabricTestReply) ProtoMessage() {}
|
||||
|
||||
func (x *FabricTestReply) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[32]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[33]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3484,7 +3606,7 @@ func (x *FabricTestReply) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use FabricTestReply.ProtoReflect.Descriptor instead.
|
||||
func (*FabricTestReply) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{32}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{33}
|
||||
}
|
||||
|
||||
func (x *FabricTestReply) GetRequestId() string {
|
||||
@@ -3544,7 +3666,7 @@ type UpgradeAgent struct {
|
||||
|
||||
func (x *UpgradeAgent) Reset() {
|
||||
*x = UpgradeAgent{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[33]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[34]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3556,7 +3678,7 @@ func (x *UpgradeAgent) String() string {
|
||||
func (*UpgradeAgent) ProtoMessage() {}
|
||||
|
||||
func (x *UpgradeAgent) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[33]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[34]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3569,7 +3691,7 @@ func (x *UpgradeAgent) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use UpgradeAgent.ProtoReflect.Descriptor instead.
|
||||
func (*UpgradeAgent) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{33}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{34}
|
||||
}
|
||||
|
||||
func (x *UpgradeAgent) GetRequestId() string {
|
||||
@@ -3606,7 +3728,7 @@ type UpgradeReply struct {
|
||||
|
||||
func (x *UpgradeReply) Reset() {
|
||||
*x = UpgradeReply{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[34]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[35]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -3618,7 +3740,7 @@ func (x *UpgradeReply) String() string {
|
||||
func (*UpgradeReply) ProtoMessage() {}
|
||||
|
||||
func (x *UpgradeReply) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[34]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[35]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -3631,7 +3753,7 @@ func (x *UpgradeReply) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use UpgradeReply.ProtoReflect.Descriptor instead.
|
||||
func (*UpgradeReply) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{34}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{35}
|
||||
}
|
||||
|
||||
func (x *UpgradeReply) GetRequestId() string {
|
||||
@@ -3786,7 +3908,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
|
||||
"\n" +
|
||||
"size_bytes\x18\x02 \x01(\x03R\tsizeBytes\x12\x16\n" +
|
||||
"\x06shared\x18\x03 \x01(\bR\x06shared\"\x9b\x02\n" +
|
||||
"\x06shared\x18\x03 \x01(\bR\x06shared\"\xa4\x03\n" +
|
||||
"\bNodeLoad\x12\x1f\n" +
|
||||
"\vcpu_percent\x18\x01 \x01(\x01R\n" +
|
||||
"cpuPercent\x12\x1b\n" +
|
||||
@@ -3796,7 +3918,21 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x06load15\x18\x05 \x01(\x01R\x06load15\x12*\n" +
|
||||
"\x11memory_used_bytes\x18\x06 \x01(\x03R\x0fmemoryUsedBytes\x124\n" +
|
||||
"\x16memory_available_bytes\x18\a \x01(\x03R\x14memoryAvailableBytes\x12+\n" +
|
||||
"\x06shards\x18\b \x03(\v2\x13.llmgw.v1.ShardLoadR\x06shards\"m\n" +
|
||||
"\x06shards\x18\b \x03(\v2\x13.llmgw.v1.ShardLoadR\x06shards\x12\x1c\n" +
|
||||
"\n" +
|
||||
"cpu_temp_c\x18\t \x01(\x01R\bcpuTempC\x12\x1e\n" +
|
||||
"\vcpu_power_w\x18\n" +
|
||||
" \x01(\x01R\tcpuPowerW\x12 \n" +
|
||||
"\finlet_temp_c\x18\v \x01(\x01R\n" +
|
||||
"inletTempC\x12'\n" +
|
||||
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xb5\x01\n" +
|
||||
"\tGPUSensor\x12\x14\n" +
|
||||
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
|
||||
"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
|
||||
"\rmemory_temp_c\x18\x03 \x01(\x01R\vmemoryTempC\x12\x17\n" +
|
||||
"\apower_w\x18\x04 \x01(\x01R\x06powerW\x12\"\n" +
|
||||
"\rpower_limit_w\x18\x05 \x01(\x01R\vpowerLimitW\x12\x1a\n" +
|
||||
"\bthrottle\x18\x06 \x03(\tR\bthrottle\"m\n" +
|
||||
"\tShardLoad\x12\x19\n" +
|
||||
"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
|
||||
"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
|
||||
@@ -4026,7 +4162,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 41)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 42)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -4039,91 +4175,93 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*FabricAttachment)(nil), // 8: llmgw.v1.FabricAttachment
|
||||
(*LocalWeights)(nil), // 9: llmgw.v1.LocalWeights
|
||||
(*NodeLoad)(nil), // 10: llmgw.v1.NodeLoad
|
||||
(*ShardLoad)(nil), // 11: llmgw.v1.ShardLoad
|
||||
(*IBPort)(nil), // 12: llmgw.v1.IBPort
|
||||
(*SubnetManager)(nil), // 13: llmgw.v1.SubnetManager
|
||||
(*IMEX)(nil), // 14: llmgw.v1.IMEX
|
||||
(*GPUFabricState)(nil), // 15: llmgw.v1.GPUFabricState
|
||||
(*Heartbeat)(nil), // 16: llmgw.v1.Heartbeat
|
||||
(*Provision)(nil), // 17: llmgw.v1.Provision
|
||||
(*EngineWant)(nil), // 18: llmgw.v1.EngineWant
|
||||
(*ProvisionStatus)(nil), // 19: llmgw.v1.ProvisionStatus
|
||||
(*ShardStatus)(nil), // 20: llmgw.v1.ShardStatus
|
||||
(*FetchProgress)(nil), // 21: llmgw.v1.FetchProgress
|
||||
(*StartShard)(nil), // 22: llmgw.v1.StartShard
|
||||
(*EngineProfile)(nil), // 23: llmgw.v1.EngineProfile
|
||||
(*RemoveWeights)(nil), // 24: llmgw.v1.RemoveWeights
|
||||
(*StopShard)(nil), // 25: llmgw.v1.StopShard
|
||||
(*DrainShard)(nil), // 26: llmgw.v1.DrainShard
|
||||
(*Ack)(nil), // 27: llmgw.v1.Ack
|
||||
(*LogsRequest)(nil), // 28: llmgw.v1.LogsRequest
|
||||
(*LogsReply)(nil), // 29: llmgw.v1.LogsReply
|
||||
(*FabricScanRequest)(nil), // 30: llmgw.v1.FabricScanRequest
|
||||
(*FabricScanReply)(nil), // 31: llmgw.v1.FabricScanReply
|
||||
(*FabricTestRequest)(nil), // 32: llmgw.v1.FabricTestRequest
|
||||
(*FabricTestReply)(nil), // 33: llmgw.v1.FabricTestReply
|
||||
(*UpgradeAgent)(nil), // 34: llmgw.v1.UpgradeAgent
|
||||
(*UpgradeReply)(nil), // 35: llmgw.v1.UpgradeReply
|
||||
nil, // 36: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 37: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 38: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 39: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 40: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 41: llmgw.v1.EngineProfile.EnvEntry
|
||||
(*GPUSensor)(nil), // 11: llmgw.v1.GPUSensor
|
||||
(*ShardLoad)(nil), // 12: llmgw.v1.ShardLoad
|
||||
(*IBPort)(nil), // 13: llmgw.v1.IBPort
|
||||
(*SubnetManager)(nil), // 14: llmgw.v1.SubnetManager
|
||||
(*IMEX)(nil), // 15: llmgw.v1.IMEX
|
||||
(*GPUFabricState)(nil), // 16: llmgw.v1.GPUFabricState
|
||||
(*Heartbeat)(nil), // 17: llmgw.v1.Heartbeat
|
||||
(*Provision)(nil), // 18: llmgw.v1.Provision
|
||||
(*EngineWant)(nil), // 19: llmgw.v1.EngineWant
|
||||
(*ProvisionStatus)(nil), // 20: llmgw.v1.ProvisionStatus
|
||||
(*ShardStatus)(nil), // 21: llmgw.v1.ShardStatus
|
||||
(*FetchProgress)(nil), // 22: llmgw.v1.FetchProgress
|
||||
(*StartShard)(nil), // 23: llmgw.v1.StartShard
|
||||
(*EngineProfile)(nil), // 24: llmgw.v1.EngineProfile
|
||||
(*RemoveWeights)(nil), // 25: llmgw.v1.RemoveWeights
|
||||
(*StopShard)(nil), // 26: llmgw.v1.StopShard
|
||||
(*DrainShard)(nil), // 27: llmgw.v1.DrainShard
|
||||
(*Ack)(nil), // 28: llmgw.v1.Ack
|
||||
(*LogsRequest)(nil), // 29: llmgw.v1.LogsRequest
|
||||
(*LogsReply)(nil), // 30: llmgw.v1.LogsReply
|
||||
(*FabricScanRequest)(nil), // 31: llmgw.v1.FabricScanRequest
|
||||
(*FabricScanReply)(nil), // 32: llmgw.v1.FabricScanReply
|
||||
(*FabricTestRequest)(nil), // 33: llmgw.v1.FabricTestRequest
|
||||
(*FabricTestReply)(nil), // 34: llmgw.v1.FabricTestReply
|
||||
(*UpgradeAgent)(nil), // 35: llmgw.v1.UpgradeAgent
|
||||
(*UpgradeReply)(nil), // 36: llmgw.v1.UpgradeReply
|
||||
nil, // 37: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 38: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 39: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 40: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 41: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 42: llmgw.v1.EngineProfile.EnvEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
3, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
16, // 1: llmgw.v1.AgentMessage.heartbeat:type_name -> llmgw.v1.Heartbeat
|
||||
20, // 2: llmgw.v1.AgentMessage.shard:type_name -> llmgw.v1.ShardStatus
|
||||
29, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
|
||||
31, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
|
||||
33, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
|
||||
35, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
|
||||
22, // 7: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
25, // 8: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
26, // 9: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
27, // 10: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
24, // 11: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
17, // 12: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
28, // 13: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
30, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
32, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
34, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
36, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
17, // 1: llmgw.v1.AgentMessage.heartbeat:type_name -> llmgw.v1.Heartbeat
|
||||
21, // 2: llmgw.v1.AgentMessage.shard:type_name -> llmgw.v1.ShardStatus
|
||||
30, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
|
||||
32, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
|
||||
34, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
|
||||
36, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
|
||||
23, // 7: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
26, // 8: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
27, // 9: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
28, // 10: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
25, // 11: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
18, // 12: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
29, // 13: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
31, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
33, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
35, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
37, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
7, // 18: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
8, // 19: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
37, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
38, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
6, // 21: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
9, // 22: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
5, // 23: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
4, // 24: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
14, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
15, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 26: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
38, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
11, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
39, // 29: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
13, // 30: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
5, // 31: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 32: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
40, // 33: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
19, // 34: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
10, // 35: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
12, // 36: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
8, // 37: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
15, // 38: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
14, // 39: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
18, // 40: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
0, // 41: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
21, // 42: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
23, // 43: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
41, // 44: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
1, // 45: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
2, // 46: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
46, // [46:47] is the sub-list for method output_type
|
||||
45, // [45:46] is the sub-list for method input_type
|
||||
45, // [45:45] is the sub-list for extension type_name
|
||||
45, // [45:45] is the sub-list for extension extendee
|
||||
0, // [0:45] is the sub-list for field type_name
|
||||
39, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
12, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
11, // 29: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
40, // 30: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
14, // 31: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
5, // 32: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 33: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
41, // 34: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
20, // 35: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
10, // 36: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
13, // 37: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
8, // 38: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
16, // 39: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
15, // 40: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
19, // 41: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
0, // 42: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
22, // 43: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
24, // 44: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
42, // 45: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
1, // 46: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
2, // 47: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
47, // [47:48] is the sub-list for method output_type
|
||||
46, // [46:47] is the sub-list for method input_type
|
||||
46, // [46:46] is the sub-list for extension type_name
|
||||
46, // [46:46] is the sub-list for extension extendee
|
||||
0, // [0:46] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -4158,7 +4296,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 41,
|
||||
NumMessages: 42,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -285,6 +285,25 @@ message NodeLoad {
|
||||
int64 memory_used_bytes = 6;
|
||||
int64 memory_available_bytes = 7;
|
||||
repeated ShardLoad shards = 8;
|
||||
// What the machine's own sensors say; zero is not measured. cpu_temp_c is
|
||||
// the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
|
||||
// inlet_temp_c the board's inlet or ambient sensor where it has one.
|
||||
double cpu_temp_c = 9;
|
||||
double cpu_power_w = 10;
|
||||
double inlet_temp_c = 11;
|
||||
repeated GPUSensor gpus = 12;
|
||||
}
|
||||
|
||||
// GPUSensor is one card's heat, power, and what is holding it back.
|
||||
message GPUSensor {
|
||||
int32 index = 1;
|
||||
double temp_c = 2;
|
||||
double memory_temp_c = 3;
|
||||
double power_w = 4;
|
||||
double power_limit_w = 5;
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
repeated string throttle = 6;
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
# Facility-aware placement: rack power, temperature and cooling
|
||||
|
||||
**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
|
||||
deferred** until the rack, PDU and power information of a real site is at
|
||||
hand — without it they would be designed against guesses.
|
||||
This is the design for step 5 of resource allocation, after reserved CPU cores
|
||||
and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
|
||||
and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
|
||||
what follows, only the node sensors (rollout step 0) exist in the code: they
|
||||
are read, published and shown, and change no decision.
|
||||
|
||||
## Why
|
||||
|
||||
In a large installation the scarce thing is often not a GPU but the room it
|
||||
sits in. Three failures are invisible to placement today:
|
||||
|
||||
- **A rack goes over its power budget.** Every node in it reports healthy;
|
||||
placement sees free cards and fills them; the breaker, or the facility's
|
||||
power cap, decides the rest. A rack with A and B feeds is worse: it runs
|
||||
fine until one feed fails and the other cannot carry the load.
|
||||
- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
|
||||
replica placed in a rack whose inlet air is too warm serves fewer tokens per
|
||||
second than its measured capacity says, and nothing connects the two.
|
||||
- **Correlated failure.** Two replicas of a model on two nodes in the same rack
|
||||
or on the same PDU are one failure away from zero. Placement spreads replicas
|
||||
across nodes, not across racks.
|
||||
|
||||
The goal is for placement to treat facility limits the way it already treats
|
||||
memory and cores: reserve against a budget, refuse what does not fit, say why,
|
||||
and never let missing data stop the cluster.
|
||||
|
||||
## What it builds on
|
||||
|
||||
| Already in place | Used for |
|
||||
|---|---|
|
||||
| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
|
||||
| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
|
||||
| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
|
||||
| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
|
||||
| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
|
||||
| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
|
||||
| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
|
||||
| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
|
||||
|
||||
## Concepts
|
||||
|
||||
**Location.** Each node has a place in a tree: site, room, row, rack. Power has
|
||||
its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
|
||||
feed. Both are read from node labels with well-known keys:
|
||||
|
||||
```yaml
|
||||
# agent.yaml
|
||||
labels:
|
||||
llmgw.io/site: sto1
|
||||
llmgw.io/room: hall-b
|
||||
llmgw.io/row: "12"
|
||||
llmgw.io/rack: r12-07
|
||||
llmgw.io/pdu: r12-07-a,r12-07-b # both feeds, for a dual-fed rack
|
||||
```
|
||||
|
||||
A node without these labels is its own rack of one: placement behaves as it
|
||||
does today.
|
||||
|
||||
**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
|
||||
condition. Domains are declared in a facility file (below) or discovered from
|
||||
node labels; a label naming a domain the file does not declare creates it with
|
||||
no budget and no condition.
|
||||
|
||||
**Budget.** A domain may have a power cap in watts. Placement reserves each
|
||||
replica's estimated draw against every domain its node belongs to and refuses a
|
||||
placement that would take any of them past its cap less a safety margin. For a
|
||||
dual-fed rack the cap that matters is what one feed can carry alone, since that
|
||||
is what the rack must survive.
|
||||
|
||||
**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
|
||||
or `unknown`, derived from readings against thresholds with hysteresis.
|
||||
Conditions change what placement prefers and allows; they are not amounts to
|
||||
fill.
|
||||
|
||||
**Reading.** A measured value with a time: inlet temperature, power draw, a
|
||||
cooling unit's alarm state. Readings older than their staleness limit are
|
||||
treated as absent.
|
||||
|
||||
## Estimating a replica's draw
|
||||
|
||||
What placement reserves has to be known before the replica runs, so it is an
|
||||
estimate first and a measurement later:
|
||||
|
||||
1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
|
||||
2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
|
||||
(the agent already reads the cards; the power limit is one more field) plus
|
||||
a per-node share for CPU, memory and fans; CPU profiles take their reserved
|
||||
cores times a per-core figure for the node's CPU model.
|
||||
3. **Measured.** Once replicas run, the agent reports their cards' actual draw
|
||||
(NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
|
||||
domain's headroom is computed from the larger of what is reserved and what
|
||||
is measured, so an estimate that was too low is corrected by the room it
|
||||
stops offering, not by an outage.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Power
|
||||
|
||||
- **Hard limit.** A placement that would take any domain past `cap × (1 −
|
||||
margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
|
||||
16.0 kW reserved, replica needs ~2.8, margin 10%*.
|
||||
- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
|
||||
times its derating (typically 80%), not the sum.
|
||||
- **Already over.** A domain found over its cap, because the estimate was low
|
||||
or the cap was lowered, gets no new placements, and an alert. It is not
|
||||
emptied automatically; shedding power is the facility's decision, and
|
||||
stopping inference to do it is ours only if asked (see shedding below).
|
||||
|
||||
### Temperature and cooling
|
||||
|
||||
Thresholds are per domain with defaults, and each condition needs to hold for a
|
||||
while before it changes, in both directions, so a single reading neither
|
||||
starts nor ends anything:
|
||||
|
||||
| Condition | Enters when | Placement |
|
||||
|---|---|---|
|
||||
| `ok` | below the warm threshold for 5 min | normal |
|
||||
| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
|
||||
| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
|
||||
| `unknown` | no fresh reading | as `ok`, and reported as unknown |
|
||||
|
||||
**Shedding** (optional, off by default): a domain `hot` for longer than its
|
||||
shed delay has load moved off it using step 2's machinery. The lowest class goes
|
||||
first, and replicas above their model's minimum go before any at it. A critical
|
||||
minimum is never touched, and moves are rate limited to one replica per domain
|
||||
per interval. The replica is placed elsewhere first where there is room, so
|
||||
shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
|
||||
-pl`) is a gentler alternative worth measuring against shedding: less
|
||||
throughput, no restart.
|
||||
|
||||
### Spreading
|
||||
|
||||
Replicas of one model are spread across racks, and across PDUs, the way they
|
||||
are spread across nodes today. This is a preference, dropped when it is the only
|
||||
thing standing in the way, as node spreading already is.
|
||||
|
||||
### Scoring
|
||||
|
||||
Placement today filters, then takes the first candidate in a fixed order
|
||||
(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
|
||||
the weights first). Facility signals are better as a score than as more
|
||||
filters. Candidates that pass the hard limits are ordered by:
|
||||
|
||||
1. condition: `ok` before `warm`;
|
||||
2. power headroom left in the rack after this replica, more first, so load
|
||||
levels across racks;
|
||||
3. the existing order.
|
||||
|
||||
Once measurements include power, step 4's profile preference can use tokens per
|
||||
watt instead of tokens per GPU where both are known.
|
||||
|
||||
## Where the data comes from
|
||||
|
||||
A `FacilitySource` interface, implemented several ways, chosen in
|
||||
`server.yaml`:
|
||||
|
||||
```go
|
||||
type FacilitySource interface {
|
||||
// Domains is what the source knows: budgets, thresholds, members.
|
||||
Domains(ctx) ([]Domain, error)
|
||||
// Readings is the latest value of each signal, per domain.
|
||||
Readings(ctx) ([]Reading, error)
|
||||
}
|
||||
```
|
||||
|
||||
- **Static file.** Domains, caps and thresholds, written by hand or generated
|
||||
from a DCIM export. Enough for power budgets with no live data at all.
|
||||
- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
|
||||
readings usually end up in Prometheus. The source runs configured instant
|
||||
queries and maps their labels onto domains:
|
||||
|
||||
```yaml
|
||||
facility:
|
||||
source: prometheus
|
||||
prometheus: http://monitoring:9090
|
||||
signals:
|
||||
inlet_temp_c:
|
||||
query: max by (rack) (dcim_rack_inlet_temperature_celsius)
|
||||
domain_label: rack
|
||||
stale_after: 2m
|
||||
power_watts:
|
||||
query: sum by (pdu) (pdu_outlet_power_watts)
|
||||
domain_label: pdu
|
||||
cooling_alarm:
|
||||
query: max by (row) (crac_alarm)
|
||||
domain_label: row
|
||||
```
|
||||
- **Agents (built, step 0).** What each node reads about itself, every ten
|
||||
seconds at most, sent with its heartbeat:
|
||||
- **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
|
||||
driver reports it, power draw, the enforced power limit, and what is
|
||||
holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
|
||||
`hw_thermal`, `power_brake`.
|
||||
- **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
|
||||
package temperature, or the hottest core; and the board's inlet or ambient
|
||||
sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
|
||||
- **CPU power, from RAPL:** the packages' energy counters over time,
|
||||
subdomains not counted twice, counter wrap handled.
|
||||
|
||||
A machine without a sensor reports nothing for it, never a zero; a VM has
|
||||
none of them. Published as `llmgw_gpu_temperature_celsius`,
|
||||
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
|
||||
`llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
|
||||
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
|
||||
`llmgw_node_inlet_temperature_celsius`, and on each node's page.
|
||||
|
||||
What they can and cannot say:
|
||||
- **Throttling is the strongest signal there is.** A temperature is a proxy;
|
||||
a card holding its clocks down is the fact that invalidates its measured
|
||||
capacity, and it needs no threshold to be chosen.
|
||||
- **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
|
||||
power-supply losses are not in it, so a node's draw is that plus an
|
||||
estimated remainder. A rack's is its llmgw nodes' plus a declared figure
|
||||
for everything else in it: switches and storage report nothing here.
|
||||
- **CPU and GPU temperatures follow the load, not the room.** A hot rack is
|
||||
inferred from many nodes running hot at comparable load, or from
|
||||
throttling, not from one node's CPU. Only a board inlet sensor reads the
|
||||
air, and most boards have none.
|
||||
- **No budget comes from a sensor.** A reading is a draw; the cap it is
|
||||
measured against still comes from the facility.
|
||||
- **Simulated.** Readings from a script, for tests and for rehearsing a
|
||||
thermal event on a cluster without facility data.
|
||||
|
||||
## Safety
|
||||
|
||||
- **Missing data never blocks.** A source that is down, a reading that is stale,
|
||||
a node without location labels: placement behaves as it does today, and says
|
||||
the data is missing (a metric, a line on the cluster page). Facility
|
||||
awareness can make placement more careful, never impossible.
|
||||
- **Hysteresis and hold times** on every condition, so a noisy sensor does not
|
||||
flap a rack between allowed and forbidden.
|
||||
- **Observe-only first.** A mode where every decision is computed, logged and
|
||||
shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
|
||||
for long enough to trust the thresholds before turning enforcement on.
|
||||
- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
|
||||
facility team to mark a domain unavailable, or to clear a condition they know
|
||||
to be a sensor fault.
|
||||
- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
|
||||
|
||||
## What people see
|
||||
|
||||
- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
|
||||
31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
|
||||
- `llmgw ctl facility` lists domains with budget, reserved, measured and
|
||||
condition. The cluster page gets a rack summary beside the nodes table.
|
||||
- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
|
||||
`_measured_watts`, `llmgw_domain_condition{condition}`,
|
||||
`llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
|
||||
domain hot, a domain over its cap, and facility data gone stale.
|
||||
|
||||
## Rollout
|
||||
|
||||
Each step ships on its own and is useful without the next:
|
||||
|
||||
0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
|
||||
CPU heat and power, inlet air where the board has a sensor; published and
|
||||
shown, no decision changed.
|
||||
1. **Location and spreading.** Well-known labels, domains from labels, replicas
|
||||
spread across racks and PDUs, locations on the node views. No facility data
|
||||
needed.
|
||||
2. **Power budgets from a static file.** Caps and margins, estimated draw per
|
||||
replica (declared or derived), hard limit in placement and headroom.
|
||||
3. **Readings, observe-only.** The Prometheus source beside the agents',
|
||||
conditions with hysteresis (throttling first among them), measured power
|
||||
folded into headroom; decisions shown, not enforced.
|
||||
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
|
||||
release.
|
||||
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
|
||||
preemption; measured power in profile preference.
|
||||
|
||||
## Testing
|
||||
|
||||
- The simulated source drives unit tests of every condition transition, the
|
||||
hysteresis, staleness, and the placement outcomes, with mutation checks as
|
||||
for steps 1–4.
|
||||
- Property tests: whatever the readings, placement never exceeds a declared
|
||||
cap with fresh data, and never refuses everything because data is missing.
|
||||
- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
|
||||
with a static file of made-up caps, steps 3–4 with the simulated source
|
||||
replaying a thermal event.
|
||||
- Before enforcement anywhere real: a period in observe-only mode against the
|
||||
site's actual data.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
|
||||
of it already reaches a Prometheus. The source design assumes Prometheus is
|
||||
the common ground; a site without it needs another source.
|
||||
- **Who owns the numbers.** Caps, deratings and thresholds belong to the
|
||||
facility team. Whether they edit llmgw's file, or llmgw reads theirs.
|
||||
- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
|
||||
should see phases or only the PDU total.
|
||||
- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
|
||||
the domain tree allows either, but the defaults should match what sites
|
||||
actually measure.
|
||||
- **Power capping instead of shedding.** Whether to lower GPU power limits in a
|
||||
warm rack rather than move replicas, and how that interacts with measured
|
||||
capacity from step 4, which would no longer hold at the lower limit.
|
||||
+39
-2
@@ -940,8 +940,18 @@ restart either — it rebuilds the fleet from what the agents report.
|
||||
fields, and both sides treat an absent one as unknown rather than as a default.
|
||||
Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
|
||||
installing the release binary and restarting `llmgw-agent`, and every upgrade
|
||||
after that is the command above. The mirror, which is not an agent, is
|
||||
upgraded the same way by hand.
|
||||
after that is the command above.
|
||||
|
||||
**The mirror is upgraded on its own host**, since it is not an agent and its
|
||||
service cannot replace its own binary:
|
||||
|
||||
```
|
||||
sudo llmgw upgrade v0.10.5 --service llmgw-mirror
|
||||
```
|
||||
|
||||
The same checks — the release's checksums, the old binary kept beside the new —
|
||||
then a restart of the unit, waiting for it to stay up. A mirror older than
|
||||
v0.10.5 does not have the command: install the release binary by hand once.
|
||||
|
||||
Then confirm what you have, rather than assuming:
|
||||
|
||||
@@ -956,6 +966,33 @@ replica alone on purpose, because churning one costs more than the difference.
|
||||
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
|
||||
the new one before stopping the old.
|
||||
|
||||
### Signed releases
|
||||
|
||||
Every install above checks a release against its published checksums, which
|
||||
proves the download is what the server holds — not that the server holds what
|
||||
was built. Signing closes that: CI signs the checksums, and a binary that
|
||||
trusts a key installs only releases signed with it. Off until a key exists;
|
||||
turning it on, once:
|
||||
|
||||
```
|
||||
llmgw release keygen
|
||||
```
|
||||
|
||||
1. Put the **private** key in the repository's Actions secret
|
||||
`RELEASE_SIGNING_KEY`. From the next tag, releases carry
|
||||
`checksums.txt.sig`.
|
||||
2. Put the **public** key in `internal/release/trusted.pub` and release. Every
|
||||
binary built from then on carries it and refuses releases that are not
|
||||
signed with it — `ctl upgrade`, agents upgrading themselves, and
|
||||
`llmgw upgrade` alike.
|
||||
|
||||
In that order: a build that trusts a key refuses unsigned releases, so the
|
||||
secret has to be signing before the key is trusted. The first trusting release
|
||||
is itself installed by a binary that trusts nothing; every one after is checked
|
||||
by the one before. `LLMGW_RELEASE_KEY` overrides the built-in key, for rotating
|
||||
it. The onboarding script fetches with `curl` and checks checksums only; a node
|
||||
is checked from its first upgrade on.
|
||||
|
||||
## Backing up
|
||||
|
||||
`llmgw server backup` writes one archive holding everything a control plane
|
||||
|
||||
@@ -21,6 +21,7 @@ rather than repeated.
|
||||
| `band_header` | `string` | [Admission](#admission) |
|
||||
| `queue_ttl` | `time.Duration` | [Admission](#admission) |
|
||||
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
|
||||
| `queue_capacity_total` | `int` | [Admission](#admission) |
|
||||
| `interval` | `time.Duration` | [Admission](#admission) |
|
||||
| `enabled` | `bool` | [Auth](#auth) |
|
||||
| `store` | `string` | [Auth](#auth) |
|
||||
@@ -228,6 +229,15 @@ should not be able to promote themselves.
|
||||
|
||||
`int`
|
||||
|
||||
### `queue_capacity_total`
|
||||
|
||||
`int`
|
||||
|
||||
QueueCapacityTotal bounds everything queued across bands. Reached, a
|
||||
request is not refused while a lower band has any queued: the newest of
|
||||
the lowest band is turned away instead, with a 503 and Retry-After.
|
||||
Zero takes half of all the bands' capacities together.
|
||||
|
||||
### `interval`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
@@ -101,6 +101,9 @@ type Agent struct {
|
||||
smsErr map[string]string
|
||||
// load measures the machine between heartbeats.
|
||||
load *discovery.LoadSampler
|
||||
// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
|
||||
// from nvidia-smi.
|
||||
sensors *discovery.SensorSampler
|
||||
|
||||
// fetchWatchers are the shards waiting on each source, so progress on one
|
||||
// download reaches every shard that asked for it. A fetch is single-flight
|
||||
@@ -146,6 +149,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
logRing: ring,
|
||||
provKick: make(chan struct{}, 1),
|
||||
load: discovery.NewLoadSampler(),
|
||||
sensors: discovery.NewSensorSampler(),
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
return a, nil
|
||||
@@ -638,6 +642,16 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
|
||||
ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
|
||||
})
|
||||
}
|
||||
if a.sensors != nil {
|
||||
se := a.sensors.Sample(context.Background())
|
||||
out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
|
||||
for _, g := range se.GPUs {
|
||||
out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
|
||||
Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
|
||||
PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
|
||||
})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,242 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Sensors is what a node's own hardware says about heat and power: the CPUs,
|
||||
// the board where it has an inlet sensor, and each GPU. A zero is not
|
||||
// measured, never a reading of zero — a VM has none of these, and must not
|
||||
// look like a machine running at 0 °C on no power.
|
||||
type Sensors struct {
|
||||
// CPUTempC is the hottest CPU sensor: the package, or the hottest core
|
||||
// where no package reading exists.
|
||||
CPUTempC float64
|
||||
// CPUPowerW is the CPU packages' draw, from RAPL: the change in their
|
||||
// energy counters over the time between two samples.
|
||||
CPUPowerW float64
|
||||
// InletTempC is the air coming in, where the board has a sensor it calls
|
||||
// that. Most do not, and a CPU's temperature is not a substitute: it
|
||||
// follows the load, not the room.
|
||||
InletTempC float64
|
||||
GPUs []GPUSensor
|
||||
}
|
||||
|
||||
// GPUSensor is one card's heat, power and whether it is slowing itself down.
|
||||
type GPUSensor struct {
|
||||
Index int
|
||||
TempC float64
|
||||
MemoryTempC float64
|
||||
PowerW float64
|
||||
PowerLimitW float64
|
||||
// Throttle names what is holding the card's clocks down now:
|
||||
// sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake. Empty
|
||||
// is running free. Idle and application clock settings are left out;
|
||||
// they are not the card being held back.
|
||||
Throttle []string
|
||||
}
|
||||
|
||||
// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
|
||||
// than temperatures change, and each run costs a few hundred milliseconds.
|
||||
const sensorEvery = 10 * time.Second
|
||||
|
||||
// SensorSampler reads Sensors. The first sample has no CPU power, which needs
|
||||
// two readings of the energy counters.
|
||||
type SensorSampler struct {
|
||||
sys string // /sys, or a test's copy of it
|
||||
smi func(ctx context.Context, fields string) ([]byte, error)
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
last Sensors
|
||||
lastAt time.Time
|
||||
prevEnergy map[string]uint64
|
||||
prevAt time.Time
|
||||
}
|
||||
|
||||
// NewSensorSampler reads this machine.
|
||||
func NewSensorSampler() *SensorSampler {
|
||||
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
||||
}
|
||||
|
||||
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
//nolint:gosec // fixed binary, fields from this file
|
||||
return exec.CommandContext(ctx, "nvidia-smi", "--query-gpu="+fields,
|
||||
"--format=csv,noheader,nounits").Output()
|
||||
}
|
||||
|
||||
// Sample reads the sensors, or returns the last reading when that is recent.
|
||||
func (s *SensorSampler) Sample(ctx context.Context) Sensors {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
now := s.now()
|
||||
if !s.lastAt.IsZero() && now.Sub(s.lastAt) < sensorEvery {
|
||||
return s.last
|
||||
}
|
||||
out := Sensors{}
|
||||
out.CPUTempC, out.InletTempC = s.hwmon()
|
||||
out.CPUPowerW = s.rapl(now)
|
||||
out.GPUs = s.gpus(ctx)
|
||||
s.last, s.lastAt = out, now
|
||||
return out
|
||||
}
|
||||
|
||||
// hwmon reads the hottest CPU sensor and an inlet sensor, from every hwmon
|
||||
// device the kernel exposes — what lm_sensors reads, without it installed.
|
||||
func (s *SensorSampler) hwmon() (cpu, inlet float64) {
|
||||
devs, _ := filepath.Glob(filepath.Join(s.sys, "class/hwmon/hwmon*"))
|
||||
var pkg, core float64
|
||||
for _, d := range devs {
|
||||
name := readTrim(filepath.Join(d, "name"))
|
||||
inputs, _ := filepath.Glob(filepath.Join(d, "temp*_input"))
|
||||
for _, in := range inputs {
|
||||
milli, err := strconv.ParseFloat(readTrim(in), 64)
|
||||
if err != nil || milli <= 0 {
|
||||
continue
|
||||
}
|
||||
c := milli / 1000
|
||||
label := strings.ToLower(readTrim(strings.TrimSuffix(in, "_input") + "_label"))
|
||||
switch {
|
||||
case isCPUSensor(name) && (strings.HasPrefix(label, "package") || label == "tctl" || label == "tdie"):
|
||||
pkg = max(pkg, c)
|
||||
case isCPUSensor(name):
|
||||
core = max(core, c)
|
||||
case label == "cpu":
|
||||
// A board's own reading of the CPU, as Dell's SMM gives.
|
||||
core = max(core, c)
|
||||
case strings.Contains(label, "inlet") || strings.Contains(label, "ambient") ||
|
||||
strings.Contains(label, "intake") || label == "systin":
|
||||
inlet = max(inlet, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
if pkg > 0 {
|
||||
return pkg, inlet
|
||||
}
|
||||
return core, inlet
|
||||
}
|
||||
|
||||
func isCPUSensor(name string) bool {
|
||||
switch name {
|
||||
case "coretemp", "k10temp", "zenpower", "cpu_thermal":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// rapl is the CPU packages' power from their energy counters. Package
|
||||
// domains only: the dram and core subdomains are parts of a package, and
|
||||
// adding them would count the same energy twice.
|
||||
func (s *SensorSampler) rapl(now time.Time) float64 {
|
||||
doms, _ := filepath.Glob(filepath.Join(s.sys, "class/powercap/intel-rapl:*"))
|
||||
energy := map[string]uint64{}
|
||||
wrap := map[string]uint64{}
|
||||
for _, d := range doms {
|
||||
if strings.Count(filepath.Base(d), ":") != 1 {
|
||||
continue // intel-rapl:0:0 is a subdomain of intel-rapl:0
|
||||
}
|
||||
e, err := strconv.ParseUint(readTrim(filepath.Join(d, "energy_uj")), 10, 64)
|
||||
if err != nil {
|
||||
continue // unreadable without root on newer kernels
|
||||
}
|
||||
energy[d] = e
|
||||
wrap[d], _ = strconv.ParseUint(readTrim(filepath.Join(d, "max_energy_range_uj")), 10, 64)
|
||||
}
|
||||
prev, prevAt := s.prevEnergy, s.prevAt
|
||||
s.prevEnergy, s.prevAt = energy, now
|
||||
dt := now.Sub(prevAt).Seconds()
|
||||
if prev == nil || dt <= 0 || len(energy) == 0 {
|
||||
return 0
|
||||
}
|
||||
var joules float64
|
||||
for d, e := range energy {
|
||||
p, ok := prev[d]
|
||||
if !ok {
|
||||
return 0 // a domain appeared: no fair difference this time
|
||||
}
|
||||
if e < p { // the counter wrapped
|
||||
e += wrap[d]
|
||||
}
|
||||
joules += float64(e-p) / 1e6
|
||||
}
|
||||
return joules / dt
|
||||
}
|
||||
|
||||
// The throttle reasons nvidia-smi reports, as bits. Only those that mean the
|
||||
// card is being held back are named.
|
||||
var throttleBits = []struct {
|
||||
bit uint64
|
||||
name string
|
||||
}{
|
||||
{0x4, "sw_power_cap"},
|
||||
{0x8, "hw_slowdown"},
|
||||
{0x20, "sw_thermal"},
|
||||
{0x40, "hw_thermal"},
|
||||
{0x80, "power_brake"},
|
||||
}
|
||||
|
||||
// gpus asks nvidia-smi for each card's heat and power. The memory temperature
|
||||
// is not known to every driver, and a field nvidia-smi does not know fails the
|
||||
// whole query, so it is asked for first and dropped on failure.
|
||||
func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
|
||||
out, err := s.smi(ctx, withMem)
|
||||
if err != nil {
|
||||
if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
var gs []GPUSensor
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, ",")
|
||||
if len(f) < 5 {
|
||||
continue
|
||||
}
|
||||
for i := range f {
|
||||
f[i] = strings.TrimSpace(f[i])
|
||||
}
|
||||
idx, err := strconv.Atoi(f[0])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
g := GPUSensor{Index: idx, TempC: num(f[1]), PowerW: num(f[2]), PowerLimitW: num(f[3])}
|
||||
if bits, err := strconv.ParseUint(strings.TrimPrefix(f[4], "0x"), 16, 64); err == nil {
|
||||
for _, b := range throttleBits {
|
||||
if bits&b.bit != 0 {
|
||||
g.Throttle = append(g.Throttle, b.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(f) > 5 {
|
||||
g.MemoryTempC = num(f[5])
|
||||
}
|
||||
gs = append(gs, g)
|
||||
}
|
||||
return gs
|
||||
}
|
||||
|
||||
// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
|
||||
func num(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || v < 0 {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func readTrim(path string) string {
|
||||
b, err := os.ReadFile(path) //nolint:gosec // sysfs
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func put(t *testing.T, root, rel, body string) {
|
||||
t.Helper()
|
||||
p := filepath.Join(root, rel)
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(body+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// dellHost is the layout of the Proxmox host these tests were written
|
||||
// against: Dell's SMM with a CPU and SODIMM sensors, coretemp with a package
|
||||
// and cores, and RAPL with a package and its dram subdomain.
|
||||
func dellHost(t *testing.T) string {
|
||||
root := t.TempDir()
|
||||
put(t, root, "class/hwmon/hwmon0/name", "dell_smm")
|
||||
put(t, root, "class/hwmon/hwmon0/temp1_input", "55000")
|
||||
put(t, root, "class/hwmon/hwmon0/temp1_label", "CPU")
|
||||
put(t, root, "class/hwmon/hwmon0/temp2_input", "47000")
|
||||
put(t, root, "class/hwmon/hwmon0/temp2_label", "SODIMM")
|
||||
put(t, root, "class/hwmon/hwmon1/name", "coretemp")
|
||||
put(t, root, "class/hwmon/hwmon1/temp1_input", "58000")
|
||||
put(t, root, "class/hwmon/hwmon1/temp1_label", "Package id 0")
|
||||
put(t, root, "class/hwmon/hwmon1/temp10_input", "61000")
|
||||
put(t, root, "class/hwmon/hwmon1/temp10_label", "Core 8")
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1000000000")
|
||||
put(t, root, "class/powercap/intel-rapl:0/max_energy_range_uj", "262143328850")
|
||||
put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "5000000")
|
||||
return root
|
||||
}
|
||||
|
||||
func sampler(sys string, smi func(context.Context, string) ([]byte, error), clock *time.Time) *SensorSampler {
|
||||
return &SensorSampler{sys: sys, smi: smi, now: func() time.Time { return *clock }}
|
||||
}
|
||||
|
||||
func noSMI(context.Context, string) ([]byte, error) { return nil, errors.New("not found") }
|
||||
|
||||
func TestAHostsCPUHeatAndPowerAreReadAsLmSensorsWould(t *testing.T) {
|
||||
root := dellHost(t)
|
||||
now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
s := sampler(root, noSMI, &now)
|
||||
|
||||
first := s.Sample(t.Context())
|
||||
// The package reading, not the hotter core: the package is what the
|
||||
// chip itself reports as its temperature.
|
||||
if first.CPUTempC != 58 {
|
||||
t.Errorf("CPU %.1f °C, expected the package's 58", first.CPUTempC)
|
||||
}
|
||||
if first.CPUPowerW != 0 {
|
||||
t.Error("power from one reading of a counter is invented")
|
||||
}
|
||||
if first.InletTempC != 0 || first.GPUs != nil {
|
||||
t.Errorf("sensors this host does not have were reported: %+v", first)
|
||||
}
|
||||
|
||||
// 10 s later the package has used 650 J, and its dram subdomain some of
|
||||
// that — counted once.
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1650000000")
|
||||
put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "99000000")
|
||||
now = now.Add(10 * time.Second)
|
||||
if got := s.Sample(t.Context()).CPUPowerW; got < 64.9 || got > 65.1 {
|
||||
t.Errorf("CPU power %.2f W, expected 65", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACounterThatWrapsIsNotANegativeDraw(t *testing.T) {
|
||||
root := dellHost(t)
|
||||
now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "262143000000")
|
||||
s := sampler(root, noSMI, &now)
|
||||
s.Sample(t.Context())
|
||||
// 328,850 µJ to the wrap, then 499,671,150 past it: 500 J in 10 s.
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "499671150")
|
||||
now = now.Add(10 * time.Second)
|
||||
if got := s.Sample(t.Context()).CPUPowerW; got < 49.9 || got > 50.1 {
|
||||
t.Errorf("across a wrap: %.2f W, expected 50", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAVMHasNoSensorsAndSaysNothing(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
put(t, root, "class/thermal/cooling_device0/type", "Processor")
|
||||
now := time.Now()
|
||||
got := sampler(root, noSMI, &now).Sample(t.Context())
|
||||
if got.CPUTempC != 0 || got.CPUPowerW != 0 || got.InletTempC != 0 || got.GPUs != nil {
|
||||
t.Errorf("a machine with no sensors reported %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABoardsInletSensorIsTheRoomNotTheCPU(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
put(t, root, "class/hwmon/hwmon2/name", "nct6798")
|
||||
put(t, root, "class/hwmon/hwmon2/temp1_input", "24500")
|
||||
put(t, root, "class/hwmon/hwmon2/temp1_label", "SYSTIN")
|
||||
put(t, root, "class/hwmon/hwmon3/name", "k10temp")
|
||||
put(t, root, "class/hwmon/hwmon3/temp1_input", "71250")
|
||||
put(t, root, "class/hwmon/hwmon3/temp1_label", "Tctl")
|
||||
now := time.Now()
|
||||
got := sampler(root, noSMI, &now).Sample(t.Context())
|
||||
if got.InletTempC != 24.5 || got.CPUTempC != 71.25 {
|
||||
t.Errorf("inlet %.2f, CPU %.2f", got.InletTempC, got.CPUTempC)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachGPUSaysHowHotHowHungryAndWhatHoldsItBack(t *testing.T) {
|
||||
var asked []string
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
asked = append(asked, fields)
|
||||
return []byte(
|
||||
"0, 71, 612.45, 700.00, 0x0000000000000000, 84\n" +
|
||||
"1, 88, 698.10, 700.00, 0x0000000000000024, 95\n" + // sw_power_cap + sw_thermal
|
||||
"2, 45, [N/A], 700.00, 0x0000000000000001, [N/A]\n"), nil // idle only
|
||||
}
|
||||
now := time.Now()
|
||||
ss := sampler(t.TempDir(), smi, &now)
|
||||
got := ss.Sample(t.Context())
|
||||
if len(got.GPUs) != 3 {
|
||||
t.Fatalf("%+v", got.GPUs)
|
||||
}
|
||||
g0, g1, g2 := got.GPUs[0], got.GPUs[1], got.GPUs[2]
|
||||
if g0.TempC != 71 || g0.PowerW != 612.45 || g0.PowerLimitW != 700 || g0.MemoryTempC != 84 || g0.Throttle != nil {
|
||||
t.Errorf("card 0 %+v", g0)
|
||||
}
|
||||
if strings.Join(g1.Throttle, ",") != "sw_power_cap,sw_thermal" {
|
||||
t.Errorf("card 1 throttled by %v", g1.Throttle)
|
||||
}
|
||||
if g2.PowerW != 0 || g2.MemoryTempC != 0 || g2.Throttle != nil {
|
||||
t.Errorf("card 2: N/A is not a reading, and idle is not being held back: %+v", g2)
|
||||
}
|
||||
// Within ten seconds the last reading is reused rather than running
|
||||
// nvidia-smi on every heartbeat; after, it is asked again.
|
||||
now = now.Add(5 * time.Second)
|
||||
ss.Sample(t.Context())
|
||||
if len(asked) != 1 {
|
||||
t.Errorf("nvidia-smi was asked %d times within ten seconds", len(asked))
|
||||
}
|
||||
now = now.Add(6 * time.Second)
|
||||
ss.Sample(t.Context())
|
||||
if len(asked) != 2 {
|
||||
t.Errorf("nvidia-smi was asked %d times after eleven seconds, expected 2", len(asked))
|
||||
}
|
||||
}
|
||||
|
||||
func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T) {
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if strings.Contains(fields, "temperature.memory") {
|
||||
return nil, errors.New(`Field "temperature.memory" is not a valid field to query.`)
|
||||
}
|
||||
return []byte("0, 66, 250.0, 300.0, 0x0000000000000040\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context())
|
||||
if len(got.GPUs) != 1 || got.GPUs[0].TempC != 66 || strings.Join(got.GPUs[0].Throttle, ",") != "hw_thermal" {
|
||||
t.Errorf("%+v", got.GPUs)
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -137,7 +137,7 @@ func newCtlCmd() *cobra.Command {
|
||||
// apply to it; it lives here because this is where operators look.
|
||||
newBenchCmd(client, &asJSON),
|
||||
newStatusCmd(client, &asJSON),
|
||||
newModelCmd(client),
|
||||
newModelCmd(client, &asJSON),
|
||||
newScaleCmd(client),
|
||||
newRestartCmd(client),
|
||||
newRetryCmd(client),
|
||||
|
||||
@@ -172,6 +172,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
|
||||
// The one machine this does not reach. Said every time, because it is
|
||||
// exactly the kind of thing that stays three versions behind unnoticed.
|
||||
fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
|
||||
" sudo llmgw upgrade %s --service llmgw-mirror\n", v)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -24,13 +24,14 @@ import (
|
||||
// minutes of outage. For a fixed production set that is a reasonable trade; for
|
||||
// a platform whose promise is that people can try things, it is the difference
|
||||
// between trying one and not bothering.
|
||||
func newModelCmd(client func() *ctlClient) *cobra.Command {
|
||||
func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "model",
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newTargetCmd(client, asJSON),
|
||||
newModelServeCmd(client))
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
@@ -162,3 +163,72 @@ func newOwnerCmd(client func() *ctlClient) *cobra.Command {
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var tps float64
|
||||
var apply bool
|
||||
c := &cobra.Command{
|
||||
Use: "target <model> --tokens-per-sec N [--apply]",
|
||||
Short: "How many replicas a model needs for a throughput, and whether they fit",
|
||||
Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
|
||||
"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
|
||||
"--apply makes that the number it always runs, if the cluster has room.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if tps <= 0 {
|
||||
return fmt.Errorf("--tokens-per-sec is required")
|
||||
}
|
||||
var v adminapi.TargetView
|
||||
path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
|
||||
strconv.FormatFloat(tps, 'f', -1, 64)
|
||||
if err := client().get(cmd.Context(), path, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
|
||||
v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
|
||||
switch {
|
||||
case v.More == 0:
|
||||
fmt.Fprintln(out, "It already has them.")
|
||||
case v.Fits && len(v.Preempts) > 0:
|
||||
fmt.Fprintf(out, "%d more; there is room once replicas of %s are stopped, which its class may do.\n",
|
||||
v.More, strings.Join(v.Preempts, ", "))
|
||||
case v.Fits:
|
||||
fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
|
||||
default:
|
||||
fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
|
||||
for _, b := range v.Blocked {
|
||||
fmt.Fprintln(out, " "+b)
|
||||
}
|
||||
}
|
||||
if !apply || v.More == 0 {
|
||||
return nil
|
||||
}
|
||||
if !v.Fits {
|
||||
return fmt.Errorf("not applied: the cluster has no room for them")
|
||||
}
|
||||
var models []adminapi.ModelView
|
||||
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
|
||||
return err
|
||||
}
|
||||
top := v.Needed
|
||||
for _, m := range models {
|
||||
if m.Name == v.Model {
|
||||
top = max(top, m.Max)
|
||||
}
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
|
||||
c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
|
||||
return c
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
)
|
||||
|
||||
func newReleaseCmd() *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "release",
|
||||
Short: "Sign releases, and make the key they are signed with",
|
||||
}
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "keygen",
|
||||
Short: "Make a release signing key pair",
|
||||
Long: "Prints a public key and a private key. The private key goes into the CI\n" +
|
||||
"secret RELEASE_SIGNING_KEY and nowhere else; the public key into\n" +
|
||||
"internal/release/trusted.pub, which every build from then on carries\n" +
|
||||
"and checks releases against. Run once, on a machine you trust.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
pub, priv, err := release.GenerateKey()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "public: %s\nprivate: %s\n", pub, priv)
|
||||
fmt.Fprintln(cmd.ErrOrStderr(), "\nThe private key signs every release: put it in the CI secret "+
|
||||
"RELEASE_SIGNING_KEY, then clear this terminal's scrollback.")
|
||||
return nil
|
||||
},
|
||||
})
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "sign FILE",
|
||||
Short: "Sign a release's checksums, writing FILE.sig",
|
||||
Long: "Reads the private key from RELEASE_SIGNING_KEY, never from an argument,\n" +
|
||||
"so it does not land in a shell history or a process list.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
key := os.Getenv("RELEASE_SIGNING_KEY")
|
||||
if key == "" {
|
||||
return errors.New("RELEASE_SIGNING_KEY is not set")
|
||||
}
|
||||
data, err := os.ReadFile(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sig, err := release.Sign(key, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if pub := release.TrustedKey(); pub != "" {
|
||||
// Signed with a key this build would not accept is a release
|
||||
// every upgraded machine refuses: stopped here instead.
|
||||
if err := release.Verify(pub, data, sig); err != nil {
|
||||
return fmt.Errorf("the signing key does not match this build's trusted key: %w", err)
|
||||
}
|
||||
}
|
||||
return os.WriteFile(args[0]+".sig", []byte(sig+"\n"), 0o644) //nolint:gosec // public
|
||||
},
|
||||
})
|
||||
return c
|
||||
}
|
||||
@@ -47,6 +47,8 @@ func Root() *cobra.Command {
|
||||
newSimEngineCmd(),
|
||||
newDoctorCmd(),
|
||||
newNodeCmd(),
|
||||
newSelfUpgradeCmd(),
|
||||
newReleaseCmd(),
|
||||
)
|
||||
return root
|
||||
}
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
|
||||
// newSelfUpgradeCmd upgrades this machine's llmgw on its own, for the machines
|
||||
// `llmgw ctl upgrade` does not reach: a mirror host, above all, whose service
|
||||
// runs unprivileged with the filesystem read-only and so cannot replace its own
|
||||
// binary the way an agent does.
|
||||
func newSelfUpgradeCmd() *cobra.Command {
|
||||
var releases, service string
|
||||
c := &cobra.Command{
|
||||
Use: "upgrade VERSION [--service UNIT]",
|
||||
Short: "Upgrade this machine's llmgw, and restart the service that runs it",
|
||||
Long: "For a machine the control plane does not upgrade — the model mirror's host.\n" +
|
||||
"The control plane and the nodes are upgraded with `llmgw ctl upgrade`.\n\n" +
|
||||
"Fetches the release for this platform, checks it against the release's\n" +
|
||||
"checksums, keeps the old binary beside the new one, and restarts --service,\n" +
|
||||
"waiting for it to be running again. Run it with sudo.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
v := args[0]
|
||||
out := cmd.OutOrStdout()
|
||||
if !release.Tag.MatchString(v) {
|
||||
return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
|
||||
}
|
||||
if version.Version == v {
|
||||
fmt.Fprintf(out, "this machine already runs %s\n", v)
|
||||
return nil
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return errors.New("replacing this machine's binary needs root: run it with sudo")
|
||||
}
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
backup, err := release.Install(cmd.Context(), nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%s -> %s: installed, checksum verified; the old binary is %s\n",
|
||||
version.Version, v, backup)
|
||||
if service == "" {
|
||||
return nil
|
||||
}
|
||||
//nolint:gosec // the unit name is the operator's own flag
|
||||
if b, err := exec.CommandContext(cmd.Context(), "systemctl", "restart", service).CombinedOutput(); err != nil {
|
||||
return fmt.Errorf("restarting %s: %v: %s", service, err, strings.TrimSpace(string(b)))
|
||||
}
|
||||
// Running, and still running a few seconds on: a unit that starts
|
||||
// and dies on a bad flag is "active" for a moment first.
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
steady := 0
|
||||
for time.Now().Before(deadline) {
|
||||
//nolint:gosec // as above
|
||||
b, _ := exec.CommandContext(cmd.Context(), "systemctl", "is-active", service).Output()
|
||||
if strings.TrimSpace(string(b)) == "active" {
|
||||
if steady++; steady >= 3 {
|
||||
fmt.Fprintf(out, "%s restarted and running\n", service)
|
||||
return nil
|
||||
}
|
||||
} else {
|
||||
steady = 0
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
return fmt.Errorf("%s is not running after the upgrade; see journalctl -u %s, and %s holds the previous binary",
|
||||
service, service, backup)
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from")
|
||||
c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
|
||||
return c
|
||||
}
|
||||
@@ -488,6 +488,9 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
// The tenancy tree's capacity quotas: what each part is
|
||||
// guaranteed and may hold.
|
||||
ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
|
||||
// And what each model was measured to serve, for scaling and for
|
||||
// choosing between profiles.
|
||||
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
|
||||
@@ -378,7 +378,12 @@ type Admission struct {
|
||||
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
|
||||
Interval time.Duration `koanf:"interval"`
|
||||
// QueueCapacityTotal bounds everything queued across bands. Reached, a
|
||||
// request is not refused while a lower band has any queued: the newest of
|
||||
// the lowest band is turned away instead, with a 503 and Retry-After.
|
||||
// Zero takes half of all the bands' capacities together.
|
||||
QueueCapacityTotal int `koanf:"queue_capacity_total"`
|
||||
Interval time.Duration `koanf:"interval"`
|
||||
}
|
||||
|
||||
// Prefix configures approximate prefix routing.
|
||||
|
||||
@@ -60,6 +60,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
@@ -248,6 +249,11 @@ type LoadView struct {
|
||||
// MemPercent is used over used plus available.
|
||||
MemPercent float64 `json:"memory_percent"`
|
||||
AgeSec int64 `json:"age_seconds"`
|
||||
// What the node's sensors say; zero is not measured.
|
||||
CPUTempC float64 `json:"cpu_temp_c,omitempty"`
|
||||
CPUPowerW float64 `json:"cpu_power_w,omitempty"`
|
||||
InletTempC float64 `json:"inlet_temp_c,omitempty"`
|
||||
GPUs []inventory.GPUSensor `json:"gpu_sensors,omitempty"`
|
||||
}
|
||||
|
||||
// loadFresh is how old a measurement may be and still be shown as current.
|
||||
@@ -261,7 +267,8 @@ func loadView(l *inventory.Load, now time.Time) *LoadView {
|
||||
CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
|
||||
Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
|
||||
MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
|
||||
AgeSec: int64(now.Sub(l.At).Seconds()),
|
||||
AgeSec: int64(now.Sub(l.At).Seconds()),
|
||||
CPUTempC: l.CPUTempC, CPUPowerW: l.CPUPowerW, InletTempC: l.InletTempC, GPUs: l.GPUs,
|
||||
}
|
||||
if v.MemTotal > 0 {
|
||||
v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
|
||||
@@ -399,6 +406,12 @@ type HeadroomView struct {
|
||||
Limited bool `json:"limited,omitempty"`
|
||||
// Blocked says why one more does not fit, a line per node considered.
|
||||
Blocked []string `json:"blocked,omitempty"`
|
||||
// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
|
||||
TokPerSec float64 `json:"tok_per_sec,omitempty"`
|
||||
// ByPreempting is how many more would fit by stopping replicas this
|
||||
// model's class may take; Preempts the models those belong to.
|
||||
ByPreempting int `json:"by_preempting,omitempty"`
|
||||
Preempts []string `json:"preempts,omitempty"`
|
||||
}
|
||||
|
||||
// RejectionView is one rejected candidate.
|
||||
@@ -958,7 +971,8 @@ func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
|
||||
return HeadroomView{}, false
|
||||
}
|
||||
return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
|
||||
Limited: h.Limited, Blocked: h.Blocked}, true
|
||||
Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec,
|
||||
ByPreempting: h.ByPreempting, Preempts: h.Preempts}, true
|
||||
}
|
||||
|
||||
func (a *API) Explain(model string) ExplainView {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -72,38 +73,24 @@ func (a *API) registerQuotas(mux *http.ServeMux) {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(r.Context(), req.Scope)
|
||||
key, err := a.SetQuota(r.Context(), req, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if !req.Guarantee.Any() && !req.Limit.Any() {
|
||||
writeError(w, http.StatusBadRequest, "a quota needs a guarantee or a limit; to remove one, DELETE it")
|
||||
return
|
||||
}
|
||||
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit,
|
||||
UpdatedBy: actorOf(r)}
|
||||
if err := a.quotas.PutCapacityQuota(r.Context(), q); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"scope": quota.Key(scope, id)})
|
||||
writeJSON(w, http.StatusOK, map[string]string{"scope": key})
|
||||
})
|
||||
mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(r.Context(), r.PathValue("scope"))
|
||||
key, err := a.ClearQuota(r.Context(), r.PathValue("scope"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if err := a.quotas.DeleteCapacityQuota(r.Context(), scope, id); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"removed": quota.Key(scope, id)})
|
||||
writeJSON(w, http.StatusOK, map[string]string{"removed": key})
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil || a.quotas == nil {
|
||||
@@ -137,6 +124,9 @@ func (a *API) registerQuotas(mux *http.ServeMux) {
|
||||
|
||||
// QuotaViews lists every quota with what its scope holds now.
|
||||
func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
|
||||
if a.quotas == nil {
|
||||
return nil, nil
|
||||
}
|
||||
qs, err := a.quotas.CapacityQuotas(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -235,3 +225,61 @@ func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string,
|
||||
}
|
||||
return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
|
||||
}
|
||||
|
||||
// SetQuota sets a scope's capacity quota, named by path or ID, and returns
|
||||
// the scope as org:ID.
|
||||
func (a *API) SetQuota(ctx context.Context, req QuotaRequest, by string) (string, error) {
|
||||
if a.quotas == nil {
|
||||
return "", fmt.Errorf("this control plane has no store")
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(ctx, req.Scope)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !req.Guarantee.Any() && !req.Limit.Any() {
|
||||
return "", fmt.Errorf("a quota needs a guarantee or a limit; to remove one, clear it")
|
||||
}
|
||||
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit, UpdatedBy: by}
|
||||
if err := a.quotas.PutCapacityQuota(ctx, q); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return quota.Key(scope, id), nil
|
||||
}
|
||||
|
||||
// ClearQuota removes a scope's capacity quota.
|
||||
func (a *API) ClearQuota(ctx context.Context, scope string) (string, error) {
|
||||
if a.quotas == nil {
|
||||
return "", fmt.Errorf("this control plane has no store")
|
||||
}
|
||||
s, id, _, err := a.resolveScope(ctx, scope)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := a.quotas.DeleteCapacityQuota(ctx, s, id); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return quota.Key(s, id), nil
|
||||
}
|
||||
|
||||
// ScopeName is a part of the tenancy tree: org:ID and its path.
|
||||
type ScopeName struct {
|
||||
Key string `json:"scope"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// Scopes lists every organization, team and project, by path.
|
||||
func (a *API) Scopes(ctx context.Context) ([]ScopeName, error) {
|
||||
if a.quotas == nil {
|
||||
return nil, nil
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]ScopeName, 0, len(names))
|
||||
for k, n := range names {
|
||||
out = append(out, ScopeName{Key: k, Name: n})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"net/http"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// TargetView is how many replicas a model needs to serve a throughput, from
|
||||
// what one replica was measured to serve, and whether the cluster has room.
|
||||
type TargetView struct {
|
||||
Model string `json:"model"`
|
||||
// TokPerSec is what was asked for; PerReplica what one replica was
|
||||
// measured to serve at its SLO, on Profile.
|
||||
TokPerSec float64 `json:"tok_per_sec"`
|
||||
PerReplica float64 `json:"per_replica_tok_per_sec"`
|
||||
Profile string `json:"profile"`
|
||||
// Needed replicas in all; Current hold capacity now; More is how many of
|
||||
// Needed are not running, and Room how many more fit.
|
||||
Needed int `json:"needed"`
|
||||
Current int `json:"current"`
|
||||
More int `json:"more"`
|
||||
Room int `json:"room"`
|
||||
Fits bool `json:"fits"`
|
||||
Blocked []string `json:"blocked,omitempty"`
|
||||
// Preempts is whose replicas would be stopped to make the room, when
|
||||
// free room alone is not enough; Room counts that room too.
|
||||
Preempts []string `json:"preempts,omitempty"`
|
||||
}
|
||||
|
||||
// Target sizes a model for a throughput.
|
||||
func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
|
||||
meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
|
||||
if !ok || meas.TokPerSec <= 0 {
|
||||
return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
|
||||
}
|
||||
v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
|
||||
Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
|
||||
h, ok := a.Headroom(model, v.Needed)
|
||||
if !ok {
|
||||
return TargetView{}, fmt.Errorf("no model %q", model)
|
||||
}
|
||||
v.Current = h.Current
|
||||
v.More = max(v.Needed-h.Current, 0)
|
||||
v.Room = h.More + h.ByPreempting
|
||||
v.Fits = v.More <= v.Room || h.Limited
|
||||
if v.More > h.More {
|
||||
v.Preempts = h.Preempts
|
||||
}
|
||||
if !v.Fits {
|
||||
v.Blocked = h.Blocked
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func (a *API) target(w http.ResponseWriter, r *http.Request) {
|
||||
tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
|
||||
if err != nil || tps <= 0 {
|
||||
writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
|
||||
return
|
||||
}
|
||||
v, err := a.Target(r.PathValue("name"), tps)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
|
||||
// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
|
||||
a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
|
||||
if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
|
||||
t.Errorf("an unmeasured model should say how to measure it: %v", err)
|
||||
}
|
||||
ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
|
||||
return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
|
||||
})
|
||||
v, err := a.Target("llama", 1200)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
|
||||
t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
|
||||
}
|
||||
v, _ = a.Target("llama", 2000)
|
||||
if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
|
||||
t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
|
||||
}
|
||||
}
|
||||
@@ -545,6 +545,13 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
|
||||
CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
|
||||
}
|
||||
}
|
||||
out.CPUTempC, out.CPUPowerW, out.InletTempC = l.GetCpuTempC(), l.GetCpuPowerW(), l.GetInletTempC()
|
||||
for _, g := range l.GetGpus() {
|
||||
out.GPUs = append(out.GPUs, inventory.GPUSensor{
|
||||
Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
|
||||
PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
|
||||
@@ -74,6 +74,13 @@ type Collector struct {
|
||||
gpuMemoryUsed *prometheus.Desc
|
||||
gpuUtil *prometheus.Desc
|
||||
gpuTemp *prometheus.Desc
|
||||
gpuMemTemp *prometheus.Desc
|
||||
gpuPower *prometheus.Desc
|
||||
gpuPowerLimit *prometheus.Desc
|
||||
gpuThrottled *prometheus.Desc
|
||||
nodeCPUTemp *prometheus.Desc
|
||||
nodeCPUPower *prometheus.Desc
|
||||
nodeInletTemp *prometheus.Desc
|
||||
gpuHealthy *prometheus.Desc
|
||||
gpuNVLink *prometheus.Desc
|
||||
gpuECC *prometheus.Desc
|
||||
@@ -142,6 +149,21 @@ func New(src Source) *Collector {
|
||||
"How busy a GPU was over the last sample period. Absent when the driver does not "+
|
||||
"report it, which is not the same as idle. Memory used says a model is loaded; "+
|
||||
"this says whether it is doing anything.", gpu, nil),
|
||||
gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
|
||||
"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
|
||||
gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
|
||||
"A GPU's power draw now.", gpu, nil),
|
||||
gpuPowerLimit: prometheus.NewDesc("llmgw_gpu_power_limit_watts",
|
||||
"The power limit a GPU is held to.", gpu, nil),
|
||||
gpuThrottled: prometheus.NewDesc("llmgw_gpu_throttled",
|
||||
"1 while something holds a GPU's clocks down: sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.",
|
||||
append(gpu, "reason"), nil),
|
||||
nodeCPUTemp: prometheus.NewDesc("llmgw_node_cpu_temperature_celsius",
|
||||
"A node's hottest CPU sensor.", node, nil),
|
||||
nodeCPUPower: prometheus.NewDesc("llmgw_node_cpu_power_watts",
|
||||
"A node's CPU packages' power draw, from RAPL.", node, nil),
|
||||
nodeInletTemp: prometheus.NewDesc("llmgw_node_inlet_temperature_celsius",
|
||||
"A node's inlet or ambient air sensor, where its board has one.", node, nil),
|
||||
gpuTemp: prometheus.NewDesc("llmgw_gpu_temperature_celsius",
|
||||
"A GPU's reported temperature.", gpu, nil),
|
||||
gpuHealthy: prometheus.NewDesc("llmgw_gpu_healthy",
|
||||
@@ -250,6 +272,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
|
||||
c.gpuNVLJoined, c.imexUp,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
|
||||
@@ -297,6 +320,16 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
}
|
||||
|
||||
if l := n.Load; l != nil && now.Sub(l.At) < loadFresh {
|
||||
// Each only when measured: a VM has none of these, and a zero
|
||||
// would read as a machine at 0 °C drawing nothing.
|
||||
for _, v := range []struct {
|
||||
d *prometheus.Desc
|
||||
val float64
|
||||
}{{c.nodeCPUTemp, l.CPUTempC}, {c.nodeCPUPower, l.CPUPowerW}, {c.nodeInletTemp, l.InletTempC}} {
|
||||
if v.val > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, n.ID)
|
||||
}
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.nodeCPU, prometheus.GaugeValue, l.CPUPercent, n.ID)
|
||||
if l.CPUCount > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.nodeCPUs, prometheus.GaugeValue, float64(l.CPUCount), n.ID)
|
||||
@@ -366,7 +399,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
}
|
||||
if g.Health.TemperatureC > 0 {
|
||||
// The sensors' fresh reading when there is one; the temperature
|
||||
// read at registration otherwise, which is all an older agent sends.
|
||||
if s, ok := sensorFor(n.Load, g.Index, now); ok {
|
||||
c.gpuSensor(ch, s, labels)
|
||||
} else if g.Health.TemperatureC > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
|
||||
float64(g.Health.TemperatureC), labels...)
|
||||
}
|
||||
@@ -518,3 +555,30 @@ func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sensorFor is a GPU's fresh sensor reading, if the node sent one.
|
||||
func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor, bool) {
|
||||
if l == nil || now.Sub(l.At) >= loadFresh {
|
||||
return inventory.GPUSensor{}, false
|
||||
}
|
||||
for _, g := range l.GPUs {
|
||||
if g.Index == index {
|
||||
return g, true
|
||||
}
|
||||
}
|
||||
return inventory.GPUSensor{}, false
|
||||
}
|
||||
|
||||
func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
for _, v := range []struct {
|
||||
d *prometheus.Desc
|
||||
val float64
|
||||
}{{c.gpuTemp, s.TempC}, {c.gpuMemTemp, s.MemoryTempC}, {c.gpuPower, s.PowerW}, {c.gpuPowerLimit, s.PowerLimitW}} {
|
||||
if v.val > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, labels...)
|
||||
}
|
||||
}
|
||||
for _, reason := range s.Throttle {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package fleetmetrics
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
gpuNode := inventory.Node{ID: "g1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559, Health: discovery.GPUHealth{TemperatureC: 40}}},
|
||||
Load: &inventory.Load{At: time.Now(), CPUTempC: 58, CPUPowerW: 65,
|
||||
GPUs: []inventory.GPUSensor{{Index: 0, TempC: 88, PowerW: 698, PowerLimitW: 700,
|
||||
Throttle: []string{"sw_power_cap", "sw_thermal"}}}}}
|
||||
vm := inventory.Node{ID: "vm1", State: inventory.Ready, Load: &inventory.Load{At: time.Now(), CPUPercent: 3}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{gpuNode, vm}})
|
||||
|
||||
for _, want := range []string{
|
||||
"llmgw_node_cpu_temperature_celsius node=g1 -> 58",
|
||||
"llmgw_node_cpu_power_watts node=g1 -> 65",
|
||||
"llmgw_gpu_power_watts class=h100-80g gpu=0 node=g1 -> 698",
|
||||
"llmgw_gpu_power_limit_watts class=h100-80g gpu=0 node=g1 -> 700",
|
||||
"llmgw_gpu_throttled class=h100-80g gpu=0 node=g1 reason=sw_thermal -> 1",
|
||||
// The fresh reading, not the 40 °C from registration.
|
||||
"llmgw_gpu_temperature_celsius class=h100-80g gpu=0 node=g1 -> 88",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %q", want)
|
||||
}
|
||||
}
|
||||
for _, absent := range []string{"node=vm1 -> 0", "llmgw_node_inlet_temperature_celsius", "llmgw_gpu_memory_temperature_celsius"} {
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if strings.Contains(line, absent) && (strings.Contains(line, "temperature") || strings.Contains(line, "power")) {
|
||||
t.Errorf("published what was not measured: %s", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -304,6 +304,21 @@ type Load struct {
|
||||
MemAvailBytes int64
|
||||
// Shards is each engine's use, by shard ID.
|
||||
Shards map[string]ShardLoad
|
||||
// What the node's sensors say; zero is not measured.
|
||||
CPUTempC float64
|
||||
CPUPowerW float64
|
||||
InletTempC float64
|
||||
GPUs []GPUSensor
|
||||
}
|
||||
|
||||
// GPUSensor is one card's heat and power, and what is throttling it.
|
||||
type GPUSensor struct {
|
||||
Index int `json:"index"`
|
||||
TempC float64 `json:"temp_c,omitempty"`
|
||||
MemoryTempC float64 `json:"memory_temp_c,omitempty"`
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerLimitW float64 `json:"power_limit_w,omitempty"`
|
||||
Throttle []string `json:"throttle,omitempty"`
|
||||
}
|
||||
|
||||
// ShardLoad is one engine's use: cores busy, and resident memory.
|
||||
|
||||
@@ -2,11 +2,14 @@ package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -50,3 +53,43 @@ func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
|
||||
return cur
|
||||
}
|
||||
}
|
||||
|
||||
// MeasurementStore is where benchmark results are kept.
|
||||
type MeasurementStore interface {
|
||||
Capacities(ctx context.Context) ([]store.StoredCapacity, error)
|
||||
}
|
||||
|
||||
// Measurements returns what the reconciler asks for benchmark results, keyed
|
||||
// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
|
||||
// keeps the last good answer when the store fails.
|
||||
func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
|
||||
var mu sync.Mutex
|
||||
var cur map[string]reconciler.Measured
|
||||
var at time.Time
|
||||
return func() map[string]reconciler.Measured {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if cur != nil && time.Since(at) < quotaRefresh {
|
||||
return cur
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
at = time.Now()
|
||||
rows, err := st.Capacities(ctx)
|
||||
if err != nil {
|
||||
log.Warn("could not read benchmark results; keeping the last ones", "err", err)
|
||||
return cur
|
||||
}
|
||||
next := map[string]reconciler.Measured{}
|
||||
for _, row := range rows {
|
||||
var c bench.Capacity
|
||||
if json.Unmarshal(row.Measurement, &c) != nil {
|
||||
continue
|
||||
}
|
||||
next[row.Model+"/"+row.Profile] = reconciler.Measured{
|
||||
MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
|
||||
}
|
||||
cur = next
|
||||
return cur
|
||||
}
|
||||
}
|
||||
@@ -157,8 +157,31 @@ type Load struct {
|
||||
// the last one is still starting: on a small CPU fleet each replica is minutes
|
||||
// of weights and gigabytes of memory, and a spike that is over by the time it
|
||||
// loads should not leave three behind.
|
||||
//
|
||||
// With a benchmark stored for the profile a model runs on, its measured
|
||||
// capacity is used too: more requests per replica than were measured to meet
|
||||
// the SLO is busy, even with the engines' queues empty — an engine will take
|
||||
// requests into its slots long after latency has gone — and a replica is only
|
||||
// taken back when the rest would still be under scaleDownHeadroom of what they
|
||||
// were measured to hold, so scaling down does not land on the edge.
|
||||
func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
now := r.now()
|
||||
// The profile each model's replicas run on, for its measurement; read
|
||||
// before the loop, which holds the lock.
|
||||
running := map[string]string{}
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State.Counts() {
|
||||
running[rep.Model] = rep.Profile
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
measured := map[string]Measured{}
|
||||
for model, profile := range running {
|
||||
if m, ok := r.measured(model, profile); ok {
|
||||
measured[model] = m
|
||||
}
|
||||
}
|
||||
for _, m := range r.cat.Load().Models() {
|
||||
base := r.baseFor(m)
|
||||
r.mu.Lock()
|
||||
@@ -178,8 +201,13 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
l := loads[m.Name]
|
||||
perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
|
||||
extra := r.extra[m.Name]
|
||||
meas, hasMeas := measured[m.Name]
|
||||
overMeasured := hasMeas &&
|
||||
float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
|
||||
fitsInFewer := !hasMeas ||
|
||||
float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
|
||||
switch {
|
||||
case perReplica >= r.cfg.ScaleUpWaiting:
|
||||
case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
|
||||
delete(r.quietSince, m.Name)
|
||||
since, ok := r.busySince[m.Name]
|
||||
if !ok {
|
||||
@@ -190,9 +218,10 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
r.extra[m.Name] = extra + 1
|
||||
r.busySince[m.Name] = now
|
||||
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
|
||||
"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
|
||||
"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
|
||||
"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
|
||||
}
|
||||
case l.Waiting == 0 && extra > 0:
|
||||
case l.Waiting == 0 && extra > 0 && fitsInFewer:
|
||||
delete(r.busySince, m.Name)
|
||||
since, ok := r.quietSince[m.Name]
|
||||
if !ok {
|
||||
@@ -212,3 +241,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
r.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// scaleDownHeadroom is how full, of their measured capacity, the replicas left
|
||||
// after a scale-down may be. Below one so that the load that was there a
|
||||
// minute ago does not put them straight back over.
|
||||
const scaleDownHeadroom = 0.7
|
||||
@@ -25,6 +25,14 @@ type Headroom struct {
|
||||
// Blocked is why one more than More does not fit, a line per node that
|
||||
// was considered. Empty when Limited.
|
||||
Blocked []string
|
||||
// TokPerSec is what one more replica was measured to serve, on the
|
||||
// profile placement would give it; zero when it was never measured.
|
||||
TokPerSec float64
|
||||
// ByPreempting is how many more, beyond More, would fit by stopping
|
||||
// replicas the model's class may take, and Preempts the models those
|
||||
// replicas belong to: what "room" means for a model that outranks others.
|
||||
ByPreempting int
|
||||
Preempts []string
|
||||
}
|
||||
|
||||
// Headroom tries placing up to limit more replicas of a model on a copy of
|
||||
@@ -58,11 +66,37 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
if pol != nil {
|
||||
usage = pol.Usage(r.Holdings())
|
||||
}
|
||||
for h.More < limit {
|
||||
res := sched.Place(m.PlacementProfiles(), avoid)
|
||||
place := func() placement.Result {
|
||||
res := sched.Place(r.orderedProfiles(m), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// As scaleUp does: spreading is a preference, not a rule.
|
||||
res = sched.Place(m.PlacementProfiles(), nil)
|
||||
res = sched.Place(r.orderedProfiles(m), nil)
|
||||
}
|
||||
return res
|
||||
}
|
||||
// What the model's class may take, in the order preemption would, each
|
||||
// released on the copy only when free room has run out.
|
||||
victims, nextVictim, preempting := r.victims(m), 0, false
|
||||
for h.More+h.ByPreempting < limit {
|
||||
res := place()
|
||||
// Named only once they have made room: candidates tried on the way
|
||||
// to finding there is none would otherwise read as whose replicas a
|
||||
// placement would stop.
|
||||
var tried []string
|
||||
for res.Placement == nil && nextVictim < len(victims) {
|
||||
v := victims[nextVictim]
|
||||
nextVictim++
|
||||
inv.ReleaseReplica(v.ID)
|
||||
tried = append(tried, v.Model)
|
||||
res = place()
|
||||
}
|
||||
if res.Placement != nil && len(tried) > 0 {
|
||||
preempting = true
|
||||
for _, m := range tried {
|
||||
if !slices.Contains(h.Preempts, m) {
|
||||
h.Preempts = append(h.Preempts, m)
|
||||
}
|
||||
}
|
||||
}
|
||||
if res.Placement == nil {
|
||||
for _, rej := range res.Rejections {
|
||||
@@ -87,6 +121,11 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
|
||||
mem, cores = prof.MinMemoryGB, cpuCores(prof)
|
||||
}
|
||||
if h.More == 0 {
|
||||
if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
|
||||
h.TokPerSec = meas.TokPerSec
|
||||
}
|
||||
}
|
||||
var nodes []string
|
||||
for _, a := range res.Placement.Nodes {
|
||||
inv.Allocate(a.NodeID, inventory.Allocation{
|
||||
@@ -95,8 +134,14 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
nodes = append(nodes, a.NodeID)
|
||||
}
|
||||
avoid = append(avoid, nodes...)
|
||||
h.Nodes = append(h.Nodes, nodes)
|
||||
h.More++
|
||||
// Once anything was stopped, what fits after is counted as fitting
|
||||
// by preempting: it may be sitting in the room that was freed.
|
||||
if preempting {
|
||||
h.ByPreempting++
|
||||
} else {
|
||||
h.Nodes = append(h.Nodes, nodes)
|
||||
h.More++
|
||||
}
|
||||
}
|
||||
h.Limited = true
|
||||
return h, true
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
)
|
||||
|
||||
// Measured is what one replica of a model on a profile was measured to serve
|
||||
// while meeting its SLO, by `llmgw ctl bench --save`.
|
||||
type Measured struct {
|
||||
// MaxConcurrency is the most requests at once that still met the SLO.
|
||||
MaxConcurrency int
|
||||
// TokPerSec is the throughput at that concurrency.
|
||||
TokPerSec float64
|
||||
}
|
||||
|
||||
// SetMeasurements supplies benchmark results, keyed model/profile, read
|
||||
// afresh each time it is called. Without them scaling goes by queue alone and
|
||||
// profiles are chosen by priority alone.
|
||||
func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.measurements = f
|
||||
}
|
||||
|
||||
// measured is a model's measurement on one profile.
|
||||
func (r *Reconciler) measured(model, profile string) (Measured, bool) {
|
||||
r.mu.Lock()
|
||||
f := r.measurements
|
||||
r.mu.Unlock()
|
||||
if f == nil {
|
||||
return Measured{}, false
|
||||
}
|
||||
m, ok := f()[model+"/"+profile]
|
||||
return m, ok && m.MaxConcurrency > 0
|
||||
}
|
||||
|
||||
// orderedProfiles is a model's profiles as placement should try them: by
|
||||
// priority, and among equal priorities the one measured to serve the most per
|
||||
// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
|
||||
// their catalog order behind measured ones of the same priority.
|
||||
//
|
||||
// Placement sorts by priority itself, so the order found here is written back
|
||||
// as priorities, descending, for it to keep.
|
||||
func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
|
||||
ps := m.PlacementProfiles()
|
||||
eff := make([]float64, len(ps))
|
||||
for i, p := range ps {
|
||||
meas, ok := r.measured(m.Name, p.Name)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
|
||||
if units == 0 {
|
||||
units = float64(max(p.CPUCores, 1))
|
||||
}
|
||||
eff[i] = meas.TokPerSec / units
|
||||
}
|
||||
idx := make([]int, len(ps))
|
||||
for i := range idx {
|
||||
idx[i] = i
|
||||
}
|
||||
sort.SliceStable(idx, func(a, b int) bool {
|
||||
pa, pb := ps[idx[a]], ps[idx[b]]
|
||||
if pa.Priority != pb.Priority {
|
||||
return pa.Priority > pb.Priority
|
||||
}
|
||||
return eff[idx[a]] > eff[idx[b]]
|
||||
})
|
||||
out := make([]placement.Profile, len(ps))
|
||||
for rank, i := range idx {
|
||||
out[rank] = ps[i]
|
||||
out[rank].Priority = len(ps) - rank
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MeasuredFor is the measurement that sizes a model: that of the profile its
|
||||
// replicas run on, or, with none running, of the first profile placement
|
||||
// would try that was measured.
|
||||
func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
|
||||
r.mu.Lock()
|
||||
profile := ""
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == model && rep.State.Counts() {
|
||||
profile = rep.Profile
|
||||
break
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if profile != "" {
|
||||
if m, ok := r.measured(model, profile); ok {
|
||||
return m, profile, true
|
||||
}
|
||||
}
|
||||
cm, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return Measured{}, "", false
|
||||
}
|
||||
for _, p := range r.orderedProfiles(cm) {
|
||||
if m, ok := r.measured(cm.Name, p.Name); ok {
|
||||
return m, p.Name, true
|
||||
}
|
||||
}
|
||||
return Measured{}, "", false
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func measure(h *harness, byKey map[string]Measured) {
|
||||
h.r.SetMeasurements(func() map[string]Measured { return byKey })
|
||||
}
|
||||
|
||||
func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling.Max = 2
|
||||
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
// Six in flight, nothing waiting at the engine: by the queue it is fine.
|
||||
load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
|
||||
hold := func() {
|
||||
h.r.RecordLoad(load)
|
||||
h.advance(61 * time.Second)
|
||||
h.r.RecordLoad(load)
|
||||
h.r.Reconcile(t.Context())
|
||||
}
|
||||
hold()
|
||||
if n := liveCount(h); n != 1 {
|
||||
t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
|
||||
}
|
||||
// Measured to meet its SLO at four at once: six is past it.
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
|
||||
hold()
|
||||
if n := liveCount(h); n != 2 {
|
||||
t.Errorf("%d replicas with 6 in flight against a measured 4", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling.Max = 2
|
||||
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.r.mu.Lock()
|
||||
h.r.extra["llama"] = 1
|
||||
h.r.mu.Unlock()
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
if liveCount(h) != 2 {
|
||||
t.Fatal("setup: not at two replicas")
|
||||
}
|
||||
quietFor := func(l Load) {
|
||||
for range 3 {
|
||||
h.r.RecordLoad(map[string]Load{"llama": l})
|
||||
h.advance(DefaultConfig().ScaleDownAfter + time.Second)
|
||||
h.r.RecordLoad(map[string]Load{"llama": l})
|
||||
h.r.Reconcile(t.Context())
|
||||
}
|
||||
}
|
||||
// Nothing waiting, but three running: one replica left would be at 3 of
|
||||
// its 4, past 70%.
|
||||
quietFor(Load{Replicas: 2, Running: 3})
|
||||
if n := liveCount(h); n != 2 {
|
||||
t.Errorf("scaled down to the edge of what one replica holds: %d", n)
|
||||
}
|
||||
quietFor(Load{Replicas: 2, Running: 2})
|
||||
if n := liveCount(h); n != 1 {
|
||||
t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
|
||||
m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{
|
||||
{Name: "tp4", GPUClass: "h100-80g", TP: 4},
|
||||
{Name: "tp2", GPUClass: "h100-80g", TP: 2},
|
||||
{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
|
||||
}}
|
||||
h := newHarness(t, m, gpuNode("1", 8))
|
||||
// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
|
||||
// the higher priority, which an operator set and a benchmark does not
|
||||
// overrule.
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
|
||||
"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
|
||||
cm, _ := h.r.cat.Load().Resolve("llama")
|
||||
var order []string
|
||||
for _, p := range h.r.orderedProfiles(cm) {
|
||||
order = append(order, p.Name)
|
||||
}
|
||||
if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
|
||||
t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,9 @@ type victim struct {
|
||||
AboveMin bool
|
||||
Tier int
|
||||
Idle time.Duration
|
||||
// Size is what stopping it frees, for choosing the smaller of two
|
||||
// otherwise equal victims.
|
||||
Size quota.Amount
|
||||
}
|
||||
|
||||
// preemptFor stops replicas of lower classes when a model cannot otherwise be
|
||||
@@ -54,28 +57,46 @@ func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []st
|
||||
if len(cands) == 0 {
|
||||
return false
|
||||
}
|
||||
trial := r.inv.Clone()
|
||||
sched := placement.New(trial)
|
||||
// Candidates in order until the model fits; then every one whose
|
||||
// stopping turns out not to be needed is kept after all. Without that
|
||||
// second step an idle replica on the wrong node, tried first, would be
|
||||
// stopped along with the one that actually made the room.
|
||||
var chosen []victim
|
||||
for _, v := range cands {
|
||||
if len(chosen) == maxVictims {
|
||||
break
|
||||
}
|
||||
trial.ReleaseReplica(v.ID)
|
||||
chosen = append(chosen, v)
|
||||
res := sched.Place(m.PlacementProfiles(), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
res = sched.Place(m.PlacementProfiles(), nil)
|
||||
}
|
||||
if res.Placement == nil {
|
||||
if !r.fitsWithout(m, avoid, chosen) {
|
||||
continue
|
||||
}
|
||||
for i := len(chosen) - 1; i >= 0 && len(chosen) > 1; i-- {
|
||||
fewer := append(append([]victim(nil), chosen[:i]...), chosen[i+1:]...)
|
||||
if r.fitsWithout(m, avoid, fewer) {
|
||||
chosen = fewer
|
||||
}
|
||||
}
|
||||
r.preempt(ctx, m, chosen)
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// fitsWithout reports whether m could be placed with these replicas gone,
|
||||
// tried on a copy of the inventory.
|
||||
func (r *Reconciler) fitsWithout(m catalog.Model, avoid []string, gone []victim) bool {
|
||||
trial := r.inv.Clone()
|
||||
for _, v := range gone {
|
||||
trial.ReleaseReplica(v.ID)
|
||||
}
|
||||
sched := placement.New(trial)
|
||||
res := sched.Place(r.orderedProfiles(m), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
res = sched.Place(r.orderedProfiles(m), nil)
|
||||
}
|
||||
return res.Placement != nil
|
||||
}
|
||||
|
||||
// preempt drains the chosen replicas and holds their models back.
|
||||
func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
|
||||
now := r.now()
|
||||
@@ -245,15 +266,32 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
|
||||
if len(nodesOf(rep)) > 1 {
|
||||
continue // see makeRoomFor: a multi-node replica is its own decision
|
||||
}
|
||||
prof, _ := other.ProfileByName(rep.Profile)
|
||||
var size quota.Amount
|
||||
for _, sh := range rep.Shards {
|
||||
size = size.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
out = append(out, victim{ID: rep.ID, Model: model, Class: class,
|
||||
AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now)})
|
||||
AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now), Size: size})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
if out[i].Tier != out[j].Tier {
|
||||
return out[i].Tier < out[j].Tier
|
||||
a, b := out[i], out[j]
|
||||
if a.Tier != b.Tier {
|
||||
return a.Tier < b.Tier
|
||||
}
|
||||
return out[i].Idle > out[j].Idle
|
||||
if a.Idle != b.Idle {
|
||||
return a.Idle > b.Idle
|
||||
}
|
||||
// The smaller first: the least served is lost if it is enough, and
|
||||
// the pruning after keeps a larger one only when it is needed.
|
||||
if a.Size.GPUs != b.Size.GPUs {
|
||||
return a.Size.GPUs < b.Size.GPUs
|
||||
}
|
||||
if a.Size.MemoryGB != b.Size.MemoryGB {
|
||||
return a.Size.MemoryGB < b.Size.MemoryGB
|
||||
}
|
||||
return a.ID < b.ID
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -235,3 +235,184 @@ func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
|
||||
t.Error("a hold outlived its time")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheReplicasThatMakeTheRoomAreStopped(t *testing.T) {
|
||||
// Two half-used nodes. "idle" on node 1 is on demand and long unused, so
|
||||
// it is first in line; but the critical model needs node 2's other half,
|
||||
// which only stopping "chat" frees.
|
||||
idle := catalog.Model{Name: "idle", Source: "/m", Scaling: catalog.Scaling{Min: 0, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
|
||||
chat := catalog.Model{Name: "chat", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp6", Engine: "vllm", GPUClass: "h100-80g", TP: 6}}}
|
||||
vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
|
||||
h := newFleetHarness(t, []catalog.Model{idle, chat}, gpuNode("1", 6), gpuNode("2", 8))
|
||||
h.r.Want("idle")
|
||||
h.settle(t, 4)
|
||||
var idleNode, chatNode string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
switch rep.Model {
|
||||
case "idle":
|
||||
idleNode = rep.Shards[0].NodeID
|
||||
case "chat":
|
||||
chatNode = rep.Shards[0].NodeID
|
||||
}
|
||||
}
|
||||
if idleNode == "" || chatNode != "2" {
|
||||
t.Fatalf("setup: idle on %q, chat on %q; chat must hold node 2: %+v", idleNode, chatNode, h.r.Replicas())
|
||||
}
|
||||
h.r.mu.Lock()
|
||||
for _, rep := range h.r.replicas {
|
||||
rep.LastUsedAt = h.r.now().Add(-time.Hour)
|
||||
}
|
||||
h.r.mu.Unlock()
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{idle, chat, vital})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
var stopped []string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor == "vital" {
|
||||
stopped = append(stopped, rep.Model)
|
||||
}
|
||||
}
|
||||
if len(stopped) != 1 || stopped[0] != "chat" {
|
||||
t.Errorf("stopped %v; only chat's replica makes room for a whole node", stopped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
|
||||
cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 31 * 1024, CPUCount: 8}
|
||||
big := catalog.Model{Name: "big", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12, CPUCores: 4}}}
|
||||
small := catalog.Model{Name: "small", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
|
||||
vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, CPUCores: 4}}}
|
||||
// 8 cores: big 4 + small 1 leaves 3, and vital needs 4. Stopping either
|
||||
// is enough; stopping big loses far more.
|
||||
for range 5 { // the choice must not depend on map order
|
||||
h := newFleetHarness(t, []catalog.Model{big, small}, cpu)
|
||||
h.settle(t, 3)
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{big, small, vital})
|
||||
h.r.cat.Store(cat)
|
||||
h.r.Reconcile(t.Context())
|
||||
var stopped []string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor == "vital" {
|
||||
stopped = append(stopped, rep.Model)
|
||||
}
|
||||
}
|
||||
if len(stopped) != 1 || stopped[0] != "small" {
|
||||
t.Fatalf("stopped %v; the small replica alone makes room", stopped)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeadroomCountsWhatAHigherClassCouldTake(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
got, _ := h.r.Headroom("vital", 3)
|
||||
if got.More != 0 || got.ByPreempting != 1 || len(got.Preempts) != 1 || got.Preempts[0] != "chat" {
|
||||
t.Errorf("a critical model should see one more by stopping chat: %+v", got)
|
||||
}
|
||||
// Tried on a copy: nothing was stopped.
|
||||
if replicasOf(h, "chat") != 1 {
|
||||
t.Fatal("asking about room stopped a replica")
|
||||
}
|
||||
// A model of chat's own class may not take its minimum, so has no such room.
|
||||
peer := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("peer", "", 1), 0)}, gpuNode("1", 8))
|
||||
peer.settle(t, 3)
|
||||
peer.advance(peer.r.cfg.MinLifetime + time.Minute)
|
||||
if got, _ := peer.r.Headroom("peer", 3); got.ByPreempting != 0 {
|
||||
t.Errorf("a standard model counted room in another's minimum: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartBorrowsALowerClassesRoomAndGivesItBack(t *testing.T) {
|
||||
// chat fills node 1; the only other node holds an opportunistic replica.
|
||||
chat := gpuModel("chat", "", 1)
|
||||
extra := gpuModel("extra", catalog.ClassOpportunistic, 1)
|
||||
h := newFleetHarness(t, []catalog.Model{chat, extra}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.settle(t, 3)
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
var old string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" {
|
||||
old = rep.ID
|
||||
}
|
||||
}
|
||||
if old == "" || replicasOf(h, "extra") != 1 {
|
||||
t.Fatalf("setup: %+v", h.r.Replicas())
|
||||
}
|
||||
|
||||
if err := h.r.Restart(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Replacement first, beside the old one, in the room extra gives up;
|
||||
// the old one keeps serving until then.
|
||||
servedThroughout := true
|
||||
for range 8 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
ready := 0
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" && rep.State == Ready {
|
||||
ready++
|
||||
}
|
||||
}
|
||||
if ready == 0 {
|
||||
servedThroughout = false
|
||||
}
|
||||
h.advance(h.r.cfg.PreemptGrace + time.Second)
|
||||
}
|
||||
if !servedThroughout {
|
||||
t.Error("chat had no ready replica during its restart")
|
||||
}
|
||||
var newChat bool
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" && rep.ID != old && rep.State == Ready {
|
||||
newChat = true
|
||||
}
|
||||
if rep.ID == old {
|
||||
t.Errorf("the old replica is still there: %+v", rep)
|
||||
}
|
||||
}
|
||||
if !newChat {
|
||||
t.Fatalf("no new chat replica: %+v", h.r.Replicas())
|
||||
}
|
||||
// And extra has its replica back, in the room the old chat left.
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "extra") != 1 {
|
||||
t.Errorf("the opportunistic model did not get its room back: %+v", h.r.Replicas())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeadroomNamesOnlyTheReplicasThatWouldMakeRoom(t *testing.T) {
|
||||
// vital needs a whole node and chat is on the only one; a second
|
||||
// critical can never fit. Asking for 3: one by stopping chat, then no
|
||||
// candidates left — nothing else may be named.
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
got, _ := h.r.Headroom("vital", 3)
|
||||
if got.ByPreempting != 1 || len(got.Preempts) != 1 {
|
||||
t.Errorf("%+v", got)
|
||||
}
|
||||
// Free room for one, and candidates that cannot make room for a second:
|
||||
// none of them may be listed.
|
||||
small := catalog.Model{Name: "small", Source: "/m", Class: catalog.ClassOpportunistic,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp2", Engine: "vllm", GPUClass: "h100-80g", TP: 2}}}
|
||||
h2 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 8))
|
||||
got2, _ := h2.r.Headroom("big", 3)
|
||||
if got2.More != 1 || got2.ByPreempting != 1 || len(got2.Preempts) != 1 {
|
||||
t.Errorf("one free, one by stopping small: %+v", got2)
|
||||
}
|
||||
h3 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 6))
|
||||
got3, _ := h3.r.Headroom("big", 3)
|
||||
if got3.ByPreempting != 0 || len(got3.Preempts) != 0 {
|
||||
t.Errorf("stopping small frees a 6-GPU node, too small for big; nobody may be named: %+v", got3)
|
||||
}
|
||||
}
|
||||
@@ -273,6 +273,8 @@ type Reconciler struct {
|
||||
preempted map[string]int
|
||||
// quotas is the tenancy tree's capacity quotas as last read; nil is none.
|
||||
quotas func() *quota.Policy
|
||||
// measurements are benchmark results by model/profile; nil is none.
|
||||
measurements func() map[string]Measured
|
||||
|
||||
wake chan struct{}
|
||||
stop chan struct{}
|
||||
@@ -753,10 +755,10 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var expired bool
|
||||
grace, preemptedFor := r.cfg.DrainGrace, ""
|
||||
if ok {
|
||||
grace := r.cfg.DrainGrace
|
||||
if rep.PreemptedFor != "" {
|
||||
grace = r.cfg.PreemptGrace
|
||||
grace, preemptedFor = r.cfg.PreemptGrace, rep.PreemptedFor
|
||||
}
|
||||
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
|
||||
}
|
||||
@@ -766,11 +768,15 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
case !ok:
|
||||
case replaced:
|
||||
r.stopReplica(ctx, id)
|
||||
case expired && preemptedFor != "":
|
||||
r.log.Info("stopping a preempted replica", "replica", id,
|
||||
"for", preemptedFor, "after", grace)
|
||||
r.stopReplica(ctx, id)
|
||||
case expired:
|
||||
// The replacement never arrived — often because nowhere else can
|
||||
// host it. Draining forever would leave the node undrainable.
|
||||
r.log.Warn("stopping a drained replica without a replacement",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
"replica", id, "after", grace)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
}
|
||||
@@ -784,12 +790,13 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// allocated, and starting several at once from the same inventory snapshot
|
||||
// would hand them the same GPUs.
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
|
||||
res := r.sched.Place(m.PlacementProfiles(), avoid)
|
||||
profiles := r.orderedProfiles(m)
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// Spreading across failure domains is a preference, not a rule. If it
|
||||
// is the only thing standing in the way, take the placement anyway:
|
||||
// a second replica on one node still survives an engine crash.
|
||||
res = r.sched.Place(m.PlacementProfiles(), nil)
|
||||
res = r.sched.Place(profiles, nil)
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
|
||||
@@ -9,6 +9,7 @@ package release
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
@@ -51,7 +52,7 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
|
||||
asset := AssetName(version, goos, goarch)
|
||||
root := strings.TrimSuffix(base, "/") + "/" + version + "/"
|
||||
|
||||
want, err := checksum(ctx, client, root+"checksums.txt", asset)
|
||||
want, err := checksum(ctx, client, root, asset)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -85,13 +86,35 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
|
||||
return backup, nil
|
||||
}
|
||||
|
||||
func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
|
||||
body, err := get(ctx, client, url)
|
||||
// checksum reads the release's checksum for asset — after checking the
|
||||
// checksums' signature, when this build trusts a signing key. A release with
|
||||
// no signature is then refused rather than taken on its checksums: a missing
|
||||
// signature is exactly what replacing a release would look like.
|
||||
func checksum(ctx context.Context, client *http.Client, root, asset string) (string, error) {
|
||||
body, err := get(ctx, client, root+"checksums.txt")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer body.Close()
|
||||
sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
|
||||
sums, err := io.ReadAll(io.LimitReader(body, 1<<20))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if key := TrustedKey(); key != "" {
|
||||
sigBody, err := get(ctx, client, root+SignatureName)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("this build only installs signed releases, and %s could not be read: %w", SignatureName, err)
|
||||
}
|
||||
sig, err := io.ReadAll(io.LimitReader(sigBody, 4096))
|
||||
sigBody.Close()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := Verify(key, sums, string(sig)); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
sc := bufio.NewScanner(bytes.NewReader(sums))
|
||||
for sc.Scan() {
|
||||
f := strings.Fields(sc.Text())
|
||||
if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
package release
|
||||
|
||||
import (
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
_ "embed"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// trustedPub is the public key releases are signed with, base64, built into
|
||||
// every binary from the repository. Empty — as it is until a key is made —
|
||||
// trusts a release on its checksums alone, as every release before this did.
|
||||
//
|
||||
// Built in rather than fetched, because the point of a signature is that the
|
||||
// server the release comes from cannot vouch for it: a machine that already
|
||||
// runs llmgw checks the next release against the key the release it runs was
|
||||
// built with, so replacing both the binaries and the checksums on the server
|
||||
// is no longer enough.
|
||||
//
|
||||
//go:embed trusted.pub
|
||||
var trustedPub string
|
||||
|
||||
// TrustedKey is the key releases are verified with: LLMGW_RELEASE_KEY when set
|
||||
// (for rotating a key, or testing), the built-in one otherwise; empty is none.
|
||||
func TrustedKey() string {
|
||||
if k := strings.TrimSpace(os.Getenv("LLMGW_RELEASE_KEY")); k != "" {
|
||||
return k
|
||||
}
|
||||
return strings.TrimSpace(trustedPub)
|
||||
}
|
||||
|
||||
// SignatureName is the file a release's checksums are signed in.
|
||||
const SignatureName = "checksums.txt.sig"
|
||||
|
||||
// GenerateKey makes a signing key pair, both base64: the private key is the
|
||||
// 32-byte seed, which is what goes into the CI secret.
|
||||
func GenerateKey() (public, private string, err error) {
|
||||
pub, priv, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
return base64.StdEncoding.EncodeToString(pub), base64.StdEncoding.EncodeToString(priv.Seed()), nil
|
||||
}
|
||||
|
||||
// Sign signs data with a base64 seed, returning the signature in base64.
|
||||
func Sign(privateSeed string, data []byte) (string, error) {
|
||||
seed, err := base64.StdEncoding.DecodeString(strings.TrimSpace(privateSeed))
|
||||
if err != nil || len(seed) != ed25519.SeedSize {
|
||||
return "", errors.New("the signing key is not a base64 ed25519 seed")
|
||||
}
|
||||
return base64.StdEncoding.EncodeToString(ed25519.Sign(ed25519.NewKeyFromSeed(seed), data)), nil
|
||||
}
|
||||
|
||||
// Verify checks a base64 signature of data against a base64 public key.
|
||||
func Verify(public string, data []byte, signature string) error {
|
||||
pub, err := base64.StdEncoding.DecodeString(strings.TrimSpace(public))
|
||||
if err != nil || len(pub) != ed25519.PublicKeySize {
|
||||
return errors.New("the trusted release key is not a base64 ed25519 public key")
|
||||
}
|
||||
sig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(signature))
|
||||
if err != nil {
|
||||
return fmt.Errorf("the release signature is not base64: %w", err)
|
||||
}
|
||||
if !ed25519.Verify(pub, data, sig) {
|
||||
return errors.New("the release's checksums are not signed by the trusted key")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package release
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// signedServer publishes a release whose checksums are signed with seed,
|
||||
// or not signed at all when seed is empty; tamper rewrites the checksums
|
||||
// after signing.
|
||||
func signedServer(t *testing.T, seed string, body []byte, tamper bool) (*httptest.Server, string) {
|
||||
t.Helper()
|
||||
asset := AssetName("v2.0.0", "linux", "amd64")
|
||||
sum := sha256.Sum256(body)
|
||||
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
|
||||
sig := ""
|
||||
if seed != "" {
|
||||
var err error
|
||||
if sig, err = Sign(seed, []byte(sums)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if tamper {
|
||||
sums += strings.Repeat("0", 64) + " extra-file\n"
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/v2.0.0/checksums.txt":
|
||||
_, _ = w.Write([]byte(sums))
|
||||
case "/v2.0.0/" + SignatureName:
|
||||
if sig == "" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(sig + "\n"))
|
||||
case "/v2.0.0/" + asset:
|
||||
_, _ = w.Write(body)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv, asset
|
||||
}
|
||||
|
||||
func TestABuildThatTrustsAKeyInstallsOnlyReleasesSignedWithIt(t *testing.T) {
|
||||
pub, priv, err := GenerateKey()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, otherPriv, _ := GenerateKey()
|
||||
t.Setenv("LLMGW_RELEASE_KEY", pub)
|
||||
install := func(srv *httptest.Server) error {
|
||||
target := filepath.Join(t.TempDir(), "llmgw")
|
||||
_ = os.WriteFile(target, []byte("old"), 0o755)
|
||||
_, err := Install(t.Context(), srv.Client(), srv.URL, "v2.0.0", "linux", "amd64", target)
|
||||
return err
|
||||
}
|
||||
body := []byte("new binary")
|
||||
|
||||
good, _ := signedServer(t, priv, body, false)
|
||||
if err := install(good); err != nil {
|
||||
t.Fatalf("a release signed with the trusted key was refused: %v", err)
|
||||
}
|
||||
for name, srv := range map[string]*httptest.Server{
|
||||
"unsigned": func() *httptest.Server { s, _ := signedServer(t, "", body, false); return s }(),
|
||||
"signed by another": func() *httptest.Server { s, _ := signedServer(t, otherPriv, body, false); return s }(),
|
||||
"checksums rewritten": func() *httptest.Server { s, _ := signedServer(t, priv, body, true); return s }(),
|
||||
} {
|
||||
if err := install(srv); err == nil {
|
||||
t.Errorf("%s: installed", name)
|
||||
}
|
||||
}
|
||||
|
||||
// A build that trusts no key installs on checksums, as before.
|
||||
t.Setenv("LLMGW_RELEASE_KEY", "")
|
||||
if TrustedKey() != "" {
|
||||
t.Skip("this build embeds a trusted key; the unsigned path cannot be tested here")
|
||||
}
|
||||
unsigned, _ := signedServer(t, "", body, false)
|
||||
if err := install(unsigned); err != nil {
|
||||
t.Errorf("without a trusted key an unsigned release was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAKeyThatIsNotOneIsSaid(t *testing.T) {
|
||||
if _, err := Sign("not-a-key", []byte("x")); err == nil {
|
||||
t.Error("signed with a malformed key")
|
||||
}
|
||||
if err := Verify("bm9wZQ==", []byte("x"), "c2ln"); err == nil || !strings.Contains(err.Error(), "public key") {
|
||||
t.Errorf("a malformed trusted key: %v", err)
|
||||
}
|
||||
}
|
||||
Whitespace-only changes.
@@ -34,6 +34,12 @@ type Config struct {
|
||||
// QueueCapacityPerBand bounds each band's queue. Rejecting immediately
|
||||
// beats accumulating silently.
|
||||
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
|
||||
// QueueCapacityTotal bounds everything queued, across bands. When it is
|
||||
// reached, a request is not refused while a lower band has requests
|
||||
// queued: the newest of the lowest band is turned away instead
|
||||
// (displacement), since the oldest have waited longest and are nearest
|
||||
// their TTL. Zero takes half of every band's capacity together.
|
||||
QueueCapacityTotal int `koanf:"queue_capacity_total"`
|
||||
// Interval is how often the dispatcher re-checks saturation. Releases also
|
||||
// wake it, so this is only the backstop for saturation falling on its own.
|
||||
Interval time.Duration `koanf:"interval"`
|
||||
@@ -94,6 +100,9 @@ func (c *Config) normalize() error {
|
||||
if c.QueueCapacityPerBand <= 0 {
|
||||
c.QueueCapacityPerBand = 2000
|
||||
}
|
||||
if c.QueueCapacityTotal <= 0 {
|
||||
c.QueueCapacityTotal = max(c.QueueCapacityPerBand*len(c.Bands)/2, 1)
|
||||
}
|
||||
if c.Interval <= 0 {
|
||||
c.Interval = 10 * time.Millisecond
|
||||
}
|
||||
@@ -123,6 +132,10 @@ var (
|
||||
ErrQueueTimeout = errors.New("admission queue timeout")
|
||||
// ErrClosed means the controller is shutting down.
|
||||
ErrClosed = errors.New("admission controller closed")
|
||||
// ErrDisplaced means a queued request was turned away to make room for
|
||||
// one of a higher band. Worth retrying: the queue is full now, not for
|
||||
// good.
|
||||
ErrDisplaced = errors.New("displaced from the admission queue by higher-priority requests")
|
||||
)
|
||||
|
||||
// Stats is a snapshot of one band's queue, for metrics.
|
||||
@@ -259,6 +272,12 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
c.mu.Unlock()
|
||||
return nil, fmt.Errorf("%w: band %q", ErrQueueFull, band.Name)
|
||||
}
|
||||
if c.queuedLocked() >= c.cfg.QueueCapacityTotal && !c.displaceBelowLocked(idx) {
|
||||
c.rejected[band.Name]++
|
||||
c.mu.Unlock()
|
||||
return nil, fmt.Errorf("%w: %d queued in all, and nothing of a lower band to make room",
|
||||
ErrQueueFull, c.cfg.QueueCapacityTotal)
|
||||
}
|
||||
|
||||
w := &waiter{
|
||||
tenant: tenantKey(r),
|
||||
@@ -279,6 +298,10 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
c.mu.Unlock()
|
||||
return nil, ErrClosed
|
||||
}
|
||||
if w.err != nil {
|
||||
c.mu.Unlock()
|
||||
return nil, w.err
|
||||
}
|
||||
c.waitTotal[band.Name] += time.Since(w.enqueued)
|
||||
c.mu.Unlock()
|
||||
return &Lease{c: c, band: band.Name, model: r.Model}, nil
|
||||
@@ -297,8 +320,9 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
func (c *Controller) remove(idx int, w *waiter, reason string) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if w.admitted {
|
||||
// It won the race; the caller will see ready closed next time round.
|
||||
if w.admitted || w.err != nil {
|
||||
// It won the race, or was displaced; the caller will see ready
|
||||
// closed next time round.
|
||||
return
|
||||
}
|
||||
c.queues[idx].remove(w)
|
||||
@@ -410,3 +434,31 @@ func tenantKey(r Request) string {
|
||||
}
|
||||
return "anonymous"
|
||||
}
|
||||
|
||||
// queuedLocked is how many requests wait in every band. Called with mu held.
|
||||
func (c *Controller) queuedLocked() int {
|
||||
n := 0
|
||||
for _, q := range c.queues {
|
||||
n += q.len()
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// displaceBelowLocked turns away the newest request of the lowest band below
|
||||
// idx that has any queued, and says whether there was one. Called with mu held.
|
||||
func (c *Controller) displaceBelowLocked(idx int) bool {
|
||||
// Bands are sorted highest priority first, so lower bands have larger
|
||||
// indices, and the lowest is last.
|
||||
for j := len(c.queues) - 1; j > idx; j-- {
|
||||
w := c.queues[j].newest()
|
||||
if w == nil {
|
||||
continue
|
||||
}
|
||||
c.queues[j].remove(w)
|
||||
w.err = fmt.Errorf("%w (band %q)", ErrDisplaced, c.cfg.Bands[j].Name)
|
||||
close(w.ready)
|
||||
c.rejected[c.cfg.Bands[j].Name+"/displaced"]++
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package admission
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// queueUntil starts an Acquire in the background and waits until the band
|
||||
// holds depth requests, returning where its result will arrive.
|
||||
func queueUntil(t *testing.T, c *Controller, band string, bandIdx, depth int) <-chan error {
|
||||
t.Helper()
|
||||
out := make(chan error, 1)
|
||||
go func() {
|
||||
l, err := c.Acquire(context.Background(), Request{Model: "m", Band: band})
|
||||
if err == nil {
|
||||
l.Release()
|
||||
}
|
||||
out <- err
|
||||
}()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for c.Depths()[bandIdx].Depth < depth && time.Now().Before(deadline) {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
if got := c.Depths()[bandIdx].Depth; got != depth {
|
||||
t.Fatalf("band %s holds %d, expected %d", band, got, depth)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestAHigherBandDisplacesTheNewestOfTheLowest(t *testing.T) {
|
||||
s := &fixedSat{}
|
||||
s.set(5) // past every ceiling: everything queues
|
||||
cfg := testConfig()
|
||||
cfg.QueueCapacityTotal = 3
|
||||
c := newController(t, cfg, s.fn)
|
||||
// Bands, highest first: platinum 0, gold 1, silver 2, bronze 3.
|
||||
oldest := queueUntil(t, c, "bronze", 3, 1)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
silver := queueUntil(t, c, "silver", 2, 1)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
newest := queueUntil(t, c, "bronze", 3, 2)
|
||||
|
||||
// Full at three. Gold arrives: the newest bronze goes, not the oldest,
|
||||
// and not silver, which outranks bronze.
|
||||
gold := queueUntil(t, c, "gold", 1, 1)
|
||||
select {
|
||||
case err := <-newest:
|
||||
if !errors.Is(err, ErrDisplaced) {
|
||||
t.Fatalf("the newest bronze got %v", err)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("nothing was displaced")
|
||||
}
|
||||
if d := c.Depths(); d[3].Depth != 1 || d[2].Depth != 1 || d[1].Depth != 1 {
|
||||
t.Errorf("depths %+v: one bronze, the silver and the gold should remain", d)
|
||||
}
|
||||
|
||||
// Bronze cannot displace anything: nothing is below it.
|
||||
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||
if !errors.Is(err, ErrQueueFull) {
|
||||
t.Errorf("a bronze request at a full queue got %v", err)
|
||||
}
|
||||
// Another gold takes the remaining bronze, then silver.
|
||||
queueUntil(t, c, "gold", 1, 2)
|
||||
if err := <-oldest; !errors.Is(err, ErrDisplaced) {
|
||||
t.Errorf("the last bronze got %v", err)
|
||||
}
|
||||
queueUntil(t, c, "gold", 1, 3)
|
||||
if err := <-silver; !errors.Is(err, ErrDisplaced) {
|
||||
t.Errorf("silver got %v", err)
|
||||
}
|
||||
// All gold now: a fourth has nobody lower to displace.
|
||||
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueFull) {
|
||||
t.Errorf("a gold request with only gold queued got %v", err)
|
||||
}
|
||||
|
||||
// And they are served once there is room.
|
||||
s.set(0)
|
||||
select {
|
||||
case err := <-gold:
|
||||
if err != nil {
|
||||
t.Errorf("the first gold was not served: %v", err)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Error("the queue did not drain")
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,9 @@ type waiter struct {
|
||||
// admitted guards the race between the dispatcher admitting a waiter and
|
||||
// the waiter giving up on its own.
|
||||
admitted bool
|
||||
// err is why the waiter was turned away while queued — displaced by a
|
||||
// higher band — set before ready is closed.
|
||||
err error
|
||||
}
|
||||
|
||||
// bandQueue holds one band's waiters, grouped by tenant.
|
||||
@@ -166,3 +169,17 @@ func (q *bandQueue) dropExpired(now time.Time, onDrop func()) {
|
||||
q.byTenant[tenant] = kept
|
||||
}
|
||||
}
|
||||
|
||||
// newest is the most recently queued waiter, across every tenant: the one
|
||||
// displacement takes, since it has invested the least waiting.
|
||||
func (q *bandQueue) newest() *waiter {
|
||||
var out *waiter
|
||||
for _, ws := range q.byTenant {
|
||||
for _, w := range ws {
|
||||
if out == nil || w.enqueued.After(out.enqueued) {
|
||||
out = w
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -88,6 +88,9 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
|
||||
{"queued too long", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
|
||||
}, 504, ""},
|
||||
{"displaced for a higher band", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
|
||||
}, 503, "5"},
|
||||
{"shutting down", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
|
||||
}, 503, ""},
|
||||
|
||||
@@ -60,9 +60,15 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
|
||||
return
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
status = http.StatusGatewayTimeout
|
||||
case errors.Is(err, admission.ErrDisplaced):
|
||||
// Turned away for higher-priority work, not for good: said so the
|
||||
// client backs off and comes back rather than giving up.
|
||||
w.Header().Set("Retry-After", "5")
|
||||
}
|
||||
reason := "unavailable"
|
||||
switch {
|
||||
case errors.Is(err, admission.ErrDisplaced):
|
||||
reason = "displaced"
|
||||
case errors.Is(err, admission.ErrQueueFull):
|
||||
reason = "queue_full"
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
@@ -185,6 +191,11 @@ func (r *Router) Handler() http.Handler {
|
||||
pages.RegisterAdminUsers(mux, r.store)
|
||||
// Quotas are written on the same tree the accounts page builds, so
|
||||
// they are mounted together.
|
||||
// Capacity quotas are the control plane's, reached through the
|
||||
// same admin API the CLI uses.
|
||||
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
|
||||
pages.SetCapacity(c)
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
|
||||
@@ -347,6 +347,7 @@ func admissionConfig(rc config.Router) admission.Config {
|
||||
},
|
||||
QueueTTL: rc.Admission.QueueTTL,
|
||||
QueueCapacityPerBand: rc.Admission.QueueCapacityPerBand,
|
||||
QueueCapacityTotal: rc.Admission.QueueCapacityTotal,
|
||||
Interval: rc.Admission.Interval,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,6 +61,8 @@ type AccountPages struct {
|
||||
hasMirror bool
|
||||
// hasQuotas is set when /admin/quotas is registered.
|
||||
hasQuotas bool
|
||||
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
|
||||
capacity CapacityAdmin
|
||||
// personalOff refuses personal keys; see SetPersonalOff.
|
||||
personalOff bool
|
||||
// personalDefaults is what personal use falls under by default.
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// CapacityAdmin sets what parts of the tenancy tree are guaranteed of the
|
||||
// cluster and may hold. The admin API implements it; the page and the CLI go
|
||||
// through the same checks.
|
||||
type CapacityAdmin interface {
|
||||
QuotaViews(ctx context.Context) ([]adminapi.QuotaView, error)
|
||||
SetQuota(ctx context.Context, req adminapi.QuotaRequest, by string) (string, error)
|
||||
ClearQuota(ctx context.Context, scope string) (string, error)
|
||||
Scopes(ctx context.Context) ([]adminapi.ScopeName, error)
|
||||
}
|
||||
|
||||
// SetCapacity gives the quotas page capacity quotas to show and set. Without
|
||||
// it the section is left off.
|
||||
func (p *AccountPages) SetCapacity(c CapacityAdmin) { p.capacity = c }
|
||||
|
||||
// capacityRow is one scope's capacity quota as the page shows it.
|
||||
type capacityRow struct {
|
||||
adminapi.QuotaView
|
||||
GPUs, Cores, Memory string
|
||||
}
|
||||
|
||||
// capacityForm is what the form starts with: a scope's quota to change, or
|
||||
// nothing for a new one.
|
||||
type capacityForm struct {
|
||||
Scope string
|
||||
GPUs, Cores, Memory string
|
||||
LGPUs, LCores, LMem string
|
||||
}
|
||||
|
||||
func (p *AccountPages) capacitySection(ctx context.Context, d *adminQuotasData, editing string) {
|
||||
if p.capacity == nil {
|
||||
return
|
||||
}
|
||||
d.HasCapacity = true
|
||||
views, err := p.capacity.QuotaViews(ctx)
|
||||
if err != nil {
|
||||
d.Error = err.Error()
|
||||
return
|
||||
}
|
||||
for _, v := range views {
|
||||
d.Capacity = append(d.Capacity, capacityRow{QuotaView: v,
|
||||
GPUs: capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
|
||||
Cores: capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
|
||||
Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)})
|
||||
if v.Scope == editing {
|
||||
d.CapForm = capacityForm{Scope: v.Scope,
|
||||
GPUs: intStr(v.Guarantee.GPUs), Cores: intStr(v.Guarantee.CPUCores), Memory: intStr(v.Guarantee.MemoryGB),
|
||||
LGPUs: intStr(v.Limit.GPUs), LCores: intStr(v.Limit.CPUCores), LMem: intStr(v.Limit.MemoryGB)}
|
||||
}
|
||||
}
|
||||
if d.Scopes, err = p.capacity.Scopes(ctx); err != nil {
|
||||
d.Error = err.Error()
|
||||
}
|
||||
}
|
||||
|
||||
// capCell is "6 of 8 (at most 16)": held, guaranteed, limit.
|
||||
func capCell(holds int, g, l *int) string {
|
||||
if g == nil && l == nil {
|
||||
return ""
|
||||
}
|
||||
out := strconv.Itoa(holds)
|
||||
if g != nil {
|
||||
out += " of " + strconv.Itoa(*g)
|
||||
}
|
||||
if l != nil {
|
||||
out += fmt.Sprintf(" (at most %d)", *l)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func intStr(v *int) string {
|
||||
if v == nil {
|
||||
return ""
|
||||
}
|
||||
return strconv.Itoa(*v)
|
||||
}
|
||||
|
||||
// applyCapacity sets or clears one scope's capacity quota from the form.
|
||||
func (p *AccountPages) applyCapacity(r *http.Request, admin store.User) (string, error) {
|
||||
scope := strings.TrimSpace(r.FormValue("scope"))
|
||||
if scope == "" {
|
||||
return "", fmt.Errorf("choose where the quota applies")
|
||||
}
|
||||
if r.FormValue("clear") != "" {
|
||||
key, err := p.capacity.ClearQuota(r.Context(), scope)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "capacity quota removed from " + key, nil
|
||||
}
|
||||
field := func(name, what string) (*int, error) {
|
||||
v := strings.TrimSpace(r.FormValue(name))
|
||||
if v == "" {
|
||||
return nil, nil
|
||||
}
|
||||
n, err := strconv.Atoi(v)
|
||||
if err != nil || n < 0 {
|
||||
return nil, fmt.Errorf("%s must be a whole number, or blank for none", what)
|
||||
}
|
||||
return &n, nil
|
||||
}
|
||||
var req adminapi.QuotaRequest
|
||||
req.Scope = scope
|
||||
var err error
|
||||
for _, f := range []struct {
|
||||
dst **int
|
||||
name string
|
||||
what string
|
||||
}{
|
||||
{&req.Guarantee.GPUs, "gpus", "guaranteed GPUs"},
|
||||
{&req.Guarantee.CPUCores, "cores", "guaranteed cores"},
|
||||
{&req.Guarantee.MemoryGB, "memory", "guaranteed memory"},
|
||||
{&req.Limit.GPUs, "limit_gpus", "the GPU limit"},
|
||||
{&req.Limit.CPUCores, "limit_cores", "the core limit"},
|
||||
{&req.Limit.MemoryGB, "limit_memory", "the memory limit"},
|
||||
} {
|
||||
if *f.dst, err = field(f.name, f.what); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
key, err := p.capacity.SetQuota(r.Context(), req, admin.Email)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "capacity quota set on " + key + "; it takes effect within half a minute", nil
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeCapacity struct {
|
||||
views []adminapi.QuotaView
|
||||
set []adminapi.QuotaRequest
|
||||
cleared []string
|
||||
}
|
||||
|
||||
func (f *fakeCapacity) QuotaViews(context.Context) ([]adminapi.QuotaView, error) { return f.views, nil }
|
||||
func (f *fakeCapacity) SetQuota(_ context.Context, r adminapi.QuotaRequest, _ string) (string, error) {
|
||||
if r.Scope == "team:nope" {
|
||||
return "", errors.New(`there is no team "nope"`)
|
||||
}
|
||||
f.set = append(f.set, r)
|
||||
return r.Scope, nil
|
||||
}
|
||||
func (f *fakeCapacity) ClearQuota(_ context.Context, s string) (string, error) {
|
||||
f.cleared = append(f.cleared, s)
|
||||
return s, nil
|
||||
}
|
||||
func (f *fakeCapacity) Scopes(context.Context) ([]adminapi.ScopeName, error) {
|
||||
return []adminapi.ScopeName{{Key: "org:o1", Name: "Acme"}, {Key: "team:t1", Name: "Acme / ml"}}, nil
|
||||
}
|
||||
|
||||
func TestCapacityQuotasAreShownAndSetFromTheQuotasPage(t *testing.T) {
|
||||
eight, sixteen := 8, 16
|
||||
fc := &fakeCapacity{views: []adminapi.QuotaView{{Scope: "team:t1", Name: "Acme / ml",
|
||||
Guarantee: store.Resources{GPUs: &eight}, Limit: store.Resources{GPUs: &sixteen},
|
||||
Holds: quota.Amount{GPUs: 6}, UpdatedBy: "ops@acme"}}}
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
|
||||
p.SetCapacity(fc)
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminQuotas(mux, &fakeQuotas{}, pricedYes{})
|
||||
|
||||
body := mirrorGet(t, mux, "/admin/quotas").Body.String()
|
||||
for _, want := range []string{`id="capacity"`, "Acme / ml", "6 of 8 (at most 16)", "ops@acme",
|
||||
`<option value="team:t1">Acme / ml</option>`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// "change" fills the form with the scope's figures.
|
||||
edit := mirrorGet(t, mux, "/admin/quotas?cap=team:t1").Body.String()
|
||||
if !strings.Contains(edit, `<option value="team:t1" selected>`) || !strings.Contains(edit, `name="limit_gpus" value="16"`) {
|
||||
t.Error("change did not open the form on the scope's quota")
|
||||
}
|
||||
|
||||
res := mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"},
|
||||
"gpus": {"8"}, "cores": {""}, "limit_memory": {"64"}}).Body.String()
|
||||
if len(fc.set) != 1 || *fc.set[0].Guarantee.GPUs != 8 || fc.set[0].Guarantee.CPUCores != nil || *fc.set[0].Limit.MemoryGB != 64 {
|
||||
t.Errorf("sent %+v", fc.set)
|
||||
}
|
||||
if !strings.Contains(res, "capacity quota set on team:t1") {
|
||||
t.Errorf("no word of it: %s", res)
|
||||
}
|
||||
for _, bad := range []url.Values{{"scope": {"team:t1"}, "gpus": {"eight"}}, {"scope": {"team:nope"}, "gpus": {"1"}}, {"gpus": {"1"}}} {
|
||||
bad.Set("csrf", "tok")
|
||||
n := len(fc.set)
|
||||
out := mirrorPost(t, mux, "/admin/quotas/capacity", bad).Body.String()
|
||||
if len(fc.set) != n || !strings.Contains(out, `class="error"`) {
|
||||
t.Errorf("%v was not refused with a reason", bad)
|
||||
}
|
||||
}
|
||||
mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"}, "clear": {"1"}})
|
||||
if len(fc.cleared) != 1 || fc.cleared[0] != "team:t1" {
|
||||
t.Errorf("cleared %v", fc.cleared)
|
||||
}
|
||||
// Without a CSRF token nothing happens.
|
||||
mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"scope": {"team:t1"}, "clear": {"1"}})
|
||||
if len(fc.cleared) != 1 {
|
||||
t.Error("a change was made without the CSRF token")
|
||||
}
|
||||
}
|
||||
@@ -71,6 +71,11 @@ type modelUse struct {
|
||||
Replicas int
|
||||
// Min and Max are its scaling as the catalog holds it.
|
||||
Min, Max int
|
||||
// Class is what it set, empty for its default; EffClass what that comes
|
||||
// to. Owner is org:ID or empty for shared. MemoryGB and CPUCores are what
|
||||
// a replica of its first profile reserves.
|
||||
Class, EffClass, Owner string
|
||||
MemoryGB, CPUCores int
|
||||
// Ready, Wanted and Nodes are how it runs now, for the row itself:
|
||||
// replicas ready of those it should have, and the nodes they are on.
|
||||
// Fleet is false where nothing could be asked, so the row says nothing
|
||||
@@ -293,7 +298,7 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
return name + " now runs all the time", nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setReplicas))
|
||||
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setModel))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
|
||||
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
@@ -463,8 +468,12 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
u := modelUse{
|
||||
Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
|
||||
Min: m.Scaling.Min, Max: m.Scaling.Max,
|
||||
Class: m.Class, EffClass: m.EffectiveClass(), Owner: m.Owner,
|
||||
Declared: p.catalog.Declared(m.Name),
|
||||
}
|
||||
if len(m.Profiles) > 0 {
|
||||
u.MemoryGB, u.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
|
||||
}
|
||||
if v, ok := views[m.Name]; ok {
|
||||
u.Fleet, u.Ready, u.Wanted = true, v.Ready, v.Desired
|
||||
u.Parked, u.Unplaceable = v.Parked, v.Unplaceable
|
||||
@@ -484,7 +493,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
for i := range d.Rows {
|
||||
d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
|
||||
if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
|
||||
d.Rows[i].Detail = p.heldDetailFor(d.Rows[i])
|
||||
d.Rows[i].Detail = p.heldDetailFor(r.Context(), d.Rows[i])
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -129,12 +130,17 @@ type adminQuotasData struct {
|
||||
// Editing is the "scope:id" whose form is open, so the template can keep
|
||||
// the link on every other row.
|
||||
Editing string
|
||||
Since string
|
||||
Message string
|
||||
Error string
|
||||
CSRF string
|
||||
Version string
|
||||
Now string
|
||||
// The capacity section: quotas on GPUs, cores and memory.
|
||||
HasCapacity bool
|
||||
Capacity []capacityRow
|
||||
Scopes []adminapi.ScopeName
|
||||
CapForm capacityForm
|
||||
Since string
|
||||
Message string
|
||||
Error string
|
||||
CSRF string
|
||||
Version string
|
||||
Now string
|
||||
}
|
||||
|
||||
// RegisterAdminQuotas mounts quota administration under /admin/quotas.
|
||||
@@ -176,6 +182,24 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
|
||||
}
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
|
||||
func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
sess, _, _ := p.session(r)
|
||||
if p.capacity == nil || !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
msg, err := p.applyCapacity(r, admin)
|
||||
if err != nil {
|
||||
p.log.Warn("a capacity quota change was refused",
|
||||
"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
|
||||
p.renderQuotas(w, r, s, "", err.Error())
|
||||
return
|
||||
}
|
||||
p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
}))
|
||||
}
|
||||
|
||||
// applyQuota writes one scope's limits, and says what it did.
|
||||
@@ -259,6 +283,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
|
||||
d.Editing = r.URL.Query().Get("edit")
|
||||
p.capacitySection(r.Context(), &d, r.FormValue("cap"))
|
||||
|
||||
limits, err := s.AllLimits(r.Context())
|
||||
if err != nil {
|
||||
|
||||
@@ -22,6 +22,11 @@ type CatalogAdmin interface {
|
||||
// Scale changes how a model added at runtime runs: always, on demand, or
|
||||
// not at all. One declared in configuration is refused.
|
||||
Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error)
|
||||
// SetClass, SetOwner and Resources change what a model's capacity is
|
||||
// worth, whose quota it counts against, and what a replica reserves.
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// serveForm is the offer to put one mirrored version into service.
|
||||
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
@@ -21,6 +22,9 @@ type fakeCatalog struct {
|
||||
scaled []string
|
||||
lastScale catalog.Scaling
|
||||
err error
|
||||
classes []string
|
||||
owners []string
|
||||
resources []string
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
|
||||
@@ -35,6 +39,21 @@ func (f *fakeCatalog) Sources(context.Context) (map[string][]catalog.Model, erro
|
||||
return f.sources, nil
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
|
||||
f.classes = append(f.classes, name+"="+class)
|
||||
return catalog.Model{Name: name, Class: class}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
|
||||
f.owners = append(f.owners, name+"="+owner)
|
||||
return catalog.Model{Name: name, Owner: owner}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) Resources(_ context.Context, name, _ string, mem, cores int, _ string) (catalog.Model, error) {
|
||||
f.resources = append(f.resources, fmt.Sprintf("%s=%d/%d", name, mem, cores))
|
||||
return catalog.Model{Name: name}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) Scale(_ context.Context, name string, sc catalog.Scaling, _ string) (catalog.Model, error) {
|
||||
f.scaled = append(f.scaled, name)
|
||||
f.lastScale = sc
|
||||
|
||||
@@ -29,6 +29,9 @@ const maxReplicas = 64
|
||||
// and room to change it.
|
||||
type heldDetail struct {
|
||||
Models []heldModel
|
||||
// Scopes are the parts of the tenancy tree a model can belong to; empty
|
||||
// when capacity quotas are not set up, and the owner choice is left off.
|
||||
Scopes []adminapi.ScopeName
|
||||
}
|
||||
|
||||
// heldModel is one catalog model running a held version.
|
||||
@@ -64,8 +67,11 @@ func openKey(model, version string) string { return model + ":" + version }
|
||||
|
||||
// heldDetailFor reads, for each model using a row, how it runs and how much
|
||||
// more room it has.
|
||||
func (p *AccountPages) heldDetailFor(row mirrorRow) *heldDetail {
|
||||
func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDetail {
|
||||
d := &heldDetail{}
|
||||
if p.capacity != nil {
|
||||
d.Scopes, _ = p.capacity.Scopes(ctx)
|
||||
}
|
||||
if p.fleet == nil {
|
||||
for _, u := range row.UsedBy {
|
||||
d.Models = append(d.Models, heldModel{modelUse: u})
|
||||
@@ -98,7 +104,7 @@ func (p *AccountPages) heldDetailFor(row mirrorRow) *heldDetail {
|
||||
if h, ok := hs.Headroom(u.Name, 16); ok {
|
||||
hm.Headroom = &h
|
||||
if !h.Limited {
|
||||
hm.MostNow = h.Current + h.More
|
||||
hm.MostNow = h.Current + h.More + h.ByPreempting
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -144,7 +150,7 @@ func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *htt
|
||||
if !ok {
|
||||
return "", fmt.Errorf("no model %q", name)
|
||||
}
|
||||
fits := h.Current + h.More
|
||||
fits := h.Current + h.More + h.ByPreempting
|
||||
if lo > fits && !h.Limited {
|
||||
why := "nothing more fits"
|
||||
if len(h.Blocked) > 0 {
|
||||
@@ -156,6 +162,11 @@ func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *htt
|
||||
if hi > fits && !h.Limited {
|
||||
note = fmt.Sprintf("; there is room for %d now, so it grows past that only if room is made", fits)
|
||||
}
|
||||
// Room its class would make by stopping others: said, since it is
|
||||
// somebody else's replica that goes.
|
||||
if lo > h.Current+h.More && len(h.Preempts) > 0 {
|
||||
note += "; to make room, replicas of " + strings.Join(h.Preempts, ", ") + " will be stopped"
|
||||
}
|
||||
}
|
||||
|
||||
sc := catalog.Scaling{Min: lo, Max: hi, IdleTimeout: current.Scaling.IdleTimeout}
|
||||
@@ -171,7 +182,7 @@ func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *htt
|
||||
case lo == 0:
|
||||
return fmt.Sprintf("%s loads when asked for, up to %d under load%s", name, hi, note), nil
|
||||
case lo == hi:
|
||||
return fmt.Sprintf("%s runs %s", name, plural(lo, "replica", "replicas")), nil
|
||||
return fmt.Sprintf("%s runs %s%s", name, plural(lo, "replica", "replicas"), note), nil
|
||||
}
|
||||
return fmt.Sprintf("%s runs %d, and up to %d under load%s", name, lo, hi, note), nil
|
||||
}
|
||||
@@ -205,3 +216,69 @@ func age(sec int64) string {
|
||||
}
|
||||
return fmt.Sprintf("%dd", int(d.Hours()/24))
|
||||
}
|
||||
|
||||
// setModel applies the card's settings form: what a replica reserves, the
|
||||
// model's class and owner, then how many replicas it runs — in that order, so
|
||||
// the room checked for the replicas is the room its new class and reservation
|
||||
// would get. Only what changed is applied, and each change is said.
|
||||
func (p *AccountPages) setModel(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
name := r.FormValue("model")
|
||||
current, err := p.currentModel(ctx, name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if p.catalog.Declared(name) {
|
||||
return "", fmt.Errorf("%s is set in server.yaml; change it there", name)
|
||||
}
|
||||
var done []string
|
||||
if r.Form.Has("memory") && len(current.Profiles) > 0 {
|
||||
mem, err1 := strconv.Atoi(strings.TrimSpace(r.FormValue("memory")))
|
||||
cores, err2 := strconv.Atoi(strings.TrimSpace(r.FormValue("cores")))
|
||||
if err1 != nil || err2 != nil || mem < 0 || cores < 0 {
|
||||
return "", errors.New("memory and cores are whole numbers")
|
||||
}
|
||||
if cur := current.Profiles[0]; mem != cur.MinMemoryGB || cores != cur.CPUCores {
|
||||
if _, err := p.catalog.Resources(ctx, name, "", mem, cores, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
done = append(done, fmt.Sprintf("a replica reserves %d GB and %d cores (running ones keep theirs until restarted)", mem, cores))
|
||||
}
|
||||
}
|
||||
if r.Form.Has("class") {
|
||||
if c := r.FormValue("class"); c != current.Class {
|
||||
m, err := p.catalog.SetClass(ctx, name, c, admin.Email)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
done = append(done, "is "+m.EffectiveClass())
|
||||
}
|
||||
}
|
||||
if r.Form.Has("owner") && p.capacity != nil {
|
||||
if o := r.FormValue("owner"); o != current.Owner {
|
||||
if _, err := p.catalog.SetOwner(ctx, name, o, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if o == "" {
|
||||
done = append(done, "is shared")
|
||||
} else {
|
||||
done = append(done, "counts against "+o)
|
||||
}
|
||||
}
|
||||
}
|
||||
lo, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("min")))
|
||||
hi, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("max")))
|
||||
if r.Form.Has("min") && (lo != current.Scaling.Min || hi != current.Scaling.Max) {
|
||||
msg, err := p.setReplicas(ctx, admin, r)
|
||||
if err != nil {
|
||||
if len(done) > 0 {
|
||||
return "", fmt.Errorf("%s %s, but not the replicas: %w", name, strings.Join(done, ", "), err)
|
||||
}
|
||||
return "", err
|
||||
}
|
||||
done = append(done, strings.TrimPrefix(msg, name+" "))
|
||||
}
|
||||
if len(done) == 0 {
|
||||
return name + ": nothing changed", nil
|
||||
}
|
||||
return name + " " + strings.Join(done, "; "), nil
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -8,14 +10,17 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// roomyFleet runs one replica of "coder" on n2 and has room for `more`.
|
||||
type roomyFleet struct {
|
||||
fakeFleet
|
||||
more int
|
||||
blocked []string
|
||||
asked int
|
||||
more int
|
||||
blocked []string
|
||||
asked int
|
||||
preempt int
|
||||
preempts []string
|
||||
}
|
||||
|
||||
func (f *roomyFleet) Nodes() []adminapi.NodeView {
|
||||
@@ -41,6 +46,7 @@ func (f *roomyFleet) Headroom(model string, limit int) (adminapi.HeadroomView, b
|
||||
} else {
|
||||
h.Limited = true
|
||||
}
|
||||
h.ByPreempting, h.Preempts = f.preempt, f.preempts
|
||||
return h, true
|
||||
}
|
||||
|
||||
@@ -181,3 +187,85 @@ func TestNodeNamesLoseOnlyWhatEveryNodeShares(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func settingsPage(t *testing.T, cat *fakeCatalog, withQuotas bool) *http.ServeMux {
|
||||
t.Helper()
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
|
||||
if withQuotas {
|
||||
p.SetCapacity(&fakeCapacity{})
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
|
||||
}}, Catalog: cat, Fleet: &roomyFleet{more: 3}})
|
||||
return mux
|
||||
}
|
||||
|
||||
func TestACardSetsClassOwnerAndReservationAndOnlyWhatChanged(t *testing.T) {
|
||||
cat := coderCatalog()
|
||||
cat.sources["http://mirror:8081/coder/v1/c.gguf"][0].Profiles = []catalog.Profile{{Name: "cpu", MinMemoryGB: 12, CPUCores: 4}}
|
||||
mux := settingsPage(t, cat, true)
|
||||
|
||||
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
|
||||
for _, want := range []string{`name="class"`, `<option value="" selected>default (standard)</option>`,
|
||||
`name="owner"`, `<option value="team:t1">Acme / ml</option>`, `name="memory" value="12"`, `name="cores" value="4"`} {
|
||||
if !strings.Contains(card, want) {
|
||||
t.Errorf("the card lacks %q", want)
|
||||
}
|
||||
}
|
||||
post := func(v url.Values) string {
|
||||
v.Set("csrf", "tok")
|
||||
v.Set("model", "coder")
|
||||
v.Set("open", "coder:v1")
|
||||
return mirrorPost(t, mux, "/admin/mirror/replicas", v).Body.String()
|
||||
}
|
||||
same := url.Values{"min": {"1"}, "max": {"1"}, "class": {""}, "owner": {""}, "memory": {"12"}, "cores": {"4"}}
|
||||
if body := post(same); !strings.Contains(body, "coder: nothing changed") ||
|
||||
len(cat.classes)+len(cat.owners)+len(cat.resources)+len(cat.scaled) != 0 {
|
||||
t.Errorf("an unchanged form changed something: %v %v %v %v", cat.classes, cat.owners, cat.resources, cat.scaled)
|
||||
}
|
||||
changed := url.Values{"min": {"1"}, "max": {"1"}, "class": {"critical"}, "owner": {"team:t1"}, "memory": {"14"}, "cores": {"4"}}
|
||||
body := post(changed)
|
||||
if strings.Join(cat.resources, ",") != "coder=14/4" || strings.Join(cat.classes, ",") != "coder=critical" ||
|
||||
strings.Join(cat.owners, ",") != "coder=team:t1" || len(cat.scaled) != 0 {
|
||||
t.Errorf("applied %v %v %v %v", cat.resources, cat.classes, cat.owners, cat.scaled)
|
||||
}
|
||||
if !strings.Contains(body, "a replica reserves 14 GB and 4 cores") || !strings.Contains(body, "counts against team:t1") {
|
||||
t.Errorf("the change was not said: %s", body)
|
||||
}
|
||||
|
||||
// Without capacity quotas there is no owner to choose, and none is sent.
|
||||
plain := mirrorGet(t, settingsPage(t, coderCatalog(), false), "/admin/mirror?open=coder:v1").Body.String()
|
||||
if strings.Contains(plain, `name="owner"`) {
|
||||
t.Error("an owner choice with no quotas to count against")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRoomAHigherClassWouldMakeIsOfferedAndSaid(t *testing.T) {
|
||||
fleet := &roomyFleet{more: 0, preempt: 1, preempts: []string{"chat"}, blocked: []string{"n1: full"}}
|
||||
cat := coderCatalog()
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
|
||||
}}, Catalog: cat, Fleet: fleet}, true)
|
||||
|
||||
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
|
||||
if !strings.Contains(card, "No free room for another replica now.") ||
|
||||
!strings.Contains(card, "1 by stopping replicas its class outranks: chat.") ||
|
||||
!strings.Contains(card, `name="min" value="1" min="0" max="2"`) {
|
||||
t.Errorf("the card should offer the room preemption makes, and say whose")
|
||||
}
|
||||
body := mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
|
||||
"model": {"coder"}, "min": {"2"}, "max": {"2"}, "open": {"coder:v1"}}).Body.String()
|
||||
if len(cat.scaled) != 1 || cat.lastScale.Min != 2 {
|
||||
t.Fatalf("two were refused though preemption makes room: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "replicas of chat will be stopped") {
|
||||
t.Errorf("stopping somebody else's replica should be said: %s", body)
|
||||
}
|
||||
body = mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
|
||||
"model": {"coder"}, "min": {"3"}, "max": {"3"}, "open": {"coder:v1"}}).Body.String()
|
||||
if len(cat.scaled) != 1 || !strings.Contains(body, "at most 2") {
|
||||
t.Errorf("three is past even what preemption makes: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
@@ -174,3 +175,33 @@ func TestTheNVLinkPageDrawsEachDomainAsARack(t *testing.T) {
|
||||
t.Error("the cluster page does not lead to the racks")
|
||||
}
|
||||
}
|
||||
|
||||
// hotSource is a GPU node whose sensors report, one card held back.
|
||||
type hotSource struct{ fakeSource }
|
||||
|
||||
func (hotSource) Nodes() []adminapi.NodeView {
|
||||
n := fakeSource{}.Nodes()[0]
|
||||
n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
|
||||
CPUTempC: 58, CPUPowerW: 312, InletTempC: 24.5,
|
||||
GPUs: []inventory.GPUSensor{
|
||||
{Index: 0, TempC: 71, MemoryTempC: 84, PowerW: 612, PowerLimitW: 700},
|
||||
{Index: 1, TempC: 88, PowerW: 698, PowerLimitW: 700, Throttle: []string{"sw_power_cap", "sw_thermal"}},
|
||||
}}
|
||||
return []adminapi.NodeView{n}
|
||||
}
|
||||
|
||||
func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
|
||||
_, body := get(t, nodeMux(hotSource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, want := range []string{"58 °C", "312 W", "24.5 °C", "inlet air", "84 °C", "612 W", "of 700",
|
||||
`<span class="bad">sw_power_cap, sw_thermal</span>`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
_, vm := get(t, nodeMux(busySource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, absent := range []string{"°C", "<h2>GPUs</h2>", "inlet air"} {
|
||||
if strings.Contains(vm, absent) {
|
||||
t.Errorf("a node with no sensors shows %q", absent)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -855,3 +855,6 @@ table.explore .from { margin-left: .5rem; font-size: .78rem; color: var(--muted)
|
||||
.clip-inline { display: inline-block; max-width: 24rem; overflow: hidden; text-overflow: ellipsis;
|
||||
vertical-align: bottom; font-size: .8rem; color: var(--muted); margin-left: .4rem; }
|
||||
table.rows td.warn { color: var(--warn); cursor: help; }
|
||||
.held-card .replicas-form select, .capacity-form select { display: block; margin-top: .25rem; }
|
||||
.held-card .replicas-form .field-row { flex-wrap: wrap; gap: .75rem 1rem; }
|
||||
.quota-tree td.q-act form.inline { margin-left: .75rem; }
|
||||
@@ -443,7 +443,7 @@ computes once it has it.</p>
|
||||
{{end}}
|
||||
|
||||
{{/* The card beneath a held version: what runs it, where, and how many. */}}
|
||||
{{define "mirror_held_detail"}}{{$row := .Row}}{{$csrf := .CSRF}}
|
||||
{{define "mirror_held_detail"}}{{$row := .Row}}{{$csrf := .CSRF}}{{$scopes := .D.Scopes}}
|
||||
<div class="held-card">
|
||||
<p class="repo-meta">
|
||||
<span>{{if gt $row.Files 1}}{{$row.Files}} files{{else}}<code>{{$row.File}}</code>{{end}}</span>
|
||||
@@ -482,7 +482,8 @@ computes once it has it.</p>
|
||||
<p class="headroom">
|
||||
{{if .Limited}}Room for at least {{.More}} more replicas.
|
||||
{{else if .More}}Room for {{.More}} more now{{if .Nodes}}, on {{range $i, $n := .Nodes}}{{if $i}}, {{end}}{{join $n}}{{end}}{{end}}.
|
||||
{{else}}No room for another replica now.{{end}}
|
||||
{{else}}No free room for another replica now.{{end}}
|
||||
{{if .ByPreempting}}<br><span class="note">{{if .More}}And {{end}}{{.ByPreempting}} {{if .More}}more {{end}}by stopping replicas its class outranks: {{join .Preempts}}.</span>{{end}}
|
||||
</p>
|
||||
{{if and (not .Limited) .Blocked}}<details class="blocked"><summary>Why not {{if .More}}more{{else}}another{{end}}</summary>
|
||||
<ul>{{range .Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
|
||||
@@ -500,11 +501,30 @@ computes once it has it.</p>
|
||||
<input type="number" name="min" value="{{.Min}}" min="0"{{if .MostNow}} max="{{.MostNow}}"{{end}} required></label>
|
||||
<label>Up to, under load
|
||||
<input type="number" name="max" value="{{.Max}}" min="0" max="64" required></label>
|
||||
<button type="submit">Update</button>
|
||||
<label>Class
|
||||
<select name="class">
|
||||
<option value=""{{if eq .Class ""}} selected{{end}}>default ({{if eq .Mode "ondemand"}}opportunistic{{else}}standard{{end}})</option>
|
||||
<option value="critical"{{if eq .Class "critical"}} selected{{end}}>critical</option>
|
||||
<option value="standard"{{if eq .Class "standard"}} selected{{end}}>standard</option>
|
||||
<option value="opportunistic"{{if eq .Class "opportunistic"}} selected{{end}}>opportunistic</option>
|
||||
</select></label>
|
||||
{{if $scopes}}{{$owner := .Owner}}<label>Owner
|
||||
<select name="owner">
|
||||
<option value=""{{if eq $owner ""}} selected{{end}}>shared</option>
|
||||
{{range $scopes}}<option value="{{.Key}}"{{if eq .Key $owner}} selected{{end}}>{{.Name}}</option>{{end}}
|
||||
</select></label>{{end}}
|
||||
</div>
|
||||
<p class="note">{{if .MostNow}}At most {{.MostNow}} can always run with what is free now. {{end}}0 always running with a
|
||||
<div class="field-row">
|
||||
<label>Memory GB <span class="opt">per replica</span>
|
||||
<input type="number" name="memory" value="{{.MemoryGB}}" min="0"></label>
|
||||
<label>Cores <span class="opt">per replica, its threads</span>
|
||||
<input type="number" name="cores" value="{{.CPUCores}}" min="0"></label>
|
||||
<button type="submit">Save</button>
|
||||
</div>
|
||||
<p class="note">{{if .MostNow}}At most {{.MostNow}} can always run now{{if .Headroom}}{{if .Headroom.ByPreempting}}, counting what its class may stop to make room{{end}}{{end}}. {{end}}0 always running with a
|
||||
higher limit loads it when asked for; 0 and 0 stops it. A replica is added under load when requests wait, and
|
||||
taken away after ten quiet minutes.</p>
|
||||
taken away after ten quiet minutes. A higher class may stop replicas of lower ones to make room; memory and
|
||||
cores apply to replicas started after the change.</p>
|
||||
</form>
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
@@ -147,6 +147,60 @@ keys, projects and the chat page. {{.Personal}}</p>
|
||||
</table>
|
||||
{{else}}<p class="empty">Nobody has been approved yet.</p>{{end}}
|
||||
|
||||
{{if .HasCapacity}}
|
||||
<h2 id="capacity">Capacity</h2>
|
||||
<p class="note">What a part of the tree is guaranteed of the cluster, and the most
|
||||
it may hold: GPUs, CPU cores and memory, counted from the replicas of the models
|
||||
it owns and of those beneath it. What it holds within its guarantee is never taken
|
||||
for someone else's model; beyond it, it borrows free room and gives it back when
|
||||
asked. A model's owner is set on its card on the mirror page.</p>
|
||||
{{if .Capacity}}
|
||||
<table class="rows quota-tree">
|
||||
<tr><th>Where</th><th class="num">GPUs</th><th class="num">Cores</th><th class="num">Memory GB</th><th>Set by</th><th></th></tr>
|
||||
{{range .Capacity}}<tr>
|
||||
<td class="q-where"><span class="q-name">{{or .Name .Scope}}</span></td>
|
||||
<td class="num">{{or .GPUs "—"}}</td>
|
||||
<td class="num">{{or .Cores "—"}}</td>
|
||||
<td class="num">{{or .Memory "—"}}</td>
|
||||
<td class="note">{{.UpdatedBy}}</td>
|
||||
<td class="q-act"><a href="/admin/quotas?cap={{.Scope}}#capacity">change</a>
|
||||
<form method="post" action="/admin/quotas/capacity" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="scope" value="{{.Scope}}">
|
||||
<button type="submit" name="clear" value="1" class="link danger">remove</button>
|
||||
</form></td>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<p class="q-legend">Each figure is held of guaranteed (at most the limit); — is not governed.</p>
|
||||
{{else}}<p class="empty">No capacity quotas: every model shares the cluster by its class alone.</p>{{end}}
|
||||
|
||||
{{if .Scopes}}
|
||||
<form method="post" action="/admin/quotas/capacity" class="q-form capacity-form">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<p class="q-form-title"><strong>{{if .CapForm.Scope}}Change{{else}}Set{{end}} a capacity quota.</strong>
|
||||
A blank field is not governed; 0 is none at all.</p>
|
||||
<label>Where
|
||||
<select name="scope" required>
|
||||
<option value="">choose…</option>
|
||||
{{range .Scopes}}<option value="{{.Key}}"{{if eq .Key $.CapForm.Scope}} selected{{end}}>{{.Name}}</option>{{end}}
|
||||
</select></label>
|
||||
<div class="field-row">
|
||||
<label>Guaranteed GPUs <input type="text" name="gpus" value="{{.CapForm.GPUs}}" placeholder="—"></label>
|
||||
<label>Cores <input type="text" name="cores" value="{{.CapForm.Cores}}" placeholder="—"></label>
|
||||
<label>Memory GB <input type="text" name="memory" value="{{.CapForm.Memory}}" placeholder="—"></label>
|
||||
</div>
|
||||
<div class="field-row">
|
||||
<label>At most GPUs <input type="text" name="limit_gpus" value="{{.CapForm.LGPUs}}" placeholder="—"></label>
|
||||
<label>Cores <input type="text" name="limit_cores" value="{{.CapForm.LCores}}" placeholder="—"></label>
|
||||
<label>Memory GB <input type="text" name="limit_memory" value="{{.CapForm.LMem}}" placeholder="—"></label>
|
||||
</div>
|
||||
<div class="q-form-actions"><button type="submit">Save</button>
|
||||
{{if .CapForm.Scope}}<a href="/admin/quotas#capacity">Cancel</a>{{end}}</div>
|
||||
</form>
|
||||
{{else}}<p class="empty">Nothing to give a capacity quota to yet. Make an organization on the
|
||||
<a href="/admin/users">people page</a> first.</p>{{end}}
|
||||
{{end}}
|
||||
|
||||
<details class="explain">
|
||||
<summary>How limits combine</summary>
|
||||
<p>Along a branch the most restrictive value wins, so a team cannot raise what
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
<div class="tile"><div class="v">{{printf "%.0f" .CPUPercent}}%</div><div class="k">CPU{{if .CPUs}} · {{.CPUs}} cores{{end}}</div>{{template "minibar" pctbar .CPUPercent}}</div>
|
||||
<div class="tile"><div class="v">{{bytes .MemUsed}}</div><div class="k">RAM in use · of {{bytes .MemTotal}}</div>{{template "minibar" pctbar .MemPercent}}</div>
|
||||
<div class="tile"><div class="v">{{printf "%.2f" .Load1}}</div><div class="k">load · {{printf "%.2f" .Load5}} 5m · {{printf "%.2f" .Load15}} 15m</div></div>
|
||||
{{/* Only what this machine has sensors for: a VM has none, and a tile
|
||||
saying 0 °C would be a reading nobody took. */}}
|
||||
{{if or .CPUTempC .CPUPowerW}}<div class="tile"><div class="v">{{if .CPUTempC}}{{printf "%.0f" .CPUTempC}} °C{{else}}—{{end}}</div><div class="k">CPU{{if .CPUPowerW}} · {{printf "%.0f" .CPUPowerW}} W{{end}}</div></div>{{end}}
|
||||
{{if .InletTempC}}<div class="tile"><div class="v">{{printf "%.1f" .InletTempC}} °C</div><div class="k">inlet air</div></div>{{end}}
|
||||
{{else}}
|
||||
<div class="tile"><div class="v">—</div><div class="k">CPU and RAM not measured lately</div></div>
|
||||
{{end}}
|
||||
@@ -29,6 +33,17 @@
|
||||
</div>
|
||||
{{if .Warnings}}{{range .Warnings}}<p class="note bad">{{.}}</p>{{end}}{{end}}
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{with .Load}}{{if .GPUs}}
|
||||
<h2>GPUs</h2>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
|
||||
{{range .GPUs}}<tr>
|
||||
<td class="num">{{.Index}}</td>
|
||||
<td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{end}}{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if or .Node.IBPorts .Node.Fabrics}}
|
||||
|
||||
@@ -109,6 +109,13 @@ var funcs = template.FuncMap{
|
||||
return p
|
||||
},
|
||||
"ago": ago,
|
||||
// percent is part of whole, 0 to 100, for a bar.
|
||||
"percent": func(part, whole float64) float64 {
|
||||
if whole <= 0 {
|
||||
return 0
|
||||
}
|
||||
return part * 100 / whole
|
||||
},
|
||||
// shortNode is a node's name without what every node's name starts with.
|
||||
"shortNode": strings.TrimPrefix,
|
||||
"age": age,
|
||||
|
||||
+15
-4
@@ -1463,8 +1463,8 @@ The launch path is now built too:
|
||||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||||
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
|
||||
rather than printed
|
||||
- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
|
||||
and the autoscaler and profile choice use what `ctl bench --save` measured
|
||||
- [x] OTel tracing: an inference span with admission, schedule and each
|
||||
upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
|
||||
set, and the traceparent carried on to the engine
|
||||
@@ -1477,10 +1477,21 @@ The launch path is now built too:
|
||||
### Phase 6 — v2 ideas
|
||||
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
|
||||
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
||||
- [ ] Admission displacement (dropping requests from lower bands)
|
||||
- [x] Admission displacement: a total queue bound, and the newest request of
|
||||
the lowest band turned away (503, Retry-After) for a higher one
|
||||
- [x] Autoscaling on load: a replica added while requests wait, up to a
|
||||
model's max, and taken back after ten quiet minutes
|
||||
- [ ] Priority-based preemption
|
||||
- [x] Priority-based preemption: capacity classes (critical, standard,
|
||||
opportunistic), preemption tried on a copy of the inventory first
|
||||
- [x] Capacity quotas per organization, team and project: guarantees kept,
|
||||
borrowing taken back, limits enforced
|
||||
- [x] CPU cores reserved like memory, and memory required of CPU models
|
||||
- [x] Node sensors collected: GPU temperature, power and throttling, CPU
|
||||
temperature and RAPL power, board inlet air; published and shown
|
||||
- [ ] Rack power, temperature and cooling as placement signals: planned in
|
||||
[docs/facility-aware-placement.md](docs/facility-aware-placement.md),
|
||||
deferred — it needs each site's rack, PDU and power data first
|
||||
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
||||
- [ ] Control plane HA (Raft)
|
||||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||||
checked against their digest, and reclaiming what nothing uses
|
||||
|
||||
Reference in new issue
Block a user