Compare commits

...
2 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 5cf0003e74 Release v0.10.5
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m12s
CI / test (push) Successful in 3m53s
Collects rc.1 to rc.14: the mirror page in tabs with a card per model,
one line per record in every admin table, reserved CPU cores and required
memory, capacity classes and preemption, tenant capacity quotas, sizing by
measured throughput, and node sensor readings. Adds `llmgw upgrade` for the
mirror host, which `ctl upgrade` does not reach, and says so when it ends.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:26:19 +02:00
Ilker ManapandClaude Opus 5.5 d10fad064e Mark facility steps 1-5 deferred; node sensors collected
CI / test (push) Successful in 3m45s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 20:25:50 +02:00
7 changed files with 131 additions and 7 deletions

No files matched your search

+21 -3
View File
@@ -20,10 +20,21 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
## v0.10.5 — 2026-09-30
Changes since v0.10.4, published as release candidates (v0.10.5-rc.N) until
there are enough for v0.10.5.
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
format check and has no binaries.
After upgrading:
- **Declare what CPU models need.** A CPU model added from now on must say its
memory; one stored before this still loads and is flagged in `ctl models`
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
are then reserved per replica and become the engine's thread count, for
replicas started after the change: `llmgw ctl restart --model MODEL`.
- **Upgrade the mirror on its own host**, now with one command:
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
mirror older than this). `llmgw ctl upgrade` says so at the end.
### CPU cores are reserved, and memory has to be declared
@@ -132,6 +143,13 @@ decisions rather than only being printed:
A model never measured behaves exactly as before. Measurements are read from
the store every half minute.
### Upgrading a machine the control plane does not reach
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
the mirror's host above all, whose service runs unprivileged and cannot replace
its own binary as an agent does. The release's checksums are checked, the old
binary kept, the unit restarted and watched until it stays up.
### Nodes report their own heat and power
The first step of facility-aware placement (docs/facility-aware-placement.md),
+3 -1
View File
@@ -1,6 +1,8 @@
# Facility-aware placement: rack power, temperature and cooling
**Status: planned; its first step, reading the nodes' own sensors, is built.**
**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
deferred** until the rack, PDU and power information of a real site is at
hand — without it they would be designed against guesses.
This is the design for step 5 of resource allocation, after reserved CPU cores
and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
+12 -2
View File
@@ -940,8 +940,18 @@ restart either — it rebuilds the fleet from what the agents report.
fields, and both sides treat an absent one as unknown rather than as a default.
Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
installing the release binary and restarting `llmgw-agent`, and every upgrade
after that is the command above. The mirror, which is not an agent, is
upgraded the same way by hand.
after that is the command above.
**The mirror is upgraded on its own host**, since it is not an agent and its
service cannot replace its own binary:
```
sudo llmgw upgrade v0.10.5 --service llmgw-mirror
```
The same checks — the release's checksums, the old binary kept beside the new —
then a restart of the unit, waiting for it to stay up. A mirror older than
v0.10.5 does not have the command: install the release binary by hand once.
Then confirm what you have, rather than assuming:
+4
View File
@@ -172,6 +172,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
}
}
fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
// The one machine this does not reach. Said every time, because it is
// exactly the kind of thing that stays three versions behind unnoticed.
fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
" sudo llmgw upgrade %s --service llmgw-mirror\n", v)
return nil
}
+1
View File
@@ -47,6 +47,7 @@ func Root() *cobra.Command {
newSimEngineCmd(),
newDoctorCmd(),
newNodeCmd(),
newSelfUpgradeCmd(),
)
return root
}
+87
View File
@@ -0,0 +1,87 @@
package cli
import (
"errors"
"fmt"
"os"
"os/exec"
"runtime"
"strings"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
)
// newSelfUpgradeCmd upgrades this machine's llmgw on its own, for the machines
// `llmgw ctl upgrade` does not reach: a mirror host, above all, whose service
// runs unprivileged with the filesystem read-only and so cannot replace its own
// binary the way an agent does.
func newSelfUpgradeCmd() *cobra.Command {
var releases, service string
c := &cobra.Command{
Use: "upgrade VERSION [--service UNIT]",
Short: "Upgrade this machine's llmgw, and restart the service that runs it",
Long: "For a machine the control plane does not upgrade — the model mirror's host.\n" +
"The control plane and the nodes are upgraded with `llmgw ctl upgrade`.\n\n" +
"Fetches the release for this platform, checks it against the release's\n" +
"checksums, keeps the old binary beside the new one, and restarts --service,\n" +
"waiting for it to be running again. Run it with sudo.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
v := args[0]
out := cmd.OutOrStdout()
if !release.Tag.MatchString(v) {
return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
}
if version.Version == v {
fmt.Fprintf(out, "this machine already runs %s\n", v)
return nil
}
if os.Geteuid() != 0 {
return errors.New("replacing this machine's binary needs root: run it with sudo")
}
self, err := os.Executable()
if err != nil {
return err
}
backup, err := release.Install(cmd.Context(), nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
if err != nil {
return err
}
fmt.Fprintf(out, "%s -> %s: installed, checksum verified; the old binary is %s\n",
version.Version, v, backup)
if service == "" {
return nil
}
//nolint:gosec // the unit name is the operator's own flag
if b, err := exec.CommandContext(cmd.Context(), "systemctl", "restart", service).CombinedOutput(); err != nil {
return fmt.Errorf("restarting %s: %v: %s", service, err, strings.TrimSpace(string(b)))
}
// Running, and still running a few seconds on: a unit that starts
// and dies on a bad flag is "active" for a moment first.
deadline := time.Now().Add(30 * time.Second)
steady := 0
for time.Now().Before(deadline) {
//nolint:gosec // as above
b, _ := exec.CommandContext(cmd.Context(), "systemctl", "is-active", service).Output()
if strings.TrimSpace(string(b)) == "active" {
if steady++; steady >= 3 {
fmt.Fprintf(out, "%s restarted and running\n", service)
return nil
}
} else {
steady = 0
}
time.Sleep(2 * time.Second)
}
return fmt.Errorf("%s is not running after the upgrade; see journalctl -u %s, and %s holds the previous binary",
service, service, backup)
},
}
c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from")
c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
return c
}
+3 -1
View File
@@ -1485,9 +1485,11 @@ The launch path is now built too:
- [x] Capacity quotas per organization, team and project: guarantees kept,
borrowing taken back, limits enforced
- [x] CPU cores reserved like memory, and memory required of CPU models
- [x] Node sensors collected: GPU temperature, power and throttling, CPU
temperature and RAPL power, board inlet air; published and shown
- [ ] Rack power, temperature and cooling as placement signals: planned in
[docs/facility-aware-placement.md](docs/facility-aware-placement.md),
in five steps from rack spreading to opt-in shedding
deferred — it needs each site's rack, PDU and power data first
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
- [ ] Control plane HA (Raft)
- [x] Weight downloading and cache management: the mirror, agent fetches