v0.16.2: changelog and README

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A set wanted by several replicas on one node is fetched once
2026-11-02 08:32:34 +01:00 · 2026-11-02 08:29:25 +01:00 · 2026-11-01 17:30:08 +01:00 · 2026-11-01 16:51:54 +01:00 · 2026-11-01 13:47:47 +01:00 · 2026-11-01 09:51:26 +01:00
99 changed files with 4171 additions and 223 deletions
@@ -0,0 +1,20 @@
+# Fails when an engine llmgw follows release by release (FreeToken) has
+# published a version newer than the one DefaultEngines installs: the failed
+# run is the notice to check the new version and move to it. See
+# scripts/engine_releases.py.
+name: Engine releases
+
+on:
+  schedule:
+    # Mondays 06:00 UTC.
+    - cron: "0 6 * * 1"
+  workflow_dispatch:
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Compare with PyPI
+        run: python3 scripts/engine_releases.py
@@ -46,6 +46,8 @@ providers, of L5 cache-write and audio prices. L6 and L7 are llmgw-80's; L8 (A2A
 cluster's hardware holds, with CPU and GPU tok/s). Next M2 (every download pinned and checked against Hugging
 Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.

+**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
+
 **Governance.**
 7. **C1**: a tenant-admin role, and forms hidden from viewers.
 8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
@@ -1686,6 +1688,8 @@ how to fix it by hand (`ctl model resources`, then `ctl restart`).
 Rate limits are counted in each control plane's memory. A failover starts
 every key's minute afresh. Documented in install.md; worth a shared count in
 the store only if a customer's limits must hold across a failover.
+With `router.active_active` (O2, 2026-11-01) the counts are shared every 2 s,
+so a standby taking over keeps the minute; without it, as above.

 ### J7. nginx in front of several control planes (S)
 `deploy/proxy/Caddyfile` sends to whichever control plane answers `/leader`.
@@ -1888,6 +1892,10 @@ served only from a definition written by hand. In order:
  a large MoE that fits no card is not "fits nowhere". Done 2026-10-30, without a speed (one measurement so far).
 - N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
 - N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
+  `ctl model context` takes Strata (setup's `--context`), and `--max-output` changes the longest answer alone,
+  in place, with no restart: done 2026-11-01 (31f440a). Open: a Strata model served without `--context` still runs
+  at what Strata's setup picks (4096 on cuda01) while the catalog says the model's trained context (32768), so
+  serve should write `--context` from `context_length`; and `--parallel`.

 - N7b. The mirror page's Strata tab: every family and size, fetch (GGUF) or serve (packs Strata fetches itself:
  NVFP4, DeepSeek V4.1). Done 2026-10-30. Community `strata-ple` packs Strata 0.2.7 does not list stay unservable;
@@ -1910,6 +1918,70 @@ served only from a definition written by hand. In order:
 - N16. Versions apart from llmgw's releases (Strata's signed update feed, `--rollback-engine`); offline bundles.
 - N17. `cache_salt` for KB isolation if Strata caches prefixes; `ctl bench` with its metrics path.

+## O. More than one place answering clients (written 2026-10-30)
+
+Every client request goes through the leader's router today: the floating address is on the leader, and a standby
+passes `/v1` on to it (a standby's router sees an empty cluster). That is failover, not scale: the cluster serves what
+one control plane's router can. In order:
+
+- O1. Measured 2026-10-30/31 on the PoC: the leader (2 vCPU, 2 GB, pve1 at load 6–10 of 28 threads) in front of
+  a fake engine on node06 (`sim-engine`, 500 tok/s a stream, no batch slowdown, as an external model), 64-token
+  streams from mirror 146, the default `observe` guardrails and usage written to PostgreSQL:
+
+  | at once | req/s | tok/s | first token p50/p95 | leader llmgw CPU |
+  |---|---|---|---|---|
+  | 64 | 465 | 29,752 | 9 / 34 ms | 159% |
+  | 128 | 572 | 36,607 | 91 / 167 ms | 181% |
+  | 256 | 577 | 36,900 | 310 / 392 ms | 182% |
+  | 512 | 551 | 35,240 | 793 / 883 ms | 181% |
+
+  The ceiling is the router's CPU: ~575 requests/s, ~37k streamed tokens/s on two cores, no errors; past it the
+  first token waits. Found on the way and fixed: an external model held to 3–4 at once by admission; a valid key
+  throttled as "unrecognised" when its cache entry expired under load.
+
+  Profiled under 128 streams (2026-10-31): a third of the CPU was one write a token to the client; joining the
+  events already in hand into one write took the ceiling to ~700 req/s and ~45k tok/s, first token p50 45 ms at 128.
+  Now the largest part is the output guardrail of the default `observe` policy, ~30%: decoding each event (~20%;
+  a token-level reader with encoding/json/jsontext was measured slower than json.Unmarshal, 1.66 vs 1.27 µs) and
+  the PII patterns (~12%). Ways on, a guardrail decision: judge shadow output in larger windows, or a sample of it.
+  Next: the same run on a machine the size of a customer's control plane, then O2.
+- O2. Every control plane answers `/v1` itself (active-active):
+  - [x] the leader's replica table followed live by the standbys, so their routers route (2026-11-01, behind
+    `router.active_active`, off by default). Every 2 s a standby posts what its router used to the leader's
+    `POST /api/v1/routing/activity` over the agent port with its control-plane certificate, and is answered with
+    the Ready replicas: no keys travel, each control plane derives a replica's key from the cluster secret. The
+    leader takes the last uses in (else a replica busy behind a standby is evicted as idle) and the on-demand models
+    asked for; a catalog changed on the leader is read again from the store. Unheard from the leader for a minute,
+    a standby routes nothing and passes `/v1` on as before. The portal, `/admin` and `/mcp` still go to the leader.
+  - [x] budgets: each router raises its counters to the store's every 30 s, so another's spend is seen within that.
+  - [x] rate limits cluster-wide (2026-11-01): each router's per-counter request and token counts, this minute's
+    and the last as absolute figures, ride the same 2 s exchange; the leader answers each standby with everyone
+    else's sum, its own included, and puts the standbys' sum in force on its own router. Each limiter adds the
+    others' counts to its sliding window. Overshoot is bounded by what N routers allow in 2 s; a standby quiet for
+    10 s stops counting. (J6's single-process case was never the problem.)
+  - [ ] a session's requests to one router and one replica: the session ring already picks the same replica on
+    every router for the same replica set; what is left is the same router, which is the balancer's (source hash).
+  - [ ] in front: IPVS on the keepalived the setup already installs, DNS round-robin, or the customer's balancer
+    (J7, L9). Until then the standbys' own addresses are what spread the load.
+  - [x] tried on the PoC (2026-11-01, v0.16.1-14-g8b7eefe, setting on since): 140 leads, 150 follows. A request
+    through 150 is answered and logged by 150; a model added on the leader is routable on 150 within a tick; a
+    CPU replica used only through 150 shows idle 6 s on the leader. Load from 146, 64-token streams, a sim engine on
+    node06 at 5,000 tok/s a stream (2,300 req/s on its own; at 500 tok/s it was the ceiling, ~490 req/s, routers or
+    not), pve1 at load 14 of 28 threads:
+
+    | streams | leader alone | split over 140 + 150 | first token p50, alone / split |
+    |---|---|---|---|
+    | 128 | 971 req/s (140 at 176%) | 832 + 692 = 1,524 req/s (178% / 166%) | 115 / 59–75 ms |
+    | 256 | 990 req/s (177%) | 816 + 677 = 1,493 req/s (177% / 164%) | 243 / 135–166 ms |
+
+    1.5× with two routers on an oversubscribed host, no errors. Not 2×: both routers' VMs and the load generator
+    share pve1's cores.
+  - Known and accepted for now, per router: the precise KV index is empty on a standby (agents stream to the leader
+    only), a revoked key works on a standby until `auth.cache_ttl`, canary promotion sees the leader's share only,
+    the answer cache and prefix tree are split, an MCP caller's rate is per router.
+- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
+  control plane, as many as the traffic needs.
+
 ## D. Facts worth knowing before starting

 - Admin API: `127.0.0.1:9090`, behind the token in
@@ -20,12 +20,106 @@ is only released once its **exit criterion** is met, not when its code lands.
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
 | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
 | v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
-| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs) |
+| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |

 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.

+## v0.16.2 — 2026-11-02
+
+Standby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
+engine for models larger than the card; a served model's longest answer changed in place; and fixes found putting
+the router under load and fetching large models: a multi-file model fetched once per node however many replicas
+start there, downloads that carry on where they were cut off, and a router that does twice the work on two cores.
+
+**Upgrading.**
+- No migrations. `llmgw ctl upgrade v0.16.2`: the control planes carry active-active routing (off until
+  `llmgw ctl config set router.active_active true`) and the router fixes; the agents carry the set fetched once
+  per node and the set's progress and checking; the mirror carries the resumed downloads.
+
+### Added
+
+- **Standbys that answer the client API themselves.** With
+  `router.active_active: true` in the cluster's settings, a standby control
+  plane routes `/v1` to the leader's replicas instead of passing every
+  request to the leader, so more machines carry the traffic. It hears the
+  replicas from the leader every two seconds over the agent port (no engine
+  keys travel) and reports back what it used, so the leader does not evict
+  a replica busy behind a standby, and the on-demand models it was asked
+  for. Rate limits are the cluster's: each router's request and token
+  counts reach the others through the leader every two seconds, so a key
+  spread over several control planes gets its limit, not that many times
+  it. Budgets are brought up to the store's every 30 seconds. Off by
+  default; the portal, `/admin` and `/mcp` still go to the leader.
+- **A served model's longest answer, changed in place.** `llmgw ctl model
+  context <model> --max-output N` (and `max_output` on `POST
+  /api/v1/models/{name}/context`, and the console's Context length… form)
+  sets the longest answer on its own: the catalog's limit, in force at once,
+  with no replica started again. With a context it is set with it. Strata's
+  context can now be changed too (its setup's `--context`), so neither
+  needs the model removed and added again.
+- **FreeToken, an engine for models larger than the card.** Mixture-of-experts
+  models several times a card's size, from one card, with the experts in RAM
+  and streamed over PCIe — read from Hugging Face safetensors checkpoints as
+  published (DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8, gpt-oss, Gemma 4,
+  MiniMax), with batching. Installed like vLLM on every node it runs on: a
+  card of compute capability 8.0 or newer, driver 580+, x86-64, and the CUDA 13
+  toolkit it compiles kernels with; older cards are passed over (tried on a
+  Turing card, where 0.2.1 has no kernels). Profiles are one card
+  (`engine: freetoken`); the key goes in its environment, and flags that read
+  files or fetch are refused. Readiness now also waits for an engine whose
+  `/health` answers 200 while saying `loading` in its body. FreeToken is
+  followed release by release: a weekly workflow fails when PyPI has a version
+  newer than `DefaultEngines` installs.
+- **How much of each GPU works, beside whether anything does.** The busy
+  figure is nvidia-smi's `utilization.gpu` — the share of time any kernel
+  ran, which a kernel on one SM fills — and it is now said to be that: on
+  the dashboard, the node page, the metric's help and Grafana. Beside it,
+  each card's draw against its power limit (the efficiency table, the fleet
+  and node charts, the console's GPU work chart), and where the card's
+  profiling counters can be read, its SMs, SM occupancy, tensor cores and
+  memory interface: `llmgw_gpu_sm_active_percent`,
+  `llmgw_gpu_sm_occupancy_percent`, `llmgw_gpu_tensor_active_percent`,
+  `llmgw_gpu_dram_active_percent`. The agent reads them from NVML's GPM on
+  Hopper and later, and from a dcgm-exporter on the node elsewhere
+  (`discovery.dcgm_exporter`, default `http://127.0.0.1:9400/metrics`, `off`
+  for none). The GPU map colours by this "work" by default — SMs active
+  where read, power against the limit otherwise — and keeps busy as a mode.
+  On cuda01 decode read 92% busy with the SMs at work 69% of the time.
+
+### Fixed
+
+- **Several replicas of a multi-file model on one node fetched it once
+  each, over one another.** Four replicas placed on a node's four cards
+  wanted the same safetensors set at the same moment, each wrote the same
+  `.part` files, and all but one failed to start ("rename … no such file")
+  until a retry found the set finished. A set is now fetched once per node
+  and every replica waiting on it gets it, as single files always were.
+- **A model of several files showed one file's download as the whole.** A node fetching a set reported the part it
+  was on — "59% of 46.1 GB" for a 188 GB model — instead of the set: it now reports the set's bytes against its
+  whole size.
+- **A set's files checked with nothing said, one at a time.** After the last byte arrived a node checked each file
+  against the mirror's checksum, 195 GB on one core for eight minutes, while the replica still read "fetching 100%".
+  It now checks four at once and says so: `checking 40% of 195 GB` in `ctl replicas`, "Checking the weights" in the
+  chat page; and a check that takes long is not taken for a stuck download.
+- **A mirror download the registry cut off started again from nothing, or not at all.** Hugging Face's CDN ended a
+  60 GB transfer at 23 GB (`stream error … CANCEL`), the mirror deleted what had arrived, and `ctl model serve`
+  failed. The mirror now tries again up to six times, carrying on from where it got to (`Range`), keeps what arrived
+  when it gives up, and carries on from it after a restart too.
+- **A model from an outside provider took three or four requests at once.** Admission read every request in flight to
+  an endpoint it does not measure as a queue, so an external model (OpenAI, another llmgw cluster) was held to what a
+  saturation of 0.85 allows of "in flight / 4"; the rest waited and timed out with 504. An outside provider queues for
+  itself and says 429 when full, which its breaker already hears: it no longer counts towards saturation.
+- **A valid key told to slow down under load, every thirty seconds.** When a busy key's cached entry expired, every
+  request in flight missed the cache together and each took one of the address's allowance for unknown keys: past a
+  hundred or so streams at once, a valid key got 429 "too many unrecognised api keys". Requests missing the same key
+  at once now share one read and one allowance.
+- **A streamed answer was one write to the client a token.** Each event was flushed alone, a system call each: a
+  third of a busy router's CPU. Events already received from the engine now go out in one write, with no token
+  waiting for another; on the PoC's two-core leader the ceiling rose from ~575 to ~700 streams' requests a second,
+  and the first token at 128 at once from 91 to 45 ms.
+
 ## v0.16.1 — 2026-10-30

 The mirror page's Strata tab, Strata's ready-made packs served with Strata fetching them, and two fixes found
@@ -42,7 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
 | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
 | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
 | v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
-| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs) |
+| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) |
 | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |

 As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -2219,12 +2219,27 @@ type GPUSensor struct {
 	NvlinkReplayErrors        int64 `protobuf:"varint,23,opt,name=nvlink_replay_errors,json=nvlinkReplayErrors,proto3" json:"nvlink_replay_errors,omitempty"`
 	NvlinkRecoveryErrors      int64 `protobuf:"varint,24,opt,name=nvlink_recovery_errors,json=nvlinkRecoveryErrors,proto3" json:"nvlink_recovery_errors,omitempty"`
 	// xid_errors are the driver's error events since the agent started.
-	XidErrors     int64  `protobuf:"varint,25,opt,name=xid_errors,json=xidErrors,proto3" json:"xid_errors,omitempty"`
-	LastXid       int32  `protobuf:"varint,26,opt,name=last_xid,json=lastXid,proto3" json:"last_xid,omitempty"`
-	LastXidText   string `protobuf:"bytes,27,opt,name=last_xid_text,json=lastXidText,proto3" json:"last_xid_text,omitempty"`
-	LastXidUnix   int64  `protobuf:"varint,28,opt,name=last_xid_unix,json=lastXidUnix,proto3" json:"last_xid_unix,omitempty"`
-	unknownFields protoimpl.UnknownFields
-	sizeCache     protoimpl.SizeCache
+	XidErrors   int64  `protobuf:"varint,25,opt,name=xid_errors,json=xidErrors,proto3" json:"xid_errors,omitempty"`
+	LastXid     int32  `protobuf:"varint,26,opt,name=last_xid,json=lastXid,proto3" json:"last_xid,omitempty"`
+	LastXidText string `protobuf:"bytes,27,opt,name=last_xid_text,json=lastXidText,proto3" json:"last_xid_text,omitempty"`
+	LastXidUnix int64  `protobuf:"varint,28,opt,name=last_xid_unix,json=lastXidUnix,proto3" json:"last_xid_unix,omitempty"`
+	// How much of the chip the work on it uses, which utilization_percent
+	// does not say: that is the share of time any kernel ran, and a kernel on
+	// one SM of a hundred counts in full. has_profile says these are present;
+	// -1 is not reported. Read from NVML's GPM on Hopper and later, or from a
+	// dcgm-exporter on the node; profile_source says which ("gpm", "dcgm").
+	HasProfile bool `protobuf:"varint,29,opt,name=has_profile,json=hasProfile,proto3" json:"has_profile,omitempty"`
+	// sm_active_percent is the share of time the SMs had work, averaged over
+	// all of them; sm_occupancy_percent how full those SMs were with warps.
+	SmActivePercent    float64 `protobuf:"fixed64,30,opt,name=sm_active_percent,json=smActivePercent,proto3" json:"sm_active_percent,omitempty"`
+	SmOccupancyPercent float64 `protobuf:"fixed64,31,opt,name=sm_occupancy_percent,json=smOccupancyPercent,proto3" json:"sm_occupancy_percent,omitempty"`
+	// tensor_active_percent is the tensor cores; dram_active_percent the
+	// memory interface, which decode is bound by.
+	TensorActivePercent float64 `protobuf:"fixed64,32,opt,name=tensor_active_percent,json=tensorActivePercent,proto3" json:"tensor_active_percent,omitempty"`
+	DramActivePercent   float64 `protobuf:"fixed64,33,opt,name=dram_active_percent,json=dramActivePercent,proto3" json:"dram_active_percent,omitempty"`
+	ProfileSource       string  `protobuf:"bytes,34,opt,name=profile_source,json=profileSource,proto3" json:"profile_source,omitempty"`
+	unknownFields       protoimpl.UnknownFields
+	sizeCache           protoimpl.SizeCache
 }

 func (x *GPUSensor) Reset() {
@@ -2453,6 +2468,48 @@ func (x *GPUSensor) GetLastXidUnix() int64 {
 	return 0
 }

+func (x *GPUSensor) GetHasProfile() bool {
+	if x != nil {
+		return x.HasProfile
+	}
+	return false
+}
+
+func (x *GPUSensor) GetSmActivePercent() float64 {
+	if x != nil {
+		return x.SmActivePercent
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetSmOccupancyPercent() float64 {
+	if x != nil {
+		return x.SmOccupancyPercent
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetTensorActivePercent() float64 {
+	if x != nil {
+		return x.TensorActivePercent
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetDramActivePercent() float64 {
+	if x != nil {
+		return x.DramActivePercent
+	}
+	return 0
+}
+
+func (x *GPUSensor) GetProfileSource() string {
+	if x != nil {
+		return x.ProfileSource
+	}
+	return ""
+}
+
 // ShardLoad is one engine process group's use.
 type ShardLoad struct {
 	state   protoimpl.MessageState `protogen:"open.v1"`
@@ -3586,7 +3643,10 @@ type FetchProgress struct {
 	// bytes_total is 0 when the server did not say how large the file is, which
 	// is a real answer: a percentage would then be invented, and an invented
 	// percentage that stops moving is worse than a byte count that keeps rising.
-	BytesTotal    int64 `protobuf:"varint,3,opt,name=bytes_total,json=bytesTotal,proto3" json:"bytes_total,omitempty"`
+	BytesTotal int64 `protobuf:"varint,3,opt,name=bytes_total,json=bytesTotal,proto3" json:"bytes_total,omitempty"`
+	// phase is what the bytes are: empty while they arrive, "checking" while the
+	// node checks a set's files against the mirror's checksums.
+	Phase         string `protobuf:"bytes,4,opt,name=phase,proto3" json:"phase,omitempty"`
 	unknownFields protoimpl.UnknownFields
 	sizeCache     protoimpl.SizeCache
 }
@@ -3642,6 +3702,13 @@ func (x *FetchProgress) GetBytesTotal() int64 {
 	return 0
 }

+func (x *FetchProgress) GetPhase() string {
+	if x != nil {
+		return x.Phase
+	}
+	return ""
+}
+
 // StartShard launches one shard. It is idempotent by shard_id: the control
 // plane retries, and a repeat must not produce a second process.
 type StartShard struct {
@@ -5367,7 +5434,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	" \x01(\x01R\tcpuPowerW\x12 \n" +
 	"\finlet_temp_c\x18\v \x01(\x01R\n" +
 	"inletTempC\x12'\n" +
-	"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xde\b\n" +
+	"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xe8\n" +
+	"\n" +
 	"\tGPUSensor\x12\x14\n" +
 	"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
 	"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
@@ -5399,7 +5467,14 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"xid_errors\x18\x19 \x01(\x03R\txidErrors\x12\x19\n" +
 	"\blast_xid\x18\x1a \x01(\x05R\alastXid\x12\"\n" +
 	"\rlast_xid_text\x18\x1b \x01(\tR\vlastXidText\x12\"\n" +
-	"\rlast_xid_unix\x18\x1c \x01(\x03R\vlastXidUnix\"m\n" +
+	"\rlast_xid_unix\x18\x1c \x01(\x03R\vlastXidUnix\x12\x1f\n" +
+	"\vhas_profile\x18\x1d \x01(\bR\n" +
+	"hasProfile\x12*\n" +
+	"\x11sm_active_percent\x18\x1e \x01(\x01R\x0fsmActivePercent\x120\n" +
+	"\x14sm_occupancy_percent\x18\x1f \x01(\x01R\x12smOccupancyPercent\x122\n" +
+	"\x15tensor_active_percent\x18  \x01(\x01R\x13tensorActivePercent\x12.\n" +
+	"\x13dram_active_percent\x18! \x01(\x01R\x11dramActivePercent\x12%\n" +
+	"\x0eprofile_source\x18\" \x01(\tR\rprofileSource\"m\n" +
 	"\tShardLoad\x12\x19\n" +
 	"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
 	"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
@@ -5515,13 +5590,14 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
 	"\vSTATE_READY\x10\x04\x12\x12\n" +
 	"\x0eSTATE_DRAINING\x10\x05\x12\x11\n" +
 	"\rSTATE_STOPPED\x10\x06\x12\x10\n" +
-	"\fSTATE_FAILED\x10\a\"g\n" +
+	"\fSTATE_FAILED\x10\a\"}\n" +
 	"\rFetchProgress\x12\x16\n" +
 	"\x06source\x18\x01 \x01(\tR\x06source\x12\x1d\n" +
 	"\n" +
 	"bytes_done\x18\x02 \x01(\x03R\tbytesDone\x12\x1f\n" +
 	"\vbytes_total\x18\x03 \x01(\x03R\n" +
-	"bytesTotal\"\x93\x04\n" +
+	"bytesTotal\x12\x14\n" +
+	"\x05phase\x18\x04 \x01(\tR\x05phase\"\x93\x04\n" +
 	"\n" +
 	"StartShard\x12\x1d\n" +
 	"\n" +
@@ -448,6 +448,21 @@ message GPUSensor {
  int32 last_xid = 26;
  string last_xid_text = 27;
  int64 last_xid_unix = 28;
+  // How much of the chip the work on it uses, which utilization_percent
+  // does not say: that is the share of time any kernel ran, and a kernel on
+  // one SM of a hundred counts in full. has_profile says these are present;
+  // -1 is not reported. Read from NVML's GPM on Hopper and later, or from a
+  // dcgm-exporter on the node; profile_source says which ("gpm", "dcgm").
+  bool has_profile = 29;
+  // sm_active_percent is the share of time the SMs had work, averaged over
+  // all of them; sm_occupancy_percent how full those SMs were with warps.
+  double sm_active_percent = 30;
+  double sm_occupancy_percent = 31;
+  // tensor_active_percent is the tensor cores; dram_active_percent the
+  // memory interface, which decode is bound by.
+  double tensor_active_percent = 32;
+  double dram_active_percent = 33;
+  string profile_source = 34;
 }

 // ShardLoad is one engine process group's use.
@@ -705,6 +720,9 @@ message FetchProgress {
  // is a real answer: a percentage would then be invented, and an invented
  // percentage that stops moving is worse than a byte count that keeps rising.
  int64 bytes_total = 3;
+  // phase is what the bytes are: empty while they arrive, "checking" while the
+  // node checks a set's files against the mirror's checksums.
+  string phase = 4;
 }

 // StartShard launches one shard. It is idempotent by shard_id: the control
@@ -504,7 +504,7 @@
    {
      "type": "timeseries",
      "title": "GPU busy, by card",
-      "description": "Share of each sample period a kernel was running, sampled every heartbeat.",
+      "description": "Share of each sample period in which any kernel ran (nvidia-smi's utilization.gpu), sampled every heartbeat. Not how much of the chip: one SM busy reads 100%. The SMs panel and power against its limit say how much.",
      "datasource": {
        "type": "prometheus",
        "uid": "llmgw-prom"
@@ -908,6 +908,51 @@
          "mode": "multi"
        }
      }
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU SMs and tensor cores active, by card",
+      "description": "From the cards' profiling counters: NVML's GPM on Hopper and later, or a dcgm-exporter on the node. SMs active is the share of time the SMs had work, averaged over all of them; the gap to busy is chip a running kernel leaves idle. Tensor cores stay low in decode at batch one; prefill and batches are where they rise. Empty for cards nothing can profile.",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "llmgw-prom"
+      },
+      "gridPos": {
+        "x": 0,
+        "y": 73,
+        "w": 24,
+        "h": 7
+      },
+      "id": 25,
+      "targets": [
+        {
+          "expr": "llmgw_gpu_sm_active_percent",
+          "legendFormat": "{{node}} gpu {{gpu}} SMs",
+          "refId": "A"
+        },
+        {
+          "expr": "llmgw_gpu_tensor_active_percent",
+          "legendFormat": "{{node}} gpu {{gpu}} tensor",
+          "refId": "B"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100
+        },
+        "overrides": []
+      },
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      }
    }
  ]
 }
@@ -110,6 +110,9 @@
    {
      "name": "restart"
    },
+    {
+      "name": "routing"
+    },
    {
      "name": "settings"
    },
@@ -2730,7 +2733,7 @@
    "/api/v1/models/{name}/context": {
      "post": {
        "operationId": "postModelsNameContext",
-        "summary": "Change a served model's context: the catalog, its engines' arguments and memory, then its replicas restarted",
+        "summary": "Change a served model's context and longest answer: the catalog, its engines' arguments and memory, then its replicas restarted; max_output alone changes only the catalog and restarts nothing",
        "tags": [
          "models"
        ],
@@ -4793,6 +4796,77 @@
        }
      }
    },
+    "/api/v1/routing": {
+      "get": {
+        "operationId": "getRouting",
+        "summary": "The Ready replicas a standby routes to",
+        "description": "Read by a standby control plane when router.active_active is on: each replica's address, model, profile, pool and weight, without its key, and which catalog is in force.",
+        "tags": [
+          "routing"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/control.Routing"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/routing/activity": {
+      "post": {
+        "operationId": "postRoutingActivity",
+        "summary": "A standby's router reports what it used",
+        "description": "When it last sent each replica a request, and the on-demand models it was asked for, so the leader does not evict a replica busy behind a standby. Answered with the routing, so one request a tick does both.",
+        "tags": [
+          "routing"
+        ],
+        "parameters": [
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/control.Activity"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/control.Routing"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
    "/api/v1/settings": {
      "get": {
        "operationId": "getSettings",
@@ -6485,6 +6559,10 @@
          "percent": {
            "type": "integer",
            "description": "Percent is absent rather than zero when the total is unknown: an invented percentage that stops moving is worse than a byte count that keeps rising."
+          },
+          "phase": {
+            "type": "string",
+            "description": "Phase is empty while the bytes arrive, \"checking\" while a set's files are checked against the mirror's checksums."
          }
        }
      },
@@ -9692,6 +9770,42 @@
          }
        }
      },
+      "accounting.Counts": {
+        "type": "object",
+        "description": "Counts are one limiter's counts by counter: this minute's and the one before, as absolute figures, so reporting them twice changes nothing.",
+        "properties": {
+          "minute": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "current": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "integer",
+              "format": "int64"
+            }
+          },
+          "previous": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "integer",
+              "format": "int64"
+            }
+          }
+        }
+      },
+      "accounting.Rates": {
+        "type": "object",
+        "description": "Rates are an accountant's two limiters' counts.",
+        "properties": {
+          "requests": {
+            "$ref": "#/components/schemas/accounting.Counts"
+          },
+          "tokens": {
+            "$ref": "#/components/schemas/accounting.Counts"
+          }
+        }
+      },
      "bench.Capacity": {
        "type": "object",
        "description": "Capacity is what one replica was measured to serve, kept so the measurement outlives the terminal it was printed in.\n\nThe conclusion of a run, not the run. The ramp itself — every concurrency level, its latencies, its errors — is worth reading once, while deciding whether the run was sound; it is not worth carrying around afterwards, and storing it would grow without bound as a model is re-benchmarked. What a later reader needs is the number and enough provenance to judge whether it is still true.",
@@ -10550,6 +10664,77 @@
          }
        }
      },
+      "control.Activity": {
+        "type": "object",
+        "description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
+        "properties": {
+          "from": {
+            "type": "string",
+            "description": "From names the standby, so its counts replace its own last ones and are not given back to it."
+          },
+          "last_used": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string",
+              "format": "date-time"
+            }
+          },
+          "want": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "rates": {
+            "$ref": "#/components/schemas/accounting.Rates",
+            "description": "Rates are its router's own counts against each limit."
+          }
+        }
+      },
+      "control.RoutedReplica": {
+        "type": "object",
+        "description": "RoutedReplica is a Ready replica as a router needs it: where it is, what it serves and how much it may take. Not its key — every control plane derives that from the replica's id and the cluster's secret, so none travels.",
+        "properties": {
+          "id": {
+            "type": "string"
+          },
+          "model": {
+            "type": "string"
+          },
+          "profile": {
+            "type": "string"
+          },
+          "pool": {
+            "type": "string"
+          },
+          "endpoint": {
+            "type": "string"
+          },
+          "weight": {
+            "type": "number"
+          }
+        }
+      },
+      "control.Routing": {
+        "type": "object",
+        "description": "Routing is what a standby that answers requests itself reads from the leader: the replicas it may send to, and which catalog they belong to.",
+        "properties": {
+          "replicas": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/control.RoutedReplica"
+            }
+          },
+          "catalog": {
+            "type": "string",
+            "description": "Catalog changes whenever the leader puts a catalog in force. Only compared with the last one a standby saw, never with its own: a model added, scaled or given a canary is in the store, and the standby reads it again from there."
+          },
+          "rates": {
+            "$ref": "#/components/schemas/accounting.Rates",
+            "description": "Rates are what every other control plane's router has counted against each limit, the leader's among them: added to the asker's own, they make a key's limits the cluster's."
+          }
+        }
+      },
      "eval.Live": {
        "type": "object",
        "description": "Live is a run as it stands: one running, with how many of its cases are done, or one finished.",
@@ -11015,6 +11200,30 @@
          }
        }
      },
+      "inventory.GPUProfile": {
+        "type": "object",
+        "description": "GPUProfile is a card's profiling counters, in percent; -1 is not reported. UtilPercent is the share of time any kernel ran, and a kernel on one SM of a hundred counts in full; these say how much of the chip it ran on.",
+        "properties": {
+          "sm_active": {
+            "type": "number",
+            "description": "SMActive is the share of time the SMs had work, averaged over all of them; SMOccupancy how full those SMs were with warps."
+          },
+          "sm_occupancy": {
+            "type": "number"
+          },
+          "tensor_active": {
+            "type": "number",
+            "description": "TensorActive is the tensor cores, DRAMActive the memory interface."
+          },
+          "dram_active": {
+            "type": "number"
+          },
+          "source": {
+            "type": "string",
+            "description": "Source is what read them: \"gpm\" (NVML, Hopper and later) or \"dcgm\"."
+          }
+        }
+      },
      "inventory.GPUSensor": {
        "type": "object",
        "description": "GPUSensor is one card's heat and power, and what is throttling it.",
@@ -11062,6 +11271,10 @@
          "wear": {
            "$ref": "#/components/schemas/inventory.GPUWear",
            "description": "Wear is what wears the card out, read every minute; nil from an agent that does not read it."
+          },
+          "profile": {
+            "$ref": "#/components/schemas/inventory.GPUProfile",
+            "description": "Profile is how much of the chip the work uses, which UtilPercent does not say; nil where nothing on the node can read it."
          }
        }
      },
@@ -11241,6 +11454,13 @@
          "max_output": {
            "type": "integer"
          },
+          "output_only": {
+            "type": "boolean",
+            "description": "OutputOnly is a change of the longest answer alone, MaxOutputBefore what it was (0 no limit); nothing was started again."
+          },
+          "max_output_before": {
+            "type": "integer"
+          },
          "trained": {
            "type": "integer"
          },
@@ -11285,13 +11505,16 @@
      },
      "modelserve.ContextRequest": {
        "type": "object",
-        "description": "ContextRequest is the context to change to: tokens one request may use, and for llama.cpp how many at once (its pool is the two multiplied).",
+        "description": "ContextRequest is the context to change to: tokens one request may use, and for llama.cpp how many at once (its pool is the two multiplied). MaxOutput is the longest answer, which may be changed with it or alone: Tokens 0 with a MaxOutput changes only that, in the catalog, which the router reads it from — nothing is started again.",
        "properties": {
          "tokens": {
            "type": "integer"
          },
          "slots": {
            "type": "integer"
+          },
+          "max_output": {
+            "type": "integer"
          }
        }
      },
@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
 | vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
 | SGLang | only when you ask for it | the same way as vLLM |
 | TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
+| FreeToken | nodes with a GPU of compute capability 8.0 or newer (RTX 30, A100 and later), driver 580+, x86-64 and the CUDA 13 toolkit | a Python environment built by uv from PyPI (`freetoken[accel]`); see [FreeToken](#freetoken-models-larger-than-the-card) |
 | Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |

 vLLM is resolved as PyPI stood a day after that vLLM was published, so every
@@ -785,6 +786,61 @@ in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
 first start prepares the model on the node, which takes minutes, and fetches a
 draft layer of about 5 GB from the internet.

+#### FreeToken: models larger than the card
+
+[FreeToken](https://github.com/FlashML-org/FreeToken) serves mixture-of-experts
+models several times a card's size from one card: the experts stay in the
+node's RAM, the ones used most are cached in VRAM, and the rest stream over
+PCIe. Unlike Strata it reads Hugging Face safetensors checkpoints as they are
+published — DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8 (Flash-Next among them),
+gpt-oss, Gemma 4, MiniMax M2.5/M3 — in FP8, NVFP4, MXFP4 or BF16, and batches
+several requests (`--max-running-requests`, 4 by default).
+
+Every node that can run it installs it, as with vLLM: an NVIDIA GPU of compute
+capability 8.0 or newer, driver 580 or newer, x86-64. Older cards are passed
+over without a word — on a Quadro RTX 6000 (7.5) version 0.2.1 had no kernel
+image for the card's first prefill, and in fp16 its first request ended in an
+illegal memory access. Its FP8 checkpoints need 8.9 (Ada, Hopper) and its NVFP4
+ones 12.0 (RTX 50, RTX PRO 6000 Blackwell). It compiles some kernels the first
+time a model loads, so the node also needs the **CUDA 13 toolkit** (`nvcc` in
+`CUDA_HOME`, `/usr/local/cuda` or `/usr/local/cuda-13.*`); a node of new cards
+without one says so on the cluster page instead of installing an engine that
+fails on its first model.
+
+A FreeToken profile runs on one card of one node (`llmgw ctl model add -f`):
+
+```yaml
+name: glm-5.2
+source: /var/lib/llmgw-weights/GLM-5.2-NVFP4   # a safetensors checkpoint's directory
+profiles:
+  - name: one-card
+    engine: freetoken
+    gpu_class: rtx-pro-6000-96g
+    min_memory_gb: 200            # the experts live in RAM: say how much
+    args: ["--max-running-requests", "8"]
+```
+
+A Hugging Face id as the source works too — FreeToken downloads the
+repository itself — but it is not pinned to a commit or checked against
+Hugging Face's checksums as vLLM's are, FreeToken having no `--revision`; a
+directory the mirror fetched, or one put on the node, is.
+
+The agent gives it the model, its address, its card (`--gpu`) and the key (in
+`FREETOKEN_API_KEY`, not on the command line); `--mmproj`, media paths and
+domains, `--hf-overrides` and `--moe-cache-split` are refused as they read
+files or fetch. `ctl model context` sets `--max-seq-len-override`. Readiness
+waits for its `/health` to stop saying `loading`, since it answers 200 from the
+moment its HTTP server is up. It serves no Prometheus metrics, so the router
+counts its requests itself.
+
+llmgw follows FreeToken release by release: the version is
+`DefaultEngines["freetoken"]`, and a weekly workflow
+(`.gitea/workflows/engine-releases.yml`) fails when PyPI has a newer one, the
+notice to check it and move to it. A version of your own is named under
+`engines` like any other (`freetoken: "0.2.2"`), and `freetoken: ""` installs
+none; one node is turned off with its build set to *none* (console, the node's
+menu).
+
 #### Choosing a node's vLLM build

 Which build a GPU node installs is chosen on the cluster page, in the **vLLM
@@ -1711,23 +1767,37 @@ llmgw ctl model remove qwen-7b --purge   # its files on the mirror and the nodes
 The console has the same in a model's menu: Stop… or Start, Remove…, and
 Remove with its files….

-### Changing a served model's context
+### Changing a served model's context and longest answer

-How much one request may use is set when a model is served. To change it
-afterwards:
+How much one request may use, and how long an answer may be, are set when a
+model is served. To change them afterwards, in place:

 ```
-llmgw ctl model context qwen-7b 32768             # tokens a request
-llmgw ctl model context qwen-7b 32768 --slots 4   # llama.cpp: four at once
+llmgw ctl model context qwen-7b 32768                     # tokens a request
+llmgw ctl model context qwen-7b 32768 --slots 4           # llama.cpp: four at once
+llmgw ctl model context qwen-7b 32768 --max-output 8192   # and the longest answer
+llmgw ctl model context qwen-7b --max-output 4096         # the longest answer alone
 ```

-The catalog, the engine's arguments (`--ctx-size` and `--parallel` for
-llama.cpp, `--max-model-len` for vLLM, `--context-length` for SGLang) and
-the memory its KV cache takes change together. A context longer than the
+The longest answer is the gateway's: a request asking for more `max_tokens`
+is refused, `/v1/models` lists it as `max_output_tokens` for clients to read,
+and the account page's chat asks for no more. Changed alone it is only
+the catalog's, in force at once, and no replica is started again. With a
+context it is set with it; left out, it grows and shrinks with the context
+as before, unless it was set to something else. It never exceeds the context.
+
+For a context, the catalog, the engine's arguments (`--ctx-size` and
+`--parallel` for llama.cpp, `--max-model-len` for vLLM, `--context-length` for
+SGLang, `--context` for Strata's setup, `--max-seq-len-override` for
+FreeToken) and the memory its KV cache takes change together. Strata and
+FreeToken size their caches to the card themselves, so their cards are not
+checked here. A context longer than the
 model was trained for is refused, and so is one its cards cannot hold — with
 the figures, so you know how far to go. Its replicas are then started again on
 it, one by one, a new one ready before the old one stops. The console has the
-same under a model's menu, Context length…. A model declared in the cluster's
+same under a model's menu, Context length…, with the longest answer beside
+it. The admin API is `POST /api/v1/models/{name}/context` with `tokens`,
+`slots` and `max_output`, any of them left out. A model declared in the cluster's
 settings is changed there (`llmgw ctl config edit`).

 ### Moving a served model to more cards
@@ -2019,12 +2089,31 @@ dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
 to run it, which is not the control plane: monitoring that dies with the thing
 it monitors tells you nothing at the moment you need it.

-Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
+Three figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
 rather than zero when the driver does not report it — a flat zero across a
 fleet means the measurement is missing, not that the fleet is idle. And
 `llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
 which is what a binary built without the nvml tag reports.

+The third is what "busy" means. `llmgw_gpu_utilization_percent` is
+nvidia-smi's `utilization.gpu`: the share of time *any* kernel ran, however
+little of the chip it ran on — a kernel on one SM of 142 reads 100%. A Quadro
+RTX 6000 decoding read 92% busy while its SMs had work 69% of the time and
+issued an instruction on 18% of their cycles. How much of the chip is at work
+is in `llmgw_gpu_sm_active_percent`, `llmgw_gpu_sm_occupancy_percent`,
+`llmgw_gpu_tensor_active_percent` and `llmgw_gpu_dram_active_percent`, from
+the card's profiling counters. The agent reads them from NVML's GPM on
+Hopper and later, with nothing to install; on older cards it reads them from
+a [dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter) on the node, at
+`http://127.0.0.1:9400/metrics` unless `discovery.dcgm_exporter` in
+agent.yaml says otherwise (`off` reads none). Where neither is there, the
+dashboard's GPU map and the console colour cards by their draw against the
+power limit (`llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts`), which
+rises with how much of the chip works rather than with whether anything
+does. Low tensor-core activity in decode is not waste: a quantised
+matrix-vector product at batch one runs on the integer units, and it is
+prefill and batches that use the tensor cores.
+
 ### Token by token, and SLOs

 Beside time to the first token, every streamed answer is timed per token:
@@ -2732,9 +2821,16 @@ Caddy doing that for three control planes, and
  `server add` checks, and says what to add to the mirror's unit if it
  refuses the new control plane — which would otherwise find out the first
  time it leads and fetches a model.
- **Limits are per process.** Rate limits are counted in memory, and clients
-  reach the leader through the floating address, so a failover starts each
-  key's minute afresh.
+- **Standbys can answer clients too.** With `router.active_active: true`
+  (`llmgw ctl config set router.active_active true`) a standby answers `/v1`
+  itself, routing to the leader's replicas, instead of passing it on: more
+  machines carry the traffic. Point clients at every control plane — DNS,
+  or your own load balancer checking `/ready` — since the floating address
+  stays on the leader. Rate limits stay the cluster's: each router's counts
+  reach the others every two seconds, and a standby that becomes the leader
+  keeps the minute it was counting. Without the setting, rate limits are
+  counted in the leader's memory and a failover starts each key's minute
+  afresh.

 [deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
 their own load balancer in front.
@@ -126,6 +126,7 @@ rather than repeated.
 | `name` | `string` | [Federation](#federation) |
 | `interval` | `time.Duration` | [Federation](#federation) |
 | `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
+| `binary` | `string` | [FreeTokenConfig](#freetokenconfig) |
 | `min_compute_capability` | `float64` | [GPUNeed](#gpuneed) |
 | `min_memory_gb` | `int` | [GPUNeed](#gpuneed) |
 | `format` | `string` | [GuardModel](#guardmodel) |
@@ -234,6 +235,7 @@ rather than repeated.
 | `ib_root` | `string` | [Options](#options) |
 | `prefer_nvml` | `bool` | [Options](#options) |
 | `skip_gpus` | `bool` | [Options](#options) |
+| `dcgm_exporter` | `string` | [Options](#options) |
 | `repo` | `string` | [Origin](#origin) |
 | `revision` | `string` | [Origin](#origin) |
 | `files` | `[]OriginFile` | [Origin](#origin) |
@@ -318,6 +320,7 @@ rather than repeated.
 | `batch` | `Batch` | [Router](#router) |
 | `cache` | `Cache` | [Router](#router) |
 | `trusted_proxies` | `[]string` | [Router](#router) |
+| `active_active` | `bool` | [Router](#router) |
 | `timeouts` | `Timeouts` | [Router](#router) |
 | `scoring` | `Scoring` | [Router](#router) |
 | `admission` | `Admission` | [Router](#router) |
@@ -481,6 +484,7 @@ rather than repeated.
 | `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
 | `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
 | `strata` | `StrataConfig` | [engines.Config](#enginesconfig) |
+| `freetoken` | `FreeTokenConfig` | [engines.Config](#enginesconfig) |
 | `enabled` | `*bool` | [guard.Config](#guardconfig) |
 | `default` | `[]string` | [guard.Config](#guardconfig) |
 | `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
@@ -1316,6 +1320,16 @@ site whose own firewall does that job; the rules are kept, and true
 puts them back. Changed in the cluster's settings, it is in force on
 every node and control plane within seconds.

+## FreeTokenConfig
+
+### `binary`
+
+`string`
+
+Binary is its ft command, in the environment it was installed into.
+The agent installs one itself on a node whose GPUs it runs on; this
+is for one installed by hand.
+
 ## GPUNeed

 ### `min_compute_capability`
@@ -2182,6 +2196,15 @@ is used, is the more exact source of the peers.
 SkipGPUs omits GPU discovery. A node without nvidia-smi is already
 found to have none, so this is for one whose driver is broken.

+### `dcgm_exporter`
+
+`string`
+
+DCGMExporter is the dcgm-exporter metrics URL read for how much of
+each card's chip is in use, on cards NVML's GPM does not cover (before
+Hopper). Empty is http://127.0.0.1:9400/metrics; "off" reads none. A
+node without one costs a refused connection every five minutes.
+
 ## Origin

 ### `repo`
@@ -2788,6 +2811,22 @@ as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
 are believed; anyone else's are not, since they decide the address
 written into the setup scripts and whose attempts are counted where.

+### `active_active`
+
+`bool`
+
+ActiveActive has standby control planes answer the client API
+themselves, routing to the leader's replicas, rather than passing
+every request to the leader. More machines carry the traffic, which
+is the point. Rate limits are the cluster's: each router's counts
+reach the others through the leader every two seconds, so a key
+spread over N control planes may pass its limit by what they allow
+in those two seconds. A budget is checked against the store every
+half minute; cached answers and prefix affinity are split between
+the routers. Off by
+default. Which machine a client reaches is the load balancer's or
+DNS's to say; the floating address stays on the leader.
+
 ### `timeouts`

 `Timeouts`
@@ -4032,6 +4071,16 @@ offered by a node whose Strata is named here, or that installed it.

 Takes a [StrataConfig](#strataconfig) block.

+### `freetoken`
+
+`FreeTokenConfig`
+
+FreeToken serves mixture-of-experts models larger than a card from
+RAM and one card of 8.0 or newer; installed by the agent where the
+cards allow it, or named here.
+
+Takes a [FreeTokenConfig](#freetokenconfig) block.
+
 ## guard.Config

 ### `enabled`
@@ -78,10 +78,41 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
 	if start.IsZero() {
 		return nil
 	}
-	rows, err := s.SpendSinceByScope(ctx, start)
+	spend, keys, err := storedSpend(ctx, s, start)
 	if err != nil {
 		return fmt.Errorf("restoring budgets: %w", err)
 	}
+	a.budgets.Restore(period, spend)
+	a.log.Info("restored budgets from the store",
+		"keys", keys, "counters", len(spend), "since", start)
+	return nil
+}
+
+// CatchUpBudgets raises each counter to what the store has recorded, where
+// that is more. Run every so often by a router that is one of several taking
+// requests: the others' spend reaches it only through the store. Never
+// lowers a counter, which leads the store by whatever this process has not
+// written yet.
+func (a *Accountant) CatchUpBudgets(ctx context.Context, s store.Store, period store.Period) error {
+	start := PeriodStart(period, time.Now())
+	if start.IsZero() {
+		return nil
+	}
+	spend, _, err := storedSpend(ctx, s, start)
+	if err != nil {
+		return fmt.Errorf("reading budgets: %w", err)
+	}
+	a.budgets.Raise(period, spend)
+	return nil
+}
+
+// storedSpend is the spend since start the store has recorded, by counter,
+// and how many keys it came from.
+func storedSpend(ctx context.Context, s store.Store, start time.Time) (map[string]int64, int, error) {
+	rows, err := s.SpendSinceByScope(ctx, start)
+	if err != nil {
+		return nil, 0, err
+	}
 	// Every scope a key sits under, not just the key. A budget owned by a team
 	// is one counter fed by every key beneath it, and restoring only per-key
 	// totals would bring the control plane back up believing the team had spent
@@ -113,10 +144,7 @@ func (a *Accountant) RestoreBudgets(ctx context.Context, s store.Store, period s
 			}
 		}
 	}
-	a.budgets.Restore(period, spend)
-	a.log.Info("restored budgets from the store",
-		"keys", len(rows), "counters", len(spend), "since", start)
-	return nil
+	return spend, len(rows), nil
 }

 // Reservation is what Check returns: the estimate it charged, so Record can
@@ -216,6 +216,22 @@ func TestBudgetSurvivesARestart(t *testing.T) {
 	}
 }

+func TestCatchingUpRaisesButNeverLowers(t *testing.T) {
+	// Another router's spend arrives through the store; this one's own, not
+	// written yet, must not be wiped by a store that is behind.
+	b := NewBudgets()
+	b.Add("mine", store.PeriodMonth, 900)
+	b.Add("shared", store.PeriodMonth, 100)
+
+	b.Raise(store.PeriodMonth, map[string]int64{"mine": 400, "shared": 700, "theirs": 50})
+
+	for key, want := range map[string]int64{"mine": 900, "shared": 700, "theirs": 50} {
+		if got := b.Spent(key, store.PeriodMonth); got != want {
+			t.Errorf("%s: spent %d, want %d", key, got, want)
+		}
+	}
+}
+
 func TestBudgetsResetAtThePeriodBoundary(t *testing.T) {
 	b := NewBudgets()
 	now := time.Date(2026, 1, 31, 23, 59, 0, 0, time.UTC)
@@ -61,6 +61,19 @@ func (b *Budgets) Restore(period store.Period, spend map[string]int64) {
 	}
 }

+// Raise lifts counters to the spend given, where it is more.
+func (b *Budgets) Raise(period store.Period, spend map[string]int64) {
+	start := PeriodStart(period, b.now())
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	for key, micros := range spend {
+		if c := b.currentLocked(key, start); micros > c.micros {
+			c.micros = micros
+			b.spend[key] = c
+		}
+	}
+}
+
 // Spent returns a key's spend in the current period, resetting it if the period
 // has rolled over.
 func (b *Budgets) Spent(keyID string, period store.Period) int64 {
@@ -25,7 +25,11 @@ type Limiter struct {
 type window struct {
 	// current and previous are counts in this minute and the one before.
 	current, previous int64
-	start             time.Time
+	// othersCurrent and othersPrevious are the same counts on the other
+	// control planes taking requests, as last heard: what makes a limit the
+	// cluster's rather than each process's.
+	othersCurrent, othersPrevious int64
+	start                         time.Time
 }

 // NewLimiter builds an empty limiter.
@@ -104,21 +108,24 @@ func (w *window) roll(now time.Time) {
 		return
 	case start.Sub(w.start) == windowLen:
 		w.previous, w.current = w.current, 0
+		w.othersPrevious, w.othersCurrent = w.othersCurrent, 0
 	default:
 		// More than one window has passed with no traffic.
 		w.previous, w.current = 0, 0
+		w.othersPrevious, w.othersCurrent = 0, 0
 	}
 	w.start = start
 }

 // rate weights the previous window by how much of it is still in view.
 func (w *window) rate(now time.Time) float64 {
+	cur, prev := w.current+w.othersCurrent, w.previous+w.othersPrevious
 	elapsed := now.Sub(w.start)
 	if elapsed >= windowLen {
-		return float64(w.current)
+		return float64(cur)
 	}
 	remaining := 1 - float64(elapsed)/float64(windowLen)
-	return float64(w.previous)*remaining + float64(w.current)
+	return float64(prev)*remaining + float64(cur)
 }

 // Forget drops a key's window. Used when a key is revoked; keeping it would
@@ -0,0 +1,110 @@
+package accounting
+
+import "time"
+
+// Counts are one limiter's counts by counter: this minute's and the one
+// before, as absolute figures, so reporting them twice changes nothing.
+type Counts struct {
+	Minute   time.Time        `json:"minute"`
+	Current  map[string]int64 `json:"current,omitempty"`
+	Previous map[string]int64 `json:"previous,omitempty"`
+}
+
+// Add sums other into c, aligned to c's minute: counts of a minute that is
+// c's previous one land there, older ones are dropped. Clocks a second apart
+// at a minute's turn are why.
+func (c *Counts) Add(other Counts) {
+	switch {
+	case other.Minute.Equal(c.Minute):
+		c.Current = addInto(c.Current, other.Current)
+		c.Previous = addInto(c.Previous, other.Previous)
+	case other.Minute.Add(windowLen).Equal(c.Minute):
+		c.Previous = addInto(c.Previous, other.Current)
+	}
+}
+
+func addInto(dst, src map[string]int64) map[string]int64 {
+	if len(src) == 0 {
+		return dst
+	}
+	if dst == nil {
+		dst = make(map[string]int64, len(src))
+	}
+	for k, v := range src {
+		dst[k] += v
+	}
+	return dst
+}
+
+// Counts reads this limiter's own counts, without the others'.
+func (l *Limiter) Counts() Counts {
+	now := l.now()
+	out := Counts{Minute: now.Truncate(windowLen), Current: map[string]int64{}, Previous: map[string]int64{}}
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	for k, w := range l.keys {
+		w.roll(now)
+		if w.current > 0 {
+			out.Current[k] = w.current
+		}
+		if w.previous > 0 {
+			out.Previous[k] = w.previous
+		}
+	}
+	return out
+}
+
+// SetOthers replaces what the other control planes have counted. A counter
+// they did not mention has nothing from them.
+func (l *Limiter) SetOthers(c Counts) {
+	now := l.now()
+	aligned := Counts{Minute: now.Truncate(windowLen)}
+	aligned.Add(c)
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	for k, w := range l.keys {
+		w.roll(now)
+		w.othersCurrent, w.othersPrevious = aligned.Current[k], aligned.Previous[k]
+		delete(aligned.Current, k)
+		delete(aligned.Previous, k)
+	}
+	// Counters only others have used yet: a key spread over the cluster is
+	// limited here from its first request.
+	for k := range aligned.Current {
+		w := l.windowLocked(k, now)
+		w.othersCurrent, w.othersPrevious = aligned.Current[k], aligned.Previous[k]
+		delete(aligned.Previous, k)
+	}
+	for k, v := range aligned.Previous {
+		l.windowLocked(k, now).othersPrevious = v
+	}
+}
+
+// Rates are an accountant's two limiters' counts.
+type Rates struct {
+	Requests Counts `json:"requests"`
+	Tokens   Counts `json:"tokens"`
+}
+
+// Add sums other into r.
+func (r *Rates) Add(other Rates) {
+	r.Requests.Add(other.Requests)
+	r.Tokens.Add(other.Tokens)
+}
+
+// NewRates is an empty sum for the minute now is in.
+func NewRates(now time.Time) Rates {
+	m := now.Truncate(windowLen)
+	return Rates{Requests: Counts{Minute: m}, Tokens: Counts{Minute: m}}
+}
+
+// Rates reads this process's own request and token counts.
+func (a *Accountant) Rates() Rates {
+	return Rates{Requests: a.requests.Counts(), Tokens: a.tokens.Counts()}
+}
+
+// SetOtherRates puts the other control planes' counts in force here.
+func (a *Accountant) SetOtherRates(r Rates) {
+	a.requests.SetOthers(r.Requests)
+	a.tokens.SetOthers(r.Tokens)
+}
@@ -0,0 +1,71 @@
+package accounting
+
+import (
+	"testing"
+	"time"
+)
+
+// Two control planes each allowing a key six of its ten a minute: once each
+// has heard the other's count, neither allows another.
+func TestALimitIsTheClustersOnceTheCountsAreShared(t *testing.T) {
+	now := time.Date(2026, 11, 1, 12, 0, 10, 0, time.UTC)
+	a, b := NewLimiter(), NewLimiter()
+	a.now, b.now = func() time.Time { return now }, func() time.Time { return now }
+	for range 6 {
+		if !a.Allow("k", 10, 1) || !b.Allow("k", 10, 1) {
+			t.Fatal("refused within the limit")
+		}
+	}
+	a.SetOthers(b.Counts())
+	b.SetOthers(a.Counts())
+	if a.Allow("k", 10, 1) || b.Allow("k", 10, 1) {
+		t.Error("twelve of ten allowed once both knew")
+	}
+	// Reporting the same counts again changes nothing: they are absolute.
+	a.SetOthers(b.Counts())
+	if got := a.Rate("k"); got != 12 {
+		t.Errorf("rate %v after the same counts twice", got)
+	}
+	// What the others counted is never charged back to them.
+	if c := a.Counts(); c.Current["k"] != 6 {
+		t.Errorf("own count %d", c.Current["k"])
+	}
+}
+
+// A key only the others have used is limited here from its first request,
+// and their counts move into the previous minute with the clock.
+func TestOthersCountsFollowTheMinute(t *testing.T) {
+	now := time.Date(2026, 11, 1, 12, 0, 50, 0, time.UTC)
+	a, b := NewLimiter(), NewLimiter()
+	a.now, b.now = func() time.Time { return now }, func() time.Time { return now }
+	for range 10 {
+		b.Allow("k", 10, 1)
+	}
+	a.SetOthers(b.Counts())
+	if a.Allow("k", 10, 1) {
+		t.Error("a key the cluster has used up was allowed where it had not been seen")
+	}
+	now = now.Add(20 * time.Second) // 12:01:10, a sixth into the next minute
+	if got := a.Rate("k"); got < 8 || got > 8.5 {
+		t.Errorf("rate %v: the others' last minute should weigh five sixths", got)
+	}
+	// Counts of a minute ago, sent late, land in the previous minute.
+	old := b.Counts()
+	old.Minute = now.Truncate(time.Minute).Add(-time.Minute)
+	old.Current, old.Previous = map[string]int64{"k": 3}, nil
+	a.SetOthers(old)
+	if got := a.Rate("k"); got < 2 || got > 3 {
+		t.Errorf("rate %v from late counts", got)
+	}
+}
+
+func TestRatesAddAlignsMinutes(t *testing.T) {
+	m := time.Date(2026, 11, 1, 12, 1, 0, 0, time.UTC)
+	sum := NewRates(m)
+	sum.Add(Rates{Requests: Counts{Minute: m, Current: map[string]int64{"k": 2}, Previous: map[string]int64{"k": 1}}})
+	sum.Add(Rates{Requests: Counts{Minute: m.Add(-time.Minute), Current: map[string]int64{"k": 5}}})
+	sum.Add(Rates{Requests: Counts{Minute: m.Add(-2 * time.Minute), Current: map[string]int64{"k": 100}}})
+	if sum.Requests.Current["k"] != 2 || sum.Requests.Previous["k"] != 6 {
+		t.Errorf("sum %+v", sum.Requests)
+	}
+}
@@ -187,7 +187,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 		logRing:      ring,
 		provKick:     make(chan struct{}, 1),
 		load:         discovery.NewLoadSampler(),
-		sensors:      discovery.NewSensorSampler(),
+		sensors:      discovery.NewSensorSampler().WithProfile(discovery.NewProfSampler(cfg.Discovery.DCGMExporter)),
 		wear:         discovery.NewHealthSampler(),
 		serverList:   loadControlPlanes(cfg.Servers, cfg.TLS.Enabled()),
 		kv:           kvevents.NewMirror(),
@@ -798,6 +798,11 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
 				MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
 				SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
 			}
+			if p := g.Profile; p != nil {
+				gs.HasProfile, gs.ProfileSource = true, p.Source
+				gs.SmActivePercent, gs.SmOccupancyPercent = p.SMActive, p.SMOccupancy
+				gs.TensorActivePercent, gs.DramActivePercent = p.TensorActive, p.DRAMActive
+			}
 			if w, ok := wear[g.Index]; ok {
 				gs.HasWear = true
 				gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
@@ -1132,7 +1137,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 		send(&llmgwv1.ShardStatus{
 			State: agentsvc.StateToWire(supervisor.Starting),
 			Fetch: &llmgwv1.FetchProgress{
-				Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal,
+				Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal, Phase: p.Phase,
 			},
 		})
 	})
@@ -1155,7 +1160,8 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
 		for i, m := range ms {
 			stop := a.watchFetch(m["source"], func(p weights.Progress) {
 				send(&llmgwv1.ShardStatus{State: agentsvc.StateToWire(supervisor.Starting),
-					Fetch: &llmgwv1.FetchProgress{Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal}})
+					Fetch: &llmgwv1.FetchProgress{Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal,
+						Phase: p.Phase}})
 			})
 			path, err := a.weights().Ensure(ctx, m["source"], m["allow_pickle"] == "true")
 			stop()
@@ -68,7 +68,7 @@ func (r *statusRecorder) OnShardCompute(replicaID, shardID string, c reconciler.
 	r.computes = append(r.computes, c)
 }

-func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
+func (r *statusRecorder) OnFetchProgress(replicaID, shardID, source, phase string, done, total int64) {
 	r.mu.Lock()
 	defer r.mu.Unlock()
 	r.fetches = append(r.fetches, fetchReport{replicaID, shardID, source, done, total})
@@ -0,0 +1,93 @@
+package discovery
+
+import (
+	"encoding/json"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+)
+
+// CUDAToolkit is the root of an installed CUDA toolkit of this major version
+// — the directory whose bin holds nvcc — or "" when there is none. Engines
+// that compile their kernels where they run (FreeToken, ComfyUI's compiled
+// nodes) need one matching the CUDA their torch was built for, and a
+// toolkit of another major version compiles code that torch cannot load.
+//
+// CUDA_HOME is looked at first, then /usr/local/cuda, then every
+// /usr/local/cuda-<version>, newest first.
+func CUDAToolkit(major int) string {
+	return cudaToolkitIn(major, os.Getenv("CUDA_HOME"), "/usr/local")
+}
+
+func cudaToolkitIn(major int, home, root string) string {
+	var dirs []string
+	if home != "" {
+		dirs = append(dirs, home)
+	}
+	dirs = append(dirs, filepath.Join(root, "cuda"))
+	versioned, _ := filepath.Glob(filepath.Join(root, "cuda-*"))
+	slices.SortFunc(versioned, func(a, b string) int { return versionCompare(b, a) })
+	dirs = append(dirs, versioned...)
+	for _, d := range dirs {
+		if _, err := os.Stat(filepath.Join(d, "bin", "nvcc")); err != nil {
+			continue
+		}
+		if toolkitMajor(d) == major {
+			return d
+		}
+	}
+	return ""
+}
+
+// toolkitMajor is a toolkit's major version, from the version.json every
+// toolkit since 11.1 carries, or from nvcc where it has none; 0 when neither
+// says.
+func toolkitMajor(dir string) int {
+	if b, err := os.ReadFile(filepath.Join(dir, "version.json")); err == nil { //nolint:gosec // a toolkit's own file
+		var v struct {
+			CUDA struct {
+				Version string `json:"version"`
+			} `json:"cuda"`
+		}
+		if json.Unmarshal(b, &v) == nil && v.CUDA.Version != "" {
+			n, _ := strconv.Atoi(strings.SplitN(v.CUDA.Version, ".", 2)[0])
+			return n
+		}
+	}
+	//nolint:gosec // nvcc of a toolkit found on disk
+	out, err := exec.Command(filepath.Join(dir, "bin", "nvcc"), "--version").Output()
+	if err != nil {
+		return 0
+	}
+	if m := nvccRelease.FindSubmatch(out); m != nil {
+		n, _ := strconv.Atoi(string(m[1]))
+		return n
+	}
+	return 0
+}
+
+var nvccRelease = regexp.MustCompile(`release (\d+)\.`)
+
+// versionCompare orders two paths ending in dotted versions: cuda-13.10 is
+// newer than cuda-13.4, which comparing them as text gets wrong.
+func versionCompare(a, b string) int {
+	pa := strings.Split(strings.TrimPrefix(filepath.Base(a), "cuda-"), ".")
+	pb := strings.Split(strings.TrimPrefix(filepath.Base(b), "cuda-"), ".")
+	for i := range max(len(pa), len(pb)) {
+		var x, y int
+		if i < len(pa) {
+			x, _ = strconv.Atoi(pa[i])
+		}
+		if i < len(pb) {
+			y, _ = strconv.Atoi(pb[i])
+		}
+		if x != y {
+			return x - y
+		}
+	}
+	return 0
+}
@@ -0,0 +1,48 @@
+package discovery
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func toolkit(t *testing.T, root, name, version string) string {
+	t.Helper()
+	d := filepath.Join(root, name)
+	put(t, d, "bin/nvcc", "#!/bin/sh")
+	if version != "" {
+		put(t, d, "version.json", `{"cuda":{"name":"CUDA SDK","version":"`+version+`"}}`)
+	}
+	return d
+}
+
+func TestTheNewestToolkitOfTheMajorVersionAskedForIsFound(t *testing.T) {
+	root := t.TempDir()
+	toolkit(t, root, "cuda-12.8", "12.8.1")
+	toolkit(t, root, "cuda-13.4", "13.4.0")
+	want := toolkit(t, root, "cuda-13.10", "13.10.0")
+	// /usr/local/cuda is a 12: a toolkit of the wrong major version is not
+	// one torch built for 13 can load what it compiles.
+	toolkit(t, root, "cuda", "12.8.1")
+	if got := cudaToolkitIn(13, "", root); got != want {
+		t.Errorf("got %q, want %q", got, want)
+	}
+	if got := cudaToolkitIn(12, "", root); got != filepath.Join(root, "cuda") {
+		t.Errorf("12: got %q", got)
+	}
+	// CUDA_HOME first, when it is one.
+	home := toolkit(t, t.TempDir(), "mine", "13.0.2")
+	if got := cudaToolkitIn(13, home, root); got != home {
+		t.Errorf("CUDA_HOME: got %q", got)
+	}
+	if got := cudaToolkitIn(14, "", root); got != "" {
+		t.Errorf("14: got %q", got)
+	}
+	// A directory without nvcc is not a toolkit.
+	if err := os.MkdirAll(filepath.Join(root, "cuda-13.99"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if got := cudaToolkitIn(13, "", root); got != want {
+		t.Errorf("with an empty cuda-13.99: got %q", got)
+	}
+}
@@ -28,6 +28,11 @@ type Options struct {
 	// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
 	// found to have none, so this is for one whose driver is broken.
 	SkipGPUs bool `koanf:"skip_gpus"`
+	// DCGMExporter is the dcgm-exporter metrics URL read for how much of
+	// each card's chip is in use, on cards NVML's GPM does not cover (before
+	// Hopper). Empty is http://127.0.0.1:9400/metrics; "off" reads none. A
+	// node without one costs a refused connection every five minutes.
+	DCGMExporter string `koanf:"dcgm_exporter"`
 	// ModelDirs are scanned for weights. Empty means do not look.
 	ModelDirs []string `koanf:"-"`
 }
@@ -0,0 +1,6 @@
+//go:build !nvml
+
+package discovery
+
+// newGPM has nothing to read GPM with in a build without the nvml tag.
+func newGPM() gpmSource { return nil }
@@ -0,0 +1,124 @@
+//go:build nvml
+
+package discovery
+
+import (
+	"sync"
+
+	"github.com/NVIDIA/go-nvml/pkg/nvml"
+)
+
+// nvmlGPM reads GPM, NVML's own profiling counters, on the cards that have
+// them: Hopper and later. Each read takes a sample and works out the metrics
+// against the one before, so NVML stays initialised between reads.
+type nvmlGPM struct {
+	mu     sync.Mutex
+	opened bool
+	failed bool
+	cards  map[int]*gpmCard
+}
+
+type gpmCard struct {
+	dev          nvml.Device
+	prev, cur    nvml.GpmSample
+	havePrevious bool
+}
+
+func newGPM() gpmSource { return &nvmlGPM{} }
+
+// The GPM metrics read, in the order they are asked for.
+var gpmMetrics = []struct {
+	id  nvml.GpmMetricId
+	set func(*GPUProfile, float64)
+}{
+	{nvml.GPM_METRIC_SM_UTIL, func(p *GPUProfile, v float64) { p.SMActive = v }},
+	{nvml.GPM_METRIC_SM_OCCUPANCY, func(p *GPUProfile, v float64) { p.SMOccupancy = v }},
+	{nvml.GPM_METRIC_ANY_TENSOR_UTIL, func(p *GPUProfile, v float64) { p.TensorActive = v }},
+	{nvml.GPM_METRIC_DRAM_BW_UTIL, func(p *GPUProfile, v float64) { p.DRAMActive = v }},
+}
+
+func (g *nvmlGPM) read() map[int]GPUProfile {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	if g.failed {
+		return nil
+	}
+	if !g.opened && !g.open() {
+		g.failed = true
+		return nil
+	}
+	out := map[int]GPUProfile{}
+	for idx, c := range g.cards {
+		if c.cur.Get(c.dev) != nvml.SUCCESS {
+			// A card in MIG mode, or one that has gone: no sample, and the
+			// next must not be measured against an old one.
+			c.havePrevious = false
+			continue
+		}
+		if c.havePrevious {
+			if p, ok := gpmBetween(c.prev, c.cur); ok {
+				out[idx] = p
+			}
+		}
+		c.prev, c.cur = c.cur, c.prev
+		c.havePrevious = true
+	}
+	return out
+}
+
+// open finds the cards with GPM. False is NVML not starting, or no card
+// having it, and the reader then never tries again.
+func (g *nvmlGPM) open() bool {
+	if nvml.Init() != nvml.SUCCESS {
+		return false
+	}
+	g.cards = map[int]*gpmCard{}
+	n, ret := nvml.DeviceGetCount()
+	if ret != nvml.SUCCESS {
+		n = 0
+	}
+	for i := range n {
+		dev, ret := nvml.DeviceGetHandleByIndex(i)
+		if ret != nvml.SUCCESS {
+			continue
+		}
+		sup, ret := dev.GpmQueryDeviceSupport()
+		if ret != nvml.SUCCESS || sup.IsSupportedDevice == 0 {
+			continue
+		}
+		a, ret := nvml.GpmSampleAlloc()
+		if ret != nvml.SUCCESS {
+			continue
+		}
+		b, ret := nvml.GpmSampleAlloc()
+		if ret != nvml.SUCCESS {
+			_ = a.Free()
+			continue
+		}
+		g.cards[i] = &gpmCard{dev: dev, prev: a, cur: b}
+	}
+	if len(g.cards) == 0 {
+		_ = nvml.Shutdown()
+		return false
+	}
+	g.opened = true
+	return true
+}
+
+// gpmBetween is the profile over the time between two samples.
+func gpmBetween(earlier, later nvml.GpmSample) (GPUProfile, bool) {
+	q := nvml.GpmMetricsGetType{NumMetrics: uint32(len(gpmMetrics)), Sample1: earlier, Sample2: later}
+	for i, m := range gpmMetrics {
+		q.Metrics[i].MetricId = uint32(m.id)
+	}
+	if nvml.GpmMetricsGet(&q) != nvml.SUCCESS {
+		return GPUProfile{}, false
+	}
+	p := GPUProfile{SMActive: -1, SMOccupancy: -1, TensorActive: -1, DRAMActive: -1, Source: "gpm"}
+	for i, m := range gpmMetrics {
+		if r := q.Metrics[i]; r.NvmlReturn == uint32(nvml.SUCCESS) && r.Value >= 0 {
+			m.set(&p, r.Value)
+		}
+	}
+	return p, true
+}
@@ -0,0 +1,176 @@
+package discovery
+
+import (
+	"context"
+	"io"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// GPUProfile is how much of a card's chip the work on it uses. nvidia-smi's
+// utilization.gpu cannot say: it is the share of time any kernel ran, and a
+// kernel keeping one SM of a hundred busy reads 100%. Measured on a Quadro
+// RTX 6000 under decode, it read 92% while the SMs had work 69% of the time
+// and issued an instruction on 18% of their cycles.
+//
+// These come from the card's profiling counters, which only two things read
+// without stopping the work: NVML's GPM, on Hopper and later, and DCGM, which
+// a site may run as dcgm-exporter. Percentages; -1 is not reported.
+type GPUProfile struct {
+	// SMActive is the share of time the SMs had work, averaged over all of
+	// them; SMOccupancy how full those SMs were with warps.
+	SMActive, SMOccupancy float64
+	// TensorActive is the tensor cores. Decode at batch one hardly uses them
+	// even when nothing is wrong: a quantised matrix-vector product runs on
+	// the integer units. Prefill and batches are where they show.
+	TensorActive float64
+	// DRAMActive is the memory interface, which decode is bound by.
+	DRAMActive float64
+	// Source is "gpm" or "dcgm".
+	Source string
+}
+
+// DefaultDCGMExporter is where dcgm-exporter listens unless told otherwise.
+const DefaultDCGMExporter = "http://127.0.0.1:9400/metrics"
+
+// dcgmRetry is how long a dcgm-exporter that did not answer is left alone.
+// Most nodes run none, and asking every ten seconds would be asking nobody.
+const dcgmRetry = 5 * time.Minute
+
+// gpmSource reads GPM, where this build and the card have it.
+type gpmSource interface {
+	read() map[int]GPUProfile
+}
+
+// ProfSampler reads each card's GPUProfile: from GPM where the card has it,
+// otherwise from a dcgm-exporter on the node.
+type ProfSampler struct {
+	gpm  gpmSource
+	dcgm string
+	get  func(ctx context.Context, url string) ([]byte, error)
+	now  func() time.Time
+
+	dcgmAfter time.Time
+}
+
+// NewProfSampler reads this machine. dcgmURL is the dcgm-exporter to ask:
+// empty is DefaultDCGMExporter, "off" asks none.
+func NewProfSampler(dcgmURL string) *ProfSampler {
+	switch dcgmURL {
+	case "":
+		dcgmURL = DefaultDCGMExporter
+	case "off":
+		dcgmURL = ""
+	}
+	return &ProfSampler{gpm: newGPM(), dcgm: dcgmURL, get: httpGet, now: time.Now}
+}
+
+// Sample is each card's profile by index; a card missing has none. GPM needs
+// two samples, so its cards appear from the second call.
+func (p *ProfSampler) Sample(ctx context.Context) map[int]GPUProfile {
+	out := map[int]GPUProfile{}
+	if p.gpm != nil {
+		for i, v := range p.gpm.read() {
+			out[i] = v
+		}
+	}
+	if p.dcgm == "" || p.now().Before(p.dcgmAfter) {
+		return out
+	}
+	body, err := p.get(ctx, p.dcgm)
+	if err != nil {
+		p.dcgmAfter = p.now().Add(dcgmRetry)
+		return out
+	}
+	for i, v := range parseDCGM(body) {
+		if _, ok := out[i]; !ok {
+			out[i] = v
+		}
+	}
+	return out
+}
+
+func httpGet(ctx context.Context, url string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
+	defer cancel()
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = resp.Body.Close() }()
+	if resp.StatusCode != http.StatusOK {
+		return nil, io.ErrUnexpectedEOF
+	}
+	return io.ReadAll(io.LimitReader(resp.Body, 4<<20))
+}
+
+// The dcgm-exporter fields read, as ratios of one. Its default field list
+// has them on every card DCGM can profile.
+var dcgmFields = map[string]func(*GPUProfile, float64){
+	"DCGM_FI_PROF_SM_ACTIVE":          func(p *GPUProfile, v float64) { p.SMActive = v },
+	"DCGM_FI_PROF_SM_OCCUPANCY":       func(p *GPUProfile, v float64) { p.SMOccupancy = v },
+	"DCGM_FI_PROF_PIPE_TENSOR_ACTIVE": func(p *GPUProfile, v float64) { p.TensorActive = v },
+	"DCGM_FI_PROF_DRAM_ACTIVE":        func(p *GPUProfile, v float64) { p.DRAMActive = v },
+}
+
+// parseDCGM reads dcgm-exporter's text exposition: a line per field and
+// card, the card in the gpu label. A card with none of the fields — DCGM
+// cannot profile it — is left out. A MIG instance's lines carry a
+// GPU_I_ID and are left out too: these are whole cards.
+func parseDCGM(body []byte) map[int]GPUProfile {
+	out := map[int]GPUProfile{}
+	for _, line := range strings.Split(string(body), "\n") {
+		if line == "" || line[0] == '#' {
+			continue
+		}
+		open := strings.IndexByte(line, '{')
+		end := strings.LastIndexByte(line, '}')
+		if open < 0 || end < open {
+			continue
+		}
+		set, ok := dcgmFields[line[:open]]
+		if !ok {
+			continue
+		}
+		labels := line[open+1 : end]
+		if strings.Contains(labels, `GPU_I_ID="`) && !strings.Contains(labels, `GPU_I_ID=""`) {
+			continue
+		}
+		idx, err := strconv.Atoi(labelValue(labels, "gpu"))
+		if err != nil {
+			continue
+		}
+		val := strings.Fields(line[end+1:])
+		if len(val) == 0 {
+			continue
+		}
+		v, err := strconv.ParseFloat(val[0], 64)
+		if err != nil || v < 0 {
+			continue
+		}
+		p, seen := out[idx]
+		if !seen {
+			p = GPUProfile{SMActive: -1, SMOccupancy: -1, TensorActive: -1, DRAMActive: -1, Source: "dcgm"}
+		}
+		set(&p, 100*v)
+		out[idx] = p
+	}
+	return out
+}
+
+// labelValue is one label's value from a Prometheus label set.
+func labelValue(labels, name string) string {
+	for _, kv := range strings.Split(labels, ",") {
+		k, v, ok := strings.Cut(strings.TrimSpace(kv), "=")
+		if ok && k == name {
+			return strings.Trim(v, `"`)
+		}
+	}
+	return ""
+}
@@ -0,0 +1,107 @@
+package discovery
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+)
+
+// dcgmBody is dcgm-exporter's output for two cards and a MIG instance:
+// card 0 profiled, card 1 a model DCGM cannot profile, which has the device
+// fields and none of the PROF ones.
+const dcgmBody = `# HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).
+# TYPE DCGM_FI_DEV_GPU_UTIL gauge
+DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 100
+DCGM_FI_DEV_GPU_UTIL{gpu="1",UUID="GPU-b",device="nvidia1",modelName="Quadro RTX 6000",Hostname="n1"} 92
+# HELP DCGM_FI_PROF_SM_ACTIVE The ratio of cycles an SM has at least 1 warp assigned.
+# TYPE DCGM_FI_PROF_SM_ACTIVE gauge
+DCGM_FI_PROF_SM_ACTIVE{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 0.007042
+DCGM_FI_PROF_SM_ACTIVE{gpu="2",UUID="GPU-c",device="nvidia2",modelName="NVIDIA A100",GPU_I_PROFILE="3g.40gb",GPU_I_ID="1",Hostname="n1"} 0.5
+DCGM_FI_PROF_SM_OCCUPANCY{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 0.25
+DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0",UUID="GPU-a",device="nvidia0",modelName="NVIDIA L40S",Hostname="n1"} 0
+`
+
+func TestDCGMExporterSaysHowMuchOfEachChipIsInUse(t *testing.T) {
+	got := parseDCGM([]byte(dcgmBody))
+	if len(got) != 1 {
+		t.Fatalf("want card 0 alone (card 1 unprofiled, card 2 a MIG instance): %+v", got)
+	}
+	p := got[0]
+	// The LinkedIn L40S: 100% busy, one SM of 142 at work.
+	if p.SMActive < 0.70 || p.SMActive > 0.71 || p.SMOccupancy != 25 || p.TensorActive != 0 || p.Source != "dcgm" {
+		t.Errorf("card 0 %+v", p)
+	}
+	// A field the exporter does not send is unknown, not zero.
+	if p.DRAMActive != -1 {
+		t.Errorf("DRAM active not sent, read as %v", p.DRAMActive)
+	}
+}
+
+type fakeGPM map[int]GPUProfile
+
+func (f fakeGPM) read() map[int]GPUProfile { return f }
+
+func TestGPMIsPreferredAndDCGMFillsTheCardsItDoesNotCover(t *testing.T) {
+	calls := 0
+	p := &ProfSampler{
+		gpm:  fakeGPM{1: {SMActive: 40, Source: "gpm"}},
+		dcgm: DefaultDCGMExporter,
+		get: func(context.Context, string) ([]byte, error) {
+			calls++
+			return []byte(`DCGM_FI_PROF_SM_ACTIVE{gpu="0"} 0.5` + "\n" + `DCGM_FI_PROF_SM_ACTIVE{gpu="1"} 0.9` + "\n"), nil
+		},
+		now: time.Now,
+	}
+	got := p.Sample(t.Context())
+	if got[0].Source != "dcgm" || got[0].SMActive != 50 {
+		t.Errorf("card 0 %+v", got[0])
+	}
+	if got[1].Source != "gpm" || got[1].SMActive != 40 {
+		t.Errorf("GPM's card was taken from DCGM: %+v", got[1])
+	}
+	if calls != 1 {
+		t.Errorf("%d calls", calls)
+	}
+}
+
+func TestANodeWithoutDCGMIsNotAskedEverySample(t *testing.T) {
+	now := time.Now()
+	calls := 0
+	p := &ProfSampler{dcgm: DefaultDCGMExporter, now: func() time.Time { return now },
+		get: func(context.Context, string) ([]byte, error) { calls++; return nil, errors.New("connection refused") }}
+	for range 5 {
+		if got := p.Sample(t.Context()); len(got) != 0 {
+			t.Fatalf("%+v", got)
+		}
+		now = now.Add(10 * time.Second)
+	}
+	if calls != 1 {
+		t.Errorf("asked %d times in 50 s", calls)
+	}
+	now = now.Add(dcgmRetry)
+	p.Sample(t.Context())
+	if calls != 2 {
+		t.Errorf("not asked again after %v", dcgmRetry)
+	}
+	if off := NewProfSampler("off"); off.dcgm != "" {
+		t.Errorf("off still asks %q", off.dcgm)
+	}
+}
+
+func TestEachSampleCarriesTheProfileOfTheCardsThatHaveOne(t *testing.T) {
+	smi := func(context.Context, string) ([]byte, error) {
+		return []byte("0, 71, 239, 260, 0x0, 92, 30, 23000, 24576, 1878, 0, 0\n" +
+			"1, 30, 20, 260, 0x0, 0, 0, 512, 24576, 300, 0, 0\n"), nil
+	}
+	now := time.Now()
+	s := sampler(t.TempDir(), smi, &now).WithProfile(&ProfSampler{
+		gpm: fakeGPM{0: {SMActive: 69, SMOccupancy: 45, TensorActive: 1, DRAMActive: 33, Source: "gpm"}}, now: time.Now})
+	got := s.Sample(t.Context()).GPUs
+	if got[0].Profile == nil || got[0].Profile.SMActive != 69 || got[0].UtilPercent != 92 {
+		t.Errorf("card 0 %+v", got[0])
+	}
+	if got[1].Profile != nil {
+		t.Errorf("card 1 has a profile nobody read: %+v", got[1].Profile)
+	}
+}
@@ -49,6 +49,9 @@ type GPUSensor struct {
 	MemoryUsedMB, MemoryTotalMB float64
 	SMClockMHz                  float64
 	ECCUncorrected              float64
+	// Profile is how much of the chip the work uses, where something on the
+	// node can say: nil on most cards. UtilPercent is not that.
+	Profile *GPUProfile
 }

 // sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
@@ -61,6 +64,8 @@ type SensorSampler struct {
 	sys string // /sys, or a test's copy of it
 	smi func(ctx context.Context, fields string) ([]byte, error)
 	now func() time.Time
+	// prof reads the cards' profiling counters; nil reads none.
+	prof *ProfSampler

 	mu         sync.Mutex
 	last       Sensors
@@ -74,6 +79,12 @@ func NewSensorSampler() *SensorSampler {
 	return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
 }

+// WithProfile has each sample carry the cards' profiles as well.
+func (s *SensorSampler) WithProfile(p *ProfSampler) *SensorSampler {
+	s.prof = p
+	return s
+}
+
 // SampleGPUs reads every card's heat, power and how busy it is now, without
 // a sampler's ten seconds of memory: for a screen that asks every few seconds
 // and wants what it asked for. Nil on a machine without nvidia-smi or a card.
@@ -101,6 +112,14 @@ func (s *SensorSampler) Sample(ctx context.Context) Sensors {
 	out.CPUTempC, out.InletTempC = s.hwmon()
 	out.CPUPowerW = s.rapl(now)
 	out.GPUs = s.gpus(ctx)
+	if s.prof != nil {
+		profs := s.prof.Sample(ctx)
+		for i := range out.GPUs {
+			if p, ok := profs[out.GPUs[i].Index]; ok {
+				out.GPUs[i].Profile = &p
+			}
+		}
+	}
 	s.last, s.lastAt = out, now
 	return out
 }
@@ -223,6 +223,7 @@ func NewRegistry(cfg Config) *Registry {
 	r.Add(&Sim{cfg: cfg.Sim})
 	r.Add(&ComfyUI{cfg: cfg.ComfyUI})
 	r.Add(&Strata{cfg: cfg.Strata})
+	r.Add(&FreeToken{cfg: cfg.FreeToken})
 	return r
 }

@@ -276,6 +277,10 @@ type Config struct {
 	// Strata serves a few mixture-of-experts models from RAM and one card;
 	// offered by a node whose Strata is named here, or that installed it.
 	Strata StrataConfig `koanf:"strata"`
+	// FreeToken serves mixture-of-experts models larger than a card from
+	// RAM and one card of 8.0 or newer; installed by the agent where the
+	// cards allow it, or named here.
+	FreeToken FreeTokenConfig `koanf:"freetoken"`
 }

 // --- shared helpers ---
@@ -477,7 +477,7 @@ func TestValidateRequiresModelPath(t *testing.T) {

 func TestRegistry(t *testing.T) {
 	r := NewRegistry(Config{})
-	if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "strata", "trtllm", "vllm"}) {
+	if got := r.Names(); !slices.Equal(got, []string{"comfyui", "freetoken", "llamacpp", "sglang", "sim", "strata", "trtllm", "vllm"}) {
 		t.Errorf("registry contents: %v", got)
 	}
 	if _, err := r.Get("vllm"); err != nil {
@@ -0,0 +1,137 @@
+package engines
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync/atomic"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// FreeTokenConfig locates a FreeToken installation.
+type FreeTokenConfig struct {
+	// Binary is its ft command, in the environment it was installed into.
+	// The agent installs one itself on a node whose GPUs it runs on; this
+	// is for one installed by hand.
+	Binary string `koanf:"binary"`
+}
+
+// freetokenToolkit finds the CUDA 13 toolkit; a variable for tests.
+var freetokenToolkit = func() string { return discovery.CUDAToolkit(13) }
+
+// FreeToken runs FreeToken (github.com/FlashML-org/FreeToken): a
+// mixture-of-experts engine that keeps a model's experts in host RAM, the
+// ones used most in VRAM, and streams the rest over PCIe, so a model several
+// times a card's size serves from one card. It reads Hugging Face
+// safetensors checkpoints as they are published (FP8, NVFP4, MXFP4, BF16)
+// and serves the OpenAI and Anthropic APIs. One card of compute capability
+// 8.0 or newer; it compiles some of its kernels on first start, with the
+// CUDA 13 toolkit.
+type FreeToken struct {
+	cfg       FreeTokenConfig
+	installed atomic.Pointer[string]
+}
+
+// Name identifies the engine in the catalog.
+func (*FreeToken) Name() string { return "freetoken" }
+
+// HealthPath answers before the model is loaded too, with its progress in
+// the body; the supervisor waits for the status code, so it counts as ready
+// only once FreeToken's own startup — loading, warm-up, CUDA graphs — is
+// over and the API server is accepting.
+func (*FreeToken) HealthPath() string { return "/health" }
+
+// MetricsPath is empty: FreeToken serves its numbers as JSON (/v1/stats),
+// not in Prometheus' text, so the router counts its requests itself.
+func (*FreeToken) MetricsPath() string { return "" }
+
+// ServesHTTP is true: a FreeToken replica is one process on one node.
+func (*FreeToken) ServesHTTP(ShardSpec) bool { return true }
+
+// UseInstalled points the engine at the ft the agent installed.
+func (f *FreeToken) UseInstalled(path string) { f.installed.Store(&path) }
+
+// Configured reports whether the configuration names a FreeToken.
+func (f *FreeToken) Configured() bool { return f.cfg.Binary != "" }
+
+func (f *FreeToken) binary() string {
+	if p := f.installed.Load(); p != nil {
+		return *p
+	}
+	return f.cfg.Binary
+}
+
+// HelpCommand is ft serve's own help.
+func (f *FreeToken) HelpCommand(Command) (string, []string) {
+	return f.binary(), []string{"serve", "--help"}
+}
+
+// Probe reports the FreeToken a configuration names, by what it says its
+// version is.
+func (f *FreeToken) Probe(ctx context.Context) (string, error) {
+	bin := f.binary()
+	if bin == "" {
+		return "", fmt.Errorf("no FreeToken is configured")
+	}
+	out, err := exec.CommandContext(ctx, bin, "--version").Output() //nolint:gosec // the configured engine
+	if err != nil {
+		return "", fmt.Errorf("%s does not run: %w", bin, err)
+	}
+	// "freetoken version 0.2.1"
+	if fs := strings.Fields(string(out)); len(fs) > 0 {
+		return fs[len(fs)-1], nil
+	}
+	return Unknown, nil
+}
+
+// Validate checks the profile: one card of 8.0 or newer, on one node.
+// FreeToken has no tensor parallel; it picks one device.
+func (*FreeToken) Validate(p Profile, gpus []GPUInfo) error {
+	if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
+		return fmt.Errorf("profile %q: freetoken runs on one GPU of one node", p.Name)
+	}
+	if err := requireComputeCapability(p, gpus, 8.0, "freetoken"); err != nil {
+		return err
+	}
+	return validateCommon(p, gpus)
+}
+
+// Commands is ft serve. The card is the shard's, by CUDA_VISIBLE_DEVICES,
+// so it is device 0 to FreeToken; its key goes in the environment, where it
+// reads it from, rather than on a command line every user can read.
+func (f *FreeToken) Commands(sh ShardSpec) ([]Command, error) {
+	bin := f.binary()
+	if bin == "" {
+		return nil, fmt.Errorf("freetoken: no FreeToken is installed or configured (engines.freetoken.binary)")
+	}
+	toolkit := freetokenToolkit()
+	if toolkit == "" {
+		return nil, fmt.Errorf("freetoken compiles its kernels on first start and this node has no CUDA 13 toolkit " +
+			"(nvcc in CUDA_HOME, /usr/local/cuda or /usr/local/cuda-13.*)")
+	}
+	p := sh.Profile
+	args := []string{"serve",
+		"--model", p.ModelPath,
+		"--host", sh.Host(),
+		"--port", strconv.Itoa(sh.HTTPPort),
+		"--gpu", "0",
+	}
+	if p.ServedModelName != "" {
+		args = append(args, "--served-model-name", p.ServedModelName)
+	}
+	args = append(args, p.Args...)
+	env := baseEnv(sh)
+	env["CUDA_HOME"] = toolkit
+	// nvcc for the kernels it compiles, and the environment's own bin
+	// first, where ninja is.
+	env["PATH"] = strings.Join([]string{filepath.Dir(bin), filepath.Join(toolkit, "bin"), os.Getenv("PATH")}, ":")
+	if sh.APIKey != "" {
+		env["FREETOKEN_API_KEY"] = sh.APIKey
+	}
+	return []Command{{Name: "freetoken", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
+}
@@ -0,0 +1,112 @@
+package engines
+
+import (
+	"slices"
+	"strings"
+	"testing"
+)
+
+func withToolkit(t *testing.T, dir string) {
+	t.Helper()
+	old := freetokenToolkit
+	freetokenToolkit = func() string { return dir }
+	t.Cleanup(func() { freetokenToolkit = old })
+}
+
+func TestFreeTokenServesOnTheShardsCardWithTheKeyInTheEnvironment(t *testing.T) {
+	withToolkit(t, "/usr/local/cuda-13.4")
+	f := &FreeToken{}
+	f.UseInstalled("/opt/llmgw/engines/freetoken/0.2.1-cu13/bin/ft")
+	spec := ShardSpec{
+		ReplicaID: "r1", HTTPPort: 18091, BindAddr: "10.0.0.160", GPUIndices: []int{3},
+		APIKey: "secret-key",
+		Profile: Profile{Name: "p", Engine: "freetoken", TP: 1, ModelPath: "/var/lib/llmgw-weights/glm-5.2-nvfp4",
+			ServedModelName: "glm-5.2", Args: []string{"--max-running-requests", "8"}},
+	}
+	cmds, err := f.Commands(spec)
+	if err != nil {
+		t.Fatal(err)
+	}
+	c := cmds[0]
+	if c.Path != "/opt/llmgw/engines/freetoken/0.2.1-cu13/bin/ft" {
+		t.Errorf("path %s", c.Path)
+	}
+	args := strings.Join(c.Args, " ")
+	for _, want := range []string{
+		"serve --model /var/lib/llmgw-weights/glm-5.2-nvfp4", "--host 10.0.0.160", "--port 18091",
+		// The shard's card is device 0 once CUDA_VISIBLE_DEVICES pins it.
+		"--gpu 0", "--served-model-name glm-5.2", "--max-running-requests 8",
+	} {
+		if !strings.Contains(args, want) {
+			t.Errorf("args lack %q: %s", want, args)
+		}
+	}
+	if strings.Contains(args, "secret-key") {
+		t.Error("the key is on the command line, which every user on the node can read")
+	}
+	for _, want := range []string{"FREETOKEN_API_KEY=secret-key", "CUDA_VISIBLE_DEVICES=3", "CUDA_HOME=/usr/local/cuda-13.4"} {
+		if !slices.Contains(c.Env, want) {
+			t.Errorf("env lacks %s", want)
+		}
+	}
+	var path string
+	for _, e := range c.Env {
+		if p, ok := strings.CutPrefix(e, "PATH="); ok {
+			path = p
+		}
+	}
+	if !strings.HasPrefix(path, "/opt/llmgw/engines/freetoken/0.2.1-cu13/bin:/usr/local/cuda-13.4/bin:") {
+		t.Errorf("nvcc and the environment's tools are not first on PATH: %s", path)
+	}
+}
+
+func TestFreeTokenWithoutTheToolkitSaysSoInsteadOfFailingOnItsFirstModel(t *testing.T) {
+	withToolkit(t, "")
+	f := &FreeToken{cfg: FreeTokenConfig{Binary: "/x/bin/ft"}}
+	_, err := f.Commands(ShardSpec{HTTPPort: 1, Profile: Profile{Name: "p", ModelPath: "/m"}})
+	if err == nil || !strings.Contains(err.Error(), "CUDA 13 toolkit") {
+		t.Errorf("got %v", err)
+	}
+}
+
+func TestFreeTokenRunsOnOneAmpereOrNewerCard(t *testing.T) {
+	f := &FreeToken{}
+	p := Profile{Name: "p", Engine: "freetoken", TP: 1, ModelPath: "/m"}
+	if err := f.Validate(p, []GPUInfo{{Name: "NVIDIA RTX A6000", ComputeCapability: "8.6", MemoryTotalMB: 49140}}); err != nil {
+		t.Errorf("an Ampere card was refused: %v", err)
+	}
+	// Tried on cuda01: no kernel image for the card, then an illegal memory
+	// access in fp16.
+	if err := f.Validate(p, []GPUInfo{{Name: "Quadro RTX 6000", ComputeCapability: "7.5", MemoryTotalMB: 24576}}); err == nil {
+		t.Error("a Turing card was accepted")
+	}
+	p.TP = 2
+	if err := f.Validate(p, h100(2)); err == nil {
+		t.Error("a two-card profile was accepted; FreeToken has no tensor parallel")
+	}
+}
+
+func TestFreeTokenFlagsThatReadFilesOrMoveWhatTheAgentSetsAreRefused(t *testing.T) {
+	for _, a := range []string{
+		"--gpu", "--served-model-name=x", "--mmproj", "--allowed-local-media-path",
+		// argparse takes a beginning for the whole flag.
+		"--allowed-local", "--api-key", "--port", "--hf-overrides",
+	} {
+		if UnsafeArgs("freetoken", []string{a, "v"}) == nil {
+			t.Errorf("%s was allowed", a)
+		}
+	}
+	for _, a := range []string{"--max-running-requests", "--moe-strategy", "--dtype", "--max-seq-len-override", "--moe-cache-auto"} {
+		if err := UnsafeArgs("freetoken", []string{a, "1"}); err != nil {
+			t.Errorf("%s was refused: %v", a, err)
+		}
+	}
+	for k, refuse := range map[string]bool{
+		"FREETOKEN_PIN_BUDGET_GB": false, "FREETOKEN_API_KEY": true, "FREETOKEN_API_LOG_DIR": true,
+		"FREETOKEN_MMPROJ": true, "FREETOKEN_HOST": true,
+	} {
+		if got, why := UnsafeEnv(k); got != refuse {
+			t.Errorf("%s: refused %v (%s), want %v", k, got, why, refuse)
+		}
+	}
+}
@@ -25,7 +25,7 @@ var envFamilies = []string{
 	"CUDA_", "NCCL_", "NVIDIA_", "NVTE_", "GLOO_", "UCX_", "TORCH_", "PYTORCH_",
 	"VLLM_", "SGLANG_", "GGML_", "LLAMA_", "OMP_", "MKL_", "OPENBLAS_", "KMP_",
 	"HF_", "TRANSFORMERS_", "TOKENIZERS_", "TRITON_", "FLASHINFER_",
-	"TRTLLM_", "TLLM_",
+	"TRTLLM_", "TLLM_", "FREETOKEN_",
 }

 // envExact are single variables that are tuning too.
@@ -33,6 +33,14 @@ var envExact = map[string]bool{
 	"PYTHONUNBUFFERED": true, "PYTHONHASHSEED": true, "TZ": true,
 }

+// envNeverExact are variables in an allowed family that no profile may set,
+// whose names the patterns below do not catch: what the agent sets itself,
+// and FreeToken's way of loading a projector file.
+var envNeverExact = map[string]string{
+	"FREETOKEN_API_KEY": "is set by the agent", "FREETOKEN_HOST": "is set by the agent",
+	"FREETOKEN_MMPROJ": "loads a file",
+}
+
 // envNever are names in an allowed family that load code or move where the
 // engine reads and writes.
 var envNever = []string{
@@ -59,6 +67,9 @@ func UnsafeEnv(key string) (bool, string) {
 		return true, "only the variables engines are tuned with are allowed (" +
 			strings.Join(envFamilies, "*, ") + "*)"
 	}
+	if why, ok := envNeverExact[k]; ok {
+		return true, "it " + why
+	}
 	for _, n := range envNever {
 		if strings.Contains(k, n) && !envExact[k] {
 			return true, "it names a path, a plugin, a library or an address, which is a way to make the engine " +
@@ -157,6 +168,16 @@ var flagsNever = map[string]map[string]string{
 		"--gpu-roles": "is set by the agent", "--backend": "is set by the agent",
 		"--experimental-speed-projection": "changes what the model refuses",
 	},
+	// ft serve: where it listens, its key, its model and its card are the
+	// agent's, and what reads files or fetches is refused.
+	"freetoken": {
+		"--gpu": "is set by the agent", "--served-model-name": "is set by the agent",
+		"--mmproj": "loads a file", "--mm-encoder-weights": "loads a file",
+		"--allowed-local-media-path": "serves local files to requests",
+		"--allowed-media-domains":    "lets requests have the engine fetch from those sites",
+		"--moe-cache-split":          "reads a profile file",
+		"--hf-overrides":             "rewrites the checkpoint's configuration, which can name code to load",
+	},
 	// trtllm-serve spells its flags with underscores, so the shared list's
 	// hyphenated names do not reach them.
 	"trtllm": {
@@ -223,7 +244,7 @@ func unsafeFlag(engine, flag, _ string) string {

 // argparseEngines read their flags with Python's argparse, which takes a
 // long flag's beginning for the flag.
-var argparseEngines = map[string]bool{"vllm": true, "sglang": true, "comfyui": true, "strata": true}
+var argparseEngines = map[string]bool{"vllm": true, "sglang": true, "comfyui": true, "strata": true, "freetoken": true}

 // UnsafeArgs refuses the first of a profile's arguments an engine may not be
 // given, saying why.
@@ -0,0 +1,53 @@
+package provision
+
+import (
+	"errors"
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+func TestFreeTokenIsInstalledOnlyWhereItsKernelsRun(t *testing.T) {
+	old := freetokenToolkit
+	toolkit := "/usr/local/cuda-13.4"
+	freetokenToolkit = func() string { return toolkit }
+	t.Cleanup(func() { freetokenToolkit = old })
+
+	ampere := discovery.Host{Arch: "amd64", NVIDIADriver: "580.95.05", MinComputeCapability: 8.6}
+	got, err := freetokenCandidates("0.2.1", "", ampere)
+	if err != nil {
+		t.Fatal(err)
+	}
+	b := got[0]
+	if b.Binary != "ft" || !slices.Equal(b.Requirements, []string{"freetoken[accel]==0.2.1"}) || !b.GPUCheck || b.Python == "" {
+		t.Errorf("build %+v", b)
+	}
+
+	// Not meant for these, and not reported as a fault: no GPU, a Turing
+	// card (cuda01, where 0.2.1 had no kernel image for it), an Arm node.
+	for name, h := range map[string]discovery.Host{
+		"no gpu":  {Arch: "amd64"},
+		"turing":  {Arch: "amd64", NVIDIADriver: "615.71.09", MinComputeCapability: 7.5},
+		"aarch64": {Arch: "arm64", NVIDIADriver: "580.95.05", MinComputeCapability: 9.0},
+	} {
+		_, err := freetokenCandidates("0.2.1", "", h)
+		var nh NotHere
+		if !errors.As(err, &nh) {
+			t.Errorf("%s: %v, want NotHere", name, err)
+		}
+	}
+
+	// Meant for these, and said why not.
+	if _, err := freetokenCandidates("0.2.1", "", discovery.Host{Arch: "amd64", NVIDIADriver: "570.1", MinComputeCapability: 9.0}); err == nil ||
+		!strings.Contains(err.Error(), "driver 580") {
+		t.Errorf("old driver: %v", err)
+	}
+	toolkit = ""
+	_, err = freetokenCandidates("0.2.1", "", ampere)
+	var nh NotHere
+	if err == nil || errors.As(err, &nh) || !strings.Contains(err.Error(), "CUDA 13 toolkit") {
+		t.Errorf("no toolkit: %v", err)
+	}
+}
@@ -126,12 +126,13 @@ type Recipe struct {
 // from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
 // checked against its published checksum.
 var Recipes = map[string]Recipe{
-	"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
-	"vllm":     {PyPI: "vllm", Candidates: vllmCandidates},
-	"sglang":   {PyPI: "sglang", Candidates: sglangCandidates},
-	"comfyui":  {Candidates: comfyuiCandidates},
-	"strata":   {Repo: StrataRepo, Candidates: strataCandidates},
-	"uv":       {Repo: "astral-sh/uv", Candidates: uvCandidates},
+	"llamacpp":  {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
+	"vllm":      {PyPI: "vllm", Candidates: vllmCandidates},
+	"sglang":    {PyPI: "sglang", Candidates: sglangCandidates},
+	"comfyui":   {Candidates: comfyuiCandidates},
+	"strata":    {Repo: StrataRepo, Candidates: strataCandidates},
+	"freetoken": {PyPI: "freetoken", Candidates: freetokenCandidates},
+	"uv":        {Repo: "astral-sh/uv", Candidates: uvCandidates},
 }

 // Installable are the engines this agent can install, as it tells the
@@ -758,3 +759,56 @@ print("strata version", "` + version + `", "engine", "` + rel.engine + `")`},
 		{build("cuda12"), gpuNeeds{575, 6.0}, "Strata " + version + " built for CUDA 12.9"},
 	}, h)
 }
+
+// --- FreeToken ---
+
+// FreeTokenRepo is where FreeToken is developed; its builds are on PyPI.
+const FreeTokenRepo = "FlashML-org/FreeToken"
+
+// freetokenMinCompute is the oldest GPU FreeToken runs on. It says RTX 30
+// and newer, and its kernels agree: on a Quadro RTX 6000 (7.5) version 0.2.1
+// loaded gpt-oss-20b and failed its first prefill with no kernel image for
+// the card, and in fp16 with fused experts its first request hit an illegal
+// memory access. Its FP8 kernels need 8.9 and its NVFP4 ones 12.0, which
+// the model's quantisation decides.
+const freetokenMinCompute = 8.0
+
+// freetokenToolkit finds the CUDA toolkit FreeToken compiles its kernels
+// with; a variable for tests.
+var freetokenToolkit = func() string { return discovery.CUDAToolkit(13) }
+
+// freetokenCandidates are the FreeToken builds for a machine: its PyPI
+// wheels with the kernel extras (FlashInfer, SGLang's kernels), CUDA 13,
+// x86-64 only. A node of older cards is not one it is meant for, and is not
+// reported as a fault; one of new cards without the toolkit is.
+func freetokenCandidates(version, variant string, h discovery.Host) ([]Build, error) {
+	if _, err := gpuArch("FreeToken", h); err != nil {
+		return nil, err
+	}
+	if h.Arch != "amd64" {
+		return nil, NotHere{"FreeToken publishes wheels for x86-64 only; this node is " + h.Arch}
+	}
+	if cc := h.MinComputeCapability; cc > 0 && cc < freetokenMinCompute {
+		return nil, NotHere{fmt.Sprintf("FreeToken runs on GPUs of compute capability %.1f and newer "+
+			"(RTX 30, A100 and later); this node has %.1f", freetokenMinCompute, cc)}
+	}
+	// It compiles kernels the first time a model loads, with nvcc, as its
+	// own install guide says; without a toolkit the installation looks
+	// finished and the first replica fails.
+	if freetokenToolkit() == "" {
+		return nil, fmt.Errorf("FreeToken compiles its kernels where it runs and needs the CUDA 13 toolkit's nvcc " +
+			"(in CUDA_HOME, /usr/local/cuda or /usr/local/cuda-13.*); install cuda-toolkit-13 from NVIDIA's repository")
+	}
+	b := Build{
+		Variant:      "cu13",
+		Python:       pythonForEngines,
+		Requirements: []string{"freetoken[accel]==" + version},
+		Binary:       "ft",
+		Packages:     compilerPackages,
+		FreeGB:       20,
+		GPUCheck:     true,
+	}
+	return pick("FreeToken "+version, variant, []candidate{
+		{b, gpuNeeds{minDriverForCUDA13, freetokenMinCompute}, "FreeToken " + version},
+	}, h)
+}
@@ -54,9 +54,13 @@ func NewHTTPHealth(timeout time.Duration, opts ...HTTPHealthOption) *HTTPHealth

 // Ready reports whether the engine at baseURL is serving model.
 func (h *HTTPHealth) Ready(ctx context.Context, baseURL, model, apiKey string) error {
-	if err := h.get(ctx, baseURL+h.healthPath, apiKey, nil); err != nil {
+	var body healthBody
+	if err := h.get(ctx, baseURL+h.healthPath, apiKey, &body); err != nil {
 		return fmt.Errorf("health: %w", err)
 	}
+	if body.notReady() {
+		return fmt.Errorf("health: the engine says %q", body.Status)
+	}
 	if !h.requireName || model == "" {
 		return nil
 	}
@@ -98,5 +102,29 @@ func (h *HTTPHealth) get(ctx context.Context, url, apiKey string, out any) error
 	if out == nil {
 		return nil
 	}
+	if hb, ok := out.(*healthBody); ok {
+		// Most engines answer /health with nothing, or with text: only a
+		// JSON status says anything, and a body that is not one is not a
+		// failure.
+		_ = json.NewDecoder(io.LimitReader(resp.Body, 64<<10)).Decode(hb)
+		return nil
+	}
 	return json.NewDecoder(resp.Body).Decode(out)
 }
+
+// healthBody is what an engine's /health may say in its body. Some answer
+// 200 from the moment their HTTP server is up and say in the body that the
+// model is still loading — FreeToken's {"status":"loading","phase":...} —
+// and a model that lists itself in /v1/models meanwhile would be routed to.
+type healthBody struct {
+	Status string `json:"status"`
+}
+
+// notReady is a body that says the engine cannot serve yet, or has failed.
+func (b healthBody) notReady() bool {
+	switch b.Status {
+	case "loading", "starting", "initializing", "warming_up", "error", "unhealthy":
+		return true
+	}
+	return false
+}
@@ -103,3 +103,29 @@ func TestTheHealthCheckSendsTheEnginesKey(t *testing.T) {
 		t.Error("without the key the engine refused, and that read as ready")
 	}
 }
+
+func TestNotReadyWhileTheHealthBodySaysLoading(t *testing.T) {
+	// FreeToken answers 200 from the moment its HTTP server is up and lists
+	// its model meanwhile; only the body says the weights are still loading.
+	body := `{"status":"loading","phase":"weights","progress":{"done_bytes":1,"total_bytes":9},"model":"glm"}`
+	mux := http.NewServeMux()
+	mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) { io.WriteString(w, body) })
+	mux.HandleFunc("GET /v1/models", func(w http.ResponseWriter, _ *http.Request) {
+		io.WriteString(w, `{"data":[{"id":"glm"}]}`)
+	})
+	srv := httptest.NewServer(mux)
+	t.Cleanup(srv.Close)
+	h := NewHTTPHealth(time.Second)
+	if err := h.Ready(t.Context(), srv.URL, "glm", ""); err == nil || !strings.Contains(err.Error(), "loading") {
+		t.Errorf("a loading engine was reported ready, or not said to be loading: %v", err)
+	}
+	body = `{"status":"ok","model":"glm","uptime_s":3}`
+	if err := h.Ready(t.Context(), srv.URL, "glm", ""); err != nil {
+		t.Errorf("a serving engine was reported unready: %v", err)
+	}
+	// Text, as some engines answer: not a status, and not a failure.
+	body = "OK"
+	if err := h.Ready(t.Context(), srv.URL, "glm", ""); err != nil {
+		t.Errorf("a plain-text health answer was taken for a failure: %v", err)
+	}
+}
@@ -38,8 +38,14 @@ type Progress struct {
 	Source     string
 	BytesDone  int64
 	BytesTotal int64
+	// Phase is empty while the bytes arrive, PhaseChecking while a set's
+	// files are checked against the mirror's checksums.
+	Phase string
 }

+// PhaseChecking is a set's files being checked, after they arrived.
+const PhaseChecking = "checking"
+
 // Percent is how far along, or -1 when the total is unknown.
 func (p Progress) Percent() int {
 	if p.BytesTotal <= 0 {
@@ -140,7 +146,11 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool, w
 	// file, and a set has to be handled before any of it.
 	if IsSet(source) {
 		outcome(ctx, "set")
-		return f.ensureSet(ctx, source, allowPickle)
+		// Shared like a single file: four replicas placed on one node's
+		// four cards want the same set at the same instant, and each
+		// fetching it wrote the same .part files over one another — three
+		// failed renaming a file another had already renamed.
+		return f.shared(ctx, source, func() (string, error) { return f.ensureSet(ctx, source, allowPickle) })
 	}
 	if !isRemote(source) {
 		if !isPath(source) {
@@ -172,6 +182,15 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool, w
 		// Not the file the catalog names: fetched again below, and checked.
 	}

+	return f.shared(ctx, source, func() (string, error) {
+		outcome(ctx, "downloaded")
+		return f.fetchChecked(ctx, source, target, want)
+	})
+}
+
+// shared runs fetch once for every caller asking for source at the same
+// time; the others wait for its answer.
+func (f *Fetcher) shared(ctx context.Context, source string, fetch func() (string, error)) (string, error) {
 	f.mu.Lock()
 	if f.inflight == nil {
 		f.inflight = map[string]*fetchCall{}
@@ -190,8 +209,7 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool, w
 	f.inflight[source] = call
 	f.mu.Unlock()

-	outcome(ctx, "downloaded")
-	call.path, call.err = f.fetchChecked(ctx, source, target, want)
+	call.path, call.err = fetch()
 	close(call.done)

 	f.mu.Lock()
@@ -202,7 +220,7 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool, w

 // fetch downloads a source to a target, resuming and verifying.
 func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, error) {
-	return f.fetchAs(ctx, source, source, target)
+	return f.fetchAs(ctx, source, source, target, setShare{})
 }

 // fetchChecked is fetch for a file whose sha256 is known: a download that
@@ -287,7 +305,7 @@ func ChecksumPath(file string) string { return checksumPath(file) }
 //
 // Each file is a "download" span, with how many bytes it took and where a
 // resumed one started from.
-func (f *Fetcher) fetchAs(ctx context.Context, source, reportAs, target string) (_ string, err error) {
+func (f *Fetcher) fetchAs(ctx context.Context, source, reportAs, target string, share setShare) (_ string, err error) {
 	ctx, span := obs.Tracer(tracerName).Start(ctx, "download",
 		trace.WithSpanKind(trace.SpanKindClient),
 		trace.WithAttributes(attribute.String("llmgw.weights.source", redact(source))))
@@ -351,8 +369,8 @@ func (f *Fetcher) fetchAs(ctx context.Context, source, reportAs, target string)
 	}

 	total := existing + resp.ContentLength
-	f.report(Progress{Source: reportAs, BytesDone: existing, BytesTotal: total})
-	written, err := f.copyWithProgress(ctx, out, resp.Body, reportAs, existing, total)
+	f.report(share.of(Progress{Source: reportAs, BytesDone: existing, BytesTotal: total}))
+	written, err := f.copyWithProgress(ctx, out, resp.Body, reportAs, existing, total, share)
 	span.SetAttributes(
 		attribute.Int64("llmgw.weights.resumed_from", existing),
 		attribute.Int64("llmgw.weights.bytes", written),
@@ -376,7 +394,7 @@ func (f *Fetcher) fetchAs(ctx context.Context, source, reportAs, target string)
 }

 func (f *Fetcher) copyWithProgress(ctx context.Context, dst io.Writer, src io.Reader,
-	source string, done, total int64,
+	source string, done, total int64, share setShare,
 ) (int64, error) {
 	buf := make([]byte, 4<<20)
 	last := time.Now()
@@ -394,12 +412,12 @@ func (f *Fetcher) copyWithProgress(ctx context.Context, dst io.Writer, src io.Re
 			// hundred gigabytes is twenty-five thousand callbacks, and nobody
 			// needs to be told that often.
 			if time.Since(last) > 2*time.Second {
-				f.report(Progress{Source: source, BytesDone: done, BytesTotal: total})
+				f.report(share.of(Progress{Source: source, BytesDone: done, BytesTotal: total}))
 				last = time.Now()
 			}
 		}
 		if errors.Is(err, io.EOF) {
-			f.report(Progress{Source: source, BytesDone: done, BytesTotal: total})
+			f.report(share.of(Progress{Source: source, BytesDone: done, BytesTotal: total}))
 			return done, nil
 		}
 		if err != nil {
@@ -408,6 +426,18 @@ func (f *Fetcher) copyWithProgress(ctx context.Context, dst io.Writer, src io.Re
 	}
 }

+// setShare places one file's download within its set's: the bytes of the
+// files before it, and the whole set's. Zero for a file on its own.
+type setShare struct{ before, total int64 }
+
+// of is a file's progress as the set's.
+func (s setShare) of(p Progress) Progress {
+	if s.total > 0 {
+		p.BytesDone, p.BytesTotal = s.before+p.BytesDone, s.total
+	}
+	return p
+}
+
 func (f *Fetcher) report(p Progress) {
 	if f.OnProgress != nil {
 		f.OnProgress(p)
@@ -5,12 +5,16 @@ import (
 	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"net/http"
 	"os"
 	"path/filepath"
 	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
 )

 // IsSet reports whether a source names a set of files rather than one.
@@ -89,6 +93,12 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool
 	if err := os.MkdirAll(dir, 0o755); err != nil {
 		return "", err
 	}
+	// One number for the whole set, as the shards waiting on it see it: the
+	// file being fetched was shown alone, a 46 GB part of 188.
+	var setTotal, before int64
+	for _, file := range m.Files {
+		setTotal += file.Bytes
+	}
 	for _, file := range m.Files {
 		// Rejected rather than joined: a name with a separator in it would
 		// write outside the directory the mirror said it was in, and the
@@ -98,34 +108,24 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool
 		}
 		target := filepath.Join(dir, file.Name)
 		if st, err := os.Stat(target); err == nil && st.Size() == file.Bytes {
+			before += file.Bytes
 			continue
 		}
 		// Reported under the set's own address, because that is what the
 		// shards are waiting on. Reusing fetch rather than writing a second
 		// downloader keeps resume, the .part rename and the progress callback
 		// in one place.
-		if _, err := f.fetchAs(ctx, source+file.Name, source, target); err != nil {
+		if _, err := f.fetchAs(ctx, source+file.Name, source, target, setShare{before: before, total: setTotal}); err != nil {
 			return "", err
 		}
+		before += file.Bytes
 	}

 	// Each file against the checksum the mirror took when it fetched it,
 	// before the set is called complete: a shard damaged on the way loads as
 	// a model that answers nonsense, not as an error.
-	for _, file := range m.Files {
-		if file.SHA256 == "" {
-			continue
-		}
-		target := filepath.Join(dir, file.Name)
-		got, err := fileSHA256(target)
-		if err != nil {
-			return "", err
-		}
-		if !strings.EqualFold(got, file.SHA256) {
-			_ = os.Remove(target)
-			return "", fmt.Errorf("%s%s: sha256 %s, the mirror has %s; removed, to be fetched again",
-				source, file.Name, got, file.SHA256)
-		}
+	if err := f.checkSet(ctx, source, dir, m); err != nil {
+		return "", err
 	}

 	if err := os.WriteFile(done, []byte(source+"\n"), 0o644); err != nil {
@@ -187,3 +187,101 @@ func fileSHA256(path string) (string, error) {
 	}
 	return hex.EncodeToString(h.Sum(nil)), nil
 }
+
+// checkSetAtOnce is how many of a set's files are checked at once: a set of
+// six 50 GB files took eight minutes on one core, with nothing said.
+const checkSetAtOnce = 4
+
+// checkSet checks each file of a set against the checksum the mirror took
+// when it fetched it, several at once, saying how far it has got.
+func (f *Fetcher) checkSet(ctx context.Context, source, dir string, m manifest) error {
+	var total int64
+	for _, file := range m.Files {
+		if file.SHA256 != "" {
+			total += file.Bytes
+		}
+	}
+	if total == 0 {
+		return nil // nothing the mirror took a checksum of
+	}
+	var done atomic.Int64
+	stop := make(chan struct{})
+	defer close(stop)
+	go func() {
+		t := time.NewTicker(2 * time.Second)
+		defer t.Stop()
+		for {
+			select {
+			case <-stop:
+				return
+			case <-t.C:
+				f.report(Progress{Source: source, BytesDone: done.Load(), BytesTotal: total, Phase: PhaseChecking})
+			}
+		}
+	}()
+	f.report(Progress{Source: source, BytesTotal: total, Phase: PhaseChecking})
+
+	var (
+		mu    sync.Mutex
+		first error
+		wg    sync.WaitGroup
+	)
+	slots := make(chan struct{}, checkSetAtOnce)
+	for _, file := range m.Files {
+		if file.SHA256 == "" {
+			continue
+		}
+		wg.Add(1)
+		slots <- struct{}{}
+		go func() {
+			defer wg.Done()
+			defer func() { <-slots }()
+			target := filepath.Join(dir, file.Name)
+			got, err := countedSHA256(ctx, target, &done)
+			if err == nil && !strings.EqualFold(got, file.SHA256) {
+				_ = os.Remove(target)
+				err = fmt.Errorf("%s%s: sha256 %s, the mirror has %s; removed, to be fetched again",
+					source, file.Name, got, file.SHA256)
+			}
+			if err != nil {
+				mu.Lock()
+				if first == nil {
+					first = err
+				}
+				mu.Unlock()
+			}
+		}()
+	}
+	wg.Wait()
+	if first == nil {
+		f.report(Progress{Source: source, BytesDone: total, BytesTotal: total, Phase: PhaseChecking})
+	}
+	return first
+}
+
+// countedSHA256 is a file's checksum, adding what it has read to done.
+func countedSHA256(ctx context.Context, path string, done *atomic.Int64) (string, error) {
+	file, err := os.Open(path) //nolint:gosec // a file this fetcher wrote
+	if err != nil {
+		return "", err
+	}
+	defer file.Close()
+	h := sha256.New()
+	buf := make([]byte, 4<<20)
+	for {
+		if err := ctx.Err(); err != nil {
+			return "", err
+		}
+		n, err := file.Read(buf)
+		if n > 0 {
+			h.Write(buf[:n])
+			done.Add(int64(n))
+		}
+		if errors.Is(err, io.EOF) {
+			return hex.EncodeToString(h.Sum(nil)), nil
+		}
+		if err != nil {
+			return "", err
+		}
+	}
+}
@@ -9,7 +9,10 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
+	"sync"
+	"sync/atomic"
 	"testing"
+	"time"
 )

 // mirrorOf serves a manifest at "/m/v/" and the files beneath it.
@@ -204,3 +207,103 @@ func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) {
 		t.Fatalf("got %v", err)
 	}
 }
+
+// A set's download is one number climbing against the whole set, not each
+// part's own: a 46 GB part of 188 was shown as the download.
+func TestASetsProgressIsTheWholeSets(t *testing.T) {
+	f, srv, _ := mirrorOf(t, map[string]string{
+		"m-00001-of-00002.gguf": strings.Repeat("a", 600),
+		"m-00002-of-00002.gguf": strings.Repeat("b", 400),
+	})
+	var seen []Progress
+	f.OnProgress = func(p Progress) { seen = append(seen, p) }
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil {
+		t.Fatal(err)
+	}
+	if len(seen) == 0 {
+		t.Fatal("no progress reported")
+	}
+	last := int64(-1)
+	for _, p := range seen {
+		if p.BytesTotal != 1000 {
+			t.Fatalf("reported against %d, not the set's 1000: %+v", p.BytesTotal, seen)
+		}
+		if p.BytesDone < last {
+			t.Fatalf("went back: %+v", seen)
+		}
+		last = p.BytesDone
+	}
+	if last != 1000 {
+		t.Errorf("ended at %d", last)
+	}
+}
+
+// A set's files checked after they arrive say so, as far as they have got,
+// rather than leaving the download at 100% for minutes.
+func TestASetBeingCheckedSaysSo(t *testing.T) {
+	parts := map[string]string{"a.gguf": strings.Repeat("a", 300), "b.gguf": strings.Repeat("b", 200)}
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/") {
+			var files []string
+			for n, body := range parts {
+				sum := sha256.Sum256([]byte(body))
+				files = append(files, fmt.Sprintf(`{"name":%q,"bytes":%d,"sha256":%q}`, n, len(body), hex.EncodeToString(sum[:])))
+			}
+			fmt.Fprintf(w, `{"model":"m","version":"v","files":[%s]}`, strings.Join(files, ","))
+			return
+		}
+		_, _ = w.Write([]byte(parts[strings.TrimPrefix(r.URL.Path, "/m/v/")]))
+	}))
+	t.Cleanup(srv.Close)
+	var mu sync.Mutex
+	var checking []Progress
+	f := &Fetcher{CacheDir: t.TempDir(), OnProgress: func(p Progress) {
+		if p.Phase == PhaseChecking {
+			mu.Lock()
+			checking = append(checking, p)
+			mu.Unlock()
+		}
+	}}
+	if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil {
+		t.Fatal(err)
+	}
+	if len(checking) < 2 || checking[0].BytesDone != 0 || checking[len(checking)-1].BytesDone != 500 ||
+		checking[len(checking)-1].BytesTotal != 500 {
+		t.Errorf("%+v", checking)
+	}
+}
+
+// Four replicas on one node's four cards start together and want the same
+// set: it is fetched once, and every one of them gets it. Each fetching it
+// alone wrote the same .part files over one another, and three failed.
+func TestASetWantedByFourReplicasAtOnceIsFetchedOnce(t *testing.T) {
+	body := strings.Repeat("a", 1<<20)
+	var hits atomic.Int64
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/") {
+			fmt.Fprintf(w, `{"model":"m","version":"v","files":[{"name":"a.safetensors","bytes":%d},{"name":"b.safetensors","bytes":%d}]}`,
+				len(body), len(body))
+			return
+		}
+		hits.Add(1)
+		time.Sleep(100 * time.Millisecond) // long enough for all four to ask
+		_, _ = w.Write([]byte(body))
+	}))
+	t.Cleanup(srv.Close)
+	f := &Fetcher{CacheDir: t.TempDir()}
+
+	var wg sync.WaitGroup
+	errs := make([]error, 4)
+	for i := range errs {
+		wg.Go(func() { _, errs[i] = f.Ensure(t.Context(), srv.URL+"/m/v/", false) })
+	}
+	wg.Wait()
+	for i, err := range errs {
+		if err != nil {
+			t.Errorf("replica %d: %v", i, err)
+		}
+	}
+	if n := hits.Load(); n != 2 {
+		t.Errorf("%d downloads of a set of two files", n)
+	}
+}
@@ -112,8 +112,9 @@ func (w *statusWriter) Flush() {
 }

 // notChanges are POSTs that change nothing and carry text that must not be
-// kept: asking a judge about a sentence.
-var notChanges = []string{"/api/v1/guard/judges/ask"}
+// kept: asking a judge about a sentence. And a standby's report of what its
+// router sent where, every two seconds: activity, not an operator's change.
+var notChanges = []string{"/api/v1/guard/judges/ask", "/api/v1/routing/activity"}

 // API records every change to the admin API: a request that is not a GET or
 // HEAD, under /api/. actor says who made it.
@@ -15,6 +15,7 @@ import (
 	"time"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+	"golang.org/x/sync/singleflight"
 )

 // Principal is who a request is, once authenticated.
@@ -118,6 +119,12 @@ type Authenticator struct {
 	// lookups rations the database reads of keys not in the cache, per
 	// client address.
 	lookups *buckets
+
+	// resolving shares one database read between the requests that miss the
+	// cache for the same key at once: when a busy key's entry expired, every
+	// request in flight missed together, each took from the client's ration,
+	// and a valid key was told to slow down.
+	resolving singleflight.Group
 }

 type entry struct {
@@ -166,26 +173,37 @@ func (a *Authenticator) AuthenticateFrom(ctx context.Context, presented, client
 		return e.principal, e.err
 	}

-	if client != "" && !a.lookups.allow(client, a.now()) {
-		return Principal{}, ErrThrottled
-	}
-	p, err := a.resolve(ctx, hash)
-	if err != nil && !isRejection(err) {
-		// A database problem is not an authentication answer, so it must not be
-		// cached as one. Failing the request is right — the alternative is
-		// serving traffic whose authorization is unknown.
+	// One read for every request missing this key at once, rationed once.
+	v, err, _ := a.resolving.Do(id, func() (any, error) {
+		if e, ok := a.lookup(id); ok { // filled while this waited its turn
+			return e, nil
+		}
+		if client != "" && !a.lookups.allow(client, a.now()) {
+			return nil, ErrThrottled
+		}
+		p, err := a.resolve(ctx, hash)
+		if err != nil && !isRejection(err) {
+			// A database problem is not an authentication answer, so it must
+			// not be cached as one. Failing the request is right — the
+			// alternative is serving traffic whose authorization is unknown.
+			return nil, err
+		}
+		ttl := a.cfg.TTL
+		if err != nil {
+			ttl = a.cfg.NegativeTTL
+		}
+		e := entry{principal: p, err: err, expires: a.now().Add(ttl)}
+		a.store_(id, e)
+		return e, nil
+	})
+	if err != nil {
 		return Principal{}, err
 	}
-
-	ttl := a.cfg.TTL
-	if err != nil {
-		ttl = a.cfg.NegativeTTL
+	e := v.(entry)
+	if e.err == nil {
+		a.markUsed(e.principal.KeyID)
 	}
-	a.store_(id, entry{principal: p, err: err, expires: a.now().Add(ttl)})
-	if err == nil {
-		a.markUsed(p.KeyID)
-	}
-	return p, err
+	return e.principal, e.err
 }

 func (a *Authenticator) resolve(ctx context.Context, hash []byte) (Principal, error) {
@@ -478,3 +478,41 @@ func TestTheCacheDoesNotGrowWithoutBound(t *testing.T) {
 		t.Errorf("%d entries", n)
 	}
 }
+
+// A busy key whose entry has expired: every request in flight misses at once,
+// and they share one read and one of the address's allowance. Each taking
+// its own told a valid key to slow down, every thirty seconds, under load.
+func TestABusyKeyMissingTheCacheTogetherIsReadOnceAndNotThrottled(t *testing.T) {
+	s, secret, _ := fixture(t)
+	c := &slowStore{countingStore: countingStore{Store: s}, delay: 50 * time.Millisecond}
+	a := New(c, DefaultConfig())
+	var wg sync.WaitGroup
+	var throttled atomic.Int64
+	for range 200 {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			if _, err := a.AuthenticateFrom(context.Background(), secret, "203.0.113.9"); errors.Is(err, ErrThrottled) {
+				throttled.Add(1)
+			} else if err != nil {
+				t.Error(err)
+			}
+		}()
+	}
+	wg.Wait()
+	if throttled.Load() != 0 || c.lookups.Load() != 1 {
+		t.Errorf("%d throttled, %d database reads", throttled.Load(), c.lookups.Load())
+	}
+}
+
+// slowStore is a countingStore whose reads take a while, so the requests
+// missing together are still waiting when the first one asks.
+type slowStore struct {
+	countingStore
+	delay time.Duration
+}
+
+func (s *slowStore) KeyByHash(ctx context.Context, hash []byte) (store.Key, error) {
+	time.Sleep(s.delay)
+	return s.countingStore.KeyByHash(ctx, hash)
+}
@@ -499,8 +499,12 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
 					if s.Fetch == nil {
 						continue
 					}
-					fmt.Fprintf(w, "  %s: fetching %s\t\t\t\t\t\t\t\t\n",
-						s.Node, fetchColumn(s.Fetch))
+					what := "fetching"
+					if s.Fetch.Phase != "" {
+						what = s.Fetch.Phase
+					}
+					fmt.Fprintf(w, "  %s: %s %s\t\t\t\t\t\t\t\t\n",
+						s.Node, what, fetchColumn(s.Fetch))
 				}
 			}
 			_ = w.Flush()
@@ -72,6 +72,8 @@ func newEngineCheckCmd() *cobra.Command {
 					cfg.VLLM.Bin = bin
 				case "sglang":
 					cfg.SGLang.Python = bin
+				case "freetoken":
+					cfg.FreeToken.Binary = bin
 				case "trtllm":
 					cfg.TRTLLM.Python = bin
 				case "sim":
@@ -49,6 +49,9 @@ type leaderForward struct {
 	self string
 	tls  tlsconf.Config
 	log  *slog.Logger
+	// routes, with router.active_active, is the leader's replicas this
+	// standby routes to; nil when every request goes to the leader.
+	routes *standbyRoutes
 }

 // standby wraps the admin handler: /api/ goes to the leader while this
@@ -117,7 +120,8 @@ func (f *leaderForward) public(next http.Handler, port string) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		// Once only: a request another control plane forwarded is answered
 		// here, leading or not, rather than sent on while the lease moves.
-		if publicStaysHere(r.URL.Path) || f.lead.IsLeader() || r.Header.Get(hdrForwardedBy) != "" {
+		if publicStaysHere(r.URL.Path) || f.lead.IsLeader() || r.Header.Get(hdrForwardedBy) != "" ||
+			f.answersHere(r.URL.Path) {
 			next.ServeHTTP(w, r)
 			return
 		}
@@ -154,6 +158,15 @@ func (f *leaderForward) public(next http.Handler, port string) http.Handler {
 	})
 }

+// answersHere is whether a standby answers this client request itself: the
+// API under /v1, with router.active_active, while it has the leader's
+// replicas to send it to. Not the portal, its admin pages or /mcp, whose
+// llmgw:admin tools read the cluster from memory only the leader has; and
+// not while the leader has gone unheard, when it is passed on as before.
+func (f *leaderForward) answersHere(path string) bool {
+	return f.routes != nil && strings.HasPrefix(path, "/v1/") && f.routes.serving()
+}
+
 // trustControlPlanes keeps the client port trusting the other control
 // planes' X-Forwarded-For, besides router.trusted_proxies: a standby passes
 // requests on, and the client is the address it was given, not the standby.
@@ -32,7 +32,11 @@ type forwardCluster struct {
 	leaderURL       string
 }

-func newForwardCluster(t *testing.T) *forwardCluster {
+func newForwardCluster(t *testing.T) *forwardCluster { return newForwardClusterWith(t, nil) }
+
+// newForwardClusterWith answers forwarded requests on the leader with admin,
+// or, given nil, with one that says what it was asked.
+func newForwardClusterWith(t *testing.T, leaderAdmin http.Handler) *forwardCluster {
 	t.Helper()
 	dir := t.TempDir()
 	ca, err := tlsconf.NewCA("test CA")
@@ -81,6 +85,9 @@ func newForwardCluster(t *testing.T) *forwardCluster {
 			"auth": r.Header.Get("Authorization"), "actor": r.Header.Get("X-Llmgw-Actor")})
 	})
 	mux := http.NewServeMux()
+	if leaderAdmin != nil {
+		admin = leaderAdmin.ServeHTTP
+	}
 	mux.Handle(forwardPrefix+"/", c.leader.leaderSide(admin))
 	srv := httptest.NewUnstartedServer(mux)
 	tc, err := cfgFor("ctl-a").ServerConfig()
@@ -472,10 +472,10 @@ func newAdaptersCmd(client func() *ctlClient) *cobra.Command {
 }

 func newModelContextCmd(client func() *ctlClient) *cobra.Command {
-	var slots int
+	var slots, maxOut int
 	c := &cobra.Command{
-		Use:   "context <model> <tokens>",
-		Short: "Change how much one request to a served model may use",
+		Use:   "context <model> [tokens]",
+		Short: "Change how much one request to a served model may use, and how long its answers may be",
 		Long: "The context lives in three places: the catalog (what a request may use, and\n" +
 			"what it may answer), the engine's arguments (llama.cpp's --ctx-size and\n" +
 			"--parallel, vLLM's --max-model-len), and the memory the KV cache takes. This\n" +
@@ -483,21 +483,39 @@ func newModelContextCmd(client func() *ctlClient) *cobra.Command {
 			"and a context its cards would not hold, saying what each would need. Its\n" +
 			"replicas are then started again on it, each new one before the old stops.\n\n" +
 			"--slots is how many requests a llama.cpp replica answers at once; its pool is\n" +
-			"slots × tokens. Left out, it keeps what it has.",
+			"slots × tokens. Left out, it keeps what it has.\n\n" +
+			"--max-output is the longest answer the gateway lets a request ask for. With a\n" +
+			"context it is set with it; left out, it follows the context as it did. Alone,\n" +
+			"without tokens, it changes only that, and no replica is started again.\n\n" +
+			"Strata's context is its setup's --context, and FreeToken's\n" +
+			"--max-seq-len-override; for both the card is not checked, since each sizes its\n" +
+			"caches to it.",
 		Example: "  llmgw ctl model context qwen3-coder-30b 65536\n" +
-			"  llmgw ctl model context qwen2.5-7b 16384 --slots 4",
-		Args: cobra.ExactArgs(2),
+			"  llmgw ctl model context qwen2.5-7b 16384 --slots 4\n" +
+			"  llmgw ctl model context qwen3.8-flash-next-iq2_xs 32768 --max-output 8192\n" +
+			"  llmgw ctl model context qwen3.8-flash-next-iq2_xs --max-output 4096",
+		Args: cobra.RangeArgs(1, 2),
 		RunE: func(cmd *cobra.Command, args []string) error {
-			tokens, err := strconv.Atoi(args[1])
-			if err != nil {
-				return fmt.Errorf("the context is a whole number of tokens, not %q", args[1])
+			tokens := 0
+			if len(args) == 2 {
+				var err error
+				if tokens, err = strconv.Atoi(args[1]); err != nil {
+					return fmt.Errorf("the context is a whole number of tokens, not %q", args[1])
+				}
+			} else if maxOut == 0 {
+				return fmt.Errorf("say the context's tokens, or --max-output, or both")
 			}
-			body, _ := json.Marshal(modelserve.ContextRequest{Tokens: tokens, Slots: slots})
+			body, _ := json.Marshal(modelserve.ContextRequest{Tokens: tokens, Slots: slots, MaxOutput: maxOut})
 			var v modelserve.ContextChange
 			if err := client().patient(time.Minute).post(cmd.Context(), "/api/v1/models/"+args[0]+"/context", body, &v); err != nil {
 				return err
 			}
 			out := cmd.OutOrStdout()
+			if v.OutputOnly {
+				fmt.Fprintf(out, "%s: answers up to %d tokens (was %d); its context stays %d, and no replica was started again\n",
+					v.Model, v.MaxOutput, v.MaxOutputBefore, v.After)
+				return nil
+			}
 			fmt.Fprintf(out, "%s: %d → %d tokens a request, answers up to %d", v.Model, v.Before, v.After, v.MaxOutput)
 			if v.Trained > 0 {
 				fmt.Fprintf(out, " (trained for %d)", v.Trained)
@@ -523,6 +541,7 @@ func newModelContextCmd(client func() *ctlClient) *cobra.Command {
 		},
 	}
 	c.Flags().IntVar(&slots, "slots", 0, "llama.cpp: requests a replica answers at once (default: as it is)")
+	c.Flags().IntVar(&maxOut, "max-output", 0, "the longest answer, in tokens (default: follows the context)")
 	return c
 }

@@ -691,6 +691,21 @@ func newServerCmd() *cobra.Command {
 			if admin != nil && admin.Leadership() != nil && rt.Store() != nil && cfg.AgentTLS.Enabled() {
 				self, _ := os.Hostname()
 				forward = &leaderForward{st: rt.Store(), lead: admin.Leadership(), self: self, tls: cfg.AgentTLS, log: log}
+				// And with router.active_active, a standby answers the client
+				// API itself, to the leader's replicas.
+				if cfg.Router.ActiveActive {
+					var refresh func(context.Context) error
+					if catalogSvc != nil {
+						refresh = catalogSvc.Refresh
+					}
+					routes := newStandbyRoutes(forward, rt, refresh, admin.Publish, log)
+					forward.routes = routes
+					admin.SetReplicaFeed(routes)
+					rt.SetActivator(standbyActivator{Reconciler: admin.Reconciler(), routes: routes})
+					go routes.run(ctx)
+					go rt.ShareBudgets(ctx)
+					log.Info("standbys answer the client API themselves (router.active_active)")
+				}
 			}
 			served := adminHandler
 			if forward != nil {
@@ -0,0 +1,254 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log/slog"
+	"net/http"
+	"sync"
+	"time"
+
+	"golang.org/x/net/http2"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
+)
+
+// How often a standby answering requests itself hears from the leader, and
+// how long it keeps routing to the replicas it last heard of when it does
+// not. Long enough to ride out the leader changing hands — the engines are
+// still where they were, and a dead one opens its breaker — short enough
+// that a standby cut off from the cluster stops sending to a list it cannot
+// check.
+const (
+	standbyPoll  = 2 * time.Second
+	standbyTrust = time.Minute
+)
+
+// The router is what counts against limits, and shares the counts.
+var _ control.RateSharer = (*router.Router)(nil)
+
+// standbyRoutes is a standby's copy of the leader's Ready replicas, when
+// standbys answer requests themselves (router.active_active). Each tick it
+// reports what its router used and is answered with the replicas, in one
+// request over the agent port with its control-plane certificate.
+type standbyRoutes struct {
+	fwd     *leaderForward
+	usage   control.UsageSource
+	refresh func(context.Context) error // reads the catalog again from the store
+	publish func(context.Context)       // hands the router the new list now
+	log     *slog.Logger
+	now     func() time.Time
+
+	mu      sync.Mutex
+	reps    []control.RoutedReplica
+	heard   time.Time
+	catalog string               // the leader's catalog revision last seen
+	sent    map[string]time.Time // the last use already reported, by replica
+	want    map[string]bool      // on-demand models asked for since the last report
+	kick    chan struct{}
+
+	// The connection to the leader, kept between ticks: one per leader
+	// address, the previous one's closed when the lease moves.
+	client     *http.Client
+	clientHost string
+}
+
+func newStandbyRoutes(fwd *leaderForward, usage control.UsageSource, refresh func(context.Context) error,
+	publish func(context.Context), log *slog.Logger,
+) *standbyRoutes {
+	return &standbyRoutes{fwd: fwd, usage: usage, refresh: refresh, publish: publish, log: log, now: time.Now,
+		sent: map[string]time.Time{}, want: map[string]bool{}, kick: make(chan struct{}, 1)}
+}
+
+// Ready satisfies control.ReplicaFeed.
+func (s *standbyRoutes) Ready() ([]control.RoutedReplica, bool) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.heard.IsZero() || s.now().Sub(s.heard) > standbyTrust {
+		return nil, false
+	}
+	return s.reps, true
+}
+
+// serving says whether this standby has replicas to send a request to, which
+// is when it keeps the client API rather than passing it to the leader.
+func (s *standbyRoutes) serving() bool {
+	reps, ok := s.Ready()
+	return ok && len(reps) > 0
+}
+
+// wanted queues an on-demand model for the leader, and reports at once
+// rather than on the next tick: someone is waiting on it.
+func (s *standbyRoutes) wanted(model string) {
+	s.mu.Lock()
+	s.want[model] = true
+	s.mu.Unlock()
+	select {
+	case s.kick <- struct{}{}:
+	default:
+	}
+}
+
+func (s *standbyRoutes) run(ctx context.Context) {
+	t := time.NewTicker(standbyPoll)
+	defer t.Stop()
+	for {
+		if s.fwd.lead.IsLeader() {
+			// Its own reconciler's replicas now. What it heard as a standby
+			// is forgotten, so becoming one again starts afresh.
+			s.mu.Lock()
+			s.reps, s.heard, s.catalog = nil, time.Time{}, ""
+			clear(s.sent)
+			clear(s.want)
+			s.mu.Unlock()
+		} else if err := s.exchange(ctx); err != nil {
+			s.log.Debug("could not hear from the leader", "err", err)
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		case <-s.kick:
+		}
+	}
+}
+
+// exchange reports activity and takes the replicas in return.
+func (s *standbyRoutes) exchange(ctx context.Context) error {
+	act, sent := s.activity()
+	body, err := json.Marshal(act)
+	if err != nil {
+		return err
+	}
+	target, name, err := s.fwd.leader(ctx)
+	if err != nil {
+		return err
+	}
+	client, err := s.clientFor(target.Host, target.Hostname())
+	if err != nil {
+		return err
+	}
+	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
+	defer cancel()
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost,
+		"https://"+target.Host+forwardPrefix+"/api/v1/routing/activity", bytes.NewReader(body))
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set(hdrForwardedBy, s.fwd.self)
+	resp, err := client.Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		b, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
+		return fmt.Errorf("%s answered %s: %s", name, resp.Status, bytes.TrimSpace(b))
+	}
+	var rt control.Routing
+	if err := json.NewDecoder(resp.Body).Decode(&rt); err != nil {
+		return fmt.Errorf("%s's routing: %w", name, err)
+	}
+
+	s.mu.Lock()
+	s.reps, s.heard = rt.Replicas, s.now()
+	stale := rt.Catalog != s.catalog
+	s.catalog = rt.Catalog
+	for id, at := range sent {
+		s.sent[id] = at
+	}
+	for _, m := range act.Want {
+		delete(s.want, m)
+	}
+	s.mu.Unlock()
+
+	// What the leader and the other standbys counted, so this router's
+	// limits are the cluster's.
+	if rs, ok := s.usage.(control.RateSharer); ok && rt.Rates != nil {
+		rs.SetOtherRates(*rt.Rates)
+	}
+
+	// A model added or changed through the leader is in the store: the
+	// catalog first, so the replicas published next resolve against it.
+	if stale && s.refresh != nil {
+		if err := s.refresh(ctx); err != nil {
+			s.log.Warn("could not read the catalog the leader changed", "err", err)
+			s.mu.Lock()
+			s.catalog = "" // tried again next tick
+			s.mu.Unlock()
+		}
+	}
+	if s.publish != nil {
+		s.publish(ctx)
+	}
+	return nil
+}
+
+// clientFor is the client for the leader at host, made once per leader.
+// Only the run loop calls it.
+func (s *standbyRoutes) clientFor(host, name string) (*http.Client, error) {
+	if s.client != nil && s.clientHost == host {
+		return s.client, nil
+	}
+	tc, err := s.fwd.tls.ClientConfig(name)
+	if err != nil {
+		return nil, err
+	}
+	if s.client != nil {
+		s.client.CloseIdleConnections()
+	}
+	s.client = &http.Client{Transport: &http2.Transport{TLSClientConfig: tc, ReadIdleTimeout: 30 * time.Second}}
+	s.clientHost = host
+	return s.client, nil
+}
+
+// activity is what to report: uses newer than the last reported, and the
+// models asked for. sent is what to record as reported once the leader has
+// it.
+func (s *standbyRoutes) activity() (control.Activity, map[string]time.Time) {
+	var used map[string]time.Time
+	if s.usage != nil {
+		used = s.usage.LastUsed()
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	act := control.Activity{From: s.fwd.self, LastUsed: map[string]time.Time{}}
+	if rs, ok := s.usage.(control.RateSharer); ok {
+		r := rs.Rates()
+		act.Rates = &r
+	}
+	for id, at := range used {
+		if at.After(s.sent[id]) {
+			act.LastUsed[id] = at
+		}
+	}
+	for m := range s.want {
+		act.Want = append(act.Want, m)
+	}
+	return act, act.LastUsed
+}
+
+// standbyActivator asks the leader for an on-demand model when a request for
+// it reaches a standby: the standby's own reconciler places nothing. Which
+// models load on demand, and why one cannot, are still the reconciler's to
+// answer from the catalog.
+type standbyActivator struct {
+	*reconciler.Reconciler
+	routes *standbyRoutes
+}
+
+func (a standbyActivator) Want(model string) bool {
+	if !a.Reconciler.Want(model) {
+		return false
+	}
+	if !a.routes.fwd.lead.IsLeader() {
+		a.routes.wanted(model)
+	}
+	return true
+}
@@ -0,0 +1,186 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"log/slog"
+	"net"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
+)
+
+// fakeLeaderRouting is the leader's routing API: what it was told, and the
+// replicas and catalog it answers with.
+type fakeLeaderRouting struct {
+	mu      sync.Mutex
+	got     []control.Activity
+	routing control.Routing
+}
+
+func (f *fakeLeaderRouting) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodPost || r.URL.Path != "/api/v1/routing/activity" {
+		http.NotFound(w, r)
+		return
+	}
+	var act control.Activity
+	if err := json.NewDecoder(r.Body).Decode(&act); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.got = append(f.got, act)
+	_ = json.NewEncoder(w).Encode(f.routing)
+}
+
+func (f *fakeLeaderRouting) last() control.Activity {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return f.got[len(f.got)-1]
+}
+
+type fixedUsage map[string]time.Time
+
+func (u fixedUsage) LastUsed() map[string]time.Time { return u }
+
+// sharingUsage is a router whose limits are counted with the others'.
+type sharingUsage struct {
+	fixedUsage
+	own    accounting.Rates
+	others *accounting.Rates
+}
+
+func (u *sharingUsage) Rates() accounting.Rates          { return u.own }
+func (u *sharingUsage) SetOtherRates(r accounting.Rates) { u.others = &r }
+
+// A standby reports its counts under its name and puts what the leader says
+// the rest of the cluster counted in force on its router.
+func TestAStandbysLimitsAreTheClusters(t *testing.T) {
+	theirs := accounting.NewRates(time.Now())
+	theirs.Tokens.Current = map[string]int64{"key-1": 900}
+	leader := &fakeLeaderRouting{routing: control.Routing{Rates: &theirs}}
+	c := newForwardClusterWith(t, leader)
+	u := &sharingUsage{own: accounting.NewRates(time.Now())}
+	u.own.Requests.Current = map[string]int64{"key-1": 4}
+	s := newStandbyRoutes(c.standby, u, nil, nil, slog.New(slog.DiscardHandler))
+	if err := s.exchange(t.Context()); err != nil {
+		t.Fatal(err)
+	}
+	act := leader.last()
+	if act.From != "ctl-b" || act.Rates == nil || act.Rates.Requests.Current["key-1"] != 4 {
+		t.Errorf("reported %+v", act)
+	}
+	if u.others == nil || u.others.Tokens.Current["key-1"] != 900 {
+		t.Errorf("the router was told %+v", u.others)
+	}
+}
+
+func TestAStandbyFollowsTheLeadersReplicas(t *testing.T) {
+	leader := &fakeLeaderRouting{routing: control.Routing{Catalog: "a/1",
+		Replicas: []control.RoutedReplica{{ID: "llama-r0", Model: "llama", Endpoint: "http://10.0.0.5:20001"}}}}
+	c := newForwardClusterWith(t, leader)
+	used := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC)
+	refreshed := 0
+	s := newStandbyRoutes(c.standby, fixedUsage{"llama-r0": used}, func(_ context.Context) error {
+		refreshed++
+		return nil
+	}, nil, slog.New(slog.DiscardHandler))
+
+	if _, ok := s.Ready(); ok {
+		t.Fatal("ready before hearing from the leader")
+	}
+	if err := s.exchange(t.Context()); err != nil {
+		t.Fatal(err)
+	}
+	if reps, ok := s.Ready(); !ok || len(reps) != 1 || reps[0].ID != "llama-r0" {
+		t.Fatalf("replicas %+v, %v", reps, ok)
+	}
+	if got := leader.last().LastUsed["llama-r0"]; !got.Equal(used) {
+		t.Errorf("the leader was told the replica was last used %v", got)
+	}
+	if refreshed != 1 {
+		t.Errorf("the catalog was read %d times on first hearing from the leader", refreshed)
+	}
+
+	// Only what is new is reported, and the catalog is read again only when
+	// the leader's changed.
+	s.wanted("flux")
+	if err := s.exchange(t.Context()); err != nil {
+		t.Fatal(err)
+	}
+	if act := leader.last(); len(act.LastUsed) != 0 || !slices.Equal(act.Want, []string{"flux"}) {
+		t.Errorf("second report %+v", act)
+	}
+	if refreshed != 1 {
+		t.Errorf("an unchanged catalog was read again")
+	}
+	leader.mu.Lock()
+	leader.routing.Catalog = "a/2"
+	leader.mu.Unlock()
+	if err := s.exchange(t.Context()); err != nil {
+		t.Fatal(err)
+	}
+	if act := leader.last(); len(act.Want) != 0 {
+		t.Errorf("an on-demand model reported twice: %+v", act)
+	}
+	if refreshed != 2 {
+		t.Errorf("the changed catalog was not read: %d", refreshed)
+	}
+
+	// Unheard for longer than it is trusted, it routes nothing, and requests
+	// go to the leader again.
+	c.standby.routes = s
+	if !c.standby.answersHere("/v1/chat/completions") {
+		t.Error("a standby with replicas passed the API on")
+	}
+	s.now = func() time.Time { return time.Now().Add(standbyTrust + time.Second) }
+	if _, ok := s.Ready(); ok || c.standby.answersHere("/v1/chat/completions") {
+		t.Error("a standby the leader has not answered for a minute still routes")
+	}
+}
+
+// The client API stays on a standby with replicas; the portal, its admin
+// pages and /mcp still go to the leader.
+func TestAStandbyAnswersOnlyTheAPIItself(t *testing.T) {
+	leader := &fakeLeaderRouting{routing: control.Routing{
+		Replicas: []control.RoutedReplica{{ID: "llama-r0", Model: "llama", Endpoint: "http://10.0.0.5:20001"}}}}
+	c := newForwardClusterWith(t, leader)
+	s := newStandbyRoutes(c.standby, nil, nil, nil, slog.New(slog.DiscardHandler))
+	if err := s.exchange(t.Context()); err != nil {
+		t.Fatal(err)
+	}
+	c.standby.routes = s
+
+	port := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		_, _ = w.Write([]byte("leader " + r.URL.Path))
+	}))
+	defer port.Close()
+	_, p, _ := net.SplitHostPort(strings.TrimPrefix(port.URL, "http://"))
+	local := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("standby " + r.URL.Path)) })
+	srv := httptest.NewServer(c.standby.public(local, p))
+	defer srv.Close()
+
+	for path, want := range map[string]string{
+		"/v1/chat/completions": "standby", "/v1/models": "standby",
+		"/admin/replicas": "leader", "/mcp": "leader", "/": "leader",
+	} {
+		resp, err := http.Get(srv.URL + path) //nolint:noctx // a test
+		if err != nil {
+			t.Fatal(err)
+		}
+		b, _ := io.ReadAll(resp.Body)
+		resp.Body.Close()
+		if got, _, _ := strings.Cut(string(b), " "); got != want {
+			t.Errorf("%s answered by %q, want the %s", path, b, want)
+		}
+	}
+}
@@ -346,10 +346,12 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
 		}},
 		{"Context length…", func(m *consoleModel) tea.Cmd {
 			m.openForm(&formDialog{title: "Context of " + md.Name,
-				note: "What one request may use. The engine's arguments and the memory change with it, checked against " +
-					"what the cards hold; its replicas are then started again on it, one by one.",
+				note: "What one request may use, and the longest answer. A changed context changes the engine's arguments " +
+					"and memory, checked against what the cards hold, and starts its replicas again one by one; the " +
+					"longest answer alone changes only the gateway's limit, at once.",
 				fields: []formField{
 					textF("tokens", strconv.Itoa(md.ContextLength), "a request's prompt and answer together"),
+					textF("longest answer", strconv.Itoa(md.MaxOutput), "tokens; empty follows the context"),
 					textF("slots", "", "llama.cpp: requests at once; empty keeps it"),
 				},
 				submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
@@ -357,14 +359,25 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
 					if err != nil || tokens < 256 {
 						return nil, fmt.Errorf("the context is a whole number of tokens, at least 256")
 					}
-					slots := 0
+					maxOut := 0
 					if v[1] != "" {
-						if slots, err = strconv.Atoi(v[1]); err != nil || slots < 1 {
+						if maxOut, err = strconv.Atoi(v[1]); err != nil || maxOut < 1 {
+							return nil, fmt.Errorf("the longest answer is a whole number of tokens")
+						}
+					}
+					slots := 0
+					if v[2] != "" {
+						if slots, err = strconv.Atoi(v[2]); err != nil || slots < 1 {
 							return nil, fmt.Errorf("slots is a whole number from 1")
 						}
 					}
-					return m.call("Context of "+md.Name, http.MethodPost, path+"/context",
-						modelserve.ContextRequest{Tokens: tokens, Slots: slots}, contextSays), nil
+					req := modelserve.ContextRequest{Tokens: tokens, Slots: slots, MaxOutput: maxOut}
+					if tokens == md.ContextLength && slots == 0 && maxOut > 0 {
+						// The context as it is: only the longest answer, and
+						// no replica started again for it.
+						req.Tokens = 0
+					}
+					return m.call("Context of "+md.Name, http.MethodPost, path+"/context", req, contextSays), nil
 				}})
 			return nil
 		}},
@@ -555,7 +568,7 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
 	}
 	items = append(items, m.locationItems(n)...)
 	if n.GPUTotal > 0 {
-		for _, engine := range []string{"vllm", "sglang"} {
+		for _, engine := range []string{"vllm", "sglang", "freetoken"} {
 			items = append(items, menuItem{engine + " build…", func(m *consoleModel) tea.Cmd {
 				m.openForm(&formDialog{title: engine + " on " + n.ID,
 					note:   "Which build of " + engine + " this node installs: auto follows its hardware, none installs none.",
@@ -999,6 +1012,9 @@ func contextSays(raw json.RawMessage) string {
 	if json.Unmarshal(raw, &v) != nil {
 		return "Done."
 	}
+	if v.OutputOnly {
+		return fmt.Sprintf("%s: answers up to %d tokens (was %d); no replica was started again.", v.Model, v.MaxOutput, v.MaxOutputBefore)
+	}
 	s := fmt.Sprintf("%s: %d → %d tokens a request, answers up to %d.", v.Model, v.Before, v.After, v.MaxOutput)
 	for _, p := range v.Profiles {
 		if p.After != "" {
@@ -1497,6 +1497,10 @@ func (m consoleModel) gpuRows() [][]string {
 					}
 					if !mig {
 						busy = fmt.Sprintf("%.0f%%", s.UtilPercent)
+						if p := s.Profile; p != nil && p.SMActive >= 0 {
+							// How much of the chip, beside whether anything ran.
+							busy += fmt.Sprintf(" · SM %.0f%%", p.SMActive)
+						}
 						if s.MemoryTotalMB > 0 {
 							mem = fmt.Sprintf("%.1f/%.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024)
 						}
@@ -14,6 +14,7 @@ import (
 	"github.com/charmbracelet/lipgloss"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 )

@@ -80,6 +81,9 @@ func (s *monitorState) sample(d consoleData) {
 		for _, g := range l.GPUs {
 			card := nd.ID + ":" + strconv.Itoa(g.Index)
 			add("gpu/"+card, g.UtilPercent)
+			if w := gpuWork(g); w >= 0 {
+				add("gpuwork/"+card, w)
+			}
 			if g.MemoryTotalMB > 0 {
 				add("vram/"+card, 100*g.MemoryUsedMB/g.MemoryTotalMB)
 			}
@@ -142,6 +146,7 @@ func monitorQueries(window time.Duration, points int) (time.Duration, []adminapi
 		{ID: "cpu", Expr: `avg(llmgw_node_cpu_utilization_percent)`},
 		{ID: "mem", Expr: `100 * sum(llmgw_node_memory_used_bytes) / sum(llmgw_node_memory_bytes)`},
 		{ID: "gpu", Expr: card(`llmgw_gpu_utilization_percent`), By: "card"},
+		{ID: "gpuwork", Expr: card(gpuWorkExpr("")), By: "card"},
 		{ID: "vram", Expr: card(`100 * llmgw_gpu_memory_used_bytes / llmgw_gpu_memory_total_bytes`), By: "card"},
 		{ID: "gputemp", Expr: card(`llmgw_gpu_temperature_celsius`), By: "card"},
 		{ID: "gpupow", Expr: card(`llmgw_gpu_power_watts`), By: "card"},
@@ -314,7 +319,8 @@ func (m consoleModel) monitorPanel(pw int) (string, string, string) {
 	b.WriteString(indent(side(w, box("CPU", pct("cpu"), m.series("cpu"), 100),
 		box("Memory", pct("mem"), m.series("mem"), 100)), "  ") + "\n\n")
 	if g := m.series("gpu"); len(g) > 0 || len(m.data.sum.GPUs) > 0 {
-		b.WriteString(indent(side(w, box("GPU busy", "", g, 100), box("GPU memory", "", m.series("vram"), 100)), "  ") + "\n\n")
+		b.WriteString(indent(side(w, box("GPU busy", "", g, 100), box("GPU work", "", m.series("gpuwork"), 100)), "  ") + "\n\n")
+		b.WriteString(indent(side(w, box("GPU memory", "", m.series("vram"), 100), "  "+sDim.Render(gpuWorkNote)), "  ") + "\n\n")
 	}
 	if m.mon.res.Configured {
 		rps := m.series("rps")
@@ -356,10 +362,12 @@ func (m consoleModel) monitorPanel(pw int) (string, string, string) {
 	// Each card: busy, memory, heat and power.
 	if gpus := m.series("gpu"); len(gpus) > 0 {
 		vram, temp, pow := byName(m.series("vram")), byName(m.series("gputemp")), byName(m.series("gpupow"))
+		work := byName(m.series("gpuwork"))
 		b.WriteString("\n  " + mdBold.Render("GPUs") + "\n")
 		for _, g := range gpus {
-			fmt.Fprintf(&b, "  %-16s busy %s %4.0f%%  memory %s %4.0f%%  %3.0f°C %s  %4.0f W\n", truncate(g.name, 16),
-				sparkSeries(g, sw/2, 100), last(g.values), sparkSeries(vram[g.name], sw/2, 100), last(vram[g.name].values),
+			fmt.Fprintf(&b, "  %-16s busy %s %4.0f%%  work %s %4.0f%%  memory %s %4.0f%%  %3.0f°C %s  %4.0f W\n", truncate(g.name, 16),
+				sparkSeries(g, sw/3, 100), last(g.values), sparkSeries(work[g.name], sw/3, 100), last(work[g.name].values),
+				sparkSeries(vram[g.name], sw/3, 100), last(vram[g.name].values),
 				last(temp[g.name].values), sparkSeries(temp[g.name], 8, 100), last(pow[g.name].values))
 		}
 	}
@@ -373,6 +381,29 @@ func (m consoleModel) monitorPanel(pw int) (string, string, string) {
 	return "Monitor", head.String() + b.String(), keys
 }

+// GPU busy is the driver's figure: the share of time any kernel ran, which
+// a kernel on one SM of a hundred fills. GPU work is how much of the chip:
+// the SMs' share of time with work where the card's profiling counters are
+// read (GPM, DCGM), and its draw against its power limit elsewhere.
+const gpuWorkNote = "busy: some kernel ran. work: how much of the chip —\nSMs active where the counters are read, power/limit elsewhere."
+
+// gpuWorkExpr is GPU work in PromQL, for the series matching sel.
+func gpuWorkExpr(sel string) string {
+	return `llmgw_gpu_sm_active_percent` + sel + ` or (100 * llmgw_gpu_power_watts` + sel +
+		` / llmgw_gpu_power_limit_watts` + sel + `)`
+}
+
+// gpuWork is GPU work from a sample, or -1 without the figures for it.
+func gpuWork(g inventory.GPUSensor) float64 {
+	if p := g.Profile; p != nil && p.SMActive >= 0 {
+		return p.SMActive
+	}
+	if l := g.PowerLoad(); l >= 0 {
+		return 100 * l
+	}
+	return -1
+}
+
 func named(s []chartSeries, name string) []chartSeries {
 	for i := range s {
 		s[i].name = name
@@ -410,6 +441,7 @@ func detailQueries(kind, id string, window time.Duration, points int) (time.Dura
 			{ID: "cpu", Expr: `llmgw_node_cpu_utilization_percent{node=` + q + `}`},
 			{ID: "mem", Expr: `100 * llmgw_node_memory_used_bytes{node=` + q + `} / llmgw_node_memory_bytes{node=` + q + `}`},
 			{ID: "gpu", Expr: card(`llmgw_gpu_utilization_percent{node=` + q + `}`), By: "card"},
+			{ID: "gpuwork", Expr: card(gpuWorkExpr(`{node=` + q + `}`)), By: "card"},
 			{ID: "vram", Expr: card(`100 * llmgw_gpu_memory_used_bytes{node=` + q + `} / llmgw_gpu_memory_total_bytes{node=` + q + `}`), By: "card"},
 			{ID: "netin", Expr: `sum(rate(node_network_receive_bytes_total{` + netDevs + `,node=` + q + `}[` + rw + `]))`},
 			{ID: "netout", Expr: `sum(rate(node_network_transmit_bytes_total{` + netDevs + `,node=` + q + `}[` + rw + `]))`},
@@ -520,7 +552,8 @@ func (m consoleModel) detailCharts(pw int) string {
 		b.WriteString(indent(side(w, box("CPU", fmt.Sprintf("%.0f%%", lastOf(cpu)), cpu, 100),
 			box("Memory", fmt.Sprintf("%.0f%%", lastOf(mem)), mem, 100)), "  "))
 		if g := get("gpu"); len(g) > 0 {
-			b.WriteString("\n\n" + indent(side(w, box("GPU busy", "", g, 100), box("GPU memory", "", get("vram"), 100)), "  "))
+			b.WriteString("\n\n" + indent(side(w, box("GPU busy", "", g, 100), box("GPU work", "", get("gpuwork"), 100)), "  "))
+			b.WriteString("\n\n" + indent(side(w, box("GPU memory", "", get("vram"), 100), "  "+sDim.Render(gpuWorkNote)), "  "))
 		}
 		if m.det.res.Configured {
 			net := append(named(get("netin"), "in"), named(get("netout"), "out")...)
@@ -641,6 +641,19 @@ type Router struct {
 	// written into the setup scripts and whose attempts are counted where.
 	TrustedProxies []string `koanf:"trusted_proxies"`

+	// ActiveActive has standby control planes answer the client API
+	// themselves, routing to the leader's replicas, rather than passing
+	// every request to the leader. More machines carry the traffic, which
+	// is the point. Rate limits are the cluster's: each router's counts
+	// reach the others through the leader every two seconds, so a key
+	// spread over N control planes may pass its limit by what they allow
+	// in those two seconds. A budget is checked against the store every
+	// half minute; cached answers and prefix affinity are split between
+	// the routers. Off by
+	// default. Which machine a client reaches is the load balancer's or
+	// DNS's to say; the floating address stays on the leader.
+	ActiveActive bool `koanf:"active_active"`
+
 	Timeouts  Timeouts  `koanf:"timeouts"`
 	Scoring   Scoring   `koanf:"scoring"`
 	Admission Admission `koanf:"admission"`
@@ -1069,12 +1082,18 @@ func (s Server) ControlPlaneMode() bool {
 // DefaultEngines are the engine versions this release was checked against:
 // every flag llmgw builds for them is one they take.
 //
-// vLLM is installed on every node with an NVIDIA GPU it can run on and
-// skipped elsewhere. SGLang is not here: it is opt-in, by naming a version
-// under engines.
+// vLLM and FreeToken are installed on every node with an NVIDIA GPU they
+// can run on and skipped elsewhere — FreeToken on cards of 8.0 and newer
+// with the CUDA 13 toolkit — so a node that can serve a model too large for
+// its cards has the engine for it before anyone asks. SGLang is not here:
+// it is opt-in, by naming a version under engines.
+//
+// FreeToken is followed release by release: a new one on PyPI is moved to
+// here once ft serve has been checked to take the flags llmgw gives it.
 var DefaultEngines = map[string]string{
-	"llamacpp": "b10991",
-	"vllm":     "0.30.0",
+	"llamacpp":  "b10991",
+	"vllm":      "0.30.0",
+	"freetoken": "0.2.1",
 }

 // EngineOverride gives the nodes it picks different engines. Every selector
@@ -182,6 +182,7 @@ func (a *API) Register(mux Mux) {
 	a.registerPresets(mux)
 	mux.HandleFunc("GET /api/v1/models", a.listModels)
 	mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
+	a.registerRouting(mux)
 	// Listing is a read and deleting is a separate verb, so a mistyped
 	// command cannot remove anything.
 	mux.HandleFunc("GET /api/v1/weights/orphans", a.listOrphans)
@@ -608,6 +609,9 @@ type FetchView struct {
 	// invented percentage that stops moving is worse than a byte count that
 	// keeps rising.
 	Percent *int `json:"percent,omitempty"`
+	// Phase is empty while the bytes arrive, "checking" while a set's files
+	// are checked against the mirror's checksums.
+	Phase string `json:"phase,omitempty"`
 }

 // ExplainView is why a model could not be placed.
@@ -922,7 +926,7 @@ func (a *API) Replicas(model string) []ReplicaView {
 			}
 			if f := s.Fetch; f != nil {
 				sv.Fetch = &FetchView{
-					Source: f.Source, BytesDone: f.BytesDone, BytesTotal: f.BytesTotal,
+					Source: f.Source, BytesDone: f.BytesDone, BytesTotal: f.BytesTotal, Phase: f.Phase,
 				}
 				if p, ok := f.Percent(); ok {
 					sv.Fetch.Percent = &p
@@ -6,6 +6,7 @@ import (
 	llmgw "gitea.mynodes.xyz/ilker/llmgw"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrics"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
@@ -194,6 +195,15 @@ var Routes = []Route{
 	{Method: "GET", Path: "/api/v1/replicas", Summary: "Every replica, with its shards",
 		Query:    []Param{{Name: "model", Description: "Only this model's replicas."}},
 		Response: []ReplicaView{}},
+	{Method: "GET", Path: "/api/v1/routing", Summary: "The Ready replicas a standby routes to",
+		Description: "Read by a standby control plane when router.active_active is on: each replica's address, " +
+			"model, profile, pool and weight, without its key, and which catalog is in force.",
+		Response: control.Routing{}},
+	{Method: "POST", Path: "/api/v1/routing/activity", Summary: "A standby's router reports what it used",
+		Description: "When it last sent each replica a request, and the on-demand models it was asked for, " +
+			"so the leader does not evict a replica busy behind a standby. Answered with the routing, " +
+			"so one request a tick does both.",
+		Request: control.Activity{}, Response: control.Routing{}},
 	{Method: "GET", Path: "/api/v1/explain-placement", Summary: "Why a model could not be placed",
 		Query:    []Param{{Name: "model", Required: true, Description: "The model's name or an alias."}},
 		Response: ExplainView{}},
@@ -221,7 +231,7 @@ var Routes = []Route{
 	{Method: "POST", Path: "/api/v1/models/{name}/update", Summary: "Move a model to its repository's newer weights",
 		Description: "With a quality suite, or the model's standing gate, the new weights are asked it first as a hidden canary.",
 		Request:     modelserve.Change{}, OptionalBody: true, Response: modelserve.Job{}, Status: http.StatusAccepted},
-	{Method: "POST", Path: "/api/v1/models/{name}/context", Summary: "Change a served model's context: the catalog, its engines' arguments and memory, then its replicas restarted",
+	{Method: "POST", Path: "/api/v1/models/{name}/context", Summary: "Change a served model's context and longest answer: the catalog, its engines' arguments and memory, then its replicas restarted; max_output alone changes only the catalog and restarts nothing",
 		Request: modelserve.ContextRequest{}, Response: modelserve.ContextChange{}},
 	{Method: "POST", Path: "/api/v1/models/{name}/gpus", Summary: "Spread a served model over more cards or fewer: its profile, checked against its KV heads, its cards' memory and the nodes, then its replicas restarted",
 		Description: "Tensor parallel for vLLM, SGLang and TensorRT-LLM, a layer split for llama.cpp, on one node. With no room beside the running replicas, in_place stops each first.",
@@ -0,0 +1,27 @@
+package adminapi
+
+import (
+	"encoding/json"
+	"io"
+	"net/http"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
+)
+
+// registerRouting is what a standby answering requests itself reads from the
+// leader, and reports back to it. Reached by the standby through the agent
+// port with its control-plane certificate, like the rest of the forwarded API.
+func (a *API) registerRouting(mux Mux) {
+	mux.HandleFunc("GET /api/v1/routing", func(w http.ResponseWriter, _ *http.Request) {
+		writeJSON(w, http.StatusOK, a.ctl.Routing())
+	})
+	mux.HandleFunc("POST /api/v1/routing/activity", func(w http.ResponseWriter, r *http.Request) {
+		var act control.Activity
+		if err := json.NewDecoder(io.LimitReader(r.Body, 8<<20)).Decode(&act); err != nil {
+			writeError(w, http.StatusBadRequest, err.Error())
+			return
+		}
+		a.ctl.RecordActivity(act)
+		writeJSON(w, http.StatusOK, a.ctl.RoutingFor(act.From))
+	})
+}
@@ -35,7 +35,7 @@ type StatusSink interface {
 	// OnFetchProgress is separate because it is not a transition: the shard is
 	// still starting, and treating each report as a state change would reset
 	// the timers that decide when a slow start has become a failed one.
-	OnFetchProgress(replicaID, shardID, source string, done, total int64)
+	OnFetchProgress(replicaID, shardID, source, phase string, done, total int64)
 	// OnShardCompute records where a shard's engine said it runs the model.
 	OnShardCompute(replicaID, shardID string, c reconciler.Compute)
 	Adopt(nodeID string, shards []reconciler.AdoptedShard) (unclaimed []reconciler.AdoptedShard)
@@ -461,7 +461,7 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 		// download that takes an hour would keep a start timer at zero.
 		if f := m.Shard.GetFetch(); f != nil {
 			s.status.OnFetchProgress(m.Shard.GetReplicaId(), m.Shard.GetShardId(),
-				f.GetSource(), f.GetBytesDone(), f.GetBytesTotal())
+				f.GetSource(), f.GetPhase(), f.GetBytesDone(), f.GetBytesTotal())
 			return
 		}
 		if c := m.Shard.GetCompute(); c != nil {
@@ -772,12 +772,25 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
 			UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
 			MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
 			SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
-			Wear: wearFrom(g),
+			Wear: wearFrom(g), Profile: profileFrom(g),
 		})
 	}
 	return out
 }

+// profileFrom is a card's profiling counters, or nil from an agent that
+// sent none.
+func profileFrom(g *llmgwv1.GPUSensor) *inventory.GPUProfile {
+	if !g.GetHasProfile() {
+		return nil
+	}
+	return &inventory.GPUProfile{
+		SMActive: g.GetSmActivePercent(), SMOccupancy: g.GetSmOccupancyPercent(),
+		TensorActive: g.GetTensorActivePercent(), DRAMActive: g.GetDramActivePercent(),
+		Source: g.GetProfileSource(),
+	}
+}
+
 func (s *Service) attach(ctx context.Context, nodeID string) *conn {
 	c := &conn{
 		nodeID: nodeID,
@@ -12,6 +12,7 @@ import (
 	"os"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
@@ -95,6 +96,15 @@ type Controller struct {
 	mu   sync.Mutex
 	last string // fingerprint of the last published list

+	// feed is where a standby's router learns the leader's Ready replicas,
+	// when standbys answer requests themselves; nil when they do not.
+	feed ReplicaFeed
+	// catRev changes with every catalog put in force here, so a standby
+	// following this one knows to read the catalog again.
+	catRev atomic.Uint64
+	// rates are the standbys' counts against each limit, while this leads.
+	rates sharedRates
+
 	stop chan struct{}
 	done chan struct{}
 	once sync.Once
@@ -193,6 +203,7 @@ type LoadSource interface {
 // killed.
 func (c *Controller) ReplaceCatalog(ctx context.Context, next *catalog.Catalog) {
 	c.cat.Store(next)
+	c.catRev.Add(1)
 	if cs, ok := c.sink.(CatalogSink); ok {
 		cs.SetCatalog(next)
 	}
@@ -306,6 +317,7 @@ func (c *Controller) Start(ctx context.Context) {
 				// Before publishing, so a replica's idleness is as fresh as
 				// the endpoint list it travels beside.
 				c.collectUsage()
+				c.shareRates()
 				c.Publish(ctx)
 			}
 		}
@@ -345,7 +357,7 @@ func (c *Controller) Publish(ctx context.Context) {

 // endpoints converts Ready replicas into the router's view.
 func (c *Controller) endpoints() []endpoint.Info {
-	ready := c.rec.ReadyReplicas()
+	ready := c.routable()
 	out := make([]endpoint.Info, 0, len(ready))

 	for _, rep := range ready {
@@ -96,7 +96,7 @@ func EngineBuilds(engine string) []string {
 	switch engine {
 	case "vllm":
 		return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
-	case "sglang":
+	case "sglang", "freetoken":
 		return []string{"auto", "none"}
 	}
 	return nil
@@ -84,6 +84,10 @@ type Collector struct {
 	gpuXID         *prometheus.Desc
 	gpuLastXID     *prometheus.Desc
 	gpuMemUtil     *prometheus.Desc
+	gpuSMActive    *prometheus.Desc
+	gpuSMOccupancy *prometheus.Desc
+	gpuTensor      *prometheus.Desc
+	gpuDRAMActive  *prometheus.Desc
 	gpuSMClock     *prometheus.Desc
 	gpuTemp        *prometheus.Desc
 	gpuMemTemp     *prometheus.Desc
@@ -158,9 +162,22 @@ func New(src Source) *Collector {
 		gpuMemoryUsed: prometheus.NewDesc("llmgw_gpu_memory_used_bytes",
 			"A GPU's framebuffer in use, including processes llmgw did not start.", gpu, nil),
 		gpuUtil: prometheus.NewDesc("llmgw_gpu_utilization_percent",
-			"How busy a GPU was over the last sample period. Absent when the driver does not "+
-				"report it, which is not the same as idle. Memory used says a model is loaded; "+
-				"this says whether it is doing anything.", gpu, nil),
+			"The share of the last sample period in which any kernel ran on a GPU (nvidia-smi's "+
+				"utilization.gpu). Not how much of the chip: a kernel on one SM reads 100%. Absent "+
+				"when the driver does not report it, which is not the same as idle. Memory used says "+
+				"a model is loaded; this says whether it is doing anything, and "+
+				"llmgw_gpu_sm_active_percent how much.", gpu, nil),
+		gpuSMActive: prometheus.NewDesc("llmgw_gpu_sm_active_percent",
+			"The share of time a GPU's SMs had work, averaged over all of them: how much of the chip "+
+				"the busy figure is using. From NVML's GPM (Hopper and later) or a dcgm-exporter on the node; "+
+				"absent elsewhere.", gpu, nil),
+		gpuSMOccupancy: prometheus.NewDesc("llmgw_gpu_sm_occupancy_percent",
+			"How full a GPU's active SMs were with warps, of the most they can hold.", gpu, nil),
+		gpuTensor: prometheus.NewDesc("llmgw_gpu_tensor_active_percent",
+			"The share of time a GPU's tensor cores were working. Low in decode at batch one is normal; "+
+				"prefill and batches are where it should rise.", gpu, nil),
+		gpuDRAMActive: prometheus.NewDesc("llmgw_gpu_dram_active_percent",
+			"The share of time a GPU's memory interface was moving data, from its profiling counters.", gpu, nil),
 		gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
 			"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
 		gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
@@ -310,7 +327,8 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
 		c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
 		c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
 		c.gpuNVLJoined, c.imexUp,
-		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
+		c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock,
+		c.gpuSMActive, c.gpuSMOccupancy, c.gpuTensor, c.gpuDRAMActive, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
 		c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
 		c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
 		c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
@@ -653,6 +671,19 @@ func (c *Collector) gpuBusy(ch chan<- prometheus.Metric, s inventory.GPUSensor,
 	if s.SMClockMHz > 0 {
 		ch <- prometheus.MustNewConstMetric(c.gpuSMClock, prometheus.GaugeValue, s.SMClockMHz, labels...)
 	}
+	// The card's profile, here rather than with the sensors because it is
+	// the whole card's: a MIG instance must not be published as having it.
+	if p := s.Profile; p != nil {
+		for _, v := range []struct {
+			d   *prometheus.Desc
+			val float64
+		}{{c.gpuSMActive, p.SMActive}, {c.gpuSMOccupancy, p.SMOccupancy}, {c.gpuTensor, p.TensorActive}, {c.gpuDRAMActive, p.DRAMActive}} {
+			// Zero is an idle chip; only -1, not reported, is left out.
+			if v.val >= 0 {
+				ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, labels...)
+			}
+		}
+	}
 }

 func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
@@ -16,7 +16,8 @@ func TestAMIGInstanceHasItsOwnMemoryItsCardsHeatAndNoUtilisation(t *testing.T) {
 		MIG: &discovery.MIGDevice{ParentIndex: 0, ParentMemoryTotalMB: 81920, Device: 1, Profile: "3g.40gb"}}
 	n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{inst},
 		Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 61, PowerW: 250,
-			UtilPercent: 0, MemoryUsedMB: 60000, MemoryTotalMB: 81920}}}}
+			UtilPercent: 0, MemoryUsedMB: 60000, MemoryTotalMB: 81920,
+			Profile: &inventory.GPUProfile{SMActive: 50, SMOccupancy: 30, TensorActive: 10, DRAMActive: 20}}}}}
 	out := gather(t, fakeSource{nodes: []inventory.Node{n}})
 	labels := "class=a100-80g-mig-3g.40gb gpu=1001 node=g1"
 	for _, want := range []string{
@@ -33,4 +34,8 @@ func TestAMIGInstanceHasItsOwnMemoryItsCardsHeatAndNoUtilisation(t *testing.T) {
 	if strings.Contains(out, "llmgw_gpu_utilization_percent "+labels) {
 		t.Errorf("a utilisation was published for a MIG instance:\n%s", out)
 	}
+	// The card's profile is the whole card's, not this instance's.
+	if strings.Contains(out, "llmgw_gpu_sm_active_percent "+labels) {
+		t.Errorf("the card's SM activity was published for a MIG instance:\n%s", out)
+	}
 }
@@ -127,3 +127,27 @@ func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
 		t.Errorf("a count the card does not keep was published:\n%s", out)
 	}
 }
+
+func TestHowMuchOfTheChipIsPublishedBesideHowBusy(t *testing.T) {
+	// The Quadro RTX 6000 in decode: 92% busy, its SMs at work 69% of the
+	// time. The DRAM figure was not read, and is left out rather than zero.
+	n := inventory.Node{ID: "g1", State: inventory.Ready,
+		GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
+		Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, UtilPercent: 92, MemoryTotalMB: 81559,
+			Profile: &inventory.GPUProfile{SMActive: 69, SMOccupancy: 45, TensorActive: 0, DRAMActive: -1, Source: "gpm"}}}}}
+	out := gather(t, fakeSource{nodes: []inventory.Node{n}})
+	for _, want := range []string{
+		"llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 92\n",
+		"llmgw_gpu_sm_active_percent class=h100-80g gpu=0 node=g1 -> 69\n",
+		"llmgw_gpu_sm_occupancy_percent class=h100-80g gpu=0 node=g1 -> 45\n",
+		// Idle tensor cores are a reading.
+		"llmgw_gpu_tensor_active_percent class=h100-80g gpu=0 node=g1 -> 0\n",
+	} {
+		if !strings.Contains(out, want) {
+			t.Errorf("missing %s in:\n%s", want, out)
+		}
+	}
+	if strings.Contains(out, "llmgw_gpu_dram_active_percent") {
+		t.Errorf("a figure not read was published:\n%s", out)
+	}
+}
@@ -340,6 +340,36 @@ type GPUSensor struct {
 	// Wear is what wears the card out, read every minute; nil from an agent
 	// that does not read it.
 	Wear *GPUWear `json:"wear,omitempty"`
+	// Profile is how much of the chip the work uses, which UtilPercent does
+	// not say; nil where nothing on the node can read it.
+	Profile *GPUProfile `json:"profile,omitempty"`
+}
+
+// GPUProfile is a card's profiling counters, in percent; -1 is not reported.
+// UtilPercent is the share of time any kernel ran, and a kernel on one SM of
+// a hundred counts in full; these say how much of the chip it ran on.
+type GPUProfile struct {
+	// SMActive is the share of time the SMs had work, averaged over all of
+	// them; SMOccupancy how full those SMs were with warps.
+	SMActive    float64 `json:"sm_active"`
+	SMOccupancy float64 `json:"sm_occupancy"`
+	// TensorActive is the tensor cores, DRAMActive the memory interface.
+	TensorActive float64 `json:"tensor_active"`
+	DRAMActive   float64 `json:"dram_active"`
+	// Source is what read them: "gpm" (NVML, Hopper and later) or "dcgm".
+	Source string `json:"source,omitempty"`
+}
+
+// PowerLoad is the card's draw as a share of its power limit, from 0 to 1,
+// or -1 without both figures. Unlike UtilPercent it rises with how much of
+// the chip is working, not only with whether anything is: a card reading
+// 92% busy drew 239 W of 260 W in decode and 112 W when the busy figure was
+// a kernel or two between requests.
+func (g GPUSensor) PowerLoad() float64 {
+	if g.PowerW <= 0 || g.PowerLimitW <= 0 {
+		return -1
+	}
+	return min(g.PowerW/g.PowerLimitW, 1)
 }

 // GPUWear is a card's wear and failures. Counts of -1 are not reported for
@@ -21,9 +21,13 @@ import (

 // ContextRequest is the context to change to: tokens one request may use,
 // and for llama.cpp how many at once (its pool is the two multiplied).
+// MaxOutput is the longest answer, which may be changed with it or alone:
+// Tokens 0 with a MaxOutput changes only that, in the catalog, which the
+// router reads it from — nothing is started again.
 type ContextRequest struct {
-	Tokens int `json:"tokens"`
-	Slots  int `json:"slots,omitempty"`
+	Tokens    int `json:"tokens,omitempty"`
+	Slots     int `json:"slots,omitempty"`
+	MaxOutput int `json:"max_output,omitempty"`
 }

 // ContextProfile is what one profile needs before and after.
@@ -37,13 +41,17 @@ type ContextProfile struct {

 // ContextChange is what was changed, and the replicas started again on it.
 type ContextChange struct {
-	Model     string           `json:"model"`
-	Before    int              `json:"before"`
-	After     int              `json:"after"`
-	MaxOutput int              `json:"max_output"`
-	Trained   int              `json:"trained,omitempty"`
-	Profiles  []ContextProfile `json:"profiles"`
-	Restarted int              `json:"restarted"`
+	Model     string `json:"model"`
+	Before    int    `json:"before"`
+	After     int    `json:"after"`
+	MaxOutput int    `json:"max_output"`
+	// OutputOnly is a change of the longest answer alone, MaxOutputBefore
+	// what it was (0 no limit); nothing was started again.
+	OutputOnly      bool             `json:"output_only,omitempty"`
+	MaxOutputBefore int              `json:"max_output_before,omitempty"`
+	Trained         int              `json:"trained,omitempty"`
+	Profiles        []ContextProfile `json:"profiles"`
+	Restarted       int              `json:"restarted"`
 	// Note is what could not be checked, said once.
 	Note string `json:"note,omitempty"`
 }
@@ -75,8 +83,14 @@ func (s *Server) SetContext(ctx context.Context, name string, req ContextRequest
 		return ContextChange{}, fmt.Errorf("no model %q in the catalog", name)
 	case s.deps.Models.Declared(name):
 		return ContextChange{}, fmt.Errorf("model %q is declared in the cluster's settings: change its context there (llmgw ctl config edit)", name)
+	case req.MaxOutput < 0:
+		return ContextChange{}, errors.New("the longest answer is a number of tokens")
+	case req.Tokens == 0 && req.MaxOutput > 0:
+		return s.setMaxOutput(ctx, m, req.MaxOutput, by)
 	case req.Tokens < 256:
 		return ContextChange{}, errors.New("a context of at least 256 tokens")
+	case req.MaxOutput > req.Tokens:
+		return ContextChange{}, fmt.Errorf("an answer of %d tokens does not fit a context of %d", req.MaxOutput, req.Tokens)
 	case req.Slots < 0 || req.Slots > 64:
 		return ContextChange{}, errors.New("slots are 1 to 64")
 	}
@@ -100,6 +114,9 @@ func (s *Server) SetContext(ctx context.Context, name string, req ContextRequest
 		out.MaxOutput = maxOutput(req.Tokens)
 	}
 	out.MaxOutput = min(out.MaxOutput, req.Tokens)
+	if req.MaxOutput > 0 {
+		out.MaxOutput = req.MaxOutput
+	}
 	if _, err := s.deps.Models.SetContext(ctx, name, req.Tokens, out.MaxOutput, profiles, by); err != nil {
 		return ContextChange{}, err
 	}
@@ -121,6 +138,21 @@ func (s *Server) SetContext(ctx context.Context, name string, req ContextRequest
 	return out, nil
 }

+// setMaxOutput changes only the longest answer: the catalog's, which the
+// router holds a request's max_tokens to. No engine reads it, so no replica
+// is started again.
+func (s *Server) setMaxOutput(ctx context.Context, m catalog.Model, maxOut int, by string) (ContextChange, error) {
+	if m.ContextLength > 0 && maxOut > m.ContextLength {
+		return ContextChange{}, fmt.Errorf("an answer of %d tokens does not fit %s's context of %d: change the context with it",
+			maxOut, m.Name, m.ContextLength)
+	}
+	if _, err := s.deps.Models.SetContext(ctx, m.Name, m.ContextLength, maxOut, m.Profiles, by); err != nil {
+		return ContextChange{}, err
+	}
+	return ContextChange{Model: m.Name, Before: m.ContextLength, After: m.ContextLength, MaxOutput: maxOut,
+		OutputOnly: true, MaxOutputBefore: m.MaxOutput}, nil
+}
+
 // shapeOf reads what sizes a model's context: a GGUF file's header from the
 // mirror, else the original weights' config from their repository.
 func (s *Server) shapeOf(ctx context.Context, m catalog.Model) (shape, string) {
@@ -174,6 +206,20 @@ func resizeProfile(p catalog.Profile, req ContextRequest, sh shape) (catalog.Pro
 	case "sglang":
 		pool = req.Tokens
 		p.Args = setArg(p.Args, "--context-length", strconv.Itoa(pool))
+	case "strata":
+		// Strata's setup takes the context, and writes the engine's start
+		// script with it at every start; it sizes its KV cache and expert
+		// cache to the card itself, so there is no card check to make here.
+		p.Args = setArg(p.Args, "--context", strconv.Itoa(req.Tokens))
+		change.Args = p.Args
+		return p, change, nil
+	case "freetoken":
+		// Its experts live in RAM and stream to the card, so the weights
+		// are not what has to fit there, and the card check below would
+		// refuse what it serves.
+		p.Args = setArg(p.Args, "--max-seq-len-override", strconv.Itoa(req.Tokens))
+		change.Args = p.Args
+		return p, change, nil
 	default:
 		return p, change, fmt.Errorf("a %s profile (%s) does not take a context this way", p.Engine, p.Name)
 	}
@@ -88,3 +88,57 @@ func TestEngineArgumentsAreSetInEitherSpelling(t *testing.T) {
 		t.Error("a comfyui profile took a context")
 	}
 }
+
+func TestTheLongestAnswerAloneChangesOnlyTheCatalogAndStartsNothing(t *testing.T) {
+	s, _, models, _, fleet := updSetup(t)
+	m := models.models["qwen"]
+	m.ContextLength, m.MaxOutput = 4096, 1024
+	m.Profiles[0].Args = []string{"--ctx-size", "4096"}
+	models.models["qwen"] = m
+	before := fleet.Replicas()
+
+	got, err := s.SetContext(context.Background(), "qwen", ContextRequest{MaxOutput: 3000}, "admin")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !got.OutputOnly || got.MaxOutput != 3000 || got.MaxOutputBefore != 1024 || got.Restarted != 0 {
+		t.Fatalf("%+v", got)
+	}
+	after := models.models["qwen"]
+	if after.ContextLength != 4096 || after.MaxOutput != 3000 || !slices.Equal(after.Profiles[0].Args, []string{"--ctx-size", "4096"}) {
+		t.Errorf("catalog %d/%d, args %v", after.ContextLength, after.MaxOutput, after.Profiles[0].Args)
+	}
+	for i, r := range fleet.Replicas() {
+		if r.ID != before[i].ID {
+			t.Errorf("%s was started again for a limit no engine reads", before[i].ID)
+		}
+	}
+	// An answer longer than the context is refused, alone or with one.
+	if _, err := s.SetContext(context.Background(), "qwen", ContextRequest{MaxOutput: 5000}, "admin"); err == nil {
+		t.Error("an answer longer than the context was accepted")
+	}
+	if _, err := s.SetContext(context.Background(), "qwen", ContextRequest{Tokens: 8192, MaxOutput: 9000}, "admin"); err == nil {
+		t.Error("an answer longer than the new context was accepted")
+	}
+	// With a context, the one asked for wins over the one that follows it.
+	got, err = s.SetContext(context.Background(), "qwen", ContextRequest{Tokens: 8192, MaxOutput: 6000}, "admin")
+	if err != nil || got.MaxOutput != 6000 || models.models["qwen"].MaxOutput != 6000 {
+		t.Errorf("%+v %v", got, err)
+	}
+}
+
+func TestStrataTakesItsContextInItsSetupArguments(t *testing.T) {
+	p := catalog.Profile{Name: "one-card", Engine: "strata", GPUClass: "6000-24g",
+		Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}
+	got, change, err := resizeProfile(p, ContextRequest{Tokens: 32768}, shape{kvPerToken: 1 << 20, weights: 40 << 30})
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Not refused for weights larger than the card: its experts live in RAM.
+	if !slices.Equal(got.Args, []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768"}) || !slices.Equal(change.Args, got.Args) {
+		t.Errorf("args %v", got.Args)
+	}
+	if n := contextOf(catalog.Model{ContextLength: 4096}, got); n != 32768 {
+		t.Errorf("contextOf %d", n)
+	}
+}
@@ -208,6 +208,14 @@ func contextOf(m catalog.Model, p catalog.Profile) int {
 		if n := argInt(p.Args, "--context-length"); n > 0 {
 			return n
 		}
+	case "strata":
+		if n := argInt(p.Args, "--context"); n > 0 {
+			return n
+		}
+	case "freetoken":
+		if n := argInt(p.Args, "--max-seq-len-override"); n > 0 {
+			return n
+		}
 	}
 	return m.ContextLength
 }
@@ -23,7 +23,7 @@ func TestProgressIsVisibleWithoutBeingATransition(t *testing.T) {
 	}
 	rep, shard := reps[0].ID, reps[0].Shards[0].ID

-	h.r.OnFetchProgress(rep, shard, "hf:org/repo/model.gguf", 3<<30, 12<<30)
+	h.r.OnFetchProgress(rep, shard, "hf:org/repo/model.gguf", "", 3<<30, 12<<30)

 	got := h.r.Replicas()[0].Shards[0]
 	if got.Fetch == nil {
@@ -54,7 +54,7 @@ func TestATransitionClearsTheDownload(t *testing.T) {
 	reps := h.r.Replicas()
 	rep, shard := reps[0].ID, reps[0].Shards[0].ID

-	h.r.OnFetchProgress(rep, shard, "hf:org/repo/model.gguf", 1, 2)
+	h.r.OnFetchProgress(rep, shard, "hf:org/repo/model.gguf", "", 1, 2)
 	h.r.OnShardStatus(rep, shard, supervisor.Ready, "")

 	if got := h.r.Replicas()[0].Shards[0]; got.Fetch != nil {
@@ -66,5 +66,5 @@ func TestProgressForAReplicaThatIsGoneIsIgnored(t *testing.T) {
 	// A download that finishes after its replica was torn down should not
 	// resurrect anything or panic.
 	h := newHarness(t, testModel(1), gpuNode("n1", 8))
-	h.r.OnFetchProgress("no-such-replica", "no-such-shard", "x", 1, 2)
+	h.r.OnFetchProgress("no-such-replica", "no-such-shard", "x", "", 1, 2)
 }
@@ -1320,7 +1320,7 @@ func (r *Reconciler) stopReplica(ctx context.Context, id, why string) {
 // Separate from OnShardStatus because it is not a transition: the shard is
 // still starting, and treating each progress report as a state change would
 // reset the timers that decide when a slow start becomes a failed one.
-func (r *Reconciler) OnFetchProgress(replicaID, shardID, source string, done, total int64) {
+func (r *Reconciler) OnFetchProgress(replicaID, shardID, source, phase string, done, total int64) {
 	r.mu.Lock()
 	defer r.mu.Unlock()

@@ -1332,11 +1332,13 @@ func (r *Reconciler) OnFetchProgress(replicaID, shardID, source string, done, to
 		if rep.Shards[i].ID != shardID {
 			continue
 		}
-		if prev := rep.Shards[i].Fetch; prev == nil || done > prev.BytesDone {
+		// Moving: more bytes, or on to the next phase, whose count starts
+		// again from nothing — the files checked after they arrived.
+		if prev := rep.Shards[i].Fetch; prev == nil || done > prev.BytesDone || phase != prev.Phase {
 			rep.fetchedAt = r.now()
 		}
 		rep.Shards[i].Fetch = &FetchProgress{
-			Source: source, BytesDone: done, BytesTotal: total,
+			Source: source, BytesDone: done, BytesTotal: total, Phase: phase,
 		}
 	}
 }
@@ -101,6 +101,9 @@ type FetchProgress struct {
 	// invented, and an invented one that stops moving is worse than a byte
 	// count that keeps rising.
 	BytesTotal int64
+	// Phase is empty while the bytes arrive, "checking" while they are
+	// checked against the mirror's checksums.
+	Phase string
 }

 // Percent returns progress as a percentage, and false when the total is
@@ -422,7 +422,7 @@ func TestAReplicaStillDownloadingIsNotFailed(t *testing.T) {
 	// Forty minutes of download, moving all the while.
 	for i := int64(1); i <= 8; i++ {
 		h.advance(5 * time.Minute)
-		h.r.OnFetchProgress(rep.ID, shard, "mirror", i<<35, 400<<30)
+		h.r.OnFetchProgress(rep.ID, shard, "mirror", "", i<<35, 400<<30)
 		h.r.expireStuckReplicas()
 	}
 	if got := h.r.Replicas()[0].State; got == Failed {
@@ -430,7 +430,7 @@ func TestAReplicaStillDownloadingIsNotFailed(t *testing.T) {
 	}
 	// The same figure again is not progress: stalled, it fails a timeout on.
 	h.advance(10 * time.Minute)
-	h.r.OnFetchProgress(rep.ID, shard, "mirror", 8<<35, 400<<30)
+	h.r.OnFetchProgress(rep.ID, shard, "mirror", "", 8<<35, 400<<30)
 	h.advance(11 * time.Minute)
 	h.r.expireStuckReplicas()
 	if got := h.r.Replicas()[0].State; got != Failed {
@@ -0,0 +1,186 @@
+package control
+
+import (
+	"strconv"
+	"sync"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+)
+
+// RoutedReplica is a Ready replica as a router needs it: where it is, what it
+// serves and how much it may take. Not its key — every control plane derives
+// that from the replica's id and the cluster's secret, so none travels.
+type RoutedReplica struct {
+	ID       string  `json:"id"`
+	Model    string  `json:"model"`
+	Profile  string  `json:"profile"`
+	Pool     string  `json:"pool"`
+	Endpoint string  `json:"endpoint"`
+	Weight   float64 `json:"weight"`
+}
+
+// Routing is what a standby that answers requests itself reads from the
+// leader: the replicas it may send to, and which catalog they belong to.
+type Routing struct {
+	Replicas []RoutedReplica `json:"replicas"`
+	// Catalog changes whenever the leader puts a catalog in force. Only
+	// compared with the last one a standby saw, never with its own: a model
+	// added, scaled or given a canary is in the store, and the standby reads
+	// it again from there.
+	Catalog string `json:"catalog"`
+	// Rates are what every other control plane's router has counted against
+	// each limit, the leader's among them: added to the asker's own, they
+	// make a key's limits the cluster's.
+	Rates *accounting.Rates `json:"rates,omitempty"`
+}
+
+// Activity is what a standby's router reports back: when it last sent each
+// replica a request, and the on-demand models it was asked for. The leader
+// evicts and loads by these, and without them a replica busy behind a
+// standby looks idle to the one machine that decides.
+type Activity struct {
+	// From names the standby, so its counts replace its own last ones and
+	// are not given back to it.
+	From     string               `json:"from,omitempty"`
+	LastUsed map[string]time.Time `json:"last_used,omitempty"`
+	Want     []string             `json:"want,omitempty"`
+	// Rates are its router's own counts against each limit.
+	Rates *accounting.Rates `json:"rates,omitempty"`
+}
+
+// RateSharer is a router whose limits can be counted with other routers'.
+type RateSharer interface {
+	Rates() accounting.Rates
+	SetOtherRates(accounting.Rates)
+}
+
+// ratesFresh is how long a standby's counts are believed after it last
+// reported: a standby gone quiet stops counting against anyone's limits
+// shortly after it stops taking requests.
+const ratesFresh = 10 * time.Second
+
+// sharedRates are the standbys' last counts, by name.
+type sharedRates struct {
+	mu   sync.Mutex
+	by   map[string]rateReport
+	seen bool // whether any standby has ever reported
+}
+
+type rateReport struct {
+	rates accounting.Rates
+	at    time.Time
+}
+
+// put keeps a standby's latest counts.
+func (s *sharedRates) put(from string, r accounting.Rates, now time.Time) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.by == nil {
+		s.by = map[string]rateReport{}
+	}
+	s.by[from] = rateReport{rates: r, at: now}
+	s.seen = true
+}
+
+// sum adds every fresh standby's counts but except's; stale ones are
+// forgotten. ok is false when no standby has ever reported, and there is
+// nothing to put in force.
+func (s *sharedRates) sum(except string, now time.Time) (accounting.Rates, bool) {
+	out := accounting.NewRates(now)
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	for name, r := range s.by {
+		if now.Sub(r.at) > ratesFresh {
+			delete(s.by, name)
+			continue
+		}
+		if name != except {
+			out.Add(r.rates)
+		}
+	}
+	return out, s.seen
+}
+
+// ReplicaFeed is a standby's copy of the leader's Ready replicas. ok is false
+// when it has none worth using — standbys do not answer requests, or the
+// leader has not been heard from lately — and the standby routes nothing.
+type ReplicaFeed interface {
+	Ready() (reps []RoutedReplica, ok bool)
+}
+
+// SetReplicaFeed lets a standby route to the leader's replicas.
+func (c *Controller) SetReplicaFeed(f ReplicaFeed) { c.feed = f }
+
+// Routing is this control plane's Ready replicas, for a standby to route to.
+func (c *Controller) Routing() Routing { return c.RoutingFor("") }
+
+// RoutingFor is Routing for the standby named from, with what every other
+// router has counted against each limit.
+func (c *Controller) RoutingFor(from string) Routing {
+	out := Routing{Replicas: c.ready(), Catalog: c.CatalogRev()}
+	if rs, ok := c.sink.(RateSharer); ok {
+		others, _ := c.rates.sum(from, time.Now())
+		others.Add(rs.Rates())
+		out.Rates = &others
+	}
+	return out
+}
+
+// ready is the reconciler's Ready replicas, as a router needs them.
+func (c *Controller) ready() []RoutedReplica {
+	reps := c.rec.ReadyReplicas()
+	out := make([]RoutedReplica, 0, len(reps))
+	for _, r := range reps {
+		out = append(out, RoutedReplica{
+			ID: r.ID, Model: r.Model, Profile: r.Profile, Pool: r.Pool, Endpoint: r.Endpoint, Weight: r.Weight,
+		})
+	}
+	return out
+}
+
+// shareRates puts the standbys' counts in force on this control plane's own
+// router, while it leads. Once a tick, beside publishing.
+func (c *Controller) shareRates() {
+	rs, ok := c.sink.(RateSharer)
+	if !ok || (c.lead != nil && !c.lead.IsLeader()) {
+		return
+	}
+	if others, seen := c.rates.sum("", time.Now()); seen {
+		rs.SetOtherRates(others)
+	}
+}
+
+// CatalogRev names the catalog in force here. The process's owner id with a
+// counter: a new leader, or a restarted one, is a different catalog as far as
+// any standby knows.
+func (c *Controller) CatalogRev() string {
+	owner := ""
+	if c.lead != nil {
+		owner = c.lead.Owner()
+	}
+	return owner + "/" + strconv.FormatUint(c.catRev.Load(), 10)
+}
+
+// RecordActivity takes in what a standby's router did since its last report.
+func (c *Controller) RecordActivity(a Activity) {
+	if a.From != "" && a.Rates != nil {
+		c.rates.put(a.From, *a.Rates, time.Now())
+	}
+	c.rec.RecordUsage(a.LastUsed)
+	for _, m := range a.Want {
+		c.rec.Want(m)
+	}
+}
+
+// routable is what this process's router may send to: the replicas the
+// reconciler holds Ready, or, on a standby following the leader, the leader's.
+func (c *Controller) routable() []RoutedReplica {
+	if c.feed != nil && c.lead != nil && !c.lead.IsLeader() {
+		if reps, ok := c.feed.Ready(); ok {
+			return reps
+		}
+		return nil
+	}
+	return c.ready()
+}
@@ -0,0 +1,143 @@
+package control
+
+import (
+	"log/slog"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+type fixedFeed struct {
+	reps []RoutedReplica
+	ok   bool
+}
+
+func (f fixedFeed) Ready() ([]RoutedReplica, bool) { return f.reps, f.ok }
+
+func routingController(t *testing.T, sink *recordSink) *Controller {
+	t.Helper()
+	cat, err := catalog.New([]catalog.Model{{Name: "llama", Source: "/models/llama", Aliases: []string{"ll"},
+		Scaling:  catalog.Scaling{Min: 1, Max: 1},
+		Profiles: []catalog.Profile{{Name: "one", GPUClass: "h100-80g", Engine: "vllm", Priority: 3}}}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	cfg := DefaultConfig()
+	cfg.Reconciler.EngineKey = EngineKeys([]byte("secret"))
+	return New(cfg, catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler))
+}
+
+// standby is a leadership that does not lead: one without a store always
+// would.
+func standby() *Leadership {
+	l := NewLeadership(nil, 0, nil)
+	l.leader.Store(false)
+	return l
+}
+
+// A standby following the leader routes to the leader's replicas, each with
+// the key it derives itself and the engine and priority its own catalog
+// gives the profile; a replica of a model its catalog lacks is left out.
+func TestAStandbyRoutesToTheLeadersReplicas(t *testing.T) {
+	sink := &recordSink{}
+	c := routingController(t, sink)
+	c.SetLeadership(standby())
+	c.SetReplicaFeed(fixedFeed{ok: true, reps: []RoutedReplica{
+		{ID: "llama-r0", Model: "llama", Profile: "one", Pool: "a", Endpoint: "http://10.0.0.5:20001/", Weight: 2},
+		{ID: "gone-r0", Model: "gone", Profile: "x", Endpoint: "http://10.0.0.6:20001"},
+	}})
+	c.Publish(t.Context())
+
+	if len(sink.infos) != 1 {
+		t.Fatalf("published %+v", sink.infos)
+	}
+	e := sink.infos[0]
+	if e.ID != "llama-r0" || e.URL != "http://10.0.0.5:20001" || e.Pool != "a" || e.Weight != 2 ||
+		e.Engine != "vllm" || e.Priority != 3 || len(e.Models) != 2 || e.Models[1] != "ll" {
+		t.Errorf("endpoint %+v", e)
+	}
+	if e.APIKey == "" || e.APIKey != EngineKeys([]byte("secret"))("llama-r0") {
+		t.Errorf("the key is not the one the leader gave the engine: %q", e.APIKey)
+	}
+}
+
+// One that has not heard from the leader lately routes to nothing, and the
+// leader never reads a feed: its replicas are its reconciler's.
+func TestAStandbyCutOffRoutesNothingAndTheLeaderIgnoresTheFeed(t *testing.T) {
+	sink := &recordSink{}
+	c := routingController(t, sink)
+	l := standby()
+	c.SetLeadership(l)
+	c.SetReplicaFeed(fixedFeed{ok: false, reps: []RoutedReplica{{ID: "llama-r0", Model: "llama", Profile: "one"}}})
+	if got := c.endpoints(); len(got) != 0 {
+		t.Errorf("a standby that has not heard from the leader routes to %+v", got)
+	}
+
+	l.leader.Store(true)
+	c.SetReplicaFeed(fixedFeed{ok: true, reps: []RoutedReplica{{ID: "llama-r0", Model: "llama", Profile: "one"}}})
+	if got := c.endpoints(); len(got) != 0 {
+		t.Errorf("the leader routed to a feed rather than its own replicas: %+v", got)
+	}
+}
+
+// The catalog's revision moves with every catalog put in force, so a
+// standby knows to read it again.
+func TestTheCatalogRevisionMovesWithTheCatalog(t *testing.T) {
+	c := routingController(t, &recordSink{})
+	before := c.Routing().Catalog
+	c.ReplaceCatalog(t.Context(), c.cat.Load())
+	if after := c.Routing().Catalog; after == before {
+		t.Errorf("the revision stayed %q", after)
+	}
+}
+
+// rateSink is a router whose counts can be shared: its own, and what it was
+// last told the others counted.
+type rateSink struct {
+	recordSink
+	own    accounting.Rates
+	others *accounting.Rates
+}
+
+func (s *rateSink) Rates() accounting.Rates { return s.own }
+func (s *rateSink) SetOtherRates(r accounting.Rates) {
+	s.others = &r
+}
+
+func counts(k string, n int64) accounting.Rates {
+	r := accounting.NewRates(time.Now())
+	r.Requests.Current = map[string]int64{k: n}
+	return r
+}
+
+// Each standby is given what the leader and the other standbys counted, never
+// its own; the leader puts the standbys' counts in force on its own router.
+func TestLimitsAreCountedAcrossTheControlPlanes(t *testing.T) {
+	sink := &rateSink{own: counts("key-1", 5)}
+	cat, _ := catalog.New(nil)
+	c := New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler))
+
+	b, d := counts("key-1", 3), counts("key-1", 7)
+	c.RecordActivity(Activity{From: "ctl-b", Rates: &b})
+	c.RecordActivity(Activity{From: "ctl-d", Rates: &d})
+
+	if got := c.RoutingFor("ctl-b").Rates.Requests.Current["key-1"]; got != 12 {
+		t.Errorf("ctl-b was told %d, want the leader's 5 and ctl-d's 7", got)
+	}
+	c.shareRates()
+	if sink.others == nil || sink.others.Requests.Current["key-1"] != 10 {
+		t.Errorf("the leader's router was told %+v, want the standbys' 10", sink.others)
+	}
+
+	// A standby that stopped reporting stops counting.
+	c.rates.mu.Lock()
+	r := c.rates.by["ctl-d"]
+	r.at = r.at.Add(-ratesFresh - time.Second)
+	c.rates.by["ctl-d"] = r
+	c.rates.mu.Unlock()
+	if got := c.RoutingFor("ctl-b").Rates.Requests.Current["key-1"]; got != 5 {
+		t.Errorf("a quiet standby still counts: %d", got)
+	}
+}
@@ -157,6 +157,9 @@ var Reference = []Block{
 	{Package: "config", Name: "FirewallSettings", Title: "FirewallSettings", Doc: "", Fields: []Field{
 		{Key: "enabled", Type: "bool", Doc: "Enabled keeps llmgw's internal ports — engines, node exporter, the\nagents' stream, the mirror, the store — to the cluster's networks on\nevery machine. False removes llmgw's nftables table everywhere, for a\nsite whose own firewall does that job; the rules are kept, and true\nputs them back. Changed in the cluster's settings, it is in force on\nevery node and control plane within seconds.", Nested: ""},
 	}},
+	{Package: "engines", Name: "FreeTokenConfig", Title: "FreeTokenConfig", Doc: "", Fields: []Field{
+		{Key: "binary", Type: "string", Doc: "Binary is its ft command, in the environment it was installed into.\nThe agent installs one itself on a node whose GPUs it runs on; this\nis for one installed by hand.", Nested: ""},
+	}},
 	{Package: "catalog", Name: "GPUNeed", Title: "GPUNeed", Doc: "", Fields: []Field{
 		{Key: "min_compute_capability", Type: "float64", Doc: "MinComputeCapability is the oldest architecture that runs it: 7.5 is\nTuring, 8.0 Ampere (bf16, flash-attention 2), 8.9 Ada (fp8), 9.0 Hopper.", Nested: ""},
 		{Key: "min_memory_gb", Type: "int", Doc: "MinMemoryGB is the card's own memory it needs.", Nested: ""},
@@ -296,6 +299,7 @@ var Reference = []Block{
 		{Key: "ib_root", Type: "string", Doc: "IBRoot is the sysfs InfiniBand directory.", Nested: ""},
 		{Key: "prefer_nvml", Type: "bool", Doc: "PreferNVML uses the NVML backend when this binary has one. NVLink\ntopology is read from nvidia-smi topo -m on every build; NVML, where it\nis used, is the more exact source of the peers.", Nested: ""},
 		{Key: "skip_gpus", Type: "bool", Doc: "SkipGPUs omits GPU discovery. A node without nvidia-smi is already\nfound to have none, so this is for one whose driver is broken.", Nested: ""},
+		{Key: "dcgm_exporter", Type: "string", Doc: "DCGMExporter is the dcgm-exporter metrics URL read for how much of\neach card's chip is in use, on cards NVML's GPM does not cover (before\nHopper). Empty is http://127.0.0.1:9400/metrics; \"off\" reads none. A\nnode without one costs a refused connection every five minutes.", Nested: ""},
 	}},
 	{Package: "catalog", Name: "Origin", Title: "Origin", Doc: "", Fields: []Field{
 		{Key: "repo", Type: "string", Doc: "Repo is the repository, owner/name.", Nested: ""},
@@ -402,6 +406,7 @@ var Reference = []Block{
 		{Key: "batch", Type: "Batch", Doc: "Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of\nrequests served at the lowest priority, its answers written to\nanother. Needs the store.", Nested: "config.Batch"},
 		{Key: "cache", Type: "Cache", Doc: "Cache bounds the answers kept for the projects that reuse them\n(llmgw keys cache). Which projects do, and for how long, is theirs\nto set; this is only how much memory it may take.", Nested: "config.Cache"},
 		{Key: "trusted_proxies", Type: "[]string", Doc: "TrustedProxies are the reverse proxies in front of the client port,\nas addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host\nare believed; anyone else's are not, since they decide the address\nwritten into the setup scripts and whose attempts are counted where.", Nested: ""},
+		{Key: "active_active", Type: "bool", Doc: "ActiveActive has standby control planes answer the client API\nthemselves, routing to the leader's replicas, rather than passing\nevery request to the leader. More machines carry the traffic, which\nis the point. Rate limits are the cluster's: each router's counts\nreach the others through the leader every two seconds, so a key\nspread over N control planes may pass its limit by what they allow\nin those two seconds. A budget is checked against the store every\nhalf minute; cached answers and prefix affinity are split between\nthe routers. Off by\ndefault. Which machine a client reaches is the load balancer's or\nDNS's to say; the floating address stays on the leader.", Nested: ""},
 		{Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"},
 		{Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"},
 		{Key: "admission", Type: "Admission", Doc: "", Nested: "config.Admission"},
@@ -621,6 +626,7 @@ var Reference = []Block{
 		{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
 		{Key: "comfyui", Type: "ComfyUIConfig", Doc: "ComfyUI makes pictures; offered by a node whose ComfyUI is named here.", Nested: "engines.ComfyUIConfig"},
 		{Key: "strata", Type: "StrataConfig", Doc: "Strata serves a few mixture-of-experts models from RAM and one card;\noffered by a node whose Strata is named here, or that installed it.", Nested: "engines.StrataConfig"},
+		{Key: "freetoken", Type: "FreeTokenConfig", Doc: "FreeToken serves mixture-of-experts models larger than a card from\nRAM and one card of 8.0 or newer; installed by the agent where the\ncards allow it, or named here.", Nested: "engines.FreeTokenConfig"},
 	}},
 	{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
 		{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
@@ -24,6 +24,7 @@ var Sources = []string{
 	"internal/agent/engines/sim.go",
 	"internal/agent/engines/comfyui.go",
 	"internal/agent/engines/strata.go",
+	"internal/agent/engines/freetoken.go",
 	"internal/agent/supervisor/supervisor.go",
 	"internal/tlsconf/tlsconf.go",
 	"internal/control/catalog/catalog.go",
@@ -2,11 +2,15 @@ package mirror

 import (
 	"context"
+	"errors"
 	"fmt"
 	"io"
+	"net"
 	"net/http"
 	"os"
 	"path/filepath"
+	"strings"
+	"syscall"
 	"time"
 )

@@ -54,40 +58,11 @@ func addWith(ctx context.Context, client *http.Client, root, source string, e En
 		return Added{}, err
 	}

-	req, err := http.NewRequestWithContext(ctx, http.MethodGet, source, nil)
-	if err != nil {
-		return Added{}, err
-	}
-	resp, err := client.Do(req)
-	if err != nil {
-		return Added{}, fmt.Errorf("fetching %s: %w", source, err)
-	}
-	defer resp.Body.Close()
-	if resp.StatusCode != http.StatusOK {
-		return Added{}, fmt.Errorf("%s answered %s", source, resp.Status)
-	}
-
 	part := final + partSuffix
-	f, err := os.OpenFile(part, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644) //nolint:gosec // inside the mirror root
+	written, err := downloadResuming(ctx, client, source, part, progress)
 	if err != nil {
 		return Added{}, err
 	}
-	written, err := copyWithProgress(f, resp.Body, resp.ContentLength, progress)
-	closeErr := f.Close()
-	if err != nil {
-		_ = os.Remove(part)
-		return Added{}, err
-	}
-	if closeErr != nil {
-		_ = os.Remove(part)
-		return Added{}, closeErr
-	}
-	// A truncated download that the server ended cleanly would otherwise be
-	// renamed into place and copied to every node.
-	if resp.ContentLength > 0 && written != resp.ContentLength {
-		_ = os.Remove(part)
-		return Added{}, fmt.Errorf("%s: got %d bytes of %d", source, written, resp.ContentLength)
-	}

 	sum, err := sha256File(part)
 	if err != nil {
@@ -112,6 +87,132 @@ func addWith(ctx context.Context, client *http.Client, root, source string, e En
 	return Added{Entry: e, SHA256: sum}, nil
 }

+// downloadAttempts is how many times a download is tried before it fails:
+// a registry's CDN ends a long transfer now and then ("stream error …
+// CANCEL"), and a 60 GB file failed at 23 GB and started again from nothing.
+var downloadAttempts = 6
+
+// downloadPause is the wait before the next attempt, doubled each time.
+var downloadPause = 2 * time.Second
+
+// downloadResuming fetches source into part, carrying on from what part
+// already holds — a download cut short, or one a restarted mirror began —
+// and trying again, from where it got to, when the transfer breaks. The part
+// is kept when it fails, for the next fetch of the same file to carry on.
+func downloadResuming(ctx context.Context, client *http.Client, source, part string,
+	progress func(done, total int64),
+) (int64, error) {
+	var have int64
+	if st, err := os.Stat(part); err == nil {
+		have = st.Size()
+	}
+	pause := downloadPause
+	var last error
+	for attempt := 0; attempt < downloadAttempts; attempt++ {
+		if attempt > 0 {
+			select {
+			case <-ctx.Done():
+				return have, ctx.Err()
+			case <-time.After(pause):
+			}
+			pause = min(pause*2, 30*time.Second)
+		}
+		req, err := http.NewRequestWithContext(ctx, http.MethodGet, source, nil)
+		if err != nil {
+			return have, err
+		}
+		if have > 0 {
+			req.Header.Set("Range", fmt.Sprintf("bytes=%d-", have))
+		}
+		resp, err := client.Do(req)
+		if err != nil {
+			if ctx.Err() != nil {
+				return have, ctx.Err()
+			}
+			// A refused address, a name that does not resolve: asking
+			// again changes nothing.
+			if !transient(err) {
+				return have, fmt.Errorf("fetching %s: %w", source, err)
+			}
+			last = fmt.Errorf("fetching %s: %w", source, err)
+			continue
+		}
+		flags := os.O_CREATE | os.O_WRONLY
+		switch {
+		case resp.StatusCode == http.StatusPartialContent && have > 0:
+			flags |= os.O_APPEND
+		case resp.StatusCode == http.StatusOK:
+			have = 0 // the server ignored the range: from the start
+			flags |= os.O_TRUNC
+		case resp.StatusCode == http.StatusRequestedRangeNotSatisfiable && have > 0:
+			// Everything was there already.
+			resp.Body.Close()
+			return have, nil
+		case resp.StatusCode >= 500 || resp.StatusCode == http.StatusTooManyRequests:
+			resp.Body.Close()
+			last = fmt.Errorf("%s answered %s", source, resp.Status)
+			continue
+		default:
+			resp.Body.Close()
+			return have, fmt.Errorf("%s answered %s", source, resp.Status)
+		}
+		want := resp.ContentLength
+		f, err := os.OpenFile(part, flags, 0o644) //nolint:gosec // inside the mirror root
+		if err != nil {
+			resp.Body.Close()
+			return have, err
+		}
+		offset := have
+		var report func(done, total int64)
+		if progress != nil {
+			report = func(done, total int64) {
+				if total > 0 {
+					total += offset
+				}
+				progress(offset+done, total)
+			}
+		}
+		n, err := copyWithProgress(f, resp.Body, want, report)
+		resp.Body.Close()
+		closeErr := f.Close()
+		have += n
+		if err == nil && closeErr != nil {
+			return have, closeErr
+		}
+		if err != nil {
+			if ctx.Err() != nil {
+				return have, ctx.Err()
+			}
+			last = fmt.Errorf("fetching %s after %d bytes: %w", source, have, err)
+			continue
+		}
+		// Ended cleanly but short: the rest is asked for again.
+		if want > 0 && n != want {
+			last = fmt.Errorf("%s: got %d bytes of %d", source, n, want)
+			continue
+		}
+		return have, nil
+	}
+	return have, fmt.Errorf("%w (tried %d times; what arrived is kept and the next fetch carries on from it)",
+		last, downloadAttempts)
+}
+
+// transient says whether a failure to reach a server may pass if asked
+// again: a timeout, a connection the far end reset or ended, an HTTP/2
+// stream it cancelled.
+func transient(err error) bool {
+	var ne net.Error
+	if errors.As(err, &ne) && ne.Timeout() {
+		return true
+	}
+	if errors.Is(err, syscall.ECONNRESET) || errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF) {
+		return true
+	}
+	msg := err.Error()
+	return strings.Contains(msg, "stream error") || strings.Contains(msg, "connection reset") ||
+		strings.Contains(msg, "GOAWAY")
+}
+
 // copyWithProgress copies and reports occasionally.
 //
 // Occasionally rather than continuously: a forty gigabyte download is a
@@ -0,0 +1,94 @@
+package mirror
+
+import (
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strconv"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+// cutServer serves body, honouring Range, and cuts the first transfer off
+// halfway, as a registry's CDN does now and then.
+func cutServer(t *testing.T, body string) (*httptest.Server, *atomic.Int32, *atomic.Value) {
+	t.Helper()
+	var calls atomic.Int32
+	var ranges atomic.Value
+	ranges.Store("")
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		n := calls.Add(1)
+		from := 0
+		if rg := r.Header.Get("Range"); rg != "" {
+			ranges.Store(rg)
+			from, _ = strconv.Atoi(strings.TrimSuffix(strings.TrimPrefix(rg, "bytes="), "-"))
+			w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", from, len(body)-1, len(body)))
+			w.Header().Set("Content-Length", strconv.Itoa(len(body)-from))
+			w.WriteHeader(http.StatusPartialContent)
+		} else {
+			w.Header().Set("Content-Length", strconv.Itoa(len(body)))
+		}
+		if n == 1 {
+			_, _ = w.Write([]byte(body[from : from+len(body)/2]))
+			w.(http.Flusher).Flush()
+			hj, _ := w.(http.Hijacker)
+			conn, _, _ := hj.Hijack()
+			_ = conn.Close()
+			return
+		}
+		_, _ = w.Write([]byte(body[from:]))
+	}))
+	t.Cleanup(srv.Close)
+	return srv, &calls, &ranges
+}
+
+func TestADownloadCutOffCarriesOnFromWhereItGot(t *testing.T) {
+	old := downloadPause
+	downloadPause = 10 * time.Millisecond
+	t.Cleanup(func() { downloadPause = old })
+	body := strings.Repeat("weights ", 1000)
+	srv, calls, ranges := cutServer(t, body)
+	root := t.TempDir()
+	e := Entry{Model: "m", Version: "v", File: "w.gguf"}
+	added, err := addWith(t.Context(), http.DefaultClient, root, srv.URL+"/w.gguf", e, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	got, _ := os.ReadFile(e.Path(root))
+	if string(got) != body || added.Entry.Bytes != int64(len(body)) {
+		t.Fatalf("got %d bytes", len(got))
+	}
+	if calls.Load() != 2 || ranges.Load().(string) != fmt.Sprintf("bytes=%d-", len(body)/2) {
+		t.Errorf("calls %d, range %q", calls.Load(), ranges.Load())
+	}
+}
+
+func TestADownloadARestartedMirrorBeganIsCarriedOn(t *testing.T) {
+	body := strings.Repeat("x", 4000)
+	var asked atomic.Value
+	asked.Store("")
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		rg := r.Header.Get("Range")
+		asked.Store(rg)
+		from, _ := strconv.Atoi(strings.TrimSuffix(strings.TrimPrefix(rg, "bytes="), "-"))
+		w.Header().Set("Content-Length", strconv.Itoa(len(body)-from))
+		w.WriteHeader(http.StatusPartialContent)
+		_, _ = w.Write([]byte(body[from:]))
+	}))
+	defer srv.Close()
+	root := t.TempDir()
+	e := Entry{Model: "m", Version: "v", File: "w.gguf"}
+	_ = os.MkdirAll(root+"/m/v", 0o755)
+	if err := os.WriteFile(e.Path(root)+partSuffix, []byte(body[:1500]), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := addWith(t.Context(), http.DefaultClient, root, srv.URL+"/w.gguf", e, nil); err != nil {
+		t.Fatal(err)
+	}
+	if got, _ := os.ReadFile(e.Path(root)); string(got) != body || asked.Load().(string) != "bytes=1500-" {
+		t.Errorf("%d bytes, asked %q", len(got), asked.Load())
+	}
+}
@@ -543,3 +543,20 @@ func TestABandForBulkWorkWaitsLongerThanTheRest(t *testing.T) {
 		t.Errorf("bronze gave up after %v, before its own queue_ttl", d)
 	}
 }
+
+// A provider outside the cluster queues for itself: requests in flight to it
+// are not a queue here. Counted as one, an external model took three or four
+// at once and the rest timed out waiting.
+func TestRequestsInFlightToAnExternalProviderAreNotAQueue(t *testing.T) {
+	th := Thresholds{TargetWaitingPerReplica: 4, KVCeiling: 0.9}
+	ext := snap("openai", 0, 0, 200, 0, false)
+	ext.Engine = endpoint.EngineExternal
+	if got := Saturation([]endpoint.Snapshot{ext}, 0, th); got >= 0.5 {
+		t.Errorf("200 in flight to an external provider read as saturation %v", got)
+	}
+	// An engine of the cluster with the same in flight and nothing learned
+	// of its capacity still reads as loaded.
+	if got := Saturation([]endpoint.Snapshot{snap("a", 0, 0, 200, 0, false)}, 0, th); got < 1 {
+		t.Errorf("an engine's unreported burst ignored: %v", got)
+	}
+}
@@ -53,6 +53,14 @@ func Saturation(eps []endpoint.Snapshot, unreported int, t Thresholds) float64 {
 		if e.Breaker.String() == "open" {
 			continue
 		}
+		// A provider outside the cluster queues for itself and says 429 when
+		// full, which its breaker hears: nothing here measures its queue, and
+		// counting every request in flight to it as waiting held an external
+		// model to three or four at once, the rest timing out in the queue.
+		if e.Engine == endpoint.EngineExternal {
+			n++
+			continue
+		}

 		// The engine's own queue is the truth, but it is one scrape interval
 		// old. Fill that gap with what the router knows without any lag: how
@@ -49,10 +49,13 @@ func (r *Router) loadStatus(model string) modelStatus {
 					continue
 				}
 				st.State = "downloading"
+				if sh.Fetch.Phase == "checking" {
+					st.State = "checking"
+				}
 				st.Done += sh.Fetch.BytesDone
 				st.Total += sh.Fetch.BytesTotal
 			}
-			if st.State == "downloading" && st.Total > 0 {
+			if (st.State == "downloading" || st.State == "checking") && st.Total > 0 {
 				p := int(st.Done * 100 / st.Total)
 				st.Percent = &p
 			}
@@ -0,0 +1,50 @@
+package proxy
+
+import (
+	"context"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+// flushCounter is a response writer that counts its flushes.
+type flushCounter struct {
+	*httptest.ResponseRecorder
+	flushes int
+}
+
+func (f *flushCounter) Flush() { f.flushes++; f.ResponseRecorder.Flush() }
+
+// Events the engine sent together go to the client in one write, not one
+// each: a write a token was a third of a busy router's CPU.
+func TestEventsAlreadyInHandAreFlushedTogether(t *testing.T) {
+	var b strings.Builder
+	for i := range 10 {
+		b.WriteString(`data: {"choices":[{"index":0,"delta":{"content":"t` + string(rune('0'+i)) + `"}}]}` + "\n\n")
+	}
+	b.WriteString("data: [DONE]\n\n")
+	resp := &http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"text/event-stream"}},
+		Body: io.NopCloser(strings.NewReader(b.String()))}
+	w := &flushCounter{ResponseRecorder: httptest.NewRecorder()}
+	p := New(Config{})
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	if _, err := p.relayStream(ctx, cancel, w, resp, Request{}); err != nil {
+		t.Fatal(err)
+	}
+	got := w.Body.String()
+	for i := range 10 {
+		if !strings.Contains(got, `"t`+string(rune('0'+i))+`"`) {
+			t.Fatalf("event %d missing:\n%s", i, got)
+		}
+	}
+	if !strings.HasSuffix(strings.TrimSpace(got), "data: [DONE]") {
+		t.Errorf("not ended with [DONE]: %q", got[len(got)-40:])
+	}
+	// One for the headers, and the events: far fewer than one each.
+	if w.flushes > 3 {
+		t.Errorf("%d flushes for 11 events arriving together", w.flushes)
+	}
+}
@@ -418,7 +418,7 @@ func (p *Proxy) relayStream(ctx context.Context, abort context.CancelFunc, w htt
 							emit = append(emit, "[DONE]")
 						}
 					}
-					if err := p.emit(w, flusher, &out, &reply, r, emit); err != nil {
+					if err := p.emit(w, flusher, &out, &reply, r, emit, br.Buffered() > 0 && !stop); err != nil {
 						return out, err
 					}
 					if stop {
@@ -434,7 +434,9 @@ func (p *Proxy) relayStream(ctx context.Context, abort context.CancelFunc, w htt
 				if werr != nil {
 					return out, ErrClientGone
 				}
-				flusher.Flush()
+				if br.Buffered() == 0 {
+					flusher.Flush()
+				}
 			}
 		}

@@ -442,7 +444,7 @@ func (p *Proxy) relayStream(ctx context.Context, abort context.CancelFunc, w htt
 			if errors.Is(err, io.EOF) {
 				if r.StreamFilter != nil {
 					// The engine ended without [DONE]: what was held goes out.
-					if err := p.emit(w, flusher, &out, &reply, r, r.StreamFilter.OnEnd()); err != nil {
+					if err := p.emit(w, flusher, &out, &reply, r, r.StreamFilter.OnEnd(), false); err != nil {
 						return out, err
 					}
 				}
@@ -479,8 +481,17 @@ func usageOnly(data string) bool {

 // emit writes events a StreamFilter chose, as data events, and captures the
 // reply from what the client is actually sent.
-func (p *Proxy) emit(w io.Writer, f http.Flusher, out *Outcome, reply *replyBuilder, r Request, data []string) error {
+// emit writes events to the client, and flushes them unless more has
+// already arrived from the engine (more): those go out with the next write.
+// One write a token was a third of a busy router's CPU in system calls; this
+// joins only what is already in hand, so no token waits for another.
+func (p *Proxy) emit(w io.Writer, f http.Flusher, out *Outcome, reply *replyBuilder, r Request, data []string, more bool) error {
 	if len(data) == 0 {
+		// What an earlier event left unflushed goes out now that nothing
+		// more is in hand; with nothing written, a flush writes nothing.
+		if !more {
+			f.Flush()
+		}
 		return nil
 	}
 	for _, d := range data {
@@ -493,7 +504,9 @@ func (p *Proxy) emit(w io.Writer, f http.Flusher, out *Outcome, reply *replyBuil
 			return ErrClientGone
 		}
 	}
-	f.Flush()
+	if !more {
+		f.Flush()
+	}
 	return nil
 }

@@ -726,6 +726,44 @@ func (r *Router) Stop() {
 	}
 }

+// ShareBudgets keeps budgets honest while other control planes take requests
+// too: every half minute each counter is raised to what the store recorded,
+// which includes their spend. Until ctx ends.
+func (r *Router) ShareBudgets(ctx context.Context) {
+	if r.acct == nil || r.store == nil {
+		return
+	}
+	t := time.NewTicker(30 * time.Second)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			if err := r.acct.CatchUpBudgets(ctx, r.store, storepkg.PeriodMonth); err != nil {
+				r.log.Warn("could not read the budgets other control planes spent", "err", err)
+			}
+		}
+	}
+}
+
+// Rates are this router's own request and token counts, for the other
+// control planes taking requests to add to theirs.
+func (r *Router) Rates() accounting.Rates {
+	if r.acct == nil {
+		return accounting.Rates{}
+	}
+	return r.acct.Rates()
+}
+
+// SetOtherRates puts what the other control planes counted in force here, so
+// a key's limits are the cluster's.
+func (r *Router) SetOtherRates(o accounting.Rates) {
+	if r.acct != nil {
+		r.acct.SetOtherRates(o)
+	}
+}
+
 // SetLeader tells /leader how to know whether this process leads.
 func (r *Router) SetLeader(f func() bool) { r.isLeader.Store(&f) }

@@ -606,7 +606,8 @@ func describeServe(l *Lang, m catalog.Model) string {
 		return ""
 	}
 	p := m.Profiles[0]
-	engine := map[string]string{"llamacpp": "llama.cpp", "vllm": "vLLM", "sglang": "SGLang", "trtllm": "TensorRT-LLM"}[p.Engine]
+	engine := map[string]string{"llamacpp": "llama.cpp", "vllm": "vLLM", "sglang": "SGLang", "trtllm": "TensorRT-LLM",
+		"freetoken": "FreeToken"}[p.Engine]
 	if engine == "" {
 		engine = p.Engine
 	}
@@ -27,7 +27,11 @@ type FleetCharts struct {
 	// GPUBusy and GPUMemory are the fleet's cards by class, for a fleet that
 	// has them.
 	GPUBusy, GPUMemory TimeSeries
-	Window             string
+	// GPUPowerLoad is the cards' draw against their power limit, and
+	// GPUSMActive their SMs' share of time with work, where something reads
+	// it: how much of each chip the busy figure is using.
+	GPUPowerLoad, GPUSMActive TimeSeries
+	Window                    string
 	// Err is set when the metrics store could not be read. Shown as a line of
 	// text where the charts would be: an empty panel reads as a healthy cluster
 	// with nothing happening, which is the opposite of not knowing. The
@@ -144,4 +148,12 @@ func (f *FleetCharts) LoadGPU(ctx context.Context, h History, window time.Durati
 		"class", window, step); err == nil {
 		f.GPUMemory = l.LineChart(s, "%", 6)
 	}
+	if s, err := h.RangeQuery(ctx,
+		`sum by (class) (llmgw_gpu_power_watts) / sum by (class) (llmgw_gpu_power_limit_watts)`,
+		"class", window, step); err == nil {
+		f.GPUPowerLoad = l.LineChart(s, "%", 6)
+	}
+	if s, err := h.RangeQuery(ctx, `avg by (class) (llmgw_gpu_sm_active_percent) / 100`, "class", window, step); err == nil {
+		f.GPUSMActive = l.LineChart(s, "%", 6)
+	}
 }
@@ -13,8 +13,13 @@ type modelEfficiency struct {
 	// TokensPerSec is output tokens a second; GPUs the cards it holds now.
 	TokensPerSec float64
 	GPUs         float64
-	// Watts is what those cards draw; Util their average utilisation.
-	Watts, Util float64
+	// Watts is what those cards draw; Util the average share of time a
+	// kernel ran on them, which says nothing of how much of each chip it ran
+	// on. PowerLoad (draw against the power limit) and SMActive (the SMs'
+	// share of time with work, where GPM or DCGM reads it) are how much, in
+	// percent; -1 is not known.
+	Watts, Util         float64
+	PowerLoad, SMActive float64
 	// PerGPU and PerWh are the two ways of saying how much work the cards
 	// do: output tokens a second per card, and per watt-hour drawn.
 	PerGPU, PerWh float64
@@ -34,6 +39,10 @@ var efficiencyQueries = []struct {
 		func(e *modelEfficiency, v float64) { e.Watts = v }},
 	{`avg by (model) (llmgw_gpu_utilization_percent * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
 		func(e *modelEfficiency, v float64) { e.Util = v }},
+	{`avg by (model) ((llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts) * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
+		func(e *modelEfficiency, v float64) { e.PowerLoad = 100 * v }},
+	{`avg by (model) (llmgw_gpu_sm_active_percent * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
+		func(e *modelEfficiency, v float64) { e.SMActive = v }},
 }

 // loadEfficiency is the table, best per card first; nil when the metrics
@@ -56,7 +65,7 @@ func loadEfficiency(ctx context.Context, h History) []modelEfficiency {
 			}
 			e := byModel[s.Name]
 			if e == nil {
-				e = &modelEfficiency{Model: s.Name}
+				e = &modelEfficiency{Model: s.Name, PowerLoad: -1, SMActive: -1}
 				byModel[s.Name] = e
 			}
 			q.into(e, s.Last())
@@ -8,16 +8,25 @@ import (
 	"time"

 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
 )

 // The GPU map is every card in the fleet, a square each, a row per node,
-// coloured by one figure: how busy, how hot, or how full its memory is. A
+// coloured by one figure: how much of the chip is at work, how busy, how
+// hot, or how full its memory is. A
 // hundred cards in a table are a hundred rows to read; on the map the one
 // that is not like the others is seen before anything is read.

 // gpuMapModes are what the map can be coloured by; each is named by the key
 // dash.gpumap.mode.<mode>.
-var gpuMapModes = []string{"util", "memory", "temp"}
+//
+// work is the default. util, the driver's busy figure, is the share of time
+// any kernel ran, and a kernel on one SM of a hundred reads 100%: a map of
+// it is green wherever a model is loaded and asked anything. work is the
+// SMs' share of time with work where the card's counters can be read, and
+// its power draw against its limit elsewhere, which rises with how much of
+// the chip runs rather than with whether anything does.
+var gpuMapModes = []string{"work", "util", "memory", "temp"}

 func init() {
 	for _, m := range gpuMapModes {
@@ -72,17 +81,16 @@ func buildGPUMap(l *Lang, nodes []adminapi.NodeView, reps []adminapi.ReplicaView
 	for _, k := range gpuMapModes {
 		m.Modes = append(m.Modes, struct{ Key, Label string }{k, l.T("dash.gpumap.mode." + k)})
 	}
-	if m.By == "" {
-		m.By = "util"
-	}
 	switch m.By {
+	case "util":
+		m.Legend = l.T("dash.gpumap.legend.util")
 	case "temp":
 		m.Legend = l.T("dash.gpumap.legend.temp")
 	case "memory":
 		m.Legend = l.T("dash.gpumap.legend.memory")
 	default:
-		m.By = "util"
-		m.Legend = l.T("dash.gpumap.legend.util")
+		m.By = "work"
+		m.Legend = l.T("dash.gpumap.legend.work")
 	}
 	for _, n := range nodes {
 		if n.GPUTotal == 0 {
@@ -106,13 +114,21 @@ func buildGPUMap(l *Lang, nodes []adminapi.NodeView, reps []adminapi.ReplicaView
 				if s.MemoryTotalMB > 0 && s.UtilPercent >= 0 {
 					title = append(title, l.T("dash.gpumap.busy", l.Num(s.UtilPercent, 0)))
 				}
+				if p := s.Profile; p != nil && p.SMActive >= 0 {
+					title = append(title, l.T("dash.gpumap.sm_active", l.Num(p.SMActive, 0)))
+					if p.TensorActive >= 0 {
+						title = append(title, l.T("dash.gpumap.tensor_active", l.Num(p.TensorActive, 0)))
+					}
+				}
 				if s.MemoryTotalMB > 0 {
 					title = append(title, l.T("dash.gpumap.memory", l.Num(s.MemoryUsedMB/1024, 1), l.Num(s.MemoryTotalMB/1024, 0)))
 				}
 				if s.TempC > 0 {
 					title = append(title, l.Num(s.TempC, 0)+" °C")
 				}
-				if s.PowerW > 0 {
+				if s.PowerW > 0 && s.PowerLimitW > 0 {
+					title = append(title, l.T("dash.gpumap.power_of", l.Num(s.PowerW, 0), l.Num(s.PowerLimitW, 0)))
+				} else if s.PowerW > 0 {
 					title = append(title, l.Num(s.PowerW, 0)+" W")
 				}
 				if len(s.Throttle) > 0 {
@@ -162,6 +178,20 @@ func cellColour(by string, s sensorView, sampled bool) (string, string) {
 		}
 		f := clamp01(s.MemoryUsedMB / s.MemoryTotalMB)
 		return fmt.Sprintf("background:hsl(265,45%%,%.0f%%)", 90-55*f), fmt.Sprintf("%.0f", 100*f)
+	case "work":
+		if !sampled {
+			return unknown, ""
+		}
+		var f float64
+		switch {
+		case s.Profile != nil && s.Profile.SMActive >= 0:
+			f = clamp01(s.Profile.SMActive / 100)
+		case s.PowerW > 0 && s.PowerLimitW > 0:
+			f = clamp01(s.PowerW / s.PowerLimitW)
+		default:
+			return unknown, ""
+		}
+		return fmt.Sprintf("background:hsl(200,%.0f%%,%.0f%%)", 15+55*f, 88-50*f), fmt.Sprintf("%.0f", 100*f)
 	}
 	if !sampled || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
 		return unknown, ""
@@ -286,9 +316,10 @@ func gpuKey(node string, gpu int) string { return fmt.Sprintf("%s/%d", node, gpu

 // sensorView is a card's last sample.
 type sensorView = struct {
-	TempC, PowerW, UtilPercent, MemoryUsedMB, MemoryTotalMB float64
-	Throttle                                                []string
-	Problems                                                []string
+	TempC, PowerW, PowerLimitW, UtilPercent, MemoryUsedMB, MemoryTotalMB float64
+	Throttle                                                             []string
+	Problems                                                             []string
+	Profile                                                              *inventory.GPUProfile
 }

 func sensorOf(n adminapi.NodeView, gpu int) (sensorView, bool) {
@@ -297,9 +328,9 @@ func sensorOf(n adminapi.NodeView, gpu int) (sensorView, bool) {
 	}
 	for _, s := range n.Load.GPUs {
 		if s.Index == gpu {
-			return sensorView{TempC: s.TempC, PowerW: s.PowerW, UtilPercent: s.UtilPercent,
+			return sensorView{TempC: s.TempC, PowerW: s.PowerW, PowerLimitW: s.PowerLimitW, UtilPercent: s.UtilPercent,
 				MemoryUsedMB: s.MemoryUsedMB, MemoryTotalMB: s.MemoryTotalMB, Throttle: s.Throttle,
-				Problems: s.Wear.Problems()}, true
+				Problems: s.Wear.Problems(), Profile: s.Profile}, true
 		}
 	}
 	return sensorView{}, false
@@ -17,10 +17,14 @@ type mapFleet struct{ fakeSource }

 func (mapFleet) Nodes() []adminapi.NodeView {
 	busy := func(i int, util, temp float64) inventory.GPUSensor {
-		return inventory.GPUSensor{Index: i, UtilPercent: util, TempC: temp, MemoryUsedMB: 40000, MemoryTotalMB: 81559, PowerW: 300}
+		return inventory.GPUSensor{Index: i, UtilPercent: util, TempC: temp, MemoryUsedMB: 40000, MemoryTotalMB: 81559, PowerW: 300, PowerLimitW: 700}
+	}
+	profiled := func(s inventory.GPUSensor) inventory.GPUSensor {
+		s.Profile = &inventory.GPUProfile{SMActive: 0, SMOccupancy: 0, TensorActive: 0, DRAMActive: -1, Source: "dcgm"}
+		return s
 	}
 	return []adminapi.NodeView{{ID: "gpu07", State: "ready", GPUTotal: 4, GPUFree: 1, Classes: []string{"h100-80g"},
-		Load: &adminapi.LoadView{GPUs: []inventory.GPUSensor{busy(0, 1, 35), busy(1, 0, 34), busy(2, 93, 81), worn(busy(3, 0, 30))}}}}
+		Load: &adminapi.LoadView{GPUs: []inventory.GPUSensor{profiled(busy(0, 1, 35)), busy(1, 0, 34), busy(2, 93, 81), worn(busy(3, 0, 30))}}}}
 }

 func worn(s inventory.GPUSensor) inventory.GPUSensor {
@@ -60,6 +64,20 @@ func TestTheClusterPageMapsEveryCardAndNamesTheIdleOnes(t *testing.T) {
 	if !strings.Contains(temp, "<strong>temperature</strong>") || !strings.Contains(temp, ">81</span>") {
 		t.Error("the map coloured by temperature does not say so, or show it")
 	}
+	// The default colours by how much of the chip works: card 2's draw of
+	// 300 W of 700, where nothing reads its SMs; card 0's SMs where
+	// something does. Busy stays a mode of its own.
+	if !strings.Contains(body, "<strong>work</strong>") || !strings.Contains(body, ">43</span>") ||
+		!strings.Contains(body, "300 of 700 W") {
+		t.Error("the map does not colour by work, or say the draw against the limit")
+	}
+	if !strings.Contains(body, "gpu07 GPU 0 · chat (chat-1) · 1% busy · SMs 0% active · tensor cores 0%") {
+		t.Error("a profiled card does not say how much of its chip is at work")
+	}
+	_, util := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/?gpumap=util", nil)
+	if !strings.Contains(util, "<strong>busy</strong>") || !strings.Contains(util, ">93</span>") {
+		t.Error("the map coloured by busy does not say so, or show it")
+	}
 	_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
 	if strings.Contains(cpu, "gpu-map") {
 		t.Error("a fleet without GPUs has a GPU map")
@@ -68,7 +86,7 @@ func TestTheClusterPageMapsEveryCardAndNamesTheIdleOnes(t *testing.T) {

 func TestEachModelSaysWhatItGetsFromItsCards(t *testing.T) {
 	h := &fakeHistory{byExpr: map[string][]promq.Series{}}
-	for i, vals := range [][2]float64{{120, 30}, {2, 2}, {1200, 400}, {80, 40}} {
+	for i, vals := range [][2]float64{{120, 30}, {2, 2}, {1200, 400}, {80, 40}, {0.6, 0.2}, {55, 0}} {
 		h.byExpr[efficiencyQueries[i].expr] = []promq.Series{ts("chat", vals[0], vals[0]), ts("code", vals[1], vals[1])}
 	}
 	got := loadEfficiency(context.Background(), h)
@@ -76,7 +94,7 @@ func TestEachModelSaysWhatItGetsFromItsCards(t *testing.T) {
 		t.Fatalf("%+v", got)
 	}
 	c := got[0]
-	if c.PerGPU != 60 || c.PerWh != 120*3600/1200.0 || c.Util != 80 {
+	if c.PerGPU != 60 || c.PerWh != 120*3600/1200.0 || c.Util != 80 || c.PowerLoad != 60 || c.SMActive != 55 {
 		t.Errorf("chat: %+v", c)
 	}
 	if got[1].PerGPU != 15 {
@@ -37,7 +37,7 @@ func TestTheDashboardSpeaksTurkishWhenAsked(t *testing.T) {
 		// A state is still its class, and a Turkish word.
 		`<td class="state state-ready">hazır</td>`,
 		// The GPU map: its modes, and what a card says under the pointer.
-		"<strong>meşguliyet</strong>", "gpu07 GPU 2 · code (code-1) · %93 meşgul · 39,1 / 80 GB",
+		"<strong>iş</strong>", "gpu07 GPU 2 · code (code-1) · %93 meşgul · 39,1 / 80 GB",
 		// How long a replica has sat idle, in Turkish units.
 		"30dk0sn önce",
 		// The traffic chart's peak, with Turkish thousands.
@@ -23,6 +23,7 @@
  "chat.js.download_failed": "download failed",
  "chat.js.stage.waiting": "Waiting for a node with room",
  "chat.js.stage.downloading": "Downloading weights",
+  "chat.js.stage.checking": "Checking the weights",
  "chat.js.stage.starting": "Starting the engine",
  "chat.js.stage.ready": "Ready",
  "chat.js.of": "{done} of {total}",
@@ -26,11 +26,16 @@
  "dash.gpumap.mode.util": "busy",
  "dash.gpumap.mode.memory": "memory",
  "dash.gpumap.mode.temp": "temperature",
+  "dash.gpumap.mode.work": "work",
  "dash.gpumap.legend.temp": "Blue is cool, red is 90 °C and over.",
  "dash.gpumap.legend.memory": "Pale is empty, dark is full. Engines reserve most of a card's memory when they start.",
-  "dash.gpumap.legend.util": "Pale is idle, dark green is fully busy. A dashed border is a card no replica holds.",
+  "dash.gpumap.legend.util": "Pale is idle, dark green is busy: the share of time any kernel ran, however little of the chip it used. A dashed border is a card no replica holds.",
+  "dash.gpumap.legend.work": "Pale is idle, dark blue is the whole chip at work: the SMs' share of time with work where the card's counters can be read (GPM, DCGM), its power draw against its limit elsewhere. A dashed border is a card no replica holds.",
  "dash.gpumap.free": "free",
  "dash.gpumap.busy": "%s%% busy",
+  "dash.gpumap.sm_active": "SMs %s%% active",
+  "dash.gpumap.tensor_active": "tensor cores %s%%",
+  "dash.gpumap.power_of": "%s of %s W",
  "dash.gpumap.memory": "%s of %s GB",
  "dash.gpumap.held_back": "held back by %s",
  "dash.gpumap.note": "A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.",
@@ -41,8 +46,11 @@
  "dash.eff.tps": "Output tokens/s",
  "dash.eff.per_gpu": "Per GPU",
  "dash.eff.busy": "Busy",
+  "dash.eff.power_load": "Power load",
+  "dash.eff.sm_active": "SM active",
  "dash.eff.draw": "Draw",
  "dash.eff.per_wh": "Tokens per Wh",
+  "dash.eff.note": "Busy is the share of time any kernel ran on the cards, not how much of each chip it ran on: a kernel on one SM reads 100%. Power load (draw against the power limit) and SM active (the SMs' share of time with work, where the cards' counters can be read) say how much.",

  "dash.idle.title": "Cards held and doing nothing",
  "dash.col.replica": "Replica",
@@ -150,7 +158,11 @@
  "dash.fleet.ready": "Replicas ready, by model",
  "dash.fleet.ready_note": "A model deliberately stopped has no line at all, which is why an on-demand catalogue looks quiet here.",
  "dash.fleet.gpu_busy": "GPUs busy, by class",
-  "dash.fleet.gpu_busy_note": "The average card of each class. Low here with replicas placed is capacity held and not asked for; each node's page has its cards one by one.",
+  "dash.fleet.gpu_busy_note": "The average card of each class: the share of time any kernel ran, not how much of the chip it used. Low here with replicas placed is capacity held and not asked for; each node's page has its cards one by one.",
+  "dash.fleet.gpu_power_load": "GPU power load, by class",
+  "dash.fleet.gpu_power_load_note": "The cards' draw against their power limit. It rises with how much of each chip is working, where busy only says that something is; a class busy above and low here runs small kernels or waits between them.",
+  "dash.fleet.gpu_sm_active": "SMs active, by class",
+  "dash.fleet.gpu_sm_active_note": "The share of time the cards' SMs had work, from their profiling counters (NVML GPM on Hopper and later, or a dcgm-exporter on the node). The gap to busy is the part of the chip a running kernel leaves idle.",
  "dash.fleet.gpu_memory": "GPU memory in use, by class",
  "dash.fleet.gpu_memory_note": "Engines reserve most of a card's memory when they start, so this follows replicas placed more than requests served.",
  "dash.fleet.queue": "Engine queue depth, by replica",
@@ -77,7 +77,11 @@
  "node.chart.engine_memory": "Engine memory, by model",
  "node.chart.engine_memory_note": "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use.",
  "node.chart.gpu_busy": "GPU busy, by card",
-  "node.chart.gpu_busy_note": "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking.",
+  "node.chart.gpu_busy_note": "Of each sample period, the share a kernel was running — not how much of the chip it used: one SM busy reads 100%. A card held by a replica and idle here is one nobody is asking.",
+  "node.chart.gpu_power_load": "GPU power load, by card",
+  "node.chart.gpu_power_load_note": "Draw against the card's power limit. Busy says a kernel ran; this rises with how much of the chip it ran on.",
+  "node.chart.gpu_sm_active": "SMs active, by card",
+  "node.chart.gpu_sm_active_note": "The share of time the card's SMs had work, from its profiling counters. When busy reads 90 and this 60, a third of the chip sits idle while a kernel runs.",
  "node.chart.gpu_memory": "GPU memory used, by card",
  "node.chart.gpu_memory_note": "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's.",
  "node.chart.gpu_temp": "GPU temperature, °C",
@@ -23,6 +23,7 @@
  "chat.js.download_failed": "indirme başarısız oldu",
  "chat.js.stage.waiting": "Yeri olan bir düğüm bekleniyor",
  "chat.js.stage.downloading": "Ağırlıklar indiriliyor",
+  "chat.js.stage.checking": "Ağırlıklar denetleniyor",
  "chat.js.stage.starting": "Motor başlatılıyor",
  "chat.js.stage.ready": "Hazır",
  "chat.js.of": "{done} / {total}",
@@ -26,11 +26,16 @@
  "dash.gpumap.mode.util": "meşguliyet",
  "dash.gpumap.mode.memory": "bellek",
  "dash.gpumap.mode.temp": "sıcaklık",
+  "dash.gpumap.mode.work": "iş",
  "dash.gpumap.legend.temp": "Mavi serin, kırmızı 90 °C ve üstü.",
  "dash.gpumap.legend.memory": "Açık renk boş, koyu renk dolu. Motorlar başlarken bir kartın belleğinin çoğunu ayırır.",
-  "dash.gpumap.legend.util": "Açık renk boşta, koyu yeşil tamamen meşgul. Kesikli kenarlık, hiçbir replikanın tutmadığı bir karttır.",
+  "dash.gpumap.legend.util": "Açık renk boşta, koyu yeşil meşgul demektir: yonganın ne kadarını kullanırsa kullansın, herhangi bir çekirdek işlevinin çalıştığı zaman payı. Kesik kenarlık hiçbir replikanın tutmadığı karttır.",
+  "dash.gpumap.legend.work": "Açık renk boşta, koyu mavi tüm yongası çalışan kart demektir: kartın sayaçları okunabiliyorsa (GPM, DCGM) SM'lerin iş yaptığı zaman payı, okunamıyorsa çektiği gücün güç sınırına oranı. Kesik kenarlık hiçbir replikanın tutmadığı karttır.",
  "dash.gpumap.free": "boş",
  "dash.gpumap.busy": "%%%s meşgul",
+  "dash.gpumap.sm_active": "SM'ler %%%s etkin",
+  "dash.gpumap.tensor_active": "tensör çekirdekleri %%%s",
+  "dash.gpumap.power_of": "%s / %s W",
  "dash.gpumap.memory": "%s / %s GB",
  "dash.gpumap.held_back": "kısıtlayan: %s",
  "dash.gpumap.note": "Kırmızı çerçeve, yıpranmakta olan bir karttır: XID hataları, yeniden eşlenmiş ya da sıfırlama bekleyen bellek satırları, NVLink hataları. Karttaki rakam, renklendirmede kullanılan değerdir; kartın geri kalanı imlecin altındadır.",
@@ -41,8 +46,11 @@
  "dash.eff.tps": "Çıkış token/sn",
  "dash.eff.per_gpu": "GPU başına",
  "dash.eff.busy": "Meşgul",
+  "dash.eff.power_load": "Güç yükü",
+  "dash.eff.sm_active": "Etkin SM",
  "dash.eff.draw": "Çekilen güç",
  "dash.eff.per_wh": "Wh başına token",
+  "dash.eff.note": "Meşgul, kartlarda herhangi bir çekirdek işlevinin çalıştığı zaman payıdır; her yonganın ne kadarında çalıştığı değil: tek bir SM'de çalışan işlev %100 okunur. Ne kadarını kullandığını güç yükü (çekilen gücün güç sınırına oranı) ve etkin SM (kartların sayaçları okunabildiğinde SM'lerin iş yaptığı zaman payı) söyler.",

  "dash.idle.title": "Tutulan ama hiçbir şey yapmayan kartlar",
  "dash.col.replica": "Replika",
@@ -148,7 +156,11 @@
  "dash.fleet.ready": "Hazır replikalar, modele göre",
  "dash.fleet.ready_note": "Bilerek durdurulmuş bir modelin hiç çizgisi olmaz; isteğe bağlı çalışan bir katalogun burada sessiz görünmesinin nedeni budur.",
  "dash.fleet.gpu_busy": "Meşgul GPU'lar, sınıfa göre",
-  "dash.fleet.gpu_busy_note": "Her sınıfın ortalama kartı. Replikalar yerleştirilmişken burada düşük olması, tutulan ama istenmeyen kapasite demektir; her düğümün sayfası kartlarını tek tek gösterir.",
+  "dash.fleet.gpu_busy_note": "Her sınıfın ortalama kartı: herhangi bir çekirdek işlevinin çalıştığı zaman payı, yonganın ne kadarını kullandığı değil. Replikalar yerleştirilmişken burada düşük olması, tutulan ama istenmeyen kapasite demektir; her düğümün sayfası kartlarını tek tek gösterir.",
+  "dash.fleet.gpu_power_load": "GPU güç yükü, sınıfa göre",
+  "dash.fleet.gpu_power_load_note": "Kartların çektiği gücün güç sınırına oranı. Meşgul yalnızca bir şeyin çalıştığını söylerken bu, her yonganın ne kadarının çalıştığıyla artar; yukarıda meşgul ve burada düşük görünen bir sınıf küçük çekirdek işlevleri çalıştırıyor ya da aralarında bekliyordur.",
+  "dash.fleet.gpu_sm_active": "Etkin SM'ler, sınıfa göre",
+  "dash.fleet.gpu_sm_active_note": "Kartların SM'lerinin iş yaptığı zaman payı, profil sayaçlarından (Hopper ve sonrasında NVML GPM ya da düğümdeki bir dcgm-exporter). Meşgul ile arasındaki fark, çalışan bir işlevin yongada boş bıraktığı kısımdır.",
  "dash.fleet.gpu_memory": "Kullanılan GPU belleği, sınıfa göre",
  "dash.fleet.gpu_memory_note": "Motorlar başlarken bir kartın belleğinin çoğunu ayırır; bu yüzden bu grafik sunulan isteklerden çok yerleştirilen replikaları izler.",
  "dash.fleet.queue": "Motor kuyruk derinliği, replikaya göre",
@@ -76,7 +76,11 @@
  "node.chart.engine_memory": "Motor belleği, modele göre",
  "node.chart.engine_memory_note": "Diskten eşlenen ağırlıklar dâhil yerleşik bellek. Çekirdek o sayfaları geri alabilir, bu yüzden bu değer kullanılan RAM'i aşabilir.",
  "node.chart.gpu_busy": "Meşgul GPU, karta göre",
-  "node.chart.gpu_busy_note": "Her örnekleme süresinde bir çekirdek işlevinin çalıştığı pay. Bir replikanın tuttuğu ve burada boşta görünen kart, kimsenin istemediği karttır.",
+  "node.chart.gpu_busy_note": "Her örnekleme süresinde bir çekirdek işlevinin çalıştığı pay — yonganın ne kadarını kullandığı değil: tek bir SM'nin meşgul olması %100 okunur. Bir replikanın tuttuğu ve burada boşta görünen kart, kimsenin istemediği karttır.",
+  "node.chart.gpu_power_load": "GPU güç yükü, karta göre",
+  "node.chart.gpu_power_load_note": "Çekilen gücün kartın güç sınırına oranı. Meşgul bir işlevin çalıştığını söyler; bu, yonganın ne kadarında çalıştığıyla artar.",
+  "node.chart.gpu_sm_active": "Etkin SM'ler, karta göre",
+  "node.chart.gpu_sm_active_note": "Kartın SM'lerinin iş yaptığı zaman payı, profil sayaçlarından. Meşgul %90 iken bunun %60 olması, bir işlev çalışırken yonganın üçte birinin boşta olduğu demektir.",
  "node.chart.gpu_memory": "Kullanılan GPU belleği, karta göre",
  "node.chart.gpu_memory_note": "Kullanımdaki çerçeve arabelleği: bir motorun ayırdığı ağırlıklar ve KV önbelleği, ayrıca llmgw'ye ait olmayan her şey.",
  "node.chart.gpu_temp": "GPU sıcaklığı, °C",
@@ -51,8 +51,11 @@ type nodeCharts struct {
 	CPU, Memory, Load, EngineCPU, EngineMemory, Queue TimeSeries
 	// The GPUs', one line per card; empty on a node without them.
 	GPUUtil, GPUMemory, GPUTemp, GPUPower TimeSeries
-	Window                                string
-	Err                                   string
+	// GPUPowerLoad is draw against the power limit; GPUSMActive the SMs'
+	// share of time with work, empty on a card nothing can profile.
+	GPUPowerLoad, GPUSMActive TimeSeries
+	Window                    string
+	Err                       string
 }

 // nodePanels are the charts, each a query scoped to this node. %s is the
@@ -89,6 +92,10 @@ var gpuPanels = []struct {
 		func(c *nodeCharts, t TimeSeries) { c.GPUTemp = t }},
 	{`llmgw_gpu_power_watts{node=%s}`, "",
 		func(c *nodeCharts, t TimeSeries) { c.GPUPower = t }},
+	{`llmgw_gpu_power_watts{node=%[1]s} / llmgw_gpu_power_limit_watts{node=%[1]s}`, "%",
+		func(c *nodeCharts, t TimeSeries) { c.GPUPowerLoad = t }},
+	{`llmgw_gpu_sm_active_percent{node=%s} / 100`, "%",
+		func(c *nodeCharts, t TimeSeries) { c.GPUSMActive = t }},
 }

 // NodePage shows one node. Its charts come from the metrics store when one
@@ -82,6 +82,7 @@
    downloadFailed: {{t "chat.js.download_failed"}},
    stageWaiting: {{t "chat.js.stage.waiting"}},
    stageDownloading: {{t "chat.js.stage.downloading"}},
+    stageChecking: {{t "chat.js.stage.checking"}},
    stageStarting: {{t "chat.js.stage.starting"}},
    stageReady: {{t "chat.js.stage.ready"}},
    of: {{t "chat.js.of"}},
@@ -288,6 +289,7 @@
  const stages = {
    waiting: say.stageWaiting,
    downloading: say.stageDownloading,
+    checking: say.stageChecking,
    starting: say.stageStarting,
    ready: say.stageReady,
  };
@@ -322,7 +324,7 @@
      for (;;) {
        const secs = Math.round((Date.now() - started) / 1000);
        let text = (stages[st.state] || st.state) + " — " + fill(say.seconds, { n: secs });
-        if (st.state === "downloading" && st.total) text += " · " + fill(say.of, { done: mib(st.done), total: mib(st.total) });
+        if ((st.state === "downloading" || st.state === "checking") && st.total) text += " · " + fill(say.of, { done: mib(st.done), total: mib(st.total) });
        label.textContent = text;
        // Indeterminate (no value) unless there is a real number: a bar that
        // sits at 0% for the engine start says it is stuck when it is not.
@@ -64,11 +64,13 @@
 {{/* Per card, because a model on eight cards making twice the tokens of one
     on two is the worse of the two; per watt-hour, because that is what the
     room is paying. */}}
-<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th class="num">GPUs</th><th class="num">{{t "dash.eff.tps"}}</th><th class="num">{{t "dash.eff.per_gpu"}}</th><th class="num">{{t "dash.eff.busy"}}</th><th class="num">{{t "dash.eff.draw"}}</th><th class="num">{{t "dash.eff.per_wh"}}</th></tr>
+<table class="rows"><tr><th>{{t "dash.col.model"}}</th><th class="num">GPUs</th><th class="num">{{t "dash.eff.tps"}}</th><th class="num">{{t "dash.eff.per_gpu"}}</th><th class="num">{{t "dash.eff.busy"}}</th><th class="num">{{t "dash.eff.power_load"}}</th><th class="num">{{t "dash.eff.sm_active"}}</th><th class="num">{{t "dash.eff.draw"}}</th><th class="num">{{t "dash.eff.per_wh"}}</th></tr>
 {{range .}}<tr><td>{{.Model}}</td><td class="num">{{num .GPUs 0}}</td><td class="num">{{num .TokensPerSec 1}}</td>
  <td class="num">{{num .PerGPU 1}}</td><td class="num">{{pc .Util 0}}</td>
+  <td class="num">{{if ge .PowerLoad 0.0}}{{pc .PowerLoad 0}}{{else}}—{{end}}</td><td class="num">{{if ge .SMActive 0.0}}{{pc .SMActive 0}}{{else}}—{{end}}</td>
  <td class="num">{{if .Watts}}{{num .Watts 0}} W{{else}}—{{end}}</td><td class="num">{{if .PerWh}}{{num .PerWh 0}}{{else}}—{{end}}</td></tr>
 {{end}}</table>
+<p class="note">{{t "dash.eff.note"}}</p>
 {{end}}
 {{with .IdleGPUs}}
 <h3>{{t "dash.idle.title"}}</h3>
@@ -282,6 +284,10 @@ document.querySelectorAll("form.build-choice select").forEach(function (s) {
  {{template "tschart" dict "t" .GPUBusy "title" (t "dash.fleet.gpu_busy") "note" (t "dash.fleet.gpu_busy_note")}}
  {{template "tschart" dict "t" .GPUMemory "title" (t "dash.fleet.gpu_memory") "note" (t "dash.fleet.gpu_memory_note")}}
 </div>
+<div class="chart-row">
+  {{template "tschart" dict "t" .GPUPowerLoad "title" (t "dash.fleet.gpu_power_load") "note" (t "dash.fleet.gpu_power_load_note")}}
+  {{if not .GPUSMActive.Empty}}{{template "tschart" dict "t" .GPUSMActive "title" (t "dash.fleet.gpu_sm_active") "note" (t "dash.fleet.gpu_sm_active_note")}}{{end}}
+</div>
 {{end}}
 <div class="chart-row">
  {{template "tschart" dict "t" .Queue "title" (t "dash.fleet.queue") "note" (t "dash.fleet.queue_note")}}
@@ -141,6 +141,10 @@
  {{template "tschart" dict "t" .GPUTemp "title" (t "node.chart.gpu_temp") "note" (t "node.chart.gpu_temp_note")}}
  {{template "tschart" dict "t" .GPUPower "title" (t "node.chart.gpu_power") "note" (t "node.chart.gpu_power_note")}}
 </div>
+<div class="chart-row">
+  {{template "tschart" dict "t" .GPUPowerLoad "title" (t "node.chart.gpu_power_load") "note" (t "node.chart.gpu_power_load_note")}}
+  {{if not .GPUSMActive.Empty}}{{template "tschart" dict "t" .GPUSMActive "title" (t "node.chart.gpu_sm_active") "note" (t "node.chart.gpu_sm_active_note")}}{{end}}
+</div>
 {{end}}
 <div class="chart-row">
  {{template "tschart" dict "t" .Load "title" (t "node.chart.load") "note" (t "node.chart.load_note")}}
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+"""Say which followed engines have published a version newer than the one llmgw installs.
+
+The versions llmgw installs are DefaultEngines in internal/config/config.go. An
+engine followed here is one whose releases are taken up as they come: each new
+one is checked against the flags llmgw gives it (llmgw ctl engine-check, and a
+replica on a card it runs on) and then moved to in DefaultEngines.
+
+Exits 1 when one is behind, so the scheduled workflow's failed run is the
+notice; 0 when all are current. No token needed: PyPI's JSON API is public.
+
+Usage:
+    scripts/engine_releases.py [--config internal/config/config.go]
+"""
+
+import argparse
+import json
+import re
+import sys
+import urllib.request
+
+# Engine name in DefaultEngines -> (PyPI project, where its changes are read).
+FOLLOWED = {
+    "freetoken": ("freetoken", "https://github.com/FlashML-org/FreeToken/releases"),
+}
+
+
+def pinned(config_path):
+    """The DefaultEngines map, as written in the Go source."""
+    src = open(config_path, encoding="utf-8").read()
+    block = re.search(r"var DefaultEngines = map\[string\]string\{(.*?)\n\}", src, re.S)
+    if not block:
+        sys.exit(f"no DefaultEngines in {config_path}")
+    return dict(re.findall(r'"([^"]+)":\s*"([^"]+)"', block.group(1)))
+
+
+def key(version):
+    """A version as numbers to compare; a pre-release sorts before its release."""
+    nums = [int(n) for n in re.findall(r"\d+", version.split("+")[0])]
+    pre = bool(re.search(r"(a|b|rc|dev)\d*$", version))
+    return (nums, 0 if pre else 1)
+
+
+def latest(project):
+    """The newest version PyPI has of a project that is not yanked or a pre-release."""
+    with urllib.request.urlopen(f"https://pypi.org/pypi/{project}/json", timeout=30) as r:
+        data = json.load(r)
+    versions = [
+        v for v, files in data["releases"].items()
+        if files and not all(f.get("yanked") for f in files) and not re.search(r"(a|b|rc|dev)\d*$", v)
+    ]
+    return max(versions, key=key) if versions else data["info"]["version"]
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--config", default="internal/config/config.go")
+    args = ap.parse_args()
+    pins = pinned(args.config)
+    behind = False
+    for engine, (project, changes) in FOLLOWED.items():
+        have = pins.get(engine)
+        if have is None:
+            print(f"{engine}: not in DefaultEngines")
+            behind = True
+            continue
+        new = latest(project)
+        if key(new) > key(have):
+            print(f"{engine}: {new} is out, llmgw installs {have} — {changes}")
+            behind = True
+        else:
+            print(f"{engine}: {have} is current")
+    return 1 if behind else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	220110ed0d	v0.16.2: changelog and README Release / build (push) Successful in 5m43s Details Release / notes (push) Successful in 4s Details CI / test (push) Successful in 11m11s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-11-02 08:32:34 +01:00
Ilker ManapandClaude Opus 5.5	a88624747e	A set wanted by several replicas on one node is fetched once CI / test (push) Failing after 7m23s Details Single files were shared between the shards and replicas asking for them at the same time; sets were not, and four replicas on one node's four cards wrote the same .part files over one another until all but one failed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-11-02 08:29:25 +01:00
Ilker ManapandClaude Opus 5.5	540b518fde	Rate limits counted across the control planes with router.active_active CI / test (push) Successful in 9m30s Details Each router's per-counter request and token counts, this minute's and the last, ride the standby's two-second exchange with the leader. The leader answers each standby with everyone else's sum, its own included, and puts the standbys' sum in force on its own router; each limiter adds the others' counts to its sliding window. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-11-01 17:30:08 +01:00
Ilker ManapandClaude Opus 5.5	95fb5eeb6e	BACKLOG: O2 tried on the PoC, two routers carry 1.5x the leader alone CI / test (push) Successful in 9m26s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-11-01 16:51:54 +01:00
Ilker ManapandClaude Opus 5.5	8b7eefef16	Standbys that answer the client API themselves, behind router.active_active CI / test (push) Successful in 9m35s Details A standby follows the leader's Ready replicas over the agent port, every two seconds, and reports back the replicas its router used and the on-demand models it was asked for. No engine keys travel: every control plane derives them from the cluster secret. Budgets are raised to the store's every 30 s. Unheard from the leader for a minute, a standby passes /v1 on as before. Off by default. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-11-01 13:47:47 +01:00
Ilker ManapandClaude Opus 5.5	f664592948	BACKLOG: N8 half done, ctl model context takes Strata and the longest answer alone CI / test (push) Successful in 9m21s Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 09:51:26 +01:00
Ilker ManapandClaude Opus 5.5	31f440ab6e	A served model's longest answer changed in place, and Strata's context CI / test (push) Successful in 9m29s Details max_output on POST /api/v1/models/{name}/context, --max-output on ctl model context, and a field in the console's Context length… form. Alone it changes only the catalog's limit, which the router holds max_tokens to, and starts no replica again; with a context it is set with it, never past it. Strata's context is now changed like the others', through its setup's --context, which writes the engine's start script at each start. Neither needs a model removed and added again any more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 00:15:38 +01:00
Ilker ManapandClaude Opus 5.5	f4b34dbf11	FreeToken, an engine for models larger than the card CI / test (push) Successful in 9m37s Details FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several times a card's size from one card, experts in RAM, from Hugging Face safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13 toolkit it compiles kernels with. Older cards are passed over: on cuda01's Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and in fp16 hit an illegal memory access on its first request. ft serve gets the model, address, card and served name; the key goes in FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are FREETOKEN_ variables naming paths or the agent's own settings. ctl model context sets --max-seq-len-override. Readiness now also waits for an engine whose /health answers 200 while its body says "loading", as FreeToken's does from the moment its HTTP server is up. FreeToken is followed release by release: a weekly workflow fails when PyPI has a version newer than DefaultEngines installs (scripts/engine_releases.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-31 22:10:05 +01:00
Ilker ManapandClaude Opus 5.5	c866b3a8ee	How much of each GPU works, beside whether anything does CI / test (push) Failing after 6m42s Details The busy figure is nvidia-smi's utilization.gpu: the share of time any kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy while the SMs had work 69% of the time and issued on 18% of their cycles. Busy is now said to be that, and beside it: each card's draw against its power limit (efficiency table, fleet and node charts, the console's GPU work chart), and where the profiling counters can be read, SMs active, SM occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's GPM on Hopper and later or a dcgm-exporter on the node (discovery.dcgm_exporter). The GPU map colours by work by default and keeps busy as a mode. A MIG instance is never published with its card's profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-31 20:11:26 +01:00
Ilker ManapandClaude Opus 5.5	7bfe70c8df	BACKLOG: the router profiled, the write per token removed, what is left CI / test (push) Successful in 9m22s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-31 15:50:15 +01:00
Ilker ManapandClaude Opus 5.5	5550fefaad	Stream events already in hand go to the client in one write, not one each Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-31 15:30:19 +01:00
Ilker ManapandClaude Opus 5.5	808c4724d2	BACKLOG: O1 measured, the router's ceiling on the PoC CI / test (push) Successful in 9m22s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-31 10:45:59 +01:00
Ilker ManapandClaude Opus 5.5	9ff49cad8e	Requests missing the cache for the same key at once share one read and one allowance CI / test (push) Successful in 9m51s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-31 09:30:18 +01:00
Ilker ManapandClaude Opus 5.5	524f029603	External providers left out of admission's saturation; the scale-out section in the backlog CI / test (push) Successful in 9m25s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-30 19:13:15 +01:00
Ilker ManapandClaude Opus 5.5	be54f1617b	A mirror download cut off carries on from where it got to, and is tried again CI / test (push) Successful in 10m36s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-30 17:47:32 +01:00
Ilker ManapandClaude Opus 5.5	812eb8aac1	Regenerate the admin API document for the fetch phase CI / test (push) Successful in 9m33s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-30 15:07:16 +01:00
Ilker ManapandClaude Opus 5.5	76746715ec	A set's files checked four at once, the check said as it goes and not taken for a stuck download CI / test (push) Failing after 6m51s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-30 14:58:44 +01:00
Ilker ManapandClaude Opus 5.5	d5dcf78836	A set's download reported as the whole set's, not the part being fetched CI / test (push) Successful in 9m15s Details Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-30 14:48:45 +01:00