Commit Graph
20 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 cba4897987 Deleting unused model files says what went, and sets go whole
CI / test (push) Failing after 6m44s
A sweep counted every file the nodes were asked to delete as freed as soon
as it asked, and a node refused a set's directory as not empty, so tens of
gigabytes were reported freed and nothing was. The agent removes a set's
directory whole (a link only as a link), and the control plane waits for
each node to list its files again, saying what it no longer lists and what
it kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:29:16 +01:00
Ilker ManapandClaude Opus 5.5 1a3f385be4 Every setting of a served model from the mirror page, and its replicas per profile
CI / test (push) Successful in 10m51s
The row of a served model leads to its card, which now changes its context
and longest answer, its cards, its replicas on each profile with a profile
on CPUs to add and its speed beside, stops, starts and restarts it, and
removes it with its files or without. Stopped on the row stops the model
rather than setting its scaling to nothing. The console gains Replicas per
profile, and ctl models shows the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:40:06 +01:00
Ilker ManapandClaude Opus 5.5 31f440ab6e A served model's longest answer changed in place, and Strata's context
CI / test (push) Successful in 9m29s
max_output on POST /api/v1/models/{name}/context, --max-output on ctl model
context, and a field in the console's Context length… form. Alone it changes
only the catalog's limit, which the router holds max_tokens to, and starts no
replica again; with a context it is set with it, never past it.

Strata's context is now changed like the others', through its setup's
--context, which writes the engine's start script at each start. Neither
needs a model removed and added again any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 00:15:38 +01:00
Ilker ManapandClaude Opus 5.5 f4b34dbf11 FreeToken, an engine for models larger than the card
CI / test (push) Successful in 9m37s
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.

ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.

Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.

FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 22:10:05 +01:00
Ilker ManapandClaude Opus 5.5 12464ce0f6 Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream
CI / test (push) Successful in 10m2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:14:02 +01:00
Ilker ManapandClaude Opus 5.5 59187cdae5 A model can be stopped and kept, or removed with its files from the mirror and the nodes
CI / test (push) Successful in 10m37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 20:45:48 +01:00
Ilker ManapandClaude Opus 5.5 e1cff0d4e3 A served model can be moved to more cards, or fewer
CI / test (push) Successful in 10m28s
llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:40:49 +01:00
Ilker ManapandClaude Opus 5.5 986981ac81 A served model's context can be changed, engine and memory with it
CI / test (push) Successful in 12m0s
ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context
and the console's Context length…: the catalog, the engine's arguments and
the KV memory change together, refused past the trained context or what the
cards hold, then the replicas restart one by one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 15:50:04 +01:00
Ilker ManapandClaude Opus 5.5 a8bb417618 The guardrails' judge has its own command, and its trouble is shown
llmgw ctl guard judge status lists each judge model the policies ask:
in the catalog, as a judge, replicas ready, its last failure to answer, and
what is wrong in words. enable serves the missing one through model serve -
Llama Guard 3 1B on CPUs, role guard, under the name the policies give it -
with --replace for a model of that name that is not a judge; the console has
it under Do. GET /api/v1/guard/judges, and the summary's judges, put the
problem on ctl status and the console's Overview.

A Llama Guard file served from the mirror page is added as a judge, under
the name the policies ask, rather than as a model clients use the policies
never find. ctl says plainly when the API answers with a page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 10:47:37 +02:00
Ilker ManapandClaude Opus 5.5 0d3eb82b9b Facility-aware placement: racks, PDUs, holds and power caps
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:17:49 +02:00
Ilker ManapandClaude Opus 5.5 f0477f140d Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).

- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
  non-zero below the score. Suites by name from control.eval_dir, or the
  two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
  standing gate (ctl model gate, eval: in the catalog): a promote asks the
  canary first and leaves it below the score; an update tries the new
  weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
  gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:04:43 +02:00
Ilker ManapandClaude Opus 5.5 889be70b8f Canary on the console's model menu; docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 20:33:59 +02:00
Ilker ManapandClaude Opus 5.5 a1b1663e9d A btop-like monitor in the console, from Prometheus; node exporters discovered and installed
Monitor draws CPU, memory, GPUs, requests, tokens, latency, network and disk
as Braille charts over 5m/1h/24h/7d, then per node, card and model
sparklines; node and model pages have their own. History is read from
monitoring.prometheus through POST /api/v1/metrics/range, or without it
from what the console saw. The control plane serves the cluster's machines
for Prometheus http_sd at /prometheus/targets on its metrics port, and node
add installs prometheus-node-exporter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:51:05 +02:00
Ilker ManapandClaude Opus 5.5 e12996f7dd Serve and update models in the background from the console, and follow them under Jobs
The console starts a serve or update job over the admin API and stays,
instead of closing to follow the download. Jobs lists them with their
progress and steps; the sidebar counts those running. ctl model serve and
update take --detach; ctl jobs lists jobs and follows one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:27:02 +02:00
Ilker ManapandClaude Opus 5.5 c2be429554 Tab or down from a form's Apply goes round to the first field instead of crashing the console
CI / test (push) Successful in 4m14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:12:19 +02:00
Ilker ManapandClaude Opus 5.5 f36c46a881 Trace weights fetched before origins were kept, and let an origin be given by hand
The mirror keeps where it fetched a file beside it, and reports that and the
checksum after it restarts. A model whose files the mirror cannot trace is
found on the registry: a repository holding a file of that name that has the
commit the mirror's directory is named after, with every file at the same
size, the most downloaded first. ctl model origin and the console set one by
hand, checked the same way. Commits compare shortened.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 00:24:41 +02:00
Ilker ManapandClaude Opus 5.5 44425193bd Track the registry's new weights for served models, and move a model to them or back
A model served from the registry records its origin: repository, commit and
each file's checksum; one served before is worked out from the mirror's
record of where it fetched the file. Every mirror.update_check (6h) the
control plane asks whether those files changed there. ctl model updates
lists it, ctl model update fetches beside the old files, checks, swaps and
replaces replicas one at a time, ctl model rollback goes back. The console
shows it on Models and offers the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 00:12:20 +02:00
Ilker ManapandClaude Opus 5.5 a96b1a54b2 Hugging Face in the console: models that chat when browsing, ages read at a glance, long menus scroll
CI / test (push) Successful in 6m1s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:47:26 +02:00
Ilker ManapandClaude Opus 5.5 2bf43ed90c Search Hugging Face from the console and serve what is found
CI / test (push) Successful in 5m3s
Serve a model is a view of the registry's GGUF models, trending or searched;
Enter on one lists its files, and the one chosen is served with
llmgw ctl model serve. A form of one field is applied by Enter on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:44:51 +02:00
Ilker ManapandClaude Opus 5.5 c7d5ee16b5 Act on the console's rows: scale, class, owner, resources, adapters, restart, remove; cordon, drain, engines, upgrade; quotas; cluster tasks
Enter on a row of Models, Nodes or Replicas opens what can be done to it,
as forms and questions over the admin API. A Quotas view sets and clears
capacity quotas. Do gains serving a model, upgrading the cluster,
restarting every replica and a backup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:23:12 +02:00