Commit Graph
8 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 5732a5ebda The serve forms start at the longest context that fits
CI / test (push) Successful in 9m26s
The mirror page's form began every GGUF model at 4096 tokens, the
console's at 4096 or the plan's figure. Both now start at what the model
was trained for, cut to the most whose KV cache fits beside the weights
where the form places it; the mirror page's slots are as many of that, up
to four, as the room left holds. Plan carries the trained context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:38:41 +01:00
Ilker ManapandClaude Opus 5.5 6403e61f4b How fast a model answers on the CPU, before it is served
CI / test (push) Successful in 9m31s
Each agent measures one core's memory rate beside the whole machine's; the
estimate takes a replica's threads at about one core's rate each, up to the
machine's, calibrated by llama-bench on the PoC. The serve form shows it
beside a card's, warns when slow, and asks for the CPU on purpose when a
reply would take minutes; ctl model plan, the console and serve jobs say
it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:52:15 +01:00
Ilker ManapandClaude Opus 5.5 2eb96f4872 The plan says why Strata's DeepSeek does not run here
CI / test (push) Successful in 11m38s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:09:09 +01:00
Ilker ManapandClaude Opus 5.5 845de8eef4 The mirror page's Strata tab; Strata's ready-made packs served with Strata fetching them, strata-ple named
CI / test (push) Successful in 10m35s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:06:46 +01:00
Ilker Manap d6f8f433f4 CHANGELOG: one list
CI / test (push) Successful in 9m18s
2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 8ffde2b80c A big model's download no longer runs out its startup timeout
CI / test (push) Successful in 5m18s
The startup timeout runs from when a node's download last moved, so a
model of hundreds of gigabytes on a slow mirror is not failed while it
arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO)
on every node, and a layout given by hand keeps expert parallelism for a
model of experts. docs/large-models.md: how a model is split, what each
way needs of a card, and a checklist for bringing one up on a fabric.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 09:45:31 +02:00
Ilker ManapandClaude Opus 5.5 4fe9b46451 A plan for a big model: experts spread, any number of nodes, memory said
A mixture-of-experts model on more than one card is served with vLLM's
--enable-expert-parallel; config.json's experts are read by any of their
names, and a model that also reads pictures is sized by its text_config.
Tensor parallelism past the KV heads is allowed when it is a multiple of
them (Qwen3-235B, four KV heads, runs on eight cards), which the plan
had refused. Several nodes are offered in any number, not only powers of
two. Each way says what it needs of a card, part by part, and of all its
cards; the serve form checks a context longer than planned still fits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 08:45:33 +02:00
Ilker ManapandClaude Opus 5.5 54db7aecb7 Plan a repository's files against the fleet, and serve the one that suits it the way it fits
The control plane reads the model's shape from one GGUF header or its
config.json and lists, for every file, the best way it runs here: one card,
several of a node's (by layer for llama.cpp, tensor-parallel for vLLM),
several nodes' over one fabric (vLLM, pipeline-parallel between them), or a
CPU node; what each card holds and whether it is free now. ctl model plan
prints it; model serve takes its file and way unless told, gives a GGUF on
GPUs a GPU profile, and with --fallbacks adds the next best ways. The
console shows it per file and asks for the way and the fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:03:48 +02:00