Describe v0.8.1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
20deb22dfa
commit
f90daa1ee9
1 file changed
+48
@@ -18,6 +18,54 @@ is only released once its **exit criterion** is met, not when its code lands.
|
|||||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||||
APIs change between minors without a deprecation period.
|
APIs change between minors without a deprecation period.
|
||||||
|
|
||||||
|
## v0.8.1 — 2026-09-28
|
||||||
|
|
||||||
|
Nothing here changes a configuration format: every file that worked with
|
||||||
|
v0.8.0 still works. What changes is how much of it needs writing, and how much
|
||||||
|
of running the cluster can be done from a page rather than a shell.
|
||||||
|
|
||||||
|
### A node describes itself
|
||||||
|
|
||||||
|
An agent file used to name its node, its address, the path to llama-server and
|
||||||
|
its weight directories — different on every node and kept true by hand. The
|
||||||
|
agent now finds these out: its id is the hostname, its address is the one on
|
||||||
|
the route to the control plane, llama-server is looked for on PATH and in
|
||||||
|
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
|
||||||
|
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
|
||||||
|
Memory is re-read on every heartbeat and installed engines once a minute, so a
|
||||||
|
VM given more RAM or a node given an engine becomes a candidate without an
|
||||||
|
agent restart; the control plane logs both. The agent file shrinks to the
|
||||||
|
control plane's addresses and the TLS files, the same on every node.
|
||||||
|
|
||||||
|
### Models from the mirror, without writing YAML
|
||||||
|
|
||||||
|
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
|
||||||
|
and serve a mirrored version from its row — name, source, engine and an
|
||||||
|
estimated memory requirement filled in. A model can say what it is for, and
|
||||||
|
`/v1/models` shows it. Placement no longer hands the same memory to two models
|
||||||
|
or puts an engine on a node that lacks it, and says when a replica runs an
|
||||||
|
edited profile.
|
||||||
|
|
||||||
|
### Tenancy, quotas and usage on pages
|
||||||
|
|
||||||
|
The tenancy tree, quotas and budgets are managed from the admin pages; an
|
||||||
|
organization's budget bounds its teams, a group's quota counts against the
|
||||||
|
group, and a budget that cannot be spent against says so. `/admin/usage`
|
||||||
|
charts requests, tokens and spend over time by organization, team, project or
|
||||||
|
person, and a request is priced by the model that served it.
|
||||||
|
|
||||||
|
### Watching the cluster
|
||||||
|
|
||||||
|
The cluster page has its own charts, backed by a year of Prometheus history,
|
||||||
|
with alerts that were made to fire before being trusted — including for the
|
||||||
|
disk the metrics share with the models.
|
||||||
|
|
||||||
|
### Chat
|
||||||
|
|
||||||
|
The picker offers each model once rather than once per alias, never asks for
|
||||||
|
more tokens than the model allows, and offers a model that loads on demand —
|
||||||
|
sending it a message is what loads it.
|
||||||
|
|
||||||
## v0.8.0 — 2026-09-27
|
## v0.8.0 — 2026-09-27
|
||||||
|
|
||||||
The store stops being a file on one machine's disk. PostgreSQL, a way to move
|
The store stops being a file on one machine's disk. PostgreSQL, a way to move
|
||||||
|
|||||||
Reference in new issue
Block a user