A bundle carries what a cluster would download — llmgw's release with its
signed checksums, every llama.cpp build per architecture with GitHub's
checksums, model files pinned and checked against Hugging Face's — into a
directory or a tar whose manifest sha256 is carried apart. Import checks
everything, the release against the signing key, before anything is put in
the mirror. With offline.mirror set, nodes install llama.cpp from the
mirror against checksums the control plane holds and sends over the agent
stream; releases default to the mirror for node add, ctl upgrade and agent
upgrades. server init and the setup form install from a bundle.
The Helm chart runs one control plane with SQLite on a claim or an external
PostgreSQL, Secrets for the agent certificate and admin token, Services,
Ingress and ServiceMonitor; deploy/container/Dockerfile builds its image.
GPU nodes stay outside Kubernetes, as docs/kubernetes.md explains.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mirror keeps where it fetched a file beside it, and reports that and the
checksum after it restarts. A model whose files the mirror cannot trace is
found on the registry: a repository holding a file of that name that has the
commit the mirror's directory is named after, with every file at the same
size, the most downloaded first. ctl model origin and the console set one by
hand, checked the same way. Commits compare shortened.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The explorer fetches a safetensors set with its config and tokenizer;
the mirror gives out a version's manifest only once nothing is
downloading and every shard its index names is there, with each file's
sha256, which nodes check; mirror add takes several URLs; a repository
page is refused as a source.
Refs #2 (item 9).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two places built a path from a name somebody else supplied and checked only
for slashes.
The mirror refused "a/b" as a model name but took "..", which is a single
path element too: a fetch for model ".." wrote beside the mirror root rather
than inside it. Model, version and file names now have to be one element
that does not start with a dot — the mirror's own checksum files are dotted
— on the fetch path and the API alike.
The agent took the engine version from the control plane as given and made
a directory of it under /opt/llmgw/engines, and a RemoveAll before replacing
one. A version of "../x" would have done both outside the install root, as
root. The agent now installs only versions that look like one — letters,
digits and ._+- — whatever the control plane sent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mirror would happily hold a four-shard model that nothing could serve. A
catalog entry names one source, the agent fetched one URL, and so a version of
several files had no way to be described — which is every safetensors model,
and every GGUF too large to ship whole.
A source ending in a slash names a version rather than a file. The agent fetches
every file in it into one directory and hands the engine the directory: vLLM
reads the shards and the config out of the folder, and llama.cpp is given the
first part and finds the rest beside it.
The trailing slash is the whole signal. A flag beside the source would be a
second place for the two to disagree, and a URL already distinguishes a
directory from a file everywhere else.
The mirror answers a version with a manifest instead of a file listing. A plain
file server gives HTML there, and something that has to know what a version
consists of would be left parsing it. It is on the byte listener, not the
management API: fetching weights is the one thing every node does, and making
it depend on the administrative port would undo the split that keeps that port
closeable.
Built from the directory rather than a file written beside it. A manifest on
disk is a second record of what is there, and the two disagree the first time
somebody copies a file in by hand — which the mirror explicitly allows.
The whole set or nothing. A directory with three of four shards loads as a
broken model rather than failing to load, so a completion marker is written
last and an interrupted fetch starts the missing files again rather than being
found and served. Files already there at the right size are not fetched twice.
Progress is reported under the version's address, not each file's: the shards
waiting on it asked for the version, and progress under a file's own URL
reaches nobody.
A filename from the manifest is refused if it contains a separator. The mirror
is a place an operator copies files into by hand, so what it reports is not
automatically safe to join onto a path.
An HTML listing where a manifest was expected says so. "invalid character '<'"
sends somebody looking at the wrong thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror was a directory and a discipline written down in a README. This
makes the three operations that matter into commands, on the mirror host,
where `llmgw keys` already sets the precedent of operator tooling that works
on files rather than through the API.
The gateway still never writes to the mirror. That property was chosen
deliberately — a mirror writable by the thing reading from it is a way to
replace a model with one every node then fetches without complaint — and a
person running a command does not take it away. `serve` states it rather than
inheriting it, so a handler added later cannot quietly make the mirror
writable.
`add` downloads to a temporary name and renames, for the reason the agent's
fetcher does: a mirror is copied to every node, so a partial file here is a
partial file everywhere. It verifies the length the server promised, reports a
checksum — the mirror is the one place a corrupt file reaches every node
rather than being caught once — and refuses to replace an existing version,
which is the whole point of the layout.
`unused` refuses to answer when it cannot read the catalog. An empty catalog
and an unreachable one produce the same list, everything, and the obvious next
step after that list is deletion. It compares by path rather than by spelling,
because a base typed with a different scheme or a trailing slash would
otherwise report a file in use as unused.
`remove` checks the catalog first and says to retire the entry before deleting
the file, because a node that has to refetch a deleted file cannot, and that
failure arrives later, on the node, looking like something else.
Verified end to end: Range requests answer 206, which is what makes an
interrupted fetch on a node resume; writes answer 405; a traversal attempt
answers 404.
Note: internal/agent/supervisor's TestSidecarRunsBeforeEngine fails under full
suite load on main as well as here — it asserts that one shell's echo lands
before another's, which the supervisor does not promise. Everything else
passes. Left alone rather than fixed in a commit about mirrors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
