Serve a model from the registry with one command
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m50s

Adding a model took four steps and a YAML file — plan it, fetch it into the
mirror, write its catalog entry with a memory figure somebody estimated,
scale it — and whether it would load was found out at the end.

`llmgw ctl model serve OWNER/REPO` does it as one job in the control plane,
where the registry, the mirror, the catalog and the fleet already are:

- the registry: the GGUF file set matching --quant (Q4_K_M by default) or
  --file, leaving out vision projectors and importance matrices, and naming
  what the repository does have when nothing matches; gated repositories are
  refused;
- the mirror: fetched at the repository's revision, with progress, and each
  file's sha256 compared with what the registry published — the mirror is
  the one place a corrupted download is caught once rather than loaded on
  every node;
- the check: whether the nodes' llama.cpp loads the architecture, and the
  memory it needs from its own header;
- the catalog, then a wait for a replica that stops early, with the reason,
  when the model is parked or fits on no node;
- a smoke test: one short request to the replica, with its engine key, and
  the answer and tokens per second.

ctl starts the job and prints each step as it changes; a download of hours
does not depend on the terminal. --on-demand adds the model without starting
a replica. Jobs are kept in memory until the control plane restarts.

The catalog name a repository suggests moved into the registry package, so
the mirror page and this agree on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 03:05:13 +02:00
1 parent 116e75f8cb
commit bd12b0909a
9 files changed
+946 -8

No files matched your search

+41
View File
@@ -10,6 +10,7 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"io"
"net/http"
@@ -33,6 +34,8 @@ import (
type API struct {
// releases is where release binaries are fetched from, for upgrades.
releases string
// serve runs registry-to-replica jobs; nil without a mirror and a store.
serve *modelserve.Server
models Models
capacity Capacity
@@ -83,6 +86,8 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
mux.HandleFunc("POST /api/v1/restart", a.restart)
mux.HandleFunc("POST /api/v1/models/serve", a.serveModel)
mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
}
@@ -1204,3 +1209,39 @@ func (a *API) upgradeNode(w http.ResponseWriter, r *http.Request) {
}
writeJSON(w, http.StatusOK, map[string]string{"node": r.PathValue("id"), "version": req.Version})
}
// SetServe supplies what serves a model from the registry.
func (a *API) SetServe(s *modelserve.Server) { a.serve = s }
func (a *API) serveModel(w http.ResponseWriter, r *http.Request) {
if a.serve == nil {
writeError(w, http.StatusNotImplemented,
"serving from the registry needs a mirror and a store, and this control plane lacks one")
return
}
var req modelserve.Request
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
req.By = r.Header.Get("X-Llmgw-Actor")
job, err := a.serve.Start(req)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusAccepted, job)
}
func (a *API) job(w http.ResponseWriter, r *http.Request) {
if a.serve == nil {
writeError(w, http.StatusNotFound, "no such job")
return
}
job, ok := a.serve.Get(r.PathValue("id"))
if !ok {
writeError(w, http.StatusNotFound, "no such job; jobs are kept only until the control plane restarts")
return
}
writeJSON(w, http.StatusOK, job)
}