Serve a model from the registry with one command
Adding a model took four steps and a YAML file — plan it, fetch it into the mirror, write its catalog entry with a memory figure somebody estimated, scale it — and whether it would load was found out at the end. `llmgw ctl model serve OWNER/REPO` does it as one job in the control plane, where the registry, the mirror, the catalog and the fleet already are: - the registry: the GGUF file set matching --quant (Q4_K_M by default) or --file, leaving out vision projectors and importance matrices, and naming what the repository does have when nothing matches; gated repositories are refused; - the mirror: fetched at the repository's revision, with progress, and each file's sha256 compared with what the registry published — the mirror is the one place a corrupted download is caught once rather than loaded on every node; - the check: whether the nodes' llama.cpp loads the architecture, and the memory it needs from its own header; - the catalog, then a wait for a replica that stops early, with the reason, when the model is parked or fits on no node; - a smoke test: one short request to the replica, with its engine key, and the answer and tokens per second. ctl starts the job and prints each step as it changes; a download of hours does not depend on the terminal. --on-demand adds the model without starting a replica. Jobs are kept in memory until the control plane restarts. The catalog name a repository suggests moved into the registry package, so the mirror page and this agree on it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
116e75f8cb
commit
bd12b0909a
9 files changed
+946
-8
No files matched your search
@@ -28,7 +28,8 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
|
||||
Use: "model",
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client))
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newModelServeCmd(client))
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user