A GGUF is servable when a llama.cpp that engine_overrides give some nodes loads it
CI / test (push) Failing after 9m58s
CI / test (push) Failing after 9m58s
`ctl model serve`, its plan and model updates judged a file only against the cluster's own llama.cpp, so a GPU node given a newer build could not be served an architecture only that build knows: Inkling 2 Mini (inkling2_loop) was refused for b10991 with cuda01 on b12606. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0ffd633bf6
commit
09ebe77513
8 files changed
+99
-10
No files matched your search
@@ -679,7 +679,7 @@ func newServerCmd() *cobra.Command {
|
||||
if adminAPI != nil && catalogSvc != nil {
|
||||
serve := modelserve.New(modelserve.Deps{
|
||||
Registry: registry, Mirror: mc, Catalog: catalogSvc, Fleet: admin,
|
||||
LlamaVersion: cfg.WantedEngines()["llamacpp"], Log: log,
|
||||
LlamaVersion: cfg.WantedEngines()["llamacpp"], LlamaOverrides: cfg.OverrideVersions("llamacpp"), Log: log,
|
||||
// Updates: the catalog in force, the service that
|
||||
// changes it, and what replaces replicas.
|
||||
Current: func() []catalog.Model { return admin.Catalog().Models() },
|
||||
|
||||
Reference in new issue
Block a user