A served model's longest answer changed in place, and Strata's context
CI / test (push) Successful in 9m29s
CI / test (push) Successful in 9m29s
max_output on POST /api/v1/models/{name}/context, --max-output on ctl model
context, and a field in the console's Context length… form. Alone it changes
only the catalog's limit, which the router holds max_tokens to, and starts no
replica again; with a context it is set with it, never past it.
Strata's context is now changed like the others', through its setup's
--context, which writes the engine's start script at each start. Neither
needs a model removed and added again any more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f4b34dbf11
commit
31f440ab6e
9 files changed
+200
-37
No files matched your search
@@ -221,7 +221,7 @@ var Routes = []Route{
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/update", Summary: "Move a model to its repository's newer weights",
|
||||
Description: "With a quality suite, or the model's standing gate, the new weights are asked it first as a hidden canary.",
|
||||
Request: modelserve.Change{}, OptionalBody: true, Response: modelserve.Job{}, Status: http.StatusAccepted},
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/context", Summary: "Change a served model's context: the catalog, its engines' arguments and memory, then its replicas restarted",
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/context", Summary: "Change a served model's context and longest answer: the catalog, its engines' arguments and memory, then its replicas restarted; max_output alone changes only the catalog and restarts nothing",
|
||||
Request: modelserve.ContextRequest{}, Response: modelserve.ContextChange{}},
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/gpus", Summary: "Spread a served model over more cards or fewer: its profile, checked against its KV heads, its cards' memory and the nodes, then its replicas restarted",
|
||||
Description: "Tensor parallel for vLLM, SGLang and TensorRT-LLM, a layer split for llama.cpp, on one node. With no room beside the running replicas, in_place stops each first.",
|
||||
|
||||
Reference in new issue
Block a user