serve: tool calls declared, slots that hold the context, longer answers
CI / test (push) Failing after 2m41s

A --context-length 32768 --ctx-pool 32768 model got llama.cpp's default four
slots of 8192, and refused what the gateway admitted; --parallel is now
what the pool holds of the context. max_output was 1024 for every model,
which cut an editor's agent off mid-file; it is now a quarter of the
context, from 1024 to 8192. --tools writes capabilities.tools, which the
Continue configuration now reads, as it reads max_output.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 09:30:40 +02:00
1 parent df6d243ee7
commit 481523ca46
8 files changed
+111 -8

No files matched your search

+2
View File
@@ -73,6 +73,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
f.BoolVar(&req.Tools, "tools", false,
"say it makes tool calls (for agents and editors); check a running one with llmgw ctl tool-check")
f.StringVar(&req.GuardFormat, "guard-format", "",
"serve it as the guardrails' judge (role: guard), read as llamaguard3, shieldgemma or json; hidden from clients")
f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")