2 Commits

Include renames

Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5	b5177b70f6	Stop tool-check blaming the engine for what the model did ... It had one verdict for a failed attempt — "This is the engine, not the model. Choosing a bigger model will not help" — and that verdict was wrong, in a way that sends someone off rebuilding llama.cpp when the fix is a one-line catalog change. Measured on the cluster. The engine renders the tool prompt correctly: the <tools> block and the instruction to wrap the call in <tool_call> are both in the prompt, which I confirmed through the engine's own /apply-template. The coder models simply do not use that wrapper — qwen2.5-coder-3b puts the call in a markdown fence, qwen2.5-coder-7b invents <function-call> — so the parser, which looks for <tool_call>, has nothing to find and leaves it in the content. qwen2.5-3b-instruct, same engine, same build, same flags, returns a populated tool_calls array. So it was the model all along, and the advice against trying another one was the most expensive sentence in the output. A failed attempt is now two verdicts, told apart by whether the reply used the wrapper its template asked for. Wrapped and unparsed is genuinely the engine's parser. Unwrapped is the model, and the fix is one trained for the format rather than a bigger one from the same family. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-18 08:31:09 +02:00
Ilker ManapandClaude Opus 5	84896c0ad2	Add ctl tool-check, so an afternoon becomes one command ... The catalog now declares whether a model supports tools, which makes it a claim. This is the claim that can be checked, and the one worth checking: the gateway forwards a tools array to any model, and an engine that does not return structured tool_calls looks exactly like a model that ignores its tools. Every OpenAI-compatible client then behaves as though the model is broken. It separates the two diagnoses, which have the same symptom and different fixes. The model ignoring its tools is fixed by choosing another model. The model making the call and the engine handing it back as prose is fixed by changing engine build, and no larger model will help — which is what today's investigation established only after eliminating the gateway, the model size, two chat templates and tool_choice in turn. Against the live engine it says: the model called the tool and the engine did not parse it. {"name": "get_weather", "arguments": {"city": "Istanbul"}} This is the engine, not the model. engine-check asks whether an engine still accepts our flags; this asks whether it behaves. Flags were the failure we had seen, so they were the failure we checked for. Non-zero exit, so a rollout can be gated on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-16 23:30:02 +02:00