response_header sounds like a check that the endpoint is alive, and for a
streaming request it is: headers arrive before generation starts. For a
non-streaming request the engine sends nothing until the completion is
finished, so the same setting is the total generation budget wearing a
liveness timeout's name.
At 60s that was invisible on GPUs and a hard cap of roughly a hundred
tokens on qwen2.5-coder-14b at 1.7 tokens a second. A request for 120
tokens failed at exactly one minute, and neither the setting's name nor the
error said why.
Two transports now: the short header timeout for streaming, where it still
catches a sick endpoint, and a generation budget for non-streaming,
defaulting to ten minutes. One number could not mean both things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
