Check llmgw's engine flags against real vLLM and llama.cpp
CI / test (push) Successful in 1m25s

The launch arguments had been read from documentation and never from an
engine. Now they have been:

  vllm 0.29.0    --host --port --tensor-parallel-size --served-model-name
                 --distributed-executor-backend      all accepted
  llama.cpp b10991  --model --host --port --metrics --alias --parallel
                 --n-gpu-layers                      all accepted

Getting there found three faults in the checker rather than in llmgw.

It checked every command's flags against the engine's help, but a shard can be
several programs — vLLM under Ray is `ray start` and then `vllm serve` — and
they have different vocabularies. It reported --head and --num-gpus as
missing, which is true of vllm and irrelevant, because ray is what receives
them. Each command is now checked against its own help, and one that cannot be
run is reported as skipped rather than failed.

vLLM does not give up its whole help for --help: it prints a summary of
section names and expects --help=all. Checked against the summary, all five
flags looked missing. The warning about truncated help did not fire, because
the summary happens to mention --help= while listing how to ask for more.

And vLLM refuses to print anything at all on a machine with no accelerator
unless VLLM_TARGET_DEVICE says what to assume. Asking an engine about its
flags should not require the hardware it runs on.

Two things remain unchecked and are written down as such: the Ray sidecar,
because vLLM 0.29 does not install Ray, and SGLang, because it is not
installed anywhere. `launch_mode: mp` avoids Ray entirely and is worth
preferring until someone checks it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 16:52:11 +02:00
1 parent 5b425c32e5
commit f68fbf7c49
3 files changed
+126 -46

No files matched your search

+9
View File
@@ -92,6 +92,15 @@ a flag that only appears there is exactly the one nobody notices has gone — an
checks every flag against the engine's own `--help`. No GPU required: a CLI
prints its help on any machine.
Some programs do not give up their whole help for `--help`. vLLM prints a
summary of section names and expects `--help=all`; checking against the summary
reports every flag as missing, confidently and wrongly. `engine-check` knows
that one. If you add an engine, check that what you get back is the full list.
vLLM also refuses to do anything at all on a machine with no accelerator,
including print its help, unless `VLLM_TARGET_DEVICE` says what to assume.
`engine-check` sets it.
Give it the **whole** help. Truncating llama.cpp's 725 lines to 400 reports five
flags as missing that the running cluster uses; the command warns when the text
it was given does not document `--help` itself, which is the cheapest sign it