An embedding model is served for embedding, with many slots
ctl model serve served one as a chat model: no --embedding, four slots, a 512-token batch, embeddings unset, and a chat smoke test it could not pass. A GGUF that pools (<arch>.pooling_type), or --embeddings, gets --embedding, 32 slots of 1024 tokens unless sized otherwise, a batch that holds a whole passage, the embeddings capability, and one vector asked for as its smoke test (canaries and updates too). An import of a hundred thousand chunks had gone at the pace of four slots with the card a third busy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f5eb6963af
commit
efffdb66b7
8 files changed
+142
-6
No files matched your search
@@ -36,6 +36,12 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
### Fixed
|
||||
|
||||
- `llmgw ctl model serve` served an embedding model as a chat model: no
|
||||
`--embedding`, four slots, a 512-token batch, `embeddings` unset and a
|
||||
"say hello" smoke test it could not pass. A model whose GGUF says it
|
||||
pools (or `--embeddings`) is served for embedding: 32 slots of 1024
|
||||
tokens unless sized otherwise, a batch that holds a whole passage, and
|
||||
one vector asked for as its smoke test.
|
||||
- `llmgw ctl upgrade` checks every node's version once the waves are done
|
||||
and fails naming any not on the release, and says when the control plane
|
||||
has no nodes registered; `llmgw upgrade` on a control plane says how many
|
||||
|
||||
Reference in new issue
Block a user