An embedding model is served for embedding, with many slots

ctl model serve served one as a chat model: no --embedding, four slots, a
512-token batch, embeddings unset, and a chat smoke test it could not
pass. A GGUF that pools (<arch>.pooling_type), or --embeddings, gets
--embedding, 32 slots of 1024 tokens unless sized otherwise, a batch that
holds a whole passage, the embeddings capability, and one vector asked for
as its smoke test (canaries and updates too). An import of a hundred
thousand chunks had gone at the pace of four slots with the card a third
busy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-14 09:19:57 +02:00
1 parent f5eb6963af
commit efffdb66b7
8 files changed
+142 -6

No files matched your search

+1 -1
View File
@@ -689,7 +689,7 @@ func (s *Server) replace(ctx context.Context, j *Job, name string) error {
}
s.step(j, "replicas", Done, fmt.Sprintf("%d replaced", len(old)))
s.step(j, "smoke test", Running, "asking "+ready[0].ID+" to say hello")
reply, tps, err := s.smoke(ctx, ready[0], name)
reply, tps, err := s.smoke(ctx, ready[0], name, s.embeds(name))
if err != nil {
s.step(j, "smoke test", Failed, err.Error())
return err