Replicas counted per profile: seven on cards and four on CPUs under one name

A profile's replicas, set on any of a model's profiles, is how many of its
replicas run there; the scaling becomes the sum. The reconciler starts the
ones short of their profile before stopping the ones over, so a change of
split never runs fewer. ctl model replicas sets them, adding a profile on
CPUs to a GGUF model; the router already fills the higher-priority pool
first, which the CPU profile is put below.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-03 09:21:50 +01:00
1 parent 6403e61f4b
commit e23732fbbb
13 files changed
+585 -5

No files matched your search

+11
View File
@@ -295,6 +295,7 @@ rather than repeated.
| `keep_within` | `string` | [Profile](#profile) |
| `weight` | `float64` | [Profile](#profile) |
| `priority` | `int` | [Profile](#profile) |
| `replicas` | `int` | [Profile](#profile) |
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
| `kv_events` | `bool` | [Profile](#profile) |
| `gpu` | `*GPUNeed` | [Profile](#profile) |
@@ -2611,6 +2612,16 @@ rather than a guess (plan §6.2.1).
Priority orders profiles during placement; highest that fits wins.
### `replicas`
`int`
Replicas, set on any of a model's profiles, is how many of its
replicas run on this one: seven on cards and four on CPUs, all
answering under the model's name, rather than each replica on the
best profile that fits. The model's scaling is then the sum, min and
max alike; a profile left at zero runs none.
### `startup_timeout`
`time.Duration`