Replicas counted per profile: seven on cards and four on CPUs under one name
A profile's replicas, set on any of a model's profiles, is how many of its replicas run there; the scaling becomes the sum. The reconciler starts the ones short of their profile before stopping the ones over, so a change of split never runs fewer. ctl model replicas sets them, adding a profile on CPUs to a GGUF model; the router already fills the higher-priority pool first, which the CPU profile is put below. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6403e61f4b
commit
e23732fbbb
13 files changed
+585
-5
No files matched your search
@@ -295,6 +295,7 @@ rather than repeated.
|
||||
| `keep_within` | `string` | [Profile](#profile) |
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
| `priority` | `int` | [Profile](#profile) |
|
||||
| `replicas` | `int` | [Profile](#profile) |
|
||||
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
|
||||
| `kv_events` | `bool` | [Profile](#profile) |
|
||||
| `gpu` | `*GPUNeed` | [Profile](#profile) |
|
||||
@@ -2611,6 +2612,16 @@ rather than a guess (plan §6.2.1).
|
||||
|
||||
Priority orders profiles during placement; highest that fits wins.
|
||||
|
||||
### `replicas`
|
||||
|
||||
`int`
|
||||
|
||||
Replicas, set on any of a model's profiles, is how many of its
|
||||
replicas run on this one: seven on cards and four on CPUs, all
|
||||
answering under the model's name, rather than each replica on the
|
||||
best profile that fits. The model's scaling is then the sum, min and
|
||||
max alike; a profile left at zero runs none.
|
||||
|
||||
### `startup_timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Reference in new issue
Block a user