Two of the four items were already built. A model falls through to its fallbacks on no capacity as well as on a spent budget, and the per-request routing profile header exists — it is ignored under authentication on purpose, because a header that picks a routing profile is a header that promotes a client into the top priority band. Checked against the code rather than against memory, which is how both got onto the list in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
21a502035f
commit
cf20570ff1
1 file changed
+8
-2
@@ -37,8 +37,14 @@ control plane places replicas — including ones larger than a single machine
|
|||||||
models load when someone asks for them and stop when nobody does, and keys,
|
models load when someone asks for them and stop when nobody does, and keys,
|
||||||
budgets and rate limits decide who may use any of it.
|
budgets and rate limits decide who may use any of it.
|
||||||
|
|
||||||
Before v1.0.0: routing profiles selectable per request, spillover to a fallback
|
Before v1.0.0: tracing, reference documentation, and filtering on `ctl nodes`
|
||||||
pool, tracing, and reference documentation. One thing is worth saying plainly:
|
and `ctl replicas` for fleets too large to read a whole table of. Two items
|
||||||
|
that were on this list are not gaps: a model falls through to its fallbacks
|
||||||
|
when there is no capacity as well as when a budget is spent, and routing
|
||||||
|
profiles are chosen by the key rather than by a request header on purpose —
|
||||||
|
a header would let any client promote itself into the top priority band.
|
||||||
|
|
||||||
|
One thing is worth saying plainly:
|
||||||
multi-node replicas are proven on a simulated cluster and on real hardware
|
multi-node replicas are proven on a simulated cluster and on real hardware
|
||||||
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
|
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
|
||||||
formats and failure modes those tools actually have, but they have not yet run
|
formats and failure modes those tools actually have, but they have not yet run
|
||||||
|
|||||||
Reference in new issue
Block a user