Two of the four items were already built. A model falls through to its fallbacks on no capacity as well as on a spent budget, and the per-request routing profile header exists — it is ignored under authentication on purpose, because a header that picks a routing profile is a header that promotes a client into the top priority band. Checked against the code rather than against memory, which is how both got onto the list in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
21a502035f
commit
cf20570ff1
1 file changed
+8
-2
@@ -37,8 +37,14 @@ control plane places replicas — including ones larger than a single machine
|
||||
models load when someone asks for them and stop when nobody does, and keys,
|
||||
budgets and rate limits decide who may use any of it.
|
||||
|
||||
Before v1.0.0: routing profiles selectable per request, spillover to a fallback
|
||||
pool, tracing, and reference documentation. One thing is worth saying plainly:
|
||||
Before v1.0.0: tracing, reference documentation, and filtering on `ctl nodes`
|
||||
and `ctl replicas` for fleets too large to read a whole table of. Two items
|
||||
that were on this list are not gaps: a model falls through to its fallbacks
|
||||
when there is no capacity as well as when a budget is spent, and routing
|
||||
profiles are chosen by the key rather than by a request header on purpose —
|
||||
a header would let any client promote itself into the top priority band.
|
||||
|
||||
One thing is worth saying plainly:
|
||||
multi-node replicas are proven on a simulated cluster and on real hardware
|
||||
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
|
||||
formats and failure modes those tools actually have, but they have not yet run
|
||||
|
||||
Reference in new issue
Block a user