Commit Graph
4 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 54c6b66cb9 Fixes from review: the similar cache, restore, outside services, latency
CI / test (push) Successful in 9m48s
- The similar-question cache embeds a question as its project would: a
  project kept in the cluster has external fallbacks refused, and an
  embedding model outside the cluster is not asked at all. The vectors are
  compared outside the cache's lock. The question's bound is in characters.
- The restorer puts values back in streamed tool calls' arguments (escaped
  for the JSON they are), completions' text and reasoning, each held across
  events on its own, and releases a held tail with the finish event. A
  whole answer is walked as JSON, so a value with a quote no longer breaks
  tool-call arguments.
- An outside guardrail service reads the user's turn as redacted, and a
  streamed answer once rather than at the finish chunk and again at the
  end. guardrails.md says it can stop a stream only under buffer_all.
- A replica's smoothed time to first token is forgotten after five minutes
  without an answer: one found slow and then given nothing was never tried
  again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:15:43 +01:00
Ilker ManapandClaude Opus 5.5 73e1bb6733 Chat goes to the replica that has been answering soonest
CI / test (push) Failing after 7m18s
The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:01:30 +01:00
Ilker ManapandClaude Opus 5 9d8cd5e284 Track which replicas are actually being used, and summarise the fleet
CI / test (push) Successful in 1m28s
Two things a cluster with many nodes and many GPUs needs that were missing,
and they are the same thing seen from two distances.

Idle tracking. The control plane knew what was placed and whether it was
healthy, never whether anyone wanted it — so on a fleet with more models
than cards there was no way to answer the first question anyone asks, which
is what can I stop. The router is the only thing that knows, so it now
reports it: a timestamp written in Acquire, where a request already meets an
endpoint under a lock already held, and collected once a tick as a map. That
shape is the point. Per-request notification would be a firehose carrying
one bit — the newest timestamp wins — and this costs the same on four
replicas as on four hundred.

It travels through an interface in the direction opposite EndpointSink, for
the same reason that one exists: the control plane decides what should run,
the router knows whether anyone wants it, and neither should need to be in
the other's process to say so.

Unknown is kept distinct from idle throughout. A router that has just
restarted knows nothing, and a zero timestamp would be the oldest idle
replica in every comparison — so one restart would nominate an entire fleet
for eviction. Absent from the map, "-" in the table, negative in the JSON,
and IdleFor returns a second value rather than a duration anything could
act on by mistake.

ctl status. A per-node listing is right for four nodes and wrong for two
hundred: how full am I, what is broken, what is holding cards nobody is
using are aggregates, and reading them off a long table is a chore, which
means it gets skipped. Capacity per GPU class with usage bars, free whole
nodes per fabric — the number that decides whether a multi-node replica can
land, since free cards on the wrong fabric are not capacity for a model that
spans nodes — replica and model counts, models that cannot be placed, and
what has gone unasked-for. Derived from the same state the tables show, so
it cannot disagree with them.

Unhealthy GPUs count as neither used nor free, because counting them free is
how a cluster looks like it has room it does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 00:52:34 +02:00
ilkerandClaude Opus 5 b7417e1a69 Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping
The code behind plan §7.3-7.6:

- breaker: folds cooldown and circuit breaking into one component
  (FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff
  and half-open probes.
- endpoint: live state registry. Inflight counting, staleness detection, drain
  rate derived from counters. A scrape failure does not trip the breaker — the
  engine may be serving fine while only its metrics endpoint is down.
- scrape: parses the §7.6 metric contract. Errors when any of the mandatory
  three is missing; the metric name mapping is overridable, since names drift
  between versions (plan §14).
- scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a
  set of scorer weights; chat, batch and long-context are the same code with
  different weights. Scorers: queue, kv (hard penalty above the ceiling),
  inflight, load, weight, session (bounded-load consistent hashing). Pickers:
  max-score, p2c, softmax, round-robin.

Decision carries the rationale — which scorer produced what, which filter
dropped how many. Tuning the weights without that is guesswork.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:15:21 +02:00