llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.
- proxy: SSE relayed event by event with a flush after each, cancellation
propagated upstream, and separate deadlines for dial, headers, first token,
inter-token idle and total. The idle watchdog is what distinguishes a wedged
engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
nothing has reached the client; after that, splicing two responses together
would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
Validation rejects duplicate endpoint ids and aliases that shadow a model,
because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
per-filter drops on a separate admin listener with health and pprof.
Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.
The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
23:59:59Z renders as the following day in any positive-offset timezone, so
every due date showed up a day late in the UI.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Versions track the phases in the plan: a phase is released once its exit
criterion is met, not when its code lands. That keeps "released" meaning
something — v0.1.0 ships sim-engine and the routing libraries, and says plainly
that the gateway does not serve traffic yet.
scripts/gitea/milestones.py creates the matching Gitea milestones. Idempotent
and never destructive, so it can be re-run after the roadmap shifts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runner this targets is a containerized act_runner registered with the
ubuntu-latest label, separate from maas-builder: that one runs in host mode as
root on the image build machine, which is not somewhere a test suite belongs.
workflow_dispatch lets the queued runs be retried from the UI once a runner
exists, without needing an empty commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comments, doc comments, test names, error and log messages, CLI help, README,
Makefile, CI step names and the design document are now all in English.
No behaviour changes; the full suite passes under the race detector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The code behind plan §7.3-7.6:
- breaker: folds cooldown and circuit breaking into one component
(FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff
and half-open probes.
- endpoint: live state registry. Inflight counting, staleness detection, drain
rate derived from counters. A scrape failure does not trip the breaker — the
engine may be serving fine while only its metrics endpoint is down.
- scrape: parses the §7.6 metric contract. Errors when any of the mandatory
three is missing; the metric name mapping is overridable, since names drift
between versions (plan §14).
- scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a
set of scorer weights; chat, batch and long-context are the same code with
different weights. Scorers: queue, kv (hard penalty above the ceiling),
inflight, load, weight, session (bounded-load consistent hashing). Pickers:
max-score, p2c, softmax, round-robin.
Decision carries the rationale — which scorer produced what, which filter
dropped how many. Tuning the weights without that is guesswork.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>