CHANGELOG and BACKLOG for C17-C23 and the first parts of C24; README lists the new docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
88d63fcb76
commit
a13a519386
12 files changed
+510
-494
No files matched your search
+179
-7
@@ -509,27 +509,199 @@ portal is not done — the page shows; the console and ctl change.
|
||||
**GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
|
||||
and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
|
||||
it today); per-project "never route to an External provider".
|
||||
17. **Response/semantic cache (M).** Exact-match first, per project,
|
||||
17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys
|
||||
cache --ttl/--any/--off`, console **Cache answers…**, `projects`
|
||||
columns, 30-second setting cache), reproducible answers only unless
|
||||
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
|
||||
`max_entry_bytes`, streamed answers kept and replayed as chunks,
|
||||
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit
|
||||
and the output policy on it, `no_store` and threads bypass, a hit
|
||||
recorded as `cache_hit` with no tokens and no cost (migration 0027),
|
||||
metrics, docs. Left: semantic (embedding-similarity) caching through the
|
||||
embeddings models; a cache shared across control planes (each process
|
||||
keeps its own, deliberately: a shared one would write every answer to
|
||||
the store); a hit still waits its turn in admission, because the
|
||||
guardrails judge the request after admission and the key is the request
|
||||
they let through; logprobs in a streamed answer leave it unkept; `users
|
||||
forget` does not reach into the cache (answers age out by the TTL); no
|
||||
listing or flushing of what is cached (`--off` stops it being served at
|
||||
once, the memory is freed by TTL and LRU).
|
||||
**Response/semantic cache (M).** Exact-match first, per project,
|
||||
opt-in, TTL; semantic via the embeddings models later. Before
|
||||
`dispatch` in `router/handlers.go`.
|
||||
18. **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
|
||||
18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
|
||||
(releases with signed checksums, every llama.cpp build per arch/variant
|
||||
with GitHub's checksums, starter/judge and named Hugging Face model
|
||||
files pinned and checked; directory or tar; manifest sha256), `bundle
|
||||
import` (checked, all-or-nothing, into the mirror under `_llmgw/`),
|
||||
`bundle trust`, `bundle show --verify`;
|
||||
`offline.mirror`/`offline.checksums` in server.yaml, with nodes
|
||||
installing llama.cpp from the mirror against checksums the control plane
|
||||
sends over the agent stream (proto `EngineWant.offline_*`), releases
|
||||
defaulting to the mirror for `node add`, `ctl upgrade` and agent
|
||||
upgrades, update checks off; `server init --bundle` and the setup form's
|
||||
bundle fields; docs/data.md's offline claim corrected; install.md
|
||||
"Offline sites". A Helm chart for the control plane
|
||||
(`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL,
|
||||
Secrets, Services, Ingress, ServiceMonitor) with
|
||||
`deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay
|
||||
outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of
|
||||
several GB per CUDA build and a uv `--find-links` install path); the
|
||||
NVML release binary and uv in bundles; the setup wizard's preflight
|
||||
still fails a machine without internet; `join`'s printed `curl` from the
|
||||
mirror checks nothing; models in a bundle are single files (no whole-
|
||||
repository safetensors sets yet); the chart runs one replica (a standby
|
||||
is never ready, so HA in Kubernetes needs a different readiness design),
|
||||
has not been installed on a real cluster, and ships no agent DaemonSet;
|
||||
no published container image.
|
||||
**Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
|
||||
`agent/provision`; `docs/data.md` claims offline operation but install
|
||||
assumes the internet.
|
||||
19. **OpenAPI for the admin API, Terraform provider (M).** Generate from
|
||||
19. **Done (branch `features-c2`):** `adminapi.Routes` and
|
||||
`internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
|
||||
bearer security, every route, query parameters, schemas from the Go
|
||||
types with their comments), served embedded at `GET
|
||||
/api/v1/openapi.json`, kept true by a route-table test and a staleness
|
||||
test; validated with openapi-spec-validator and openapi-generator-cli.
|
||||
`GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider-
|
||||
llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`,
|
||||
`llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`;
|
||||
client tests against a fake admin API and the OpenAPI document;
|
||||
acceptance tests with Terraform and OpenTofu against the fake; one
|
||||
apply, empty plan, in-place change and destroy against a real `llmgw
|
||||
server` run by hand. Left: organizations, teams, projects, people and
|
||||
keys are not in the admin API (they are `llmgw org`/`users`/`keys`
|
||||
against the store), so not in the provider; a model's definition beyond
|
||||
scaling, class, owner and adapters is changed by replacing it (the API
|
||||
has no edit, and `resources` is not used yet); `canary`, `slo`,
|
||||
`scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
|
||||
`llmgw_model`; models cannot be imported; the provider is not published
|
||||
to a registry and has no release build; the acceptance tests are not run
|
||||
in CI (they need Terraform); the document's schemas mark nothing
|
||||
required and give no enums; `catalog.SLO`'s fields have no JSON tags, so
|
||||
they are `TTFTp99` and `TPOTp99` on the wire.
|
||||
**OpenAPI for the admin API, Terraform provider (M).** Generate from
|
||||
`adminapi/api.go` routes via `docgen`.
|
||||
20. **Tracing beyond the router (S).** Spans for reconciler, placement,
|
||||
20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only
|
||||
when a pass does something), placement with its answer or rejections,
|
||||
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
|
||||
lost nodes; a `replica start` span from placement to serving whose
|
||||
context travels to the agent in the start command
|
||||
(`ControlMessage.trace`), where weights fetch, engine launch, rendezvous
|
||||
and readiness wait continue the trace; stops, drains, adoption and
|
||||
engine installs; modelserve jobs as a trace with a span per step linked
|
||||
to the replica's start; an agent `tracing` block; docs in install.md's
|
||||
"Tracing". Left: the router's span for a request does not link to the
|
||||
replica it was sent to's start; heartbeats, fabric scans, logs and agent
|
||||
upgrades are not traced; the supervisor has no restart span because it
|
||||
never restarts an engine itself (a restart is the reconciler's `restart
|
||||
replica` and reaches a node as a stop and a start); canary and promote
|
||||
jobs, which landed on `features-c2` after this was branched, are traced
|
||||
only through the shared step recording (as `<kind> model`), unchecked by
|
||||
a test.
|
||||
**Tracing beyond the router (S).** Spans for reconciler, placement,
|
||||
agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
|
||||
21. **Eval harness (M).** `llmgw eval` extending `internal/bench` with
|
||||
21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
|
||||
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a
|
||||
judge model with a rubric; weights and a threshold), run on the control
|
||||
plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`,
|
||||
`--history`, `--run`, `--suites`; runs and their cases kept (migration
|
||||
0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval
|
||||
SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:`
|
||||
in the catalog) — a promote asks the canary first, an update tries the
|
||||
new weights as a 0% canary and promotes or removes it; the console's
|
||||
model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in
|
||||
the binary. Left: the suite of a model's standing gate is read by name
|
||||
on the control plane at the time of the update, so two control planes
|
||||
need the same `control.eval_dir`; a gated update needs room for one more
|
||||
replica while it runs (there is no in-place gate, and a declared model
|
||||
in server.yaml cannot be updated at all); no comparison against the old
|
||||
weights' score (the gate is an absolute threshold, not "no worse than
|
||||
before"); a run is not stopped once started (Esc in the console only
|
||||
stops watching it), and the in-memory progress of a run lives on the
|
||||
control plane that started it; runs are never pruned; the JSON Schema
|
||||
support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge
|
||||
cases go through the same concurrency as the model's, and a judge's
|
||||
answers are not kept, only its score and reason; a gateway without a
|
||||
control plane cannot run suites (there is no mode that asks through the
|
||||
client API with a key).
|
||||
**Eval harness (M).** `llmgw eval` extending `internal/bench` with
|
||||
quality suites gating `model update`.
|
||||
22. **MIG (M–L).** Treat a MIG slice as a GPU class in
|
||||
22. **Done (branch `features-c2`):** MIG instances discovered on both paths
|
||||
(nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful
|
||||
fake) and reported instead of their card, each with its MIG UUID, own
|
||||
memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100
|
||||
× card + instance, shown as `card/migN`; placement on MIG classes;
|
||||
tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation
|
||||
and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and
|
||||
llama.cpp (no Vulkan device variable on MIG); per-instance memory and
|
||||
processes in heartbeats and the GPU check, card sensors under each
|
||||
instance, utilisation absent rather than zero; doctor MIG findings (`mig
|
||||
-lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model
|
||||
plan`/`serve` by class; docs/install.md MIG section. Written and tested
|
||||
against fakes only — untested on real MIG hardware. Left: run it on an
|
||||
A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG
|
||||
device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…`
|
||||
reaches vLLM's workers through Ray); instances changed while the agent
|
||||
runs are only seen when it restarts (no rescan); llmgw does not create
|
||||
or reshape instances (no mig-parted integration); quotas count an
|
||||
instance as one GPU, the same as a whole card; SGLang on MIG not
|
||||
checked; device numbers assume fewer than 100 instances per card and
|
||||
that instance numbers stay put while shards run on them.
|
||||
**MIG (M–L).** Treat a MIG slice as a GPU class in
|
||||
`agent/discovery/gpu_nvml.go` and `control/placement`.
|
||||
23. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
|
||||
23. **Done (branch `features-c2`):** engine `trtllm`
|
||||
(`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher
|
||||
of llmgw's that demands the replica's key (trtllm-serve has none) and
|
||||
writes the options file that turns on `/prometheus/metrics` and the
|
||||
iteration statistics; the scrape contract for it
|
||||
(`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
|
||||
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
|
||||
package's metadata; catalog validation (GPU class, one node, no GGUF
|
||||
source, the flags a profile may not give); `ctl engine-check trtllm`,
|
||||
which asks `trtllm-serve serve --help` through its module. Ollama/GGUF
|
||||
import: `mirror import-ollama` (manifest parsing, blob digest check,
|
||||
projector beside the model, all or nothing, a model definition printed),
|
||||
`mirror import` for any local file, and the console's **I** on the
|
||||
Mirror view. Left: TensorRT-LLM has not run on real hardware — the first
|
||||
node with one should run `engine-check` and serve a model before anyone
|
||||
relies on it; no installer for it; no multi-node TensorRT-LLM replica
|
||||
(it needs an MPI or Slurm launch from one node); Triton Inference Server
|
||||
as an engine is not done; trtllm's queue gauges are set per iteration,
|
||||
so an idle engine shows its last batch until the next request;
|
||||
`--num_serve_frontends` is refused because extra front ends would not
|
||||
demand the key. The Ollama import runs on the mirror machine only — the
|
||||
mirror's API takes no files, so an Ollama elsewhere means copying the
|
||||
blob or running the import there; Ollama models kept as separate tensors
|
||||
or as several weight layers, and its LoRA adapters, are refused or left
|
||||
with a sentence; files an import run with `sudo` writes are root's, as
|
||||
`mirror add`'s are.
|
||||
**More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
|
||||
import. `agent/engines/engine.go` interface and the metric contract in
|
||||
`router/scrape`.
|
||||
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
|
||||
vLLM, multi-cluster federation via External providers, facility-aware
|
||||
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
|
||||
portal.
|
||||
Done so far (branch `features-c2`):
|
||||
- 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
|
||||
language templates, cookie/Accept-Language/`landing.language` choice
|
||||
and a footer switcher, Turkish numbers and dates, every template and
|
||||
handler message translated, tests that no template text stays English;
|
||||
left — errors built with fmt.Errorf in other packages (store, catalog,
|
||||
mirror, adminapi) still English on Turkish pages, the embedded Latin
|
||||
font subsets lack ğ/ş/İ (latin-ext subsets needed), percent sign
|
||||
placement is mixed (%50 on limits, 93% on cluster pages), RFC3339
|
||||
footer timestamps and plain-text http.Error bodies untranslated, the
|
||||
/docs documents themselves English, a per-person language column
|
||||
(cookie only today).
|
||||
- 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark,
|
||||
GFM, heading anchors, links rewritten, no script; public,
|
||||
`landing.docs_sign_in` to require a session), and docs/clients.md
|
||||
shows the official OpenAI/Anthropic SDKs against llmgw. Left: the
|
||||
console's generated "Commands" page is not in the portal (it is built
|
||||
from the cobra tree, which internal/web cannot see); no search across
|
||||
documents.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user