CHANGELOG and BACKLOG for C17-C23 and the first parts of C24; README lists the new docs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
88d63fcb76
commit
a13a519386
12 files changed
+510
-494
No files matched your search
+179
-7
@@ -509,27 +509,199 @@ portal is not done — the page shows; the console and ctl change.
|
||||
**GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
|
||||
and rollups and erases threads/memory (`store/hierarchy.go:241` leaves
|
||||
it today); per-project "never route to an External provider".
|
||||
17. **Response/semantic cache (M).** Exact-match first, per project,
|
||||
17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys
|
||||
cache --ttl/--any/--off`, console **Cache answers…**, `projects`
|
||||
columns, 30-second setting cache), reproducible answers only unless
|
||||
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
|
||||
`max_entry_bytes`, streamed answers kept and replayed as chunks,
|
||||
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit
|
||||
and the output policy on it, `no_store` and threads bypass, a hit
|
||||
recorded as `cache_hit` with no tokens and no cost (migration 0027),
|
||||
metrics, docs. Left: semantic (embedding-similarity) caching through the
|
||||
embeddings models; a cache shared across control planes (each process
|
||||
keeps its own, deliberately: a shared one would write every answer to
|
||||
the store); a hit still waits its turn in admission, because the
|
||||
guardrails judge the request after admission and the key is the request
|
||||
they let through; logprobs in a streamed answer leave it unkept; `users
|
||||
forget` does not reach into the cache (answers age out by the TTL); no
|
||||
listing or flushing of what is cached (`--off` stops it being served at
|
||||
once, the memory is freed by TTL and LRU).
|
||||
**Response/semantic cache (M).** Exact-match first, per project,
|
||||
opt-in, TTL; semantic via the embeddings models later. Before
|
||||
`dispatch` in `router/handlers.go`.
|
||||
18. **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
|
||||
18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
|
||||
(releases with signed checksums, every llama.cpp build per arch/variant
|
||||
with GitHub's checksums, starter/judge and named Hugging Face model
|
||||
files pinned and checked; directory or tar; manifest sha256), `bundle
|
||||
import` (checked, all-or-nothing, into the mirror under `_llmgw/`),
|
||||
`bundle trust`, `bundle show --verify`;
|
||||
`offline.mirror`/`offline.checksums` in server.yaml, with nodes
|
||||
installing llama.cpp from the mirror against checksums the control plane
|
||||
sends over the agent stream (proto `EngineWant.offline_*`), releases
|
||||
defaulting to the mirror for `node add`, `ctl upgrade` and agent
|
||||
upgrades, update checks off; `server init --bundle` and the setup form's
|
||||
bundle fields; docs/data.md's offline claim corrected; install.md
|
||||
"Offline sites". A Helm chart for the control plane
|
||||
(`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL,
|
||||
Secrets, Services, Ingress, ServiceMonitor) with
|
||||
`deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay
|
||||
outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of
|
||||
several GB per CUDA build and a uv `--find-links` install path); the
|
||||
NVML release binary and uv in bundles; the setup wizard's preflight
|
||||
still fails a machine without internet; `join`'s printed `curl` from the
|
||||
mirror checks nothing; models in a bundle are single files (no whole-
|
||||
repository safetensors sets yet); the chart runs one replica (a standby
|
||||
is never ready, so HA in Kubernetes needs a different readiness design),
|
||||
has not been installed on a real cluster, and ships no agent DaemonSet;
|
||||
no published container image.
|
||||
**Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`,
|
||||
`agent/provision`; `docs/data.md` claims offline operation but install
|
||||
assumes the internet.
|
||||
19. **OpenAPI for the admin API, Terraform provider (M).** Generate from
|
||||
19. **Done (branch `features-c2`):** `adminapi.Routes` and
|
||||
`internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
|
||||
bearer security, every route, query parameters, schemas from the Go
|
||||
types with their comments), served embedded at `GET
|
||||
/api/v1/openapi.json`, kept true by a route-table test and a staleness
|
||||
test; validated with openapi-spec-validator and openapi-generator-cli.
|
||||
`GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider-
|
||||
llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`,
|
||||
`llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`;
|
||||
client tests against a fake admin API and the OpenAPI document;
|
||||
acceptance tests with Terraform and OpenTofu against the fake; one
|
||||
apply, empty plan, in-place change and destroy against a real `llmgw
|
||||
server` run by hand. Left: organizations, teams, projects, people and
|
||||
keys are not in the admin API (they are `llmgw org`/`users`/`keys`
|
||||
against the store), so not in the provider; a model's definition beyond
|
||||
scaling, class, owner and adapters is changed by replacing it (the API
|
||||
has no edit, and `resources` is not used yet); `canary`, `slo`,
|
||||
`scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
|
||||
`llmgw_model`; models cannot be imported; the provider is not published
|
||||
to a registry and has no release build; the acceptance tests are not run
|
||||
in CI (they need Terraform); the document's schemas mark nothing
|
||||
required and give no enums; `catalog.SLO`'s fields have no JSON tags, so
|
||||
they are `TTFTp99` and `TPOTp99` on the wire.
|
||||
**OpenAPI for the admin API, Terraform provider (M).** Generate from
|
||||
`adminapi/api.go` routes via `docgen`.
|
||||
20. **Tracing beyond the router (S).** Spans for reconciler, placement,
|
||||
20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only
|
||||
when a pass does something), placement with its answer or rejections,
|
||||
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
|
||||
lost nodes; a `replica start` span from placement to serving whose
|
||||
context travels to the agent in the start command
|
||||
(`ControlMessage.trace`), where weights fetch, engine launch, rendezvous
|
||||
and readiness wait continue the trace; stops, drains, adoption and
|
||||
engine installs; modelserve jobs as a trace with a span per step linked
|
||||
to the replica's start; an agent `tracing` block; docs in install.md's
|
||||
"Tracing". Left: the router's span for a request does not link to the
|
||||
replica it was sent to's start; heartbeats, fabric scans, logs and agent
|
||||
upgrades are not traced; the supervisor has no restart span because it
|
||||
never restarts an engine itself (a restart is the reconciler's `restart
|
||||
replica` and reaches a node as a stop and a start); canary and promote
|
||||
jobs, which landed on `features-c2` after this was branched, are traced
|
||||
only through the shared step recording (as `<kind> model`), unchecked by
|
||||
a test.
|
||||
**Tracing beyond the router (S).** Spans for reconciler, placement,
|
||||
agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`).
|
||||
21. **Eval harness (M).** `llmgw eval` extending `internal/bench` with
|
||||
21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
|
||||
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a
|
||||
judge model with a rubric; weights and a threshold), run on the control
|
||||
plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`,
|
||||
`--history`, `--run`, `--suites`; runs and their cases kept (migration
|
||||
0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval
|
||||
SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:`
|
||||
in the catalog) — a promote asks the canary first, an update tries the
|
||||
new weights as a 0% canary and promotes or removes it; the console's
|
||||
model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in
|
||||
the binary. Left: the suite of a model's standing gate is read by name
|
||||
on the control plane at the time of the update, so two control planes
|
||||
need the same `control.eval_dir`; a gated update needs room for one more
|
||||
replica while it runs (there is no in-place gate, and a declared model
|
||||
in server.yaml cannot be updated at all); no comparison against the old
|
||||
weights' score (the gate is an absolute threshold, not "no worse than
|
||||
before"); a run is not stopped once started (Esc in the console only
|
||||
stops watching it), and the in-memory progress of a run lives on the
|
||||
control plane that started it; runs are never pruned; the JSON Schema
|
||||
support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge
|
||||
cases go through the same concurrency as the model's, and a judge's
|
||||
answers are not kept, only its score and reason; a gateway without a
|
||||
control plane cannot run suites (there is no mode that asks through the
|
||||
client API with a key).
|
||||
**Eval harness (M).** `llmgw eval` extending `internal/bench` with
|
||||
quality suites gating `model update`.
|
||||
22. **MIG (M–L).** Treat a MIG slice as a GPU class in
|
||||
22. **Done (branch `features-c2`):** MIG instances discovered on both paths
|
||||
(nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful
|
||||
fake) and reported instead of their card, each with its MIG UUID, own
|
||||
memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100
|
||||
× card + instance, shown as `card/migN`; placement on MIG classes;
|
||||
tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation
|
||||
and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and
|
||||
llama.cpp (no Vulkan device variable on MIG); per-instance memory and
|
||||
processes in heartbeats and the GPU check, card sensors under each
|
||||
instance, utilisation absent rather than zero; doctor MIG findings (`mig
|
||||
-lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model
|
||||
plan`/`serve` by class; docs/install.md MIG section. Written and tested
|
||||
against fakes only — untested on real MIG hardware. Left: run it on an
|
||||
A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG
|
||||
device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…`
|
||||
reaches vLLM's workers through Ray); instances changed while the agent
|
||||
runs are only seen when it restarts (no rescan); llmgw does not create
|
||||
or reshape instances (no mig-parted integration); quotas count an
|
||||
instance as one GPU, the same as a whole card; SGLang on MIG not
|
||||
checked; device numbers assume fewer than 100 instances per card and
|
||||
that instance numbers stay put while shards run on them.
|
||||
**MIG (M–L).** Treat a MIG slice as a GPU class in
|
||||
`agent/discovery/gpu_nvml.go` and `control/placement`.
|
||||
23. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
|
||||
23. **Done (branch `features-c2`):** engine `trtllm`
|
||||
(`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher
|
||||
of llmgw's that demands the replica's key (trtllm-serve has none) and
|
||||
writes the options file that turns on `/prometheus/metrics` and the
|
||||
iteration statistics; the scrape contract for it
|
||||
(`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
|
||||
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
|
||||
package's metadata; catalog validation (GPU class, one node, no GGUF
|
||||
source, the flags a profile may not give); `ctl engine-check trtllm`,
|
||||
which asks `trtllm-serve serve --help` through its module. Ollama/GGUF
|
||||
import: `mirror import-ollama` (manifest parsing, blob digest check,
|
||||
projector beside the model, all or nothing, a model definition printed),
|
||||
`mirror import` for any local file, and the console's **I** on the
|
||||
Mirror view. Left: TensorRT-LLM has not run on real hardware — the first
|
||||
node with one should run `engine-check` and serve a model before anyone
|
||||
relies on it; no installer for it; no multi-node TensorRT-LLM replica
|
||||
(it needs an MPI or Slurm launch from one node); Triton Inference Server
|
||||
as an engine is not done; trtllm's queue gauges are set per iteration,
|
||||
so an idle engine shows its last batch until the next request;
|
||||
`--num_serve_frontends` is refused because extra front ends would not
|
||||
demand the key. The Ollama import runs on the mirror machine only — the
|
||||
mirror's API takes no files, so an Ollama elsewhere means copying the
|
||||
blob or running the import there; Ollama models kept as separate tensors
|
||||
or as several weight layers, and its LoRA adapters, are refused or left
|
||||
with a sentence; files an import run with `sudo` writes are root's, as
|
||||
`mirror add`'s are.
|
||||
**More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
|
||||
import. `agent/engines/engine.go` interface and the metric contract in
|
||||
`router/scrape`.
|
||||
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
|
||||
vLLM, multi-cluster federation via External providers, facility-aware
|
||||
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
|
||||
portal.
|
||||
Done so far (branch `features-c2`):
|
||||
- 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
|
||||
language templates, cookie/Accept-Language/`landing.language` choice
|
||||
and a footer switcher, Turkish numbers and dates, every template and
|
||||
handler message translated, tests that no template text stays English;
|
||||
left — errors built with fmt.Errorf in other packages (store, catalog,
|
||||
mirror, adminapi) still English on Turkish pages, the embedded Latin
|
||||
font subsets lack ğ/ş/İ (latin-ext subsets needed), percent sign
|
||||
placement is mixed (%50 on limits, 93% on cluster pages), RFC3339
|
||||
footer timestamps and plain-text http.Error bodies untranslated, the
|
||||
/docs documents themselves English, a per-person language column
|
||||
(cookie only today).
|
||||
- 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark,
|
||||
GFM, heading anchors, links rewritten, no script; public,
|
||||
`landing.docs_sign_in` to require a session), and docs/clients.md
|
||||
shows the official OpenAI/Anthropic SDKs against llmgw. Left: the
|
||||
console's generated "Commands" page is not in the portal (it is built
|
||||
from the cobra tree, which internal/web cannot see); no search across
|
||||
documents.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+313
@@ -23,6 +23,319 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Reusing answers
|
||||
|
||||
- **A response cache, per project, off until asked for.** `llmgw keys cache
|
||||
--project P --ttl 1h` has the same request — same project, endpoint,
|
||||
model by its catalog name, and body with `stream`, `stream_options`,
|
||||
`user`, `metadata`, `store` and `service_tier` left out and its keys in
|
||||
any order — answered from the gateway's memory instead of the model, for
|
||||
the TTL. `--off` stops it; without either it says how it stands. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
- **Only answers that come out the same are reused:** chat and completions
|
||||
asked at temperature 0 or with a seed, and embeddings, rerank and score.
|
||||
`--any` reuses sampled answers too. Only a whole 200 from the model asked
|
||||
for is kept — not an error, a fallback's or a canary's answer, one a
|
||||
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
|
||||
- **Streams both ways:** a streamed answer is kept put back together into
|
||||
one completion (tool calls and reasoning included), and a kept answer is
|
||||
streamed as chunks — with the usage chunk if the client asked — to a
|
||||
client that streams. `/v1/messages` and `/v1/responses` get it too.
|
||||
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
|
||||
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
|
||||
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
|
||||
- **Policies still apply:** limits and the input guardrails run before the
|
||||
cache is looked in, the output policy judges a reused answer as it would
|
||||
a new one, and a request under `no_store` or continuing a stored
|
||||
conversation is never served from it nor kept in it. A project's answers
|
||||
are never looked up by another project's keys.
|
||||
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
|
||||
marked `cache_hit` on the usage record (migration 0027), and shown by
|
||||
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
|
||||
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
|
||||
together in each gateway process, least recently used dropped first;
|
||||
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
|
||||
`llmgw_cache_bytes` and `llmgw_cache_entries`.
|
||||
|
||||
### Offline sites and Kubernetes
|
||||
|
||||
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
|
||||
collects what a cluster would otherwise download — llmgw's release for
|
||||
each `--arch` with its signed checksums, every llama.cpp build for those
|
||||
architectures (or the `--variant` ones) with the checksums GitHub
|
||||
publishes, the starter model and guardrails' judge, and any `--model
|
||||
name/version=hf:owner/repo/file` pinned to the registry's commit and its
|
||||
checksum — each checked as it arrives, into a directory or one `.tar`
|
||||
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
|
||||
bundle. Run again into the same directory, it downloads only what is
|
||||
missing. `llmgw bundle show --verify` reads one back.
|
||||
- **`llmgw bundle import`** checks every file against the manifest and
|
||||
llmgw's release against the project's signing key, and only then puts them
|
||||
into the mirror — releases and engines under `_llmgw/`, models where the
|
||||
mirror keeps models, with their checksums — so a damaged bundle leaves the
|
||||
mirror as it was; a different file under a version already there is
|
||||
refused. On a control plane it also records the engines' checksums in
|
||||
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
|
||||
does that alone, for a mirror on another machine.
|
||||
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
|
||||
outside: nodes install llama.cpp from the mirror, each file held to the
|
||||
checksum the control plane sends them over the agent's authenticated
|
||||
connection — never one from the mirror — and `releases` defaults to the
|
||||
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
|
||||
from it; no registry is asked about newer weights. A release may now come
|
||||
over plain HTTP when the build trusts a signing key, since the signature
|
||||
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
|
||||
be unsupported on an offline node rather than attempted.
|
||||
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
|
||||
fields on the setup form: installs a control plane offline — the bundle
|
||||
into the mirror, the checksums trusted, `offline:` written to server.yaml,
|
||||
the starter model and judge served from the bundle or skipped when it does
|
||||
not carry them.
|
||||
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
|
||||
it lists everything that does, and what an offline cluster changes.
|
||||
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
|
||||
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
|
||||
port's certificate and the admin token from Secrets, the API and metrics
|
||||
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
|
||||
and ServiceMonitor, `config` laid into server.yaml, and the offline
|
||||
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
|
||||
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
|
||||
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
|
||||
|
||||
### The admin API described, and a Terraform provider
|
||||
|
||||
- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path
|
||||
and query parameters, and the schema of each request and answer with the
|
||||
meaning of each field, in
|
||||
[docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by
|
||||
every control plane at `GET /api/v1/openapi.json` (with the token, like the
|
||||
rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are
|
||||
in it. Generated by `go generate ./internal/docgen` from a route table in
|
||||
`internal/control/adminapi/routes.go` and the Go types the handlers use, with
|
||||
the fields' comments as descriptions; a test fails when an endpoint is
|
||||
served and not in the table, another when the committed document is stale.
|
||||
Checked with openapi-spec-validator and openapi-generator-cli's validate.
|
||||
- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback,
|
||||
an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`,
|
||||
what a request must carry — an address in `Host`, `application/json` on
|
||||
every change, even a `DELETE` — and some calls with `curl`.
|
||||
- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in
|
||||
force, with the defaults the catalog filled in. The listing has only its
|
||||
state and its profiles' names.
|
||||
- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its
|
||||
own so the gateway does not carry its dependencies. `llmgw_model` (a model
|
||||
added at runtime: scaling, class, owner and adapters change in place, the
|
||||
rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the
|
||||
`llmgw_models` and `llmgw_nodes` data sources. Its client is checked
|
||||
against the OpenAPI document, endpoint by endpoint and field by field, so a
|
||||
change to the admin API that breaks it fails its tests; CI runs them.
|
||||
Built and installed by hand for now: see its
|
||||
[README](terraform-provider-llmgw/README.md).
|
||||
|
||||
### Tracing beyond the router
|
||||
|
||||
- **The control plane traces what it does to the fleet.** A reconcile pass
|
||||
that does something is a `reconcile` trace — passes with nothing to do are
|
||||
not traced at all — with what it did beneath it: `scale up` with `place`
|
||||
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
|
||||
candidate turned down and why, as `ctl explain-placement` would print it),
|
||||
`scale down`, `drain replica` and `stop replica` each with its reason,
|
||||
`evict`, `preempt` with its victims, `reap replica` with the error and
|
||||
when the model is tried again, `restart replica`, `restart refused`,
|
||||
`node unreachable` and `node lost`.
|
||||
- **A replica's start is one trace, down to the node.** `replica start`
|
||||
stays open from placement until the replica serves, fails or is stopped,
|
||||
with each shard's progress as an event and how long it took. The command
|
||||
to each node is a `send start` span whose context travels in the command
|
||||
(a new `trace` field on the agent protocol's `ControlMessage`), and the
|
||||
agent continues the same trace: `start shard`, with `fetch weights`
|
||||
(local, cached, downloaded, or downloaded by another shard; a `download`
|
||||
with its bytes and where a resume began, the source's credentials and
|
||||
query left out), `launch engine` (and `wait for rendezvous` on a worker
|
||||
of a multi-node replica), and `wait for engine` with how often the engine
|
||||
was asked and its last answer. "Why did qwen take four minutes to start"
|
||||
is that one trace.
|
||||
- **Stops, drains and adoptions on a node** are in the control plane's
|
||||
trace too: `stop shard` and `stop engine`, saying whether the engine had
|
||||
to be killed; `drain shard`; and `adopt` when a registering node reports
|
||||
engines already running, with how many nobody claimed. An engine install
|
||||
is an `install engine` trace on the node.
|
||||
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
|
||||
`serve model` and `update model`, with a span per step — registry,
|
||||
mirror, check, catalog, placement, smoke test, replicas — and from the
|
||||
placement step a link to the replica's own start.
|
||||
- **The agent has a `tracing` block** in `agent.yaml`, the same as
|
||||
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
|
||||
when unset). Point both at one collector. A node follows the control
|
||||
plane's sampling for every start it is sent. Traces from a node carry its
|
||||
id as `service.instance.id`, and a control plane's its host name.
|
||||
- With no endpoint nothing changes: every span is the no-op one, and a
|
||||
command carries no trace context.
|
||||
|
||||
### Quality suites, and new weights held to one
|
||||
|
||||
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
|
||||
and scores its answers: per case, then the cases weighed into one score, 0
|
||||
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
|
||||
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
|
||||
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
|
||||
and one way to score it — `exact`, `contains`, `regex`, `number` within
|
||||
a `tolerance` (the last number in the answer), `json_schema` (the answer
|
||||
read as JSON, in a code fence or a sentence or not, held to a schema), or
|
||||
`judge`: a rubric another model on the cluster grades 0 to 1. A
|
||||
reasoning model's `<think>` is not part of its answer.
|
||||
- **Run on the control plane**, through the router as its own requests,
|
||||
as a replay is: past admission, quotas, accounting and the guardrails.
|
||||
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
|
||||
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
|
||||
`deploy/eval/`, sized for small models. `--suites` lists them.
|
||||
- **Kept** in the store (migration 0028): every run with its model, the
|
||||
repository and revision of the weights that answered, the suite's name
|
||||
and hash, its score and each case's answer and reason.
|
||||
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
|
||||
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
|
||||
`model promote` take `--eval SUITE --min-score N`, or a model carries one
|
||||
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
|
||||
A promotion asks the canary first and leaves it in place below the score;
|
||||
an update runs the new weights first as a canary given none of the
|
||||
model's requests, asks it, and promotes it or takes it out again, so the
|
||||
old weights serve until the new have passed. The job's `eval` step says
|
||||
the score; `--skip-eval` moves once without the model's suite.
|
||||
- **The console** has **Ask it a quality suite…** (the score counted in as it
|
||||
comes), **Its quality runs** and **Hold its updates to a quality suite…**
|
||||
on a model's menu.
|
||||
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
|
||||
handler cut off at 64 KiB, by the audit log's reading of it.
|
||||
|
||||
### MIG instances as GPUs of their own class
|
||||
|
||||
- **A card in MIG mode is placed as its instances.** Discovery reports a card
|
||||
in MIG mode as its MIG instances instead of the whole card, each a device of
|
||||
its own with its UUID (`MIG-…`), its own memory, and a class named from the
|
||||
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
|
||||
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
|
||||
any other, and placement puts a replica on a free instance of it. Both
|
||||
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
|
||||
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
|
||||
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
|
||||
instances; `docs/install.md` has a MIG section on making them.
|
||||
- **One instance a replica.** A process opens only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer between them, so a profile on a
|
||||
MIG class asking for more than one GPU, tp or pp above 1, or more than one
|
||||
node is refused when the model is added, with that reason; placement
|
||||
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
|
||||
for a model one holds and never several together.
|
||||
- **Engines are given the instance by UUID.** Placement numbers card 0's
|
||||
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
|
||||
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
|
||||
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
|
||||
shows the device as `0/mig2`.
|
||||
- **Memory per instance, no made-up utilisation.** Each instance's memory,
|
||||
and the processes holding it, are its own — read per GPU and compute
|
||||
instance from nvidia-smi's report, so a foreign process on one instance
|
||||
keeps that instance out of placement and not its neighbours. Temperature,
|
||||
power and wear are the card's. No driver reports utilisation per instance,
|
||||
so none is published for one: the console says `n/a on MIG`, and
|
||||
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
|
||||
- **A card in MIG mode with nothing carved is kept out of placement**, with
|
||||
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
|
||||
is on.
|
||||
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
|
||||
of a card in MIG mode with no instances, of GPU instances with no compute
|
||||
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
|
||||
command for each.
|
||||
- `ctl nodes` counts a node's devices by class when it has MIG instances or
|
||||
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
|
||||
GPU list shows instances under their card with their own memory and the
|
||||
card's heat and power, and a node's page lists its classes with counts.
|
||||
`ctl status` and the plan count each device under its own class, where a
|
||||
node's cards were all put under its first class before.
|
||||
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
|
||||
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
|
||||
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
|
||||
|
||||
### Engines and the mirror
|
||||
|
||||
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
|
||||
installed it: name its interpreter in the node's `agent.yaml`
|
||||
(`engines.trtllm.python`) and the node offers it while that interpreter
|
||||
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
|
||||
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
|
||||
`--pp_size` and served name, through a few lines of its own that make it
|
||||
demand the gateway's key on every path but `/health` (trtllm-serve has no
|
||||
key option), turn on the Prometheus endpoint and iteration statistics, and
|
||||
keep usage reports to NVIDIA off. The router reads it at
|
||||
`/prometheus/metrics` — `trtllm_num_requests_running`,
|
||||
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
|
||||
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
|
||||
GPU; one node per replica. Profiles may not give it an options file,
|
||||
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
|
||||
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
|
||||
engine-check trtllm --bin <python>` checks an installed one. Written
|
||||
against TensorRT-LLM's documented command line, server and metrics and
|
||||
tested with stand-ins for them; not yet run on a datacenter GPU.
|
||||
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
|
||||
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
|
||||
model's projector beside it, each checked against the sha256 Ollama named
|
||||
its file by. With no name it lists Ollama's models; it finds Ollama's
|
||||
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
|
||||
service) or takes `--ollama-dir`. The version is the tag and the start of
|
||||
the digest, since a tag moves in Ollama. It prints a model definition to
|
||||
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
|
||||
view does the same with **I**, for the mirror beside the control plane.
|
||||
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
|
||||
mirror machine into it the way a download is kept: whole or not at all, the
|
||||
checksum beside it, a version never replaced, a `.gguf` checked to be one,
|
||||
and `--sha256` checked when given.
|
||||
|
||||
### The portal in Turkish
|
||||
|
||||
- **The web portal speaks English and Turkish** — the landing page, sign-up
|
||||
and sign-in, the account page and its editor setup, the claim page, chat,
|
||||
the about page, and every administrators' page (cluster, nodes, NVLink and
|
||||
fabrics, people, limits, usage, organizations, mirror, guardrails), and
|
||||
the docs portal around its documents, which stay English. Flash
|
||||
messages and form errors built by the server are translated too.
|
||||
- **A switcher at the foot of every page** (English · Türkçe) remembers the
|
||||
choice in a cookie; without one the browser's Accept-Language decides, and
|
||||
without that the new `landing.language` (`en` or `tr`, empty is English).
|
||||
An unknown code stops the server starting. `<html lang>` and
|
||||
Content-Language follow the page.
|
||||
- **Figures and dates in the page's language**: a Turkish page writes
|
||||
1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in
|
||||
Turkish. Form values, CSS and SVG coordinates are unchanged.
|
||||
- **Ready for more languages**: the catalogue is JSON under
|
||||
`internal/web/locales/<code>/`, and a language is a directory. Tests refuse
|
||||
a key a page uses that some language lacks, a translation with different
|
||||
arguments from English, and any template text left in English — so a new
|
||||
page cannot quietly ship English-only.
|
||||
- Messages that come from the control plane and other packages (node
|
||||
warnings, engine errors, store refusals) are still shown as written, in
|
||||
English; the API, CLI and console are English.
|
||||
|
||||
### Documentation
|
||||
|
||||
- **A docs portal at /docs.** The manual the binary carries — the pages
|
||||
`llmgw docs` prints and the console's Manual shows, the configuration
|
||||
reference among them — rendered on the client port in the portal's own
|
||||
look: a sidebar of the documents in reading order, the open one's sections
|
||||
under it, a link on every heading, and links between documents kept inside
|
||||
the portal (links to files that are not in the binary, such as `deploy/`,
|
||||
go to the repository). No script on the page, so the Content-Security-Policy
|
||||
has nothing to allow. Public by default, like the landing page; set
|
||||
`landing.docs_sign_in: true` to send visitors to sign in first. Signed in,
|
||||
the page carries the usual menu, which now has a **Docs** link.
|
||||
- **A client developer's guide, `docs/clients.md`.** How to use llmgw from
|
||||
the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is
|
||||
needed — in Python, TypeScript, Go and curl: the base URL and key,
|
||||
streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK
|
||||
against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`,
|
||||
`x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`,
|
||||
`/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index
|
||||
leads with it, and the landing page and the account page link to it.
|
||||
- The console's Manual and the portal read one list of documents in one
|
||||
order (`llmgw.Manual`), so the two cannot drift.
|
||||
|
||||
### Administration
|
||||
|
||||
- **An audit log.** Every change made through the admin API (`llmgw ctl`,
|
||||
|
||||
@@ -63,6 +63,17 @@ Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
|
||||
- **Notifications** — webhooks, signed or Slack-shaped, for a model parked,
|
||||
a node lost, a budget spent and new weights published (`notify.webhooks`,
|
||||
`ctl notify test`), and Alertmanager receiver templates.
|
||||
- **Traffic and models** — request tags, canaries of new weights with
|
||||
automatic rollback, quality suites gating updates, a response cache,
|
||||
kept and replayed requests, batches, Anthropic's Messages API, more of
|
||||
OpenAI's (structured output, images, rerank, audio, moderations),
|
||||
scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, and models
|
||||
imported from Ollama.
|
||||
- **Operations** — `server restore` and off-machine backup copies, secrets
|
||||
in Vault, forgetting a person and keeping a project's requests in the
|
||||
cluster, offline installs from a bundle, a Helm chart, tracing from the
|
||||
control plane to the engines, the admin API in OpenAPI with a Terraform
|
||||
provider, and the portal in English and Turkish with its docs at /docs.
|
||||
|
||||
Every one of these is in the console as well as on the command line:
|
||||
[docs/console.md](docs/console.md).
|
||||
@@ -95,6 +106,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
the firewall, upgrades, backups
|
||||
- [docs/node-onboarding.md](docs/node-onboarding.md) — adding nodes to a
|
||||
cluster that exists
|
||||
- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
|
||||
Kubernetes with the Helm chart, and why GPU nodes stay outside it
|
||||
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
|
||||
llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
|
||||
when a replica spanning nodes will not run
|
||||
@@ -105,6 +118,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
every view, and the command each of its actions runs
|
||||
- [docs/onboarding.md](docs/onboarding.md) — accounts, keys, limits, staff
|
||||
roles and single sign-on
|
||||
- [docs/clients.md](docs/clients.md) — using the gateway from code with the
|
||||
official OpenAI and Anthropic SDKs, and llmgw's extensions
|
||||
- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
|
||||
policies, detectors, a judge model, and where each applies
|
||||
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
|
||||
@@ -117,6 +132,9 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
|
||||
does not exist or miss one that does
|
||||
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
|
||||
what leaves the machine; written for a security review
|
||||
- [docs/admin-api.md](docs/admin-api.md) and its OpenAPI document — the
|
||||
admin API, for scripts; [terraform-provider-llmgw/](terraform-provider-llmgw/)
|
||||
manages models, quotas and guardrails with Terraform or OpenTofu
|
||||
- [deploy/](deploy/) — [a mirror](deploy/mirror/) on its own machine,
|
||||
[Caddy and nginx](deploy/proxy/) in front of the client port, [two control
|
||||
planes](deploy/ha/) behind a balancer, and [Prometheus, Alertmanager and
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Reusing answers
|
||||
|
||||
- **A response cache, per project, off until asked for.** `llmgw keys cache
|
||||
--project P --ttl 1h` has the same request — same project, endpoint,
|
||||
model by its catalog name, and body with `stream`, `stream_options`,
|
||||
`user`, `metadata`, `store` and `service_tier` left out and its keys in
|
||||
any order — answered from the gateway's memory instead of the model, for
|
||||
the TTL. `--off` stops it; without either it says how it stands. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
- **Only answers that come out the same are reused:** chat and completions
|
||||
asked at temperature 0 or with a seed, and embeddings, rerank and score.
|
||||
`--any` reuses sampled answers too. Only a whole 200 from the model asked
|
||||
for is kept — not an error, a fallback's or a canary's answer, one a
|
||||
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
|
||||
- **Streams both ways:** a streamed answer is kept put back together into
|
||||
one completion (tool calls and reasoning included), and a kept answer is
|
||||
streamed as chunks — with the usage chunk if the client asked — to a
|
||||
client that streams. `/v1/messages` and `/v1/responses` get it too.
|
||||
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
|
||||
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
|
||||
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
|
||||
- **Policies still apply:** limits and the input guardrails run before the
|
||||
cache is looked in, the output policy judges a reused answer as it would
|
||||
a new one, and a request under `no_store` or continuing a stored
|
||||
conversation is never served from it nor kept in it. A project's answers
|
||||
are never looked up by another project's keys.
|
||||
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
|
||||
marked `cache_hit` on the usage record (migration 0027), and shown by
|
||||
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
|
||||
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
|
||||
together in each gateway process, least recently used dropped first;
|
||||
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
|
||||
`llmgw_cache_bytes` and `llmgw_cache_entries`.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** exact-match answer cache per project
|
||||
(`keys cache --ttl/--any/--off`, console **Cache answers…**, `projects`
|
||||
columns, 30-second setting cache), reproducible answers only unless
|
||||
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
|
||||
`max_entry_bytes`, streamed answers kept and replayed as chunks,
|
||||
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit and
|
||||
the output policy on it, `no_store` and threads bypass, a hit recorded as
|
||||
`cache_hit` with no tokens and no cost (migration 0027), metrics, docs.
|
||||
Left: semantic (embedding-similarity) caching through the embeddings
|
||||
models; a cache shared across control planes (each process keeps its own,
|
||||
deliberately: a shared one would write every answer to the store); a hit
|
||||
still waits its turn in admission, because the guardrails judge the request
|
||||
after admission and the key is the request they let through; logprobs in a
|
||||
streamed answer leave it unkept; `users forget` does not reach into the
|
||||
cache (answers age out by the TTL); no listing or flushing of what is
|
||||
cached (`--off` stops it being served at once, the memory is freed by TTL
|
||||
and LRU).
|
||||
@@ -1,78 +0,0 @@
|
||||
# C18 — Kubernetes/Helm and an air-gapped bundle
|
||||
|
||||
README's documentation list should gain a line for `docs/kubernetes.md`
|
||||
(the control plane in Kubernetes from the Helm chart, and why GPU nodes stay
|
||||
outside it) — README.md was not edited here.
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### Offline sites and Kubernetes
|
||||
|
||||
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
|
||||
collects what a cluster would otherwise download — llmgw's release for
|
||||
each `--arch` with its signed checksums, every llama.cpp build for those
|
||||
architectures (or the `--variant` ones) with the checksums GitHub
|
||||
publishes, the starter model and guardrails' judge, and any `--model
|
||||
name/version=hf:owner/repo/file` pinned to the registry's commit and its
|
||||
checksum — each checked as it arrives, into a directory or one `.tar`
|
||||
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
|
||||
bundle. Run again into the same directory, it downloads only what is
|
||||
missing. `llmgw bundle show --verify` reads one back.
|
||||
- **`llmgw bundle import`** checks every file against the manifest and
|
||||
llmgw's release against the project's signing key, and only then puts them
|
||||
into the mirror — releases and engines under `_llmgw/`, models where the
|
||||
mirror keeps models, with their checksums — so a damaged bundle leaves the
|
||||
mirror as it was; a different file under a version already there is
|
||||
refused. On a control plane it also records the engines' checksums in
|
||||
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
|
||||
does that alone, for a mirror on another machine.
|
||||
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
|
||||
outside: nodes install llama.cpp from the mirror, each file held to the
|
||||
checksum the control plane sends them over the agent's authenticated
|
||||
connection — never one from the mirror — and `releases` defaults to the
|
||||
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
|
||||
from it; no registry is asked about newer weights. A release may now come
|
||||
over plain HTTP when the build trusts a signing key, since the signature
|
||||
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
|
||||
be unsupported on an offline node rather than attempted.
|
||||
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
|
||||
fields on the setup form: installs a control plane offline — the bundle
|
||||
into the mirror, the checksums trusted, `offline:` written to server.yaml,
|
||||
the starter model and judge served from the bundle or skipped when it does
|
||||
not carry them.
|
||||
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
|
||||
it lists everything that does, and what an offline cluster changes.
|
||||
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
|
||||
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
|
||||
port's certificate and the admin token from Secrets, the API and metrics
|
||||
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
|
||||
and ServiceMonitor, `config` laid into server.yaml, and the offline
|
||||
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
|
||||
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
|
||||
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** offline bundles — `llmgw bundle create`
|
||||
(releases with signed checksums, every llama.cpp build per arch/variant
|
||||
with GitHub's checksums, starter/judge and named Hugging Face model files
|
||||
pinned and checked; directory or tar; manifest sha256), `bundle import`
|
||||
(checked, all-or-nothing, into the mirror under `_llmgw/`), `bundle trust`,
|
||||
`bundle show --verify`; `offline.mirror`/`offline.checksums` in server.yaml,
|
||||
with nodes installing llama.cpp from the mirror against checksums the
|
||||
control plane sends over the agent stream (proto `EngineWant.offline_*`),
|
||||
releases defaulting to the mirror for `node add`, `ctl upgrade` and agent
|
||||
upgrades, update checks off; `server init --bundle` and the setup form's
|
||||
bundle fields; docs/data.md's offline claim corrected; install.md "Offline
|
||||
sites". A Helm chart for the control plane (`deploy/helm/llmgw`, one replica,
|
||||
SQLite PVC or external PostgreSQL, Secrets, Services, Ingress,
|
||||
ServiceMonitor) with `deploy/container/Dockerfile` and docs/kubernetes.md;
|
||||
GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a
|
||||
wheelhouse of several GB per CUDA build and a uv `--find-links` install
|
||||
path); the NVML release binary and uv in bundles; the setup wizard's
|
||||
preflight still fails a machine without internet; `join`'s printed `curl`
|
||||
from the mirror checks nothing; models in a bundle are single files
|
||||
(no whole-repository safetensors sets yet); the chart runs one replica (a
|
||||
standby is never ready, so HA in Kubernetes needs a different readiness
|
||||
design), has not been installed on a real cluster, and ships no agent
|
||||
DaemonSet; no published container image.
|
||||
@@ -1,57 +0,0 @@
|
||||
# C19: OpenAPI for the admin API, and a Terraform provider
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### The admin API described, and a Terraform provider
|
||||
|
||||
- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path
|
||||
and query parameters, and the schema of each request and answer with the
|
||||
meaning of each field, in
|
||||
[docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by
|
||||
every control plane at `GET /api/v1/openapi.json` (with the token, like the
|
||||
rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are
|
||||
in it. Generated by `go generate ./internal/docgen` from a route table in
|
||||
`internal/control/adminapi/routes.go` and the Go types the handlers use, with
|
||||
the fields' comments as descriptions; a test fails when an endpoint is
|
||||
served and not in the table, another when the committed document is stale.
|
||||
Checked with openapi-spec-validator and openapi-generator-cli's validate.
|
||||
- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback,
|
||||
an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`,
|
||||
what a request must carry — an address in `Host`, `application/json` on
|
||||
every change, even a `DELETE` — and some calls with `curl`.
|
||||
- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in
|
||||
force, with the defaults the catalog filled in. The listing has only its
|
||||
state and its profiles' names.
|
||||
- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its
|
||||
own so the gateway does not carry its dependencies. `llmgw_model` (a model
|
||||
added at runtime: scaling, class, owner and adapters change in place, the
|
||||
rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the
|
||||
`llmgw_models` and `llmgw_nodes` data sources. Its client is checked
|
||||
against the OpenAPI document, endpoint by endpoint and field by field, so a
|
||||
change to the admin API that breaks it fails its tests; CI runs them.
|
||||
Built and installed by hand for now: see its
|
||||
[README](terraform-provider-llmgw/README.md).
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** `adminapi.Routes` and
|
||||
`internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
|
||||
bearer security, every route, query parameters, schemas from the Go types
|
||||
with their comments), served embedded at `GET /api/v1/openapi.json`, kept true
|
||||
by a route-table test and a staleness test; validated with
|
||||
openapi-spec-validator and openapi-generator-cli. `GET /api/v1/models/{name}`.
|
||||
`docs/admin-api.md`. `terraform-provider-llmgw/` (terraform-plugin-framework):
|
||||
`llmgw_model`, `llmgw_quota`, `llmgw_guard_attachment`, data sources
|
||||
`llmgw_models` and `llmgw_nodes`; client tests against a fake admin API and
|
||||
the OpenAPI document; acceptance tests with Terraform and OpenTofu against the
|
||||
fake; one apply, empty plan, in-place change and destroy against a real
|
||||
`llmgw server` run by hand. Left: organizations, teams, projects, people and
|
||||
keys are not in the admin API (they are `llmgw org`/`users`/`keys` against the
|
||||
store), so not in the provider; a model's definition beyond scaling, class,
|
||||
owner and adapters is changed by replacing it (the API has no edit, and
|
||||
`resources` is not used yet); `canary`, `slo`, `scaling.schedule` and
|
||||
`prewarm`, `guard` role and `origin` are not in `llmgw_model`; models cannot be
|
||||
imported; the provider is not published to a registry and has no release
|
||||
build; the acceptance tests are not run in CI (they need Terraform); the
|
||||
document's schemas mark nothing required and give no enums; `catalog.SLO`'s
|
||||
fields have no JSON tags, so they are `TTFTp99` and `TPOTp99` on the wire.
|
||||
@@ -1,60 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Tracing beyond the router
|
||||
|
||||
- **The control plane traces what it does to the fleet.** A reconcile pass
|
||||
that does something is a `reconcile` trace — passes with nothing to do are
|
||||
not traced at all — with what it did beneath it: `scale up` with `place`
|
||||
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
|
||||
candidate turned down and why, as `ctl explain-placement` would print it),
|
||||
`scale down`, `drain replica` and `stop replica` each with its reason,
|
||||
`evict`, `preempt` with its victims, `reap replica` with the error and
|
||||
when the model is tried again, `restart replica`, `restart refused`,
|
||||
`node unreachable` and `node lost`.
|
||||
- **A replica's start is one trace, down to the node.** `replica start`
|
||||
stays open from placement until the replica serves, fails or is stopped,
|
||||
with each shard's progress as an event and how long it took. The command
|
||||
to each node is a `send start` span whose context travels in the command
|
||||
(a new `trace` field on the agent protocol's `ControlMessage`), and the
|
||||
agent continues the same trace: `start shard`, with `fetch weights`
|
||||
(local, cached, downloaded, or downloaded by another shard; a `download`
|
||||
with its bytes and where a resume began, the source's credentials and
|
||||
query left out), `launch engine` (and `wait for rendezvous` on a worker
|
||||
of a multi-node replica), and `wait for engine` with how often the engine
|
||||
was asked and its last answer. "Why did qwen take four minutes to start"
|
||||
is that one trace.
|
||||
- **Stops, drains and adoptions on a node** are in the control plane's
|
||||
trace too: `stop shard` and `stop engine`, saying whether the engine had
|
||||
to be killed; `drain shard`; and `adopt` when a registering node reports
|
||||
engines already running, with how many nobody claimed. An engine install
|
||||
is an `install engine` trace on the node.
|
||||
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
|
||||
`serve model` and `update model`, with a span per step — registry,
|
||||
mirror, check, catalog, placement, smoke test, replicas — and from the
|
||||
placement step a link to the replica's own start.
|
||||
- **The agent has a `tracing` block** in `agent.yaml`, the same as
|
||||
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
|
||||
when unset). Point both at one collector. A node follows the control
|
||||
plane's sampling for every start it is sent. Traces from a node carry its
|
||||
id as `service.instance.id`, and a control plane's its host name.
|
||||
- With no endpoint nothing changes: every span is the no-op one, and a
|
||||
command carries no trace context.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** spans for the reconcile pass (begun only
|
||||
when a pass does something), placement with its answer or rejections,
|
||||
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
|
||||
lost nodes; a `replica start` span from placement to serving whose context
|
||||
travels to the agent in the start command (`ControlMessage.trace`), where
|
||||
weights fetch, engine launch, rendezvous and readiness wait continue the
|
||||
trace; stops, drains, adoption and engine installs; modelserve jobs as a
|
||||
trace with a span per step linked to the replica's start; an agent
|
||||
`tracing` block; docs in install.md's "Tracing". Left: the router's span
|
||||
for a request does not link to the replica it was sent to's start;
|
||||
heartbeats, fabric scans, logs and agent upgrades are not traced; the
|
||||
supervisor has no restart span because it never restarts an engine itself
|
||||
(a restart is the reconciler's `restart replica` and reaches a node as a
|
||||
stop and a start); canary and promote jobs, which landed on `features-c2`
|
||||
after this was branched, are traced only through the shared step recording
|
||||
(as `<kind> model`), unchecked by a test.
|
||||
@@ -1,62 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Quality suites, and new weights held to one
|
||||
|
||||
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
|
||||
and scores its answers: per case, then the cases weighed into one score, 0
|
||||
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
|
||||
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
|
||||
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
|
||||
and one way to score it — `exact`, `contains`, `regex`, `number` within
|
||||
a `tolerance` (the last number in the answer), `json_schema` (the answer
|
||||
read as JSON, in a code fence or a sentence or not, held to a schema), or
|
||||
`judge`: a rubric another model on the cluster grades 0 to 1. A
|
||||
reasoning model's `<think>` is not part of its answer.
|
||||
- **Run on the control plane**, through the router as its own requests,
|
||||
as a replay is: past admission, quotas, accounting and the guardrails.
|
||||
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
|
||||
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
|
||||
`deploy/eval/`, sized for small models. `--suites` lists them.
|
||||
- **Kept** in the store (migration 0028): every run with its model, the
|
||||
repository and revision of the weights that answered, the suite's name
|
||||
and hash, its score and each case's answer and reason.
|
||||
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
|
||||
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
|
||||
`model promote` take `--eval SUITE --min-score N`, or a model carries one
|
||||
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
|
||||
A promotion asks the canary first and leaves it in place below the score;
|
||||
an update runs the new weights first as a canary given none of the
|
||||
model's requests, asks it, and promotes it or takes it out again, so the
|
||||
old weights serve until the new have passed. The job's `eval` step says
|
||||
the score; `--skip-eval` moves once without the model's suite.
|
||||
- **The console** has **Ask it a quality suite…** (the score counted in as it
|
||||
comes), **Its quality runs** and **Hold its updates to a quality suite…**
|
||||
on a model's menu.
|
||||
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
|
||||
handler cut off at 64 KiB, by the audit log's reading of it.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
|
||||
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge
|
||||
model with a rubric; weights and a threshold), run on the control plane
|
||||
through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`,
|
||||
`--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs`
|
||||
and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a
|
||||
model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote
|
||||
asks the canary first, an update tries the new weights as a 0% canary and
|
||||
promotes or removes it; the console's model menu; `arithmetic` and
|
||||
`json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a
|
||||
model's standing gate is read by name on the control plane at the time of
|
||||
the update, so two control planes need the same `control.eval_dir`; a gated
|
||||
update needs room for one more replica while it runs (there is no in-place
|
||||
gate, and a declared model in server.yaml cannot be updated at all); no
|
||||
comparison against the old weights' score (the gate is an absolute
|
||||
threshold, not "no worse than before"); a run is not stopped once started
|
||||
(Esc in the console only stops watching it), and the in-memory progress of a
|
||||
run lives on the control plane that started it; runs are never pruned; the
|
||||
JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`);
|
||||
judge cases go through the same concurrency as the model's, and a judge's
|
||||
answers are not kept, only its score and reason; a gateway without a
|
||||
control plane cannot run suites (there is no mode that asks through the
|
||||
client API with a key).
|
||||
@@ -1,54 +0,0 @@
|
||||
# C22: MIG
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### MIG instances as GPUs of their own class
|
||||
|
||||
- **A card in MIG mode is placed as its instances.** Discovery reports a card
|
||||
in MIG mode as its MIG instances instead of the whole card, each a device of
|
||||
its own with its UUID (`MIG-…`), its own memory, and a class named from the
|
||||
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
|
||||
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
|
||||
any other, and placement puts a replica on a free instance of it. Both
|
||||
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
|
||||
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
|
||||
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
|
||||
instances; `docs/install.md` has a MIG section on making them.
|
||||
- **One instance a replica.** A process opens only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer between them, so a profile on a
|
||||
MIG class asking for more than one GPU, tp or pp above 1, or more than one
|
||||
node is refused when the model is added, with that reason; placement
|
||||
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
|
||||
for a model one holds and never several together.
|
||||
- **Engines are given the instance by UUID.** Placement numbers card 0's
|
||||
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
|
||||
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
|
||||
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
|
||||
shows the device as `0/mig2`.
|
||||
- **Memory per instance, no made-up utilisation.** Each instance's memory,
|
||||
and the processes holding it, are its own — read per GPU and compute
|
||||
instance from nvidia-smi's report, so a foreign process on one instance
|
||||
keeps that instance out of placement and not its neighbours. Temperature,
|
||||
power and wear are the card's. No driver reports utilisation per instance,
|
||||
so none is published for one: the console says `n/a on MIG`, and
|
||||
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
|
||||
- **A card in MIG mode with nothing carved is kept out of placement**, with
|
||||
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
|
||||
is on.
|
||||
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
|
||||
of a card in MIG mode with no instances, of GPU instances with no compute
|
||||
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
|
||||
command for each.
|
||||
- `ctl nodes` counts a node's devices by class when it has MIG instances or
|
||||
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
|
||||
GPU list shows instances under their card with their own memory and the
|
||||
card's heat and power, and a node's page lists its classes with counts.
|
||||
`ctl status` and the plan count each device under its own class, where a
|
||||
node's cards were all put under its first class before.
|
||||
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
|
||||
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
|
||||
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** MIG instances discovered on both paths (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100 × card + instance, shown as `card/migN`; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (`mig -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model plan`/`serve` by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them.
|
||||
@@ -1,61 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Engines and the mirror
|
||||
|
||||
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
|
||||
installed it: name its interpreter in the node's `agent.yaml`
|
||||
(`engines.trtllm.python`) and the node offers it while that interpreter
|
||||
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
|
||||
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
|
||||
`--pp_size` and served name, through a few lines of its own that make it
|
||||
demand the gateway's key on every path but `/health` (trtllm-serve has no
|
||||
key option), turn on the Prometheus endpoint and iteration statistics, and
|
||||
keep usage reports to NVIDIA off. The router reads it at
|
||||
`/prometheus/metrics` — `trtllm_num_requests_running`,
|
||||
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
|
||||
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
|
||||
GPU; one node per replica. Profiles may not give it an options file,
|
||||
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
|
||||
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
|
||||
engine-check trtllm --bin <python>` checks an installed one. Written
|
||||
against TensorRT-LLM's documented command line, server and metrics and
|
||||
tested with stand-ins for them; not yet run on a datacenter GPU.
|
||||
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
|
||||
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
|
||||
model's projector beside it, each checked against the sha256 Ollama named
|
||||
its file by. With no name it lists Ollama's models; it finds Ollama's
|
||||
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
|
||||
service) or takes `--ollama-dir`. The version is the tag and the start of
|
||||
the digest, since a tag moves in Ollama. It prints a model definition to
|
||||
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
|
||||
view does the same with **I**, for the mirror beside the control plane.
|
||||
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
|
||||
mirror machine into it the way a download is kept: whole or not at all, the
|
||||
checksum beside it, a version never replaced, a `.gguf` checked to be one,
|
||||
and `--sha256` checked when given.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** engine `trtllm` (`internal/agent/engines/trtllm.go`):
|
||||
`trtllm-serve` through a launcher of llmgw's that demands the replica's key
|
||||
(trtllm-serve has none) and writes the options file that turns on
|
||||
`/prometheus/metrics` and the iteration statistics; the scrape contract for
|
||||
it (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
|
||||
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
|
||||
package's metadata; catalog validation (GPU class, one node, no GGUF source,
|
||||
the flags a profile may not give); `ctl engine-check trtllm`, which asks
|
||||
`trtllm-serve serve --help` through its module. Ollama/GGUF import: `mirror
|
||||
import-ollama` (manifest parsing, blob digest check, projector beside the
|
||||
model, all or nothing, a model definition printed), `mirror import` for any
|
||||
local file, and the console's **I** on the Mirror view. Left: TensorRT-LLM has
|
||||
not run on real hardware — the first node with one should run `engine-check`
|
||||
and serve a model before anyone relies on it; no installer for it; no
|
||||
multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one
|
||||
node); Triton Inference Server as an engine is not done; trtllm's queue gauges
|
||||
are set per iteration, so an idle engine shows its last batch until the next
|
||||
request; `--num_serve_frontends` is refused because extra front ends would not
|
||||
demand the key. The Ollama import runs on the mirror machine only — the
|
||||
mirror's API takes no files, so an Ollama elsewhere means copying the blob or
|
||||
running the import there; Ollama models kept as separate tensors or as several
|
||||
weight layers, and its LoRA adapters, are refused or left with a sentence;
|
||||
files an import run with `sudo` writes are root's, as `mirror add`'s are.
|
||||
@@ -1,30 +0,0 @@
|
||||
# C24: the SDK/docs portal
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### Documentation
|
||||
|
||||
- **A docs portal at /docs.** The manual the binary carries — the pages
|
||||
`llmgw docs` prints and the console's Manual shows, the configuration
|
||||
reference among them — rendered on the client port in the portal's own
|
||||
look: a sidebar of the documents in reading order, the open one's sections
|
||||
under it, a link on every heading, and links between documents kept inside
|
||||
the portal (links to files that are not in the binary, such as `deploy/`,
|
||||
go to the repository). No script on the page, so the Content-Security-Policy
|
||||
has nothing to allow. Public by default, like the landing page; set
|
||||
`landing.docs_sign_in: true` to send visitors to sign in first. Signed in,
|
||||
the page carries the usual menu, which now has a **Docs** link.
|
||||
- **A client developer's guide, `docs/clients.md`.** How to use llmgw from
|
||||
the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is
|
||||
needed — in Python, TypeScript, Go and curl: the base URL and key,
|
||||
streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK
|
||||
against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`,
|
||||
`x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`,
|
||||
`/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index
|
||||
leads with it, and the landing page and the account page link to it.
|
||||
- The console's Manual and the portal read one list of documents in one
|
||||
order (`llmgw.Manual`), so the two cannot drift.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [x] 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark, GFM, heading anchors, links rewritten, no script; public, `landing.docs_sign_in` to require a session), and docs/clients.md shows the official OpenAI/Anthropic SDKs against llmgw. Left: the console's generated "Commands" page is not in the portal (it is built from the cobra tree, which internal/web cannot see); no search across documents; the clients.md sections on /v1/messages, batches, tags and x-llmgw-variant were written against the features-c2 commit messages and must be checked against that code when merging (exact `x-llmgw-tags` format, `canary` as the variant value, Anthropic-shaped errors on /v1/messages).
|
||||
@@ -1,30 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### The portal in Turkish
|
||||
|
||||
- **The web portal speaks English and Turkish** — the landing page, sign-up
|
||||
and sign-in, the account page and its editor setup, the claim page, chat,
|
||||
the about page, and every administrators' page (cluster, nodes, NVLink and
|
||||
fabrics, people, limits, usage, organizations, mirror, guardrails), and
|
||||
the docs portal around its documents, which stay English. Flash
|
||||
messages and form errors built by the server are translated too.
|
||||
- **A switcher at the foot of every page** (English · Türkçe) remembers the
|
||||
choice in a cookie; without one the browser's Accept-Language decides, and
|
||||
without that the new `landing.language` (`en` or `tr`, empty is English).
|
||||
An unknown code stops the server starting. `<html lang>` and
|
||||
Content-Language follow the page.
|
||||
- **Figures and dates in the page's language**: a Turkish page writes
|
||||
1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in
|
||||
Turkish. Form values, CSS and SVG coordinates are unchanged.
|
||||
- **Ready for more languages**: the catalogue is JSON under
|
||||
`internal/web/locales/<code>/`, and a language is a directory. Tests refuse
|
||||
a key a page uses that some language lacks, a translation with different
|
||||
arguments from English, and any template text left in English — so a new
|
||||
page cannot quietly ship English-only.
|
||||
- Messages that come from the control plane and other packages (node
|
||||
warnings, engine errors, store refusals) are still shown as written, in
|
||||
English; the API, CLI and console are English.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
24 (portal i18n): built — English/Turkish catalogue with fallback, per-language templates, cookie/Accept-Language/`landing.language` choice and a footer switcher, Turkish numbers and dates, every template and handler message translated, tests that no template text stays English; left — errors built with fmt.Errorf in other packages (store, catalog, mirror, adminapi) still English on Turkish pages, the embedded Latin font subsets lack ğ/ş/İ (latin-ext subsets needed), percent sign placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 footer timestamps and plain-text http.Error bodies untranslated, the /docs documents themselves English, a per-person language column (cookie only today).
|
||||
Reference in new issue
Block a user