diff --git a/BACKLOG.md b/BACKLOG.md index 3bdb2b0..82d0a3c 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -509,27 +509,199 @@ portal is not done — the page shows; the console and ctl change. **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage and rollups and erases threads/memory (`store/hierarchy.go:241` leaves it today); per-project "never route to an External provider". -17. **Response/semantic cache (M).** Exact-match first, per project, +17. **Done (branch `features-c2`):** exact-match answer cache per project (`keys + cache --ttl/--any/--off`, console **Cache answers…**, `projects` + columns, 30-second setting cache), reproducible answers only unless + `--any`, LRU in memory bounded by `router.cache.max_bytes` and + `max_entry_bytes`, streamed answers kept and replayed as chunks, + `x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit + and the output policy on it, `no_store` and threads bypass, a hit + recorded as `cache_hit` with no tokens and no cost (migration 0027), + metrics, docs. Left: semantic (embedding-similarity) caching through the + embeddings models; a cache shared across control planes (each process + keeps its own, deliberately: a shared one would write every answer to + the store); a hit still waits its turn in admission, because the + guardrails judge the request after admission and the key is the request + they let through; logprobs in a streamed answer leave it unkept; `users + forget` does not reach into the cache (answers age out by the TTL); no + listing or flushing of what is cached (`--off` stops it being served at + once, the memory is freed by TTL and LRU). + **Response/semantic cache (M).** Exact-match first, per project, opt-in, TTL; semantic via the embeddings models later. Before `dispatch` in `router/handlers.go`. -18. **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`, +18. **Done (branch `features-c2`):** offline bundles — `llmgw bundle create` + (releases with signed checksums, every llama.cpp build per arch/variant + with GitHub's checksums, starter/judge and named Hugging Face model + files pinned and checked; directory or tar; manifest sha256), `bundle + import` (checked, all-or-nothing, into the mirror under `_llmgw/`), + `bundle trust`, `bundle show --verify`; + `offline.mirror`/`offline.checksums` in server.yaml, with nodes + installing llama.cpp from the mirror against checksums the control plane + sends over the agent stream (proto `EngineWant.offline_*`), releases + defaulting to the mirror for `node add`, `ctl upgrade` and agent + upgrades, update checks off; `server init --bundle` and the setup form's + bundle fields; docs/data.md's offline claim corrected; install.md + "Offline sites". A Helm chart for the control plane + (`deploy/helm/llmgw`, one replica, SQLite PVC or external PostgreSQL, + Secrets, Services, Ingress, ServiceMonitor) with + `deploy/container/Dockerfile` and docs/kubernetes.md; GPU nodes stay + outside Kubernetes by design. Left: vLLM/SGLang offline (a wheelhouse of + several GB per CUDA build and a uv `--find-links` install path); the + NVML release binary and uv in bundles; the setup wizard's preflight + still fails a machine without internet; `join`'s printed `curl` from the + mirror checks nothing; models in a bundle are single files (no whole- + repository safetensors sets yet); the chart runs one replica (a standby + is never ready, so HA in Kubernetes needs a different readiness design), + has not been installed on a real cluster, and ships no agent DaemonSet; + no published container image. + **Kubernetes/Helm and an air-gapped bundle (M).** `deploy/`, `agent/provision`; `docs/data.md` claims offline operation but install assumes the internet. -19. **OpenAPI for the admin API, Terraform provider (M).** Generate from +19. **Done (branch `features-c2`):** `adminapi.Routes` and + `internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1, + bearer security, every route, query parameters, schemas from the Go + types with their comments), served embedded at `GET + /api/v1/openapi.json`, kept true by a route-table test and a staleness + test; validated with openapi-spec-validator and openapi-generator-cli. + `GET /api/v1/models/{name}`. `docs/admin-api.md`. `terraform-provider- + llmgw/` (terraform-plugin-framework): `llmgw_model`, `llmgw_quota`, + `llmgw_guard_attachment`, data sources `llmgw_models` and `llmgw_nodes`; + client tests against a fake admin API and the OpenAPI document; + acceptance tests with Terraform and OpenTofu against the fake; one + apply, empty plan, in-place change and destroy against a real `llmgw + server` run by hand. Left: organizations, teams, projects, people and + keys are not in the admin API (they are `llmgw org`/`users`/`keys` + against the store), so not in the provider; a model's definition beyond + scaling, class, owner and adapters is changed by replacing it (the API + has no edit, and `resources` is not used yet); `canary`, `slo`, + `scaling.schedule` and `prewarm`, `guard` role and `origin` are not in + `llmgw_model`; models cannot be imported; the provider is not published + to a registry and has no release build; the acceptance tests are not run + in CI (they need Terraform); the document's schemas mark nothing + required and give no enums; `catalog.SLO`'s fields have no JSON tags, so + they are `TTFTp99` and `TPOTp99` on the wire. + **OpenAPI for the admin API, Terraform provider (M).** Generate from `adminapi/api.go` routes via `docgen`. -20. **Tracing beyond the router (S).** Spans for reconciler, placement, +20. **Done (branch `features-c2`):** spans for the reconcile pass (begun only + when a pass does something), placement with its answer or rejections, + scale-up/down, drain, stop, eviction, preemption, reaping, restarts and + lost nodes; a `replica start` span from placement to serving whose + context travels to the agent in the start command + (`ControlMessage.trace`), where weights fetch, engine launch, rendezvous + and readiness wait continue the trace; stops, drains, adoption and + engine installs; modelserve jobs as a trace with a span per step linked + to the replica's start; an agent `tracing` block; docs in install.md's + "Tracing". Left: the router's span for a request does not link to the + replica it was sent to's start; heartbeats, fabric scans, logs and agent + upgrades are not traced; the supervisor has no restart span because it + never restarts an engine itself (a restart is the reconciler's `restart + replica` and reaches a node as a stop and a start); canary and promote + jobs, which landed on `features-c2` after this was branched, are traced + only through the shared step recording (as ` model`), unchecked by + a test. + **Tracing beyond the router (S).** Spans for reconciler, placement, agent start (`obs/trace.go`, `control/reconciler`, `agent/supervisor`). -21. **Eval harness (M).** `llmgw eval` extending `internal/bench` with +21. **Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or + JSON lines; exact, contains, regex, number, a JSON Schema subset, and a + judge model with a rubric; weights and a threshold), run on the control + plane through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, + `--history`, `--run`, `--suites`; runs and their cases kept (migration + 0028, `eval_runs` and `eval_cases`); `model update`/`promote --eval + SUITE --min-score N`, a model's standing gate (`ctl model gate`, `eval:` + in the catalog) — a promote asks the canary first, an update tries the + new weights as a 0% canary and promotes or removes it; the console's + model menu; `arithmetic` and `json-format` in `deploy/eval/`, carried in + the binary. Left: the suite of a model's standing gate is read by name + on the control plane at the time of the update, so two control planes + need the same `control.eval_dir`; a gated update needs room for one more + replica while it runs (there is no in-place gate, and a declared model + in server.yaml cannot be updated at all); no comparison against the old + weights' score (the gate is an absolute threshold, not "no worse than + before"); a run is not stopped once started (Esc in the console only + stops watching it), and the in-memory progress of a run lives on the + control plane that started it; runs are never pruned; the JSON Schema + support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); judge + cases go through the same concurrency as the model's, and a judge's + answers are not kept, only its score and reason; a gateway without a + control plane cannot run suites (there is no mode that asks through the + client API with a key). + **Eval harness (M).** `llmgw eval` extending `internal/bench` with quality suites gating `model update`. -22. **MIG (M–L).** Treat a MIG slice as a GPU class in +22. **Done (branch `features-c2`):** MIG instances discovered on both paths + (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful + fake) and reported instead of their card, each with its MIG UUID, own + memory and class `-mig-`; device numbers 1000 + 100 + × card + instance, shown as `card/migN`; placement on MIG classes; + tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation + and again in placement; `CUDA_VISIBLE_DEVICES=MIG-` for vLLM and + llama.cpp (no Vulkan device variable on MIG); per-instance memory and + processes in heartbeats and the GPU check, card sensors under each + instance, utilisation absent rather than zero; doctor MIG findings (`mig + -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model + plan`/`serve` by class; docs/install.md MIG section. Written and tested + against fakes only — untested on real MIG hardware. Left: run it on an + A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG + device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` + reaches vLLM's workers through Ray); instances changed while the agent + runs are only seen when it restarts (no rescan); llmgw does not create + or reshape instances (no mig-parted integration); quotas count an + instance as one GPU, the same as a whole card; SGLang on MIG not + checked; device numbers assume fewer than 100 instances per card and + that instance numbers stay put while shards run on them. + **MIG (M–L).** Treat a MIG slice as a GPU class in `agent/discovery/gpu_nvml.go` and `control/placement`. -23. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF +23. **Done (branch `features-c2`):** engine `trtllm` + (`internal/agent/engines/trtllm.go`): `trtllm-serve` through a launcher + of llmgw's that demands the replica's key (trtllm-serve has none) and + writes the options file that turns on `/prometheus/metrics` and the + iteration statistics; the scrape contract for it + (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not + installed — `engines.trtllm.python` in `agent.yaml`, probed from the + package's metadata; catalog validation (GPU class, one node, no GGUF + source, the flags a profile may not give); `ctl engine-check trtllm`, + which asks `trtllm-serve serve --help` through its module. Ollama/GGUF + import: `mirror import-ollama` (manifest parsing, blob digest check, + projector beside the model, all or nothing, a model definition printed), + `mirror import` for any local file, and the console's **I** on the + Mirror view. Left: TensorRT-LLM has not run on real hardware — the first + node with one should run `engine-check` and serve a model before anyone + relies on it; no installer for it; no multi-node TensorRT-LLM replica + (it needs an MPI or Slurm launch from one node); Triton Inference Server + as an engine is not done; trtllm's queue gauges are set per iteration, + so an idle engine shows its last batch until the next request; + `--num_serve_frontends` is refused because extra front ends would not + demand the key. The Ollama import runs on the mirror machine only — the + mirror's API takes no files, so an Ollama elsewhere means copying the + blob or running the import there; Ollama models kept as separate tensors + or as several weight layers, and its LoRA adapters, are refused or left + with a sentence; files an import run with `sudo` writes are root's, as + `mirror add`'s are. + **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF import. `agent/engines/engine.go` interface and the metric contract in `router/scrape`. 24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for vLLM, multi-cluster federation via External providers, facility-aware placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs portal. + Done so far (branch `features-c2`): + - 24 (portal i18n): built — English/Turkish catalogue with fallback, per- + language templates, cookie/Accept-Language/`landing.language` choice + and a footer switcher, Turkish numbers and dates, every template and + handler message translated, tests that no template text stays English; + left — errors built with fmt.Errorf in other packages (store, catalog, + mirror, adminapi) still English on Turkish pages, the embedded Latin + font subsets lack ğ/ş/İ (latin-ext subsets needed), percent sign + placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 + footer timestamps and plain-text http.Error bodies untranslated, the + /docs documents themselves English, a per-person language column + (cookie only today). + - 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark, + GFM, heading anchors, links rewritten, no script; public, + `landing.docs_sign_in` to require a session), and docs/clients.md + shows the official OpenAI/Anthropic SDKs against llmgw. Left: the + console's generated "Commands" page is not in the portal (it is built + from the cobra tree, which internal/web cannot see); no search across + documents. --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a0d229..6dde2e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,319 @@ APIs change between minors without a deprecation period. ## Unreleased +### Reusing answers + +- **A response cache, per project, off until asked for.** `llmgw keys cache + --project P --ttl 1h` has the same request — same project, endpoint, + model by its catalog name, and body with `stream`, `stream_options`, + `user`, `metadata`, `store` and `service_tier` left out and its keys in + any order — answered from the gateway's memory instead of the model, for + the TTL. `--off` stops it; without either it says how it stands. The + console has it as **Cache answers…** on a project. +- **Only answers that come out the same are reused:** chat and completions + asked at temperature 0 or with a seed, and embeddings, rerank and score. + `--any` reuses sampled answers too. Only a whole 200 from the model asked + for is kept — not an error, a fallback's or a canary's answer, one a + guardrail changed or stopped, or one over `router.cache.max_entry_bytes`. +- **Streams both ways:** a streamed answer is kept put back together into + one completion (tool calls and reasoning included), and a kept answer is + streamed as chunks — with the usage chunk if the client asked — to a + client that streams. `/v1/messages` and `/v1/responses` get it too. +- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a + project that reuses answers, `Age` on a hit. `Cache-Control: no-cache` + or `no-store`, or `x-llmgw-cache: no`, asks the model anew. +- **Policies still apply:** limits and the input guardrails run before the + cache is looked in, the output policy judges a reused answer as it would + a new one, and a request under `no_store` or continuing a stored + conversation is never served from it nor kept in it. A project's answers + are never looked up by another project's keys. +- **A hit costs nothing:** recorded as a 200 with no tokens and no cost, + marked `cache_hit` on the usage record (migration 0027), and shown by + `llmgw ctl request`; it counts towards the requests-per-minute limit only. +- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects + together in each gateway process, least recently used dropped first; + nothing is written to the store. `llmgw_cache_requests_total{model,result}`, + `llmgw_cache_bytes` and `llmgw_cache_entries`. + +### Offline sites and Kubernetes + +- **Offline bundles.** `llmgw bundle create` (on any machine with internet) + collects what a cluster would otherwise download — llmgw's release for + each `--arch` with its signed checksums, every llama.cpp build for those + architectures (or the `--variant` ones) with the checksums GitHub + publishes, the starter model and guardrails' judge, and any `--model + name/version=hf:owner/repo/file` pinned to the registry's commit and its + checksum — each checked as it arrives, into a directory or one `.tar` + (`--tar`), with a manifest whose sha256 it prints to carry apart from the + bundle. Run again into the same directory, it downloads only what is + missing. `llmgw bundle show --verify` reads one back. +- **`llmgw bundle import`** checks every file against the manifest and + llmgw's release against the project's signing key, and only then puts them + into the mirror — releases and engines under `_llmgw/`, models where the + mirror keeps models, with their checksums — so a damaged bundle leaves the + mirror as it was; a different file under a version already there is + refused. On a control plane it also records the engines' checksums in + `offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust` + does that alone, for a mirror on another machine. +- **`offline.mirror` in server.yaml** makes the cluster reach nothing + outside: nodes install llama.cpp from the mirror, each file held to the + checksum the control plane sends them over the agent's authenticated + connection — never one from the mirror — and `releases` defaults to the + mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch + from it; no registry is asked about newer weights. A release may now come + over plain HTTP when the build trusts a signing key, since the signature + holds whatever carries it. vLLM and SGLang come from PyPI and are said to + be unsupported on an offline node rather than attempted. +- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two + fields on the setup form: installs a control plane offline — the bundle + into the mirror, the checksums trusted, `offline:` written to server.yaml, + the starter model and judge served from the bundle or skipped when it does + not carry them. +- `docs/data.md` no longer says only weights and `ctl plan` reach outside: + it lists everything that does, and what an offline cluster changes. +- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica + in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent + port's certificate and the admin token from Secrets, the API and metrics + Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress + and ServiceMonitor, `config` laid into server.yaml, and the offline + checksums from a ConfigMap; read-only root, non-root, ready on `/leader`. + `deploy/container/Dockerfile` builds the image it runs. GPU nodes stay + outside Kubernetes and join with `node add` (docs/kubernetes.md says why). + +### The admin API described, and a Terraform provider + +- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path + and query parameters, and the schema of each request and answer with the + meaning of each field, in + [docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by + every control plane at `GET /api/v1/openapi.json` (with the token, like the + rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are + in it. Generated by `go generate ./internal/docgen` from a route table in + `internal/control/adminapi/routes.go` and the Go types the handlers use, with + the fields' comments as descriptions; a test fails when an endpoint is + served and not in the table, another when the committed document is stale. + Checked with openapi-spec-validator and openapi-generator-cli's validate. +- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback, + an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`, + what a request must carry — an address in `Host`, `application/json` on + every change, even a `DELETE` — and some calls with `curl`. +- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in + force, with the defaults the catalog filled in. The listing has only its + state and its profiles' names. +- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its + own so the gateway does not carry its dependencies. `llmgw_model` (a model + added at runtime: scaling, class, owner and adapters change in place, the + rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the + `llmgw_models` and `llmgw_nodes` data sources. Its client is checked + against the OpenAPI document, endpoint by endpoint and field by field, so a + change to the admin API that breaks it fails its tests; CI runs them. + Built and installed by hand for now: see its + [README](terraform-provider-llmgw/README.md). + +### Tracing beyond the router + +- **The control plane traces what it does to the fleet.** A reconcile pass + that does something is a `reconcile` trace — passes with nothing to do are + not traced at all — with what it did beneath it: `scale up` with `place` + (the profile, engine, GPU class, pool, fabric and nodes chosen, or every + candidate turned down and why, as `ctl explain-placement` would print it), + `scale down`, `drain replica` and `stop replica` each with its reason, + `evict`, `preempt` with its victims, `reap replica` with the error and + when the model is tried again, `restart replica`, `restart refused`, + `node unreachable` and `node lost`. +- **A replica's start is one trace, down to the node.** `replica start` + stays open from placement until the replica serves, fails or is stopped, + with each shard's progress as an event and how long it took. The command + to each node is a `send start` span whose context travels in the command + (a new `trace` field on the agent protocol's `ControlMessage`), and the + agent continues the same trace: `start shard`, with `fetch weights` + (local, cached, downloaded, or downloaded by another shard; a `download` + with its bytes and where a resume began, the source's credentials and + query left out), `launch engine` (and `wait for rendezvous` on a worker + of a multi-node replica), and `wait for engine` with how often the engine + was asked and its last answer. "Why did qwen take four minutes to start" + is that one trace. +- **Stops, drains and adoptions on a node** are in the control plane's + trace too: `stop shard` and `stop engine`, saying whether the engine had + to be killed; `drain shard`; and `adopt` when a registering node reports + engines already running, with how many nobody claimed. An engine install + is an `install engine` trace on the node. +- **Model jobs are traces.** `ctl model serve` and `ctl model update` are + `serve model` and `update model`, with a span per step — registry, + mirror, check, catalog, placement, smoke test, replicas — and from the + placement step a link to the replica's own start. +- **The agent has a `tracing` block** in `agent.yaml`, the same as + `server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT` + when unset). Point both at one collector. A node follows the control + plane's sampling for every start it is sent. Traces from a node carry its + id as `service.instance.id`, and a control plane's its host name. +- With no endpoint nothing changes: every span is the no-op one, and a + command carries no trace context. + +### Quality suites, and new weights held to one + +- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite + and scores its answers: per case, then the cases weighed into one score, 0 + to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero + below it. `--concurrency`, `--judge`, `--detach`, `--json`. + - **Suites** are YAML or JSON lines: a prompt or a conversation, a weight, + and one way to score it — `exact`, `contains`, `regex`, `number` within + a `tolerance` (the last number in the answer), `json_schema` (the answer + read as JSON, in a code fence or a sentence or not, held to a schema), or + `judge`: a rubric another model on the cluster grades 0 to 1. A + reasoning model's `` is not part of its answer. + - **Run on the control plane**, through the router as its own requests, + as a replay is: past admission, quotas, accounting and the guardrails. + `--suite` is a file sent whole, or a suite's name in `control.eval_dir` + (`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in + `deploy/eval/`, sized for small models. `--suites` lists them. + - **Kept** in the store (migration 0028): every run with its model, the + repository and revision of the weights that answered, the suite's name + and hash, its score and each case's answer and reason. + `llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one. +- **Updates and promotions held to a suite.** `llmgw ctl model update` and + `model promote` take `--eval SUITE --min-score N`, or a model carries one + (`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition). + A promotion asks the canary first and leaves it in place below the score; + an update runs the new weights first as a canary given none of the + model's requests, asks it, and promotes it or takes it out again, so the + old weights serve until the new have passed. The job's `eval` step says + the score; `--skip-eval` moves once without the model's suite. +- **The console** has **Ask it a quality suite…** (the score counted in as it + comes), **Its quality runs** and **Hold its updates to a quality suite…** + on a model's menu. +- **Fixed:** a change to the admin API with a body over 64 KiB reached its + handler cut off at 64 KiB, by the audit log's reading of it. + +### MIG instances as GPUs of their own class + +- **A card in MIG mode is placed as its instances.** Discovery reports a card + in MIG mode as its MIG instances instead of the whole card, each a device of + its own with its UUID (`MIG-…`), its own memory, and a class named from the + card's and the profile: two `3g.40gb` instances of an A100 80GB are two + devices of class `a100-80g-mig-3g.40gb`. A profile names that class like + any other, and placement puts a replica on a free instance of it. Both + discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their + memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`, + `GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create + instances; `docs/install.md` has a MIG section on making them. +- **One instance a replica.** A process opens only one MIG device, and + instances have no NVLink or peer-to-peer between them, so a profile on a + MIG class asking for more than one GPU, tp or pp above 1, or more than one + node is refused when the model is added, with that reason; placement + refuses it again. `ctl model plan` and `ctl model serve` plan an instance + for a model one holds and never several together. +- **Engines are given the instance by UUID.** Placement numbers card 0's + instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns + that into `CUDA_VISIBLE_DEVICES=MIG-` for vLLM and llama.cpp, and + leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view + shows the device as `0/mig2`. +- **Memory per instance, no made-up utilisation.** Each instance's memory, + and the processes holding it, are its own — read per GPU and compute + instance from nvidia-smi's report, so a foreign process on one instance + keeps that instance out of placement and not its neighbours. Temperature, + power and wear are the card's. No driver reports utilisation per instance, + so none is published for one: the console says `n/a on MIG`, and + `llmgw_gpu_utilization_percent` has no series for it rather than a zero. +- **A card in MIG mode with nothing carved is kept out of placement**, with + the reason in `ctl nodes`, since CUDA will not open the whole card while MIG + is on. +- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns + of a card in MIG mode with no instances, of GPU instances with no compute + instance, and of a mode change waiting for a reset, with the `nvidia-smi` + command for each. +- `ctl nodes` counts a node's devices by class when it has MIG instances or + more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's + GPU list shows instances under their card with their own memory and the + card's heat and power, and a node's page lists its classes with counts. + `ctl status` and the plan count each device under its own class, where a + node's cards were all put under its first class before. +- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile, + GPU and compute instance ids); the node API's GPUs gain `label`, `class`, + `card`, `mig_profile` and memory, and nodes gain `gpus_by_class`. + +### Engines and the mirror + +- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you + installed it: name its interpreter in the node's `agent.yaml` + (`engines.trtllm.python`) and the node offers it while that interpreter + has the `tensorrt_llm` package. llmgw does not install it. The agent starts + `trtllm-serve serve` with the replica's host, port, `--tp_size`, + `--pp_size` and served name, through a few lines of its own that make it + demand the gateway's key on every path but `/health` (trtllm-serve has no + key option), turn on the Prometheus endpoint and iteration statistics, and + keep usage reports to NVIDIA off. The router reads it at + `/prometheus/metrics` — `trtllm_num_requests_running`, + `trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and + successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer + GPU; one node per replica. Profiles may not give it an options file, + `--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the + agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl + engine-check trtllm --bin ` checks an installed one. Written + against TensorRT-LLM's documented command line, server and metrics and + tested with stand-ins for them; not yet run on a datacenter GPU. +- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already + have in Ollama into the mirror for llama.cpp: its GGUF, and a vision + model's projector beside it, each checked against the sha256 Ollama named + its file by. With no name it lists Ollama's models; it finds Ollama's + directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux + service) or takes `--ollama-dir`. The version is the tag and the start of + the digest, since a tag moves in Ollama. It prints a model definition to + add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror** + view does the same with **I**, for the mirror beside the control plane. +- **`llmgw mirror import FILE... --model NAME`** copies files already on the + mirror machine into it the way a download is kept: whole or not at all, the + checksum beside it, a version never replaced, a `.gguf` checked to be one, + and `--sha256` checked when given. + +### The portal in Turkish + +- **The web portal speaks English and Turkish** — the landing page, sign-up + and sign-in, the account page and its editor setup, the claim page, chat, + the about page, and every administrators' page (cluster, nodes, NVLink and + fabrics, people, limits, usage, organizations, mirror, guardrails), and + the docs portal around its documents, which stay English. Flash + messages and form errors built by the server are translated too. +- **A switcher at the foot of every page** (English · Türkçe) remembers the + choice in a cookie; without one the browser's Accept-Language decides, and + without that the new `landing.language` (`en` or `tr`, empty is English). + An unknown code stops the server starting. `` and + Content-Language follow the page. +- **Figures and dates in the page's language**: a Turkish page writes + 1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in + Turkish. Form values, CSS and SVG coordinates are unchanged. +- **Ready for more languages**: the catalogue is JSON under + `internal/web/locales//`, and a language is a directory. Tests refuse + a key a page uses that some language lacks, a translation with different + arguments from English, and any template text left in English — so a new + page cannot quietly ship English-only. +- Messages that come from the control plane and other packages (node + warnings, engine errors, store refusals) are still shown as written, in + English; the API, CLI and console are English. + +### Documentation + +- **A docs portal at /docs.** The manual the binary carries — the pages + `llmgw docs` prints and the console's Manual shows, the configuration + reference among them — rendered on the client port in the portal's own + look: a sidebar of the documents in reading order, the open one's sections + under it, a link on every heading, and links between documents kept inside + the portal (links to files that are not in the binary, such as `deploy/`, + go to the repository). No script on the page, so the Content-Security-Policy + has nothing to allow. Public by default, like the landing page; set + `landing.docs_sign_in: true` to send visitors to sign in first. Signed in, + the page carries the usual menu, which now has a **Docs** link. +- **A client developer's guide, `docs/clients.md`.** How to use llmgw from + the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is + needed — in Python, TypeScript, Go and curl: the base URL and key, + streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK + against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`, + `x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`, + `/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index + leads with it, and the landing page and the account page link to it. +- The console's Manual and the portal read one list of documents in one + order (`llmgw.Manual`), so the two cannot drift. + ### Administration - **An audit log.** Every change made through the admin API (`llmgw ctl`, diff --git a/README.md b/README.md index 248b969..c2570e7 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,17 @@ Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md), - **Notifications** — webhooks, signed or Slack-shaped, for a model parked, a node lost, a budget spent and new weights published (`notify.webhooks`, `ctl notify test`), and Alertmanager receiver templates. +- **Traffic and models** — request tags, canaries of new weights with + automatic rollback, quality suites gating updates, a response cache, + kept and replayed requests, batches, Anthropic's Messages API, more of + OpenAI's (structured output, images, rerank, audio, moderations), + scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, and models + imported from Ollama. +- **Operations** — `server restore` and off-machine backup copies, secrets + in Vault, forgetting a person and keeping a project's requests in the + cluster, offline installs from a bundle, a Helm chart, tracing from the + control plane to the engines, the admin API in OpenAPI with a Terraform + provider, and the portal in English and Turkish with its docs at /docs. Every one of these is in the console as well as on the command line: [docs/console.md](docs/console.md). @@ -95,6 +106,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console. the firewall, upgrades, backups - [docs/node-onboarding.md](docs/node-onboarding.md) — adding nodes to a cluster that exists +- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on + Kubernetes with the Helm chart, and why GPU nodes stay outside it - [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what llmgw needs of them, checking a node with `llmgw doctor`, and what to look at when a replica spanning nodes will not run @@ -105,6 +118,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console. every view, and the command each of its actions runs - [docs/onboarding.md](docs/onboarding.md) — accounts, keys, limits, staff roles and single sign-on +- [docs/clients.md](docs/clients.md) — using the gateway from code with the + official OpenAI and Anthropic SDKs, and llmgw's extensions - [docs/guardrails.md](docs/guardrails.md) — checking requests and answers: policies, detectors, a judge model, and where each applies - [docs/choosing-models.md](docs/choosing-models.md) — which models to run, @@ -117,6 +132,9 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console. does not exist or miss one that does - [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and what leaves the machine; written for a security review +- [docs/admin-api.md](docs/admin-api.md) and its OpenAPI document — the + admin API, for scripts; [terraform-provider-llmgw/](terraform-provider-llmgw/) + manages models, quotas and guardrails with Terraform or OpenTofu - [deploy/](deploy/) — [a mirror](deploy/mirror/) on its own machine, [Caddy and nginx](deploy/proxy/) in front of the client port, [two control planes](deploy/ha/) behind a balancer, and [Prometheus, Alertmanager and diff --git a/notes/c17.md b/notes/c17.md deleted file mode 100644 index 75558d8..0000000 --- a/notes/c17.md +++ /dev/null @@ -1,55 +0,0 @@ -## CHANGELOG - -### Reusing answers - -- **A response cache, per project, off until asked for.** `llmgw keys cache - --project P --ttl 1h` has the same request — same project, endpoint, - model by its catalog name, and body with `stream`, `stream_options`, - `user`, `metadata`, `store` and `service_tier` left out and its keys in - any order — answered from the gateway's memory instead of the model, for - the TTL. `--off` stops it; without either it says how it stands. The - console has it as **Cache answers…** on a project. -- **Only answers that come out the same are reused:** chat and completions - asked at temperature 0 or with a seed, and embeddings, rerank and score. - `--any` reuses sampled answers too. Only a whole 200 from the model asked - for is kept — not an error, a fallback's or a canary's answer, one a - guardrail changed or stopped, or one over `router.cache.max_entry_bytes`. -- **Streams both ways:** a streamed answer is kept put back together into - one completion (tool calls and reasoning included), and a kept answer is - streamed as chunks — with the usage chunk if the client asked — to a - client that streams. `/v1/messages` and `/v1/responses` get it too. -- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a - project that reuses answers, `Age` on a hit. `Cache-Control: no-cache` - or `no-store`, or `x-llmgw-cache: no`, asks the model anew. -- **Policies still apply:** limits and the input guardrails run before the - cache is looked in, the output policy judges a reused answer as it would - a new one, and a request under `no_store` or continuing a stored - conversation is never served from it nor kept in it. A project's answers - are never looked up by another project's keys. -- **A hit costs nothing:** recorded as a 200 with no tokens and no cost, - marked `cache_hit` on the usage record (migration 0027), and shown by - `llmgw ctl request`; it counts towards the requests-per-minute limit only. -- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects - together in each gateway process, least recently used dropped first; - nothing is written to the store. `llmgw_cache_requests_total{model,result}`, - `llmgw_cache_bytes` and `llmgw_cache_entries`. - -## BACKLOG - -**Done (branch `features-c2`):** exact-match answer cache per project -(`keys cache --ttl/--any/--off`, console **Cache answers…**, `projects` -columns, 30-second setting cache), reproducible answers only unless -`--any`, LRU in memory bounded by `router.cache.max_bytes` and -`max_entry_bytes`, streamed answers kept and replayed as chunks, -`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit and -the output policy on it, `no_store` and threads bypass, a hit recorded as -`cache_hit` with no tokens and no cost (migration 0027), metrics, docs. -Left: semantic (embedding-similarity) caching through the embeddings -models; a cache shared across control planes (each process keeps its own, -deliberately: a shared one would write every answer to the store); a hit -still waits its turn in admission, because the guardrails judge the request -after admission and the key is the request they let through; logprobs in a -streamed answer leave it unkept; `users forget` does not reach into the -cache (answers age out by the TTL); no listing or flushing of what is -cached (`--off` stops it being served at once, the memory is freed by TTL -and LRU). diff --git a/notes/c18.md b/notes/c18.md deleted file mode 100644 index addab1c..0000000 --- a/notes/c18.md +++ /dev/null @@ -1,78 +0,0 @@ -# C18 — Kubernetes/Helm and an air-gapped bundle - -README's documentation list should gain a line for `docs/kubernetes.md` -(the control plane in Kubernetes from the Helm chart, and why GPU nodes stay -outside it) — README.md was not edited here. - -## CHANGELOG - -### Offline sites and Kubernetes - -- **Offline bundles.** `llmgw bundle create` (on any machine with internet) - collects what a cluster would otherwise download — llmgw's release for - each `--arch` with its signed checksums, every llama.cpp build for those - architectures (or the `--variant` ones) with the checksums GitHub - publishes, the starter model and guardrails' judge, and any `--model - name/version=hf:owner/repo/file` pinned to the registry's commit and its - checksum — each checked as it arrives, into a directory or one `.tar` - (`--tar`), with a manifest whose sha256 it prints to carry apart from the - bundle. Run again into the same directory, it downloads only what is - missing. `llmgw bundle show --verify` reads one back. -- **`llmgw bundle import`** checks every file against the manifest and - llmgw's release against the project's signing key, and only then puts them - into the mirror — releases and engines under `_llmgw/`, models where the - mirror keeps models, with their checksums — so a damaged bundle leaves the - mirror as it was; a different file under a version already there is - refused. On a control plane it also records the engines' checksums in - `offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust` - does that alone, for a mirror on another machine. -- **`offline.mirror` in server.yaml** makes the cluster reach nothing - outside: nodes install llama.cpp from the mirror, each file held to the - checksum the control plane sends them over the agent's authenticated - connection — never one from the mirror — and `releases` defaults to the - mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch - from it; no registry is asked about newer weights. A release may now come - over plain HTTP when the build trusts a signing key, since the signature - holds whatever carries it. vLLM and SGLang come from PyPI and are said to - be unsupported on an offline node rather than attempted. -- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two - fields on the setup form: installs a control plane offline — the bundle - into the mirror, the checksums trusted, `offline:` written to server.yaml, - the starter model and judge served from the bundle or skipped when it does - not carry them. -- `docs/data.md` no longer says only weights and `ctl plan` reach outside: - it lists everything that does, and what an offline cluster changes. -- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica - in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent - port's certificate and the admin token from Secrets, the API and metrics - Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress - and ServiceMonitor, `config` laid into server.yaml, and the offline - checksums from a ConfigMap; read-only root, non-root, ready on `/leader`. - `deploy/container/Dockerfile` builds the image it runs. GPU nodes stay - outside Kubernetes and join with `node add` (docs/kubernetes.md says why). - -## BACKLOG - -**Done (branch `features-c2`):** offline bundles — `llmgw bundle create` -(releases with signed checksums, every llama.cpp build per arch/variant -with GitHub's checksums, starter/judge and named Hugging Face model files -pinned and checked; directory or tar; manifest sha256), `bundle import` -(checked, all-or-nothing, into the mirror under `_llmgw/`), `bundle trust`, -`bundle show --verify`; `offline.mirror`/`offline.checksums` in server.yaml, -with nodes installing llama.cpp from the mirror against checksums the -control plane sends over the agent stream (proto `EngineWant.offline_*`), -releases defaulting to the mirror for `node add`, `ctl upgrade` and agent -upgrades, update checks off; `server init --bundle` and the setup form's -bundle fields; docs/data.md's offline claim corrected; install.md "Offline -sites". A Helm chart for the control plane (`deploy/helm/llmgw`, one replica, -SQLite PVC or external PostgreSQL, Secrets, Services, Ingress, -ServiceMonitor) with `deploy/container/Dockerfile` and docs/kubernetes.md; -GPU nodes stay outside Kubernetes by design. Left: vLLM/SGLang offline (a -wheelhouse of several GB per CUDA build and a uv `--find-links` install -path); the NVML release binary and uv in bundles; the setup wizard's -preflight still fails a machine without internet; `join`'s printed `curl` -from the mirror checks nothing; models in a bundle are single files -(no whole-repository safetensors sets yet); the chart runs one replica (a -standby is never ready, so HA in Kubernetes needs a different readiness -design), has not been installed on a real cluster, and ships no agent -DaemonSet; no published container image. diff --git a/notes/c19.md b/notes/c19.md deleted file mode 100644 index 76e95f3..0000000 --- a/notes/c19.md +++ /dev/null @@ -1,57 +0,0 @@ -# C19: OpenAPI for the admin API, and a Terraform provider - -## CHANGELOG - -### The admin API described, and a Terraform provider - -- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path - and query parameters, and the schema of each request and answer with the - meaning of each field, in - [docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by - every control plane at `GET /api/v1/openapi.json` (with the token, like the - rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are - in it. Generated by `go generate ./internal/docgen` from a route table in - `internal/control/adminapi/routes.go` and the Go types the handlers use, with - the fields' comments as descriptions; a test fails when an endpoint is - served and not in the table, another when the committed document is stale. - Checked with openapi-spec-validator and openapi-generator-cli's validate. -- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback, - an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`, - what a request must carry — an address in `Host`, `application/json` on - every change, even a `DELETE` — and some calls with `curl`. -- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in - force, with the defaults the catalog filled in. The listing has only its - state and its profiles' names. -- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its - own so the gateway does not carry its dependencies. `llmgw_model` (a model - added at runtime: scaling, class, owner and adapters change in place, the - rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the - `llmgw_models` and `llmgw_nodes` data sources. Its client is checked - against the OpenAPI document, endpoint by endpoint and field by field, so a - change to the admin API that breaks it fails its tests; CI runs them. - Built and installed by hand for now: see its - [README](terraform-provider-llmgw/README.md). - -## BACKLOG - -**Done (branch `features-c2`):** `adminapi.Routes` and -`internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1, -bearer security, every route, query parameters, schemas from the Go types -with their comments), served embedded at `GET /api/v1/openapi.json`, kept true -by a route-table test and a staleness test; validated with -openapi-spec-validator and openapi-generator-cli. `GET /api/v1/models/{name}`. -`docs/admin-api.md`. `terraform-provider-llmgw/` (terraform-plugin-framework): -`llmgw_model`, `llmgw_quota`, `llmgw_guard_attachment`, data sources -`llmgw_models` and `llmgw_nodes`; client tests against a fake admin API and -the OpenAPI document; acceptance tests with Terraform and OpenTofu against the -fake; one apply, empty plan, in-place change and destroy against a real -`llmgw server` run by hand. Left: organizations, teams, projects, people and -keys are not in the admin API (they are `llmgw org`/`users`/`keys` against the -store), so not in the provider; a model's definition beyond scaling, class, -owner and adapters is changed by replacing it (the API has no edit, and -`resources` is not used yet); `canary`, `slo`, `scaling.schedule` and -`prewarm`, `guard` role and `origin` are not in `llmgw_model`; models cannot be -imported; the provider is not published to a registry and has no release -build; the acceptance tests are not run in CI (they need Terraform); the -document's schemas mark nothing required and give no enums; `catalog.SLO`'s -fields have no JSON tags, so they are `TTFTp99` and `TPOTp99` on the wire. diff --git a/notes/c20.md b/notes/c20.md deleted file mode 100644 index 0e76094..0000000 --- a/notes/c20.md +++ /dev/null @@ -1,60 +0,0 @@ -## CHANGELOG - -### Tracing beyond the router - -- **The control plane traces what it does to the fleet.** A reconcile pass - that does something is a `reconcile` trace — passes with nothing to do are - not traced at all — with what it did beneath it: `scale up` with `place` - (the profile, engine, GPU class, pool, fabric and nodes chosen, or every - candidate turned down and why, as `ctl explain-placement` would print it), - `scale down`, `drain replica` and `stop replica` each with its reason, - `evict`, `preempt` with its victims, `reap replica` with the error and - when the model is tried again, `restart replica`, `restart refused`, - `node unreachable` and `node lost`. -- **A replica's start is one trace, down to the node.** `replica start` - stays open from placement until the replica serves, fails or is stopped, - with each shard's progress as an event and how long it took. The command - to each node is a `send start` span whose context travels in the command - (a new `trace` field on the agent protocol's `ControlMessage`), and the - agent continues the same trace: `start shard`, with `fetch weights` - (local, cached, downloaded, or downloaded by another shard; a `download` - with its bytes and where a resume began, the source's credentials and - query left out), `launch engine` (and `wait for rendezvous` on a worker - of a multi-node replica), and `wait for engine` with how often the engine - was asked and its last answer. "Why did qwen take four minutes to start" - is that one trace. -- **Stops, drains and adoptions on a node** are in the control plane's - trace too: `stop shard` and `stop engine`, saying whether the engine had - to be killed; `drain shard`; and `adopt` when a registering node reports - engines already running, with how many nobody claimed. An engine install - is an `install engine` trace on the node. -- **Model jobs are traces.** `ctl model serve` and `ctl model update` are - `serve model` and `update model`, with a span per step — registry, - mirror, check, catalog, placement, smoke test, replicas — and from the - placement step a link to the replica's own start. -- **The agent has a `tracing` block** in `agent.yaml`, the same as - `server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT` - when unset). Point both at one collector. A node follows the control - plane's sampling for every start it is sent. Traces from a node carry its - id as `service.instance.id`, and a control plane's its host name. -- With no endpoint nothing changes: every span is the no-op one, and a - command carries no trace context. - -## BACKLOG - -**Done (branch `features-c2`):** spans for the reconcile pass (begun only -when a pass does something), placement with its answer or rejections, -scale-up/down, drain, stop, eviction, preemption, reaping, restarts and -lost nodes; a `replica start` span from placement to serving whose context -travels to the agent in the start command (`ControlMessage.trace`), where -weights fetch, engine launch, rendezvous and readiness wait continue the -trace; stops, drains, adoption and engine installs; modelserve jobs as a -trace with a span per step linked to the replica's start; an agent -`tracing` block; docs in install.md's "Tracing". Left: the router's span -for a request does not link to the replica it was sent to's start; -heartbeats, fabric scans, logs and agent upgrades are not traced; the -supervisor has no restart span because it never restarts an engine itself -(a restart is the reconciler's `restart replica` and reaches a node as a -stop and a start); canary and promote jobs, which landed on `features-c2` -after this was branched, are traced only through the shared step recording -(as ` model`), unchecked by a test. diff --git a/notes/c21.md b/notes/c21.md deleted file mode 100644 index baa30a3..0000000 --- a/notes/c21.md +++ /dev/null @@ -1,62 +0,0 @@ -## CHANGELOG - -### Quality suites, and new weights held to one - -- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite - and scores its answers: per case, then the cases weighed into one score, 0 - to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero - below it. `--concurrency`, `--judge`, `--detach`, `--json`. - - **Suites** are YAML or JSON lines: a prompt or a conversation, a weight, - and one way to score it — `exact`, `contains`, `regex`, `number` within - a `tolerance` (the last number in the answer), `json_schema` (the answer - read as JSON, in a code fence or a sentence or not, held to a schema), or - `judge`: a rubric another model on the cluster grades 0 to 1. A - reasoning model's `` is not part of its answer. - - **Run on the control plane**, through the router as its own requests, - as a replay is: past admission, quotas, accounting and the guardrails. - `--suite` is a file sent whole, or a suite's name in `control.eval_dir` - (`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in - `deploy/eval/`, sized for small models. `--suites` lists them. - - **Kept** in the store (migration 0028): every run with its model, the - repository and revision of the weights that answered, the suite's name - and hash, its score and each case's answer and reason. - `llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one. -- **Updates and promotions held to a suite.** `llmgw ctl model update` and - `model promote` take `--eval SUITE --min-score N`, or a model carries one - (`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition). - A promotion asks the canary first and leaves it in place below the score; - an update runs the new weights first as a canary given none of the - model's requests, asks it, and promotes it or takes it out again, so the - old weights serve until the new have passed. The job's `eval` step says - the score; `--skip-eval` moves once without the model's suite. -- **The console** has **Ask it a quality suite…** (the score counted in as it - comes), **Its quality runs** and **Hold its updates to a quality suite…** - on a model's menu. -- **Fixed:** a change to the admin API with a body over 64 KiB reached its - handler cut off at 64 KiB, by the audit log's reading of it. - -## BACKLOG - -**Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or -JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge -model with a rubric; weights and a threshold), run on the control plane -through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`, -`--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs` -and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a -model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote -asks the canary first, an update tries the new weights as a 0% canary and -promotes or removes it; the console's model menu; `arithmetic` and -`json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a -model's standing gate is read by name on the control plane at the time of -the update, so two control planes need the same `control.eval_dir`; a gated -update needs room for one more replica while it runs (there is no in-place -gate, and a declared model in server.yaml cannot be updated at all); no -comparison against the old weights' score (the gate is an absolute -threshold, not "no worse than before"); a run is not stopped once started -(Esc in the console only stops watching it), and the in-memory progress of a -run lives on the control plane that started it; runs are never pruned; the -JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`); -judge cases go through the same concurrency as the model's, and a judge's -answers are not kept, only its score and reason; a gateway without a -control plane cannot run suites (there is no mode that asks through the -client API with a key). diff --git a/notes/c22.md b/notes/c22.md deleted file mode 100644 index 0907520..0000000 --- a/notes/c22.md +++ /dev/null @@ -1,54 +0,0 @@ -# C22: MIG - -## CHANGELOG - -### MIG instances as GPUs of their own class - -- **A card in MIG mode is placed as its instances.** Discovery reports a card - in MIG mode as its MIG instances instead of the whole card, each a device of - its own with its UUID (`MIG-…`), its own memory, and a class named from the - card's and the profile: two `3g.40gb` instances of an A100 80GB are two - devices of class `a100-80g-mig-3g.40gb`. A profile names that class like - any other, and placement puts a replica on a free instance of it. Both - discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their - memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`, - `GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create - instances; `docs/install.md` has a MIG section on making them. -- **One instance a replica.** A process opens only one MIG device, and - instances have no NVLink or peer-to-peer between them, so a profile on a - MIG class asking for more than one GPU, tp or pp above 1, or more than one - node is refused when the model is added, with that reason; placement - refuses it again. `ctl model plan` and `ctl model serve` plan an instance - for a model one holds and never several together. -- **Engines are given the instance by UUID.** Placement numbers card 0's - instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns - that into `CUDA_VISIBLE_DEVICES=MIG-` for vLLM and llama.cpp, and - leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view - shows the device as `0/mig2`. -- **Memory per instance, no made-up utilisation.** Each instance's memory, - and the processes holding it, are its own — read per GPU and compute - instance from nvidia-smi's report, so a foreign process on one instance - keeps that instance out of placement and not its neighbours. Temperature, - power and wear are the card's. No driver reports utilisation per instance, - so none is published for one: the console says `n/a on MIG`, and - `llmgw_gpu_utilization_percent` has no series for it rather than a zero. -- **A card in MIG mode with nothing carved is kept out of placement**, with - the reason in `ctl nodes`, since CUDA will not open the whole card while MIG - is on. -- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns - of a card in MIG mode with no instances, of GPU instances with no compute - instance, and of a mode change waiting for a reset, with the `nvidia-smi` - command for each. -- `ctl nodes` counts a node's devices by class when it has MIG instances or - more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's - GPU list shows instances under their card with their own memory and the - card's heat and power, and a node's page lists its classes with counts. - `ctl status` and the plan count each device under its own class, where a - node's cards were all put under its first class before. -- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile, - GPU and compute instance ids); the node API's GPUs gain `label`, `class`, - `card`, `mig_profile` and memory, and nodes gain `gpus_by_class`. - -## BACKLOG - -**Done (branch `features-c2`):** MIG instances discovered on both paths (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class `-mig-`; device numbers 1000 + 100 × card + instance, shown as `card/migN`; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; `CUDA_VISIBLE_DEVICES=MIG-` for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (`mig -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model plan`/`serve` by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them. diff --git a/notes/c23.md b/notes/c23.md deleted file mode 100644 index 01db2bd..0000000 --- a/notes/c23.md +++ /dev/null @@ -1,61 +0,0 @@ -## CHANGELOG - -### Engines and the mirror - -- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you - installed it: name its interpreter in the node's `agent.yaml` - (`engines.trtllm.python`) and the node offers it while that interpreter - has the `tensorrt_llm` package. llmgw does not install it. The agent starts - `trtllm-serve serve` with the replica's host, port, `--tp_size`, - `--pp_size` and served name, through a few lines of its own that make it - demand the gateway's key on every path but `/health` (trtllm-serve has no - key option), turn on the Prometheus endpoint and iteration statistics, and - keep usage reports to NVIDIA off. The router reads it at - `/prometheus/metrics` — `trtllm_num_requests_running`, - `trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and - successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer - GPU; one node per replica. Profiles may not give it an options file, - `--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the - agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl - engine-check trtllm --bin ` checks an installed one. Written - against TensorRT-LLM's documented command line, server and metrics and - tested with stand-ins for them; not yet run on a datacenter GPU. -- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already - have in Ollama into the mirror for llama.cpp: its GGUF, and a vision - model's projector beside it, each checked against the sha256 Ollama named - its file by. With no name it lists Ollama's models; it finds Ollama's - directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux - service) or takes `--ollama-dir`. The version is the tag and the start of - the digest, since a tag moves in Ollama. It prints a model definition to - add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror** - view does the same with **I**, for the mirror beside the control plane. -- **`llmgw mirror import FILE... --model NAME`** copies files already on the - mirror machine into it the way a download is kept: whole or not at all, the - checksum beside it, a version never replaced, a `.gguf` checked to be one, - and `--sha256` checked when given. - -## BACKLOG - -**Done (branch `features-c2`):** engine `trtllm` (`internal/agent/engines/trtllm.go`): -`trtllm-serve` through a launcher of llmgw's that demands the replica's key -(trtllm-serve has none) and writes the options file that turns on -`/prometheus/metrics` and the iteration statistics; the scrape contract for -it (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not -installed — `engines.trtllm.python` in `agent.yaml`, probed from the -package's metadata; catalog validation (GPU class, one node, no GGUF source, -the flags a profile may not give); `ctl engine-check trtllm`, which asks -`trtllm-serve serve --help` through its module. Ollama/GGUF import: `mirror -import-ollama` (manifest parsing, blob digest check, projector beside the -model, all or nothing, a model definition printed), `mirror import` for any -local file, and the console's **I** on the Mirror view. Left: TensorRT-LLM has -not run on real hardware — the first node with one should run `engine-check` -and serve a model before anyone relies on it; no installer for it; no -multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one -node); Triton Inference Server as an engine is not done; trtllm's queue gauges -are set per iteration, so an idle engine shows its last batch until the next -request; `--num_serve_frontends` is refused because extra front ends would not -demand the key. The Ollama import runs on the mirror machine only — the -mirror's API takes no files, so an Ollama elsewhere means copying the blob or -running the import there; Ollama models kept as separate tensors or as several -weight layers, and its LoRA adapters, are refused or left with a sentence; -files an import run with `sudo` writes are root's, as `mirror add`'s are. diff --git a/notes/c24-docs.md b/notes/c24-docs.md deleted file mode 100644 index 8026661..0000000 --- a/notes/c24-docs.md +++ /dev/null @@ -1,30 +0,0 @@ -# C24: the SDK/docs portal - -## CHANGELOG - -### Documentation - -- **A docs portal at /docs.** The manual the binary carries — the pages - `llmgw docs` prints and the console's Manual shows, the configuration - reference among them — rendered on the client port in the portal's own - look: a sidebar of the documents in reading order, the open one's sections - under it, a link on every heading, and links between documents kept inside - the portal (links to files that are not in the binary, such as `deploy/`, - go to the repository). No script on the page, so the Content-Security-Policy - has nothing to allow. Public by default, like the landing page; set - `landing.docs_sign_in: true` to send visitors to sign in first. Signed in, - the page carries the usual menu, which now has a **Docs** link. -- **A client developer's guide, `docs/clients.md`.** How to use llmgw from - the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is - needed — in Python, TypeScript, Go and curl: the base URL and key, - streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK - against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`, - `x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`, - `/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index - leads with it, and the landing page and the account page link to it. -- The console's Manual and the portal read one list of documents in one - order (`llmgw.Manual`), so the two cannot drift. - -## BACKLOG - -- [x] 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark, GFM, heading anchors, links rewritten, no script; public, `landing.docs_sign_in` to require a session), and docs/clients.md shows the official OpenAI/Anthropic SDKs against llmgw. Left: the console's generated "Commands" page is not in the portal (it is built from the cobra tree, which internal/web cannot see); no search across documents; the clients.md sections on /v1/messages, batches, tags and x-llmgw-variant were written against the features-c2 commit messages and must be checked against that code when merging (exact `x-llmgw-tags` format, `canary` as the variant value, Anthropic-shaped errors on /v1/messages). diff --git a/notes/c24-i18n.md b/notes/c24-i18n.md deleted file mode 100644 index 7902bba..0000000 --- a/notes/c24-i18n.md +++ /dev/null @@ -1,30 +0,0 @@ -## CHANGELOG - -### The portal in Turkish - -- **The web portal speaks English and Turkish** — the landing page, sign-up - and sign-in, the account page and its editor setup, the claim page, chat, - the about page, and every administrators' page (cluster, nodes, NVLink and - fabrics, people, limits, usage, organizations, mirror, guardrails), and - the docs portal around its documents, which stay English. Flash - messages and form errors built by the server are translated too. -- **A switcher at the foot of every page** (English · Türkçe) remembers the - choice in a cookie; without one the browser's Accept-Language decides, and - without that the new `landing.language` (`en` or `tr`, empty is English). - An unknown code stops the server starting. `` and - Content-Language follow the page. -- **Figures and dates in the page's language**: a Turkish page writes - 1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in - Turkish. Form values, CSS and SVG coordinates are unchanged. -- **Ready for more languages**: the catalogue is JSON under - `internal/web/locales//`, and a language is a directory. Tests refuse - a key a page uses that some language lacks, a translation with different - arguments from English, and any template text left in English — so a new - page cannot quietly ship English-only. -- Messages that come from the control plane and other packages (node - warnings, engine errors, store refusals) are still shown as written, in - English; the API, CLI and console are English. - -## BACKLOG - -24 (portal i18n): built — English/Turkish catalogue with fallback, per-language templates, cookie/Accept-Language/`landing.language` choice and a footer switcher, Turkish numbers and dates, every template and handler message translated, tests that no template text stays English; left — errors built with fmt.Errorf in other packages (store, catalog, mirror, adminapi) still English on Turkish pages, the embedded Latin font subsets lack ğ/ş/İ (latin-ext subsets needed), percent sign placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 footer timestamps and plain-text http.Error bodies untranslated, the /docs documents themselves English, a per-person language column (cookie only today).