Merge features-c2: the docs portal and its links in English and Turkish
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
e8a62b23b7
201 files changed
+26276
-630
No files matched your search
@@ -0,0 +1,55 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Reusing answers
|
||||
|
||||
- **A response cache, per project, off until asked for.** `llmgw keys cache
|
||||
--project P --ttl 1h` has the same request — same project, endpoint,
|
||||
model by its catalog name, and body with `stream`, `stream_options`,
|
||||
`user`, `metadata`, `store` and `service_tier` left out and its keys in
|
||||
any order — answered from the gateway's memory instead of the model, for
|
||||
the TTL. `--off` stops it; without either it says how it stands. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
- **Only answers that come out the same are reused:** chat and completions
|
||||
asked at temperature 0 or with a seed, and embeddings, rerank and score.
|
||||
`--any` reuses sampled answers too. Only a whole 200 from the model asked
|
||||
for is kept — not an error, a fallback's or a canary's answer, one a
|
||||
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
|
||||
- **Streams both ways:** a streamed answer is kept put back together into
|
||||
one completion (tool calls and reasoning included), and a kept answer is
|
||||
streamed as chunks — with the usage chunk if the client asked — to a
|
||||
client that streams. `/v1/messages` and `/v1/responses` get it too.
|
||||
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
|
||||
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
|
||||
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
|
||||
- **Policies still apply:** limits and the input guardrails run before the
|
||||
cache is looked in, the output policy judges a reused answer as it would
|
||||
a new one, and a request under `no_store` or continuing a stored
|
||||
conversation is never served from it nor kept in it. A project's answers
|
||||
are never looked up by another project's keys.
|
||||
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
|
||||
marked `cache_hit` on the usage record (migration 0027), and shown by
|
||||
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
|
||||
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
|
||||
together in each gateway process, least recently used dropped first;
|
||||
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
|
||||
`llmgw_cache_bytes` and `llmgw_cache_entries`.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** exact-match answer cache per project
|
||||
(`keys cache --ttl/--any/--off`, console **Cache answers…**, `projects`
|
||||
columns, 30-second setting cache), reproducible answers only unless
|
||||
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
|
||||
`max_entry_bytes`, streamed answers kept and replayed as chunks,
|
||||
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit and
|
||||
the output policy on it, `no_store` and threads bypass, a hit recorded as
|
||||
`cache_hit` with no tokens and no cost (migration 0027), metrics, docs.
|
||||
Left: semantic (embedding-similarity) caching through the embeddings
|
||||
models; a cache shared across control planes (each process keeps its own,
|
||||
deliberately: a shared one would write every answer to the store); a hit
|
||||
still waits its turn in admission, because the guardrails judge the request
|
||||
after admission and the key is the request they let through; logprobs in a
|
||||
streamed answer leave it unkept; `users forget` does not reach into the
|
||||
cache (answers age out by the TTL); no listing or flushing of what is
|
||||
cached (`--off` stops it being served at once, the memory is freed by TTL
|
||||
and LRU).
|
||||
@@ -0,0 +1,57 @@
|
||||
# C19: OpenAPI for the admin API, and a Terraform provider
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### The admin API described, and a Terraform provider
|
||||
|
||||
- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path
|
||||
and query parameters, and the schema of each request and answer with the
|
||||
meaning of each field, in
|
||||
[docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by
|
||||
every control plane at `GET /api/v1/openapi.json` (with the token, like the
|
||||
rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are
|
||||
in it. Generated by `go generate ./internal/docgen` from a route table in
|
||||
`internal/control/adminapi/routes.go` and the Go types the handlers use, with
|
||||
the fields' comments as descriptions; a test fails when an endpoint is
|
||||
served and not in the table, another when the committed document is stale.
|
||||
Checked with openapi-spec-validator and openapi-generator-cli's validate.
|
||||
- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback,
|
||||
an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`,
|
||||
what a request must carry — an address in `Host`, `application/json` on
|
||||
every change, even a `DELETE` — and some calls with `curl`.
|
||||
- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in
|
||||
force, with the defaults the catalog filled in. The listing has only its
|
||||
state and its profiles' names.
|
||||
- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its
|
||||
own so the gateway does not carry its dependencies. `llmgw_model` (a model
|
||||
added at runtime: scaling, class, owner and adapters change in place, the
|
||||
rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the
|
||||
`llmgw_models` and `llmgw_nodes` data sources. Its client is checked
|
||||
against the OpenAPI document, endpoint by endpoint and field by field, so a
|
||||
change to the admin API that breaks it fails its tests; CI runs them.
|
||||
Built and installed by hand for now: see its
|
||||
[README](terraform-provider-llmgw/README.md).
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** `adminapi.Routes` and
|
||||
`internal/docgen/openapi` generating `docs/admin-api.openapi.json` (3.1,
|
||||
bearer security, every route, query parameters, schemas from the Go types
|
||||
with their comments), served embedded at `GET /api/v1/openapi.json`, kept true
|
||||
by a route-table test and a staleness test; validated with
|
||||
openapi-spec-validator and openapi-generator-cli. `GET /api/v1/models/{name}`.
|
||||
`docs/admin-api.md`. `terraform-provider-llmgw/` (terraform-plugin-framework):
|
||||
`llmgw_model`, `llmgw_quota`, `llmgw_guard_attachment`, data sources
|
||||
`llmgw_models` and `llmgw_nodes`; client tests against a fake admin API and
|
||||
the OpenAPI document; acceptance tests with Terraform and OpenTofu against the
|
||||
fake; one apply, empty plan, in-place change and destroy against a real
|
||||
`llmgw server` run by hand. Left: organizations, teams, projects, people and
|
||||
keys are not in the admin API (they are `llmgw org`/`users`/`keys` against the
|
||||
store), so not in the provider; a model's definition beyond scaling, class,
|
||||
owner and adapters is changed by replacing it (the API has no edit, and
|
||||
`resources` is not used yet); `canary`, `slo`, `scaling.schedule` and
|
||||
`prewarm`, `guard` role and `origin` are not in `llmgw_model`; models cannot be
|
||||
imported; the provider is not published to a registry and has no release
|
||||
build; the acceptance tests are not run in CI (they need Terraform); the
|
||||
document's schemas mark nothing required and give no enums; `catalog.SLO`'s
|
||||
fields have no JSON tags, so they are `TTFTp99` and `TPOTp99` on the wire.
|
||||
@@ -0,0 +1,60 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Tracing beyond the router
|
||||
|
||||
- **The control plane traces what it does to the fleet.** A reconcile pass
|
||||
that does something is a `reconcile` trace — passes with nothing to do are
|
||||
not traced at all — with what it did beneath it: `scale up` with `place`
|
||||
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
|
||||
candidate turned down and why, as `ctl explain-placement` would print it),
|
||||
`scale down`, `drain replica` and `stop replica` each with its reason,
|
||||
`evict`, `preempt` with its victims, `reap replica` with the error and
|
||||
when the model is tried again, `restart replica`, `restart refused`,
|
||||
`node unreachable` and `node lost`.
|
||||
- **A replica's start is one trace, down to the node.** `replica start`
|
||||
stays open from placement until the replica serves, fails or is stopped,
|
||||
with each shard's progress as an event and how long it took. The command
|
||||
to each node is a `send start` span whose context travels in the command
|
||||
(a new `trace` field on the agent protocol's `ControlMessage`), and the
|
||||
agent continues the same trace: `start shard`, with `fetch weights`
|
||||
(local, cached, downloaded, or downloaded by another shard; a `download`
|
||||
with its bytes and where a resume began, the source's credentials and
|
||||
query left out), `launch engine` (and `wait for rendezvous` on a worker
|
||||
of a multi-node replica), and `wait for engine` with how often the engine
|
||||
was asked and its last answer. "Why did qwen take four minutes to start"
|
||||
is that one trace.
|
||||
- **Stops, drains and adoptions on a node** are in the control plane's
|
||||
trace too: `stop shard` and `stop engine`, saying whether the engine had
|
||||
to be killed; `drain shard`; and `adopt` when a registering node reports
|
||||
engines already running, with how many nobody claimed. An engine install
|
||||
is an `install engine` trace on the node.
|
||||
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
|
||||
`serve model` and `update model`, with a span per step — registry,
|
||||
mirror, check, catalog, placement, smoke test, replicas — and from the
|
||||
placement step a link to the replica's own start.
|
||||
- **The agent has a `tracing` block** in `agent.yaml`, the same as
|
||||
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
|
||||
when unset). Point both at one collector. A node follows the control
|
||||
plane's sampling for every start it is sent. Traces from a node carry its
|
||||
id as `service.instance.id`, and a control plane's its host name.
|
||||
- With no endpoint nothing changes: every span is the no-op one, and a
|
||||
command carries no trace context.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** spans for the reconcile pass (begun only
|
||||
when a pass does something), placement with its answer or rejections,
|
||||
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
|
||||
lost nodes; a `replica start` span from placement to serving whose context
|
||||
travels to the agent in the start command (`ControlMessage.trace`), where
|
||||
weights fetch, engine launch, rendezvous and readiness wait continue the
|
||||
trace; stops, drains, adoption and engine installs; modelserve jobs as a
|
||||
trace with a span per step linked to the replica's start; an agent
|
||||
`tracing` block; docs in install.md's "Tracing". Left: the router's span
|
||||
for a request does not link to the replica it was sent to's start;
|
||||
heartbeats, fabric scans, logs and agent upgrades are not traced; the
|
||||
supervisor has no restart span because it never restarts an engine itself
|
||||
(a restart is the reconciler's `restart replica` and reaches a node as a
|
||||
stop and a start); canary and promote jobs, which landed on `features-c2`
|
||||
after this was branched, are traced only through the shared step recording
|
||||
(as `<kind> model`), unchecked by a test.
|
||||
@@ -0,0 +1,62 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Quality suites, and new weights held to one
|
||||
|
||||
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
|
||||
and scores its answers: per case, then the cases weighed into one score, 0
|
||||
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
|
||||
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
|
||||
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
|
||||
and one way to score it — `exact`, `contains`, `regex`, `number` within
|
||||
a `tolerance` (the last number in the answer), `json_schema` (the answer
|
||||
read as JSON, in a code fence or a sentence or not, held to a schema), or
|
||||
`judge`: a rubric another model on the cluster grades 0 to 1. A
|
||||
reasoning model's `<think>` is not part of its answer.
|
||||
- **Run on the control plane**, through the router as its own requests,
|
||||
as a replay is: past admission, quotas, accounting and the guardrails.
|
||||
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
|
||||
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
|
||||
`deploy/eval/`, sized for small models. `--suites` lists them.
|
||||
- **Kept** in the store (migration 0028): every run with its model, the
|
||||
repository and revision of the weights that answered, the suite's name
|
||||
and hash, its score and each case's answer and reason.
|
||||
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
|
||||
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
|
||||
`model promote` take `--eval SUITE --min-score N`, or a model carries one
|
||||
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
|
||||
A promotion asks the canary first and leaves it in place below the score;
|
||||
an update runs the new weights first as a canary given none of the
|
||||
model's requests, asks it, and promotes it or takes it out again, so the
|
||||
old weights serve until the new have passed. The job's `eval` step says
|
||||
the score; `--skip-eval` moves once without the model's suite.
|
||||
- **The console** has **Ask it a quality suite…** (the score counted in as it
|
||||
comes), **Its quality runs** and **Hold its updates to a quality suite…**
|
||||
on a model's menu.
|
||||
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
|
||||
handler cut off at 64 KiB, by the audit log's reading of it.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** `internal/bench/eval` (suites in YAML or
|
||||
JSON lines; exact, contains, regex, number, a JSON Schema subset, and a judge
|
||||
model with a rubric; weights and a threshold), run on the control plane
|
||||
through `Router.Replay`; `llmgw ctl eval MODEL --suite FILE|NAME`, `--history`,
|
||||
`--run`, `--suites`; runs and their cases kept (migration 0028, `eval_runs`
|
||||
and `eval_cases`); `model update`/`promote --eval SUITE --min-score N`, a
|
||||
model's standing gate (`ctl model gate`, `eval:` in the catalog) — a promote
|
||||
asks the canary first, an update tries the new weights as a 0% canary and
|
||||
promotes or removes it; the console's model menu; `arithmetic` and
|
||||
`json-format` in `deploy/eval/`, carried in the binary. Left: the suite of a
|
||||
model's standing gate is read by name on the control plane at the time of
|
||||
the update, so two control planes need the same `control.eval_dir`; a gated
|
||||
update needs room for one more replica while it runs (there is no in-place
|
||||
gate, and a declared model in server.yaml cannot be updated at all); no
|
||||
comparison against the old weights' score (the gate is an absolute
|
||||
threshold, not "no worse than before"); a run is not stopped once started
|
||||
(Esc in the console only stops watching it), and the in-memory progress of a
|
||||
run lives on the control plane that started it; runs are never pruned; the
|
||||
JSON Schema support is a subset (no `$ref`, `oneOf`, `not`, `uniqueItems`);
|
||||
judge cases go through the same concurrency as the model's, and a judge's
|
||||
answers are not kept, only its score and reason; a gateway without a
|
||||
control plane cannot run suites (there is no mode that asks through the
|
||||
client API with a key).
|
||||
@@ -0,0 +1,54 @@
|
||||
# C22: MIG
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### MIG instances as GPUs of their own class
|
||||
|
||||
- **A card in MIG mode is placed as its instances.** Discovery reports a card
|
||||
in MIG mode as its MIG instances instead of the whole card, each a device of
|
||||
its own with its UUID (`MIG-…`), its own memory, and a class named from the
|
||||
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
|
||||
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
|
||||
any other, and placement puts a replica on a free instance of it. Both
|
||||
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
|
||||
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
|
||||
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
|
||||
instances; `docs/install.md` has a MIG section on making them.
|
||||
- **One instance a replica.** A process opens only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer between them, so a profile on a
|
||||
MIG class asking for more than one GPU, tp or pp above 1, or more than one
|
||||
node is refused when the model is added, with that reason; placement
|
||||
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
|
||||
for a model one holds and never several together.
|
||||
- **Engines are given the instance by UUID.** Placement numbers card 0's
|
||||
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
|
||||
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
|
||||
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
|
||||
shows the device as `0/mig2`.
|
||||
- **Memory per instance, no made-up utilisation.** Each instance's memory,
|
||||
and the processes holding it, are its own — read per GPU and compute
|
||||
instance from nvidia-smi's report, so a foreign process on one instance
|
||||
keeps that instance out of placement and not its neighbours. Temperature,
|
||||
power and wear are the card's. No driver reports utilisation per instance,
|
||||
so none is published for one: the console says `n/a on MIG`, and
|
||||
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
|
||||
- **A card in MIG mode with nothing carved is kept out of placement**, with
|
||||
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
|
||||
is on.
|
||||
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
|
||||
of a card in MIG mode with no instances, of GPU instances with no compute
|
||||
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
|
||||
command for each.
|
||||
- `ctl nodes` counts a node's devices by class when it has MIG instances or
|
||||
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
|
||||
GPU list shows instances under their card with their own memory and the
|
||||
card's heat and power, and a node's page lists its classes with counts.
|
||||
`ctl status` and the plan count each device under its own class, where a
|
||||
node's cards were all put under its first class before.
|
||||
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
|
||||
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
|
||||
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** MIG instances discovered on both paths (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100 × card + instance, shown as `card/migN`; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (`mig -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model plan`/`serve` by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them.
|
||||
@@ -0,0 +1,61 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Engines and the mirror
|
||||
|
||||
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
|
||||
installed it: name its interpreter in the node's `agent.yaml`
|
||||
(`engines.trtllm.python`) and the node offers it while that interpreter
|
||||
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
|
||||
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
|
||||
`--pp_size` and served name, through a few lines of its own that make it
|
||||
demand the gateway's key on every path but `/health` (trtllm-serve has no
|
||||
key option), turn on the Prometheus endpoint and iteration statistics, and
|
||||
keep usage reports to NVIDIA off. The router reads it at
|
||||
`/prometheus/metrics` — `trtllm_num_requests_running`,
|
||||
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
|
||||
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
|
||||
GPU; one node per replica. Profiles may not give it an options file,
|
||||
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
|
||||
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
|
||||
engine-check trtllm --bin <python>` checks an installed one. Written
|
||||
against TensorRT-LLM's documented command line, server and metrics and
|
||||
tested with stand-ins for them; not yet run on a datacenter GPU.
|
||||
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
|
||||
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
|
||||
model's projector beside it, each checked against the sha256 Ollama named
|
||||
its file by. With no name it lists Ollama's models; it finds Ollama's
|
||||
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
|
||||
service) or takes `--ollama-dir`. The version is the tag and the start of
|
||||
the digest, since a tag moves in Ollama. It prints a model definition to
|
||||
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
|
||||
view does the same with **I**, for the mirror beside the control plane.
|
||||
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
|
||||
mirror machine into it the way a download is kept: whole or not at all, the
|
||||
checksum beside it, a version never replaced, a `.gguf` checked to be one,
|
||||
and `--sha256` checked when given.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** engine `trtllm` (`internal/agent/engines/trtllm.go`):
|
||||
`trtllm-serve` through a launcher of llmgw's that demands the replica's key
|
||||
(trtllm-serve has none) and writes the options file that turns on
|
||||
`/prometheus/metrics` and the iteration statistics; the scrape contract for
|
||||
it (`scrape.TRTLLMMapping`, with a per-engine metrics path); found, not
|
||||
installed — `engines.trtllm.python` in `agent.yaml`, probed from the
|
||||
package's metadata; catalog validation (GPU class, one node, no GGUF source,
|
||||
the flags a profile may not give); `ctl engine-check trtllm`, which asks
|
||||
`trtllm-serve serve --help` through its module. Ollama/GGUF import: `mirror
|
||||
import-ollama` (manifest parsing, blob digest check, projector beside the
|
||||
model, all or nothing, a model definition printed), `mirror import` for any
|
||||
local file, and the console's **I** on the Mirror view. Left: TensorRT-LLM has
|
||||
not run on real hardware — the first node with one should run `engine-check`
|
||||
and serve a model before anyone relies on it; no installer for it; no
|
||||
multi-node TensorRT-LLM replica (it needs an MPI or Slurm launch from one
|
||||
node); Triton Inference Server as an engine is not done; trtllm's queue gauges
|
||||
are set per iteration, so an idle engine shows its last batch until the next
|
||||
request; `--num_serve_frontends` is refused because extra front ends would not
|
||||
demand the key. The Ollama import runs on the mirror machine only — the
|
||||
mirror's API takes no files, so an Ollama elsewhere means copying the blob or
|
||||
running the import there; Ollama models kept as separate tensors or as several
|
||||
weight layers, and its LoRA adapters, are refused or left with a sentence;
|
||||
files an import run with `sudo` writes are root's, as `mirror add`'s are.
|
||||
@@ -0,0 +1,30 @@
|
||||
# C24: the SDK/docs portal
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### Documentation
|
||||
|
||||
- **A docs portal at /docs.** The manual the binary carries — the pages
|
||||
`llmgw docs` prints and the console's Manual shows, the configuration
|
||||
reference among them — rendered on the client port in the portal's own
|
||||
look: a sidebar of the documents in reading order, the open one's sections
|
||||
under it, a link on every heading, and links between documents kept inside
|
||||
the portal (links to files that are not in the binary, such as `deploy/`,
|
||||
go to the repository). No script on the page, so the Content-Security-Policy
|
||||
has nothing to allow. Public by default, like the landing page; set
|
||||
`landing.docs_sign_in: true` to send visitors to sign in first. Signed in,
|
||||
the page carries the usual menu, which now has a **Docs** link.
|
||||
- **A client developer's guide, `docs/clients.md`.** How to use llmgw from
|
||||
the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is
|
||||
needed — in Python, TypeScript, Go and curl: the base URL and key,
|
||||
streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK
|
||||
against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`,
|
||||
`x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`,
|
||||
`/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index
|
||||
leads with it, and the landing page and the account page link to it.
|
||||
- The console's Manual and the portal read one list of documents in one
|
||||
order (`llmgw.Manual`), so the two cannot drift.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [x] 24 (SDK/docs portal): built — /docs renders the embedded manual (goldmark, GFM, heading anchors, links rewritten, no script; public, `landing.docs_sign_in` to require a session), and docs/clients.md shows the official OpenAI/Anthropic SDKs against llmgw. Left: the console's generated "Commands" page is not in the portal (it is built from the cobra tree, which internal/web cannot see); no search across documents; the clients.md sections on /v1/messages, batches, tags and x-llmgw-variant were written against the features-c2 commit messages and must be checked against that code when merging (exact `x-llmgw-tags` format, `canary` as the variant value, Anthropic-shaped errors on /v1/messages).
|
||||
+3
-2
@@ -5,7 +5,8 @@
|
||||
- **The web portal speaks English and Turkish** — the landing page, sign-up
|
||||
and sign-in, the account page and its editor setup, the claim page, chat,
|
||||
the about page, and every administrators' page (cluster, nodes, NVLink and
|
||||
fabrics, people, limits, usage, organizations, mirror, guardrails). Flash
|
||||
fabrics, people, limits, usage, organizations, mirror, guardrails), and
|
||||
the docs portal around its documents, which stay English. Flash
|
||||
messages and form errors built by the server are translated too.
|
||||
- **A switcher at the foot of every page** (English · Türkçe) remembers the
|
||||
choice in a cookie; without one the browser's Accept-Language decides, and
|
||||
@@ -26,4 +27,4 @@
|
||||
|
||||
## BACKLOG
|
||||
|
||||
24 (portal i18n): built — English/Turkish catalogue with fallback, per-language templates, cookie/Accept-Language/`landing.language` choice and a footer switcher, Turkish numbers and dates, every template and handler message translated, tests that no template text stays English; left — errors built with fmt.Errorf in other packages (store, catalog, mirror, adminapi) still English on Turkish pages, the embedded Latin font subsets lack ğ/ş/İ (latin-ext subsets needed), percent sign placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 footer timestamps and plain-text http.Error bodies untranslated, a per-person language column (cookie only today).
|
||||
24 (portal i18n): built — English/Turkish catalogue with fallback, per-language templates, cookie/Accept-Language/`landing.language` choice and a footer switcher, Turkish numbers and dates, every template and handler message translated, tests that no template text stays English; left — errors built with fmt.Errorf in other packages (store, catalog, mirror, adminapi) still English on Turkish pages, the embedded Latin font subsets lack ğ/ş/İ (latin-ext subsets needed), percent sign placement is mixed (%50 on limits, 93% on cluster pages), RFC3339 footer timestamps and plain-text http.Error bodies untranslated, the /docs documents themselves English, a per-person language column (cookie only today).
|
||||
Reference in new issue
Block a user