diff --git a/llmgw-plan.md b/llmgw-plan.md index 87321041..2fdba568 100644 --- a/llmgw-plan.md +++ b/llmgw-plan.md @@ -21,7 +21,8 @@ saturation) and **scheduling** (plugin-based filter/score/pick with queue, KV-cache and prefix awareness), plus fallback, retry, cooldown and circuit breaking. - Fabric awareness: a multi-node replica **never** spans two fabrics; a TP group **never** mixes GPU types. -- API keys, rate limits, token accounting, Prometheus metrics. +- API keys, rate limits, token accounting and budgets, Prometheus metrics. Persistence for these is + the only thing the store exists for (§6.5, §7.9). ### 1.2 Out of scope (at least for v1) - Inference itself. Go only orchestrates and proxies; vLLM/SGLang run the model. @@ -46,6 +47,7 @@ | **Routing Profile** | The scorer set and weights the router uses for a class of request (chat / batch / long-context). Not to be confused with the model **Profile** above. | | **Band** | A priority class (platinum/gold/silver/bronze). The unit of admission decisions. | | **Block** | The unit of KV cache accounting; a fixed number of tokens (16 in vLLM, 64 by default in the GIE indexer). | +| **Unit** | The billing unit budgets are counted in. Self-hosted models have no per-token price, so an operator defines what a unit is: GPU-seconds, a notional rate per million tokens, or 1.0 everywhere, which turns budgets into token quotas (§7.9.4). | --- @@ -82,7 +84,7 @@ semantic routing / LLM-based model selection (out of scope), and external provid │ │ (data) │◄─┤ - Inventory │ │ │ │ │ │ - Catalog │ │ │ └─────┬──────┘ │ - Scheduler │ │ - │ │ │ - Store (SQLite) │ │ + │ │ │ - Store (phase 5) │ │ │ │ └─────────▲────────────┘ │ └────────┼───────────────────┼──────────────┘ HTTP (eth) │ │ gRPC stream (mTLS, agent dials out) @@ -474,10 +476,72 @@ the replicas it knows about, which mitigates the single-point-of-failure risk in cache ages past `endpoint_cache_max_age` (say 5 min) it only raises a warning metric; it never stops traffic. -### 6.5 Store -- SQLite in WAL mode. Tables: `nodes`, `models`, `profiles`, `replicas`, `shards`, `api_keys`, `usage_rollup`, `events`. -- Inventory and replica state can be rebuilt from the agents; what genuinely needs to persist is the catalog, the keys and the usage records. -- HA (v2): Raft-replicated state, or an external Postgres/etcd. +### 6.5 What is stored, and what is not + +v0.1 of this plan listed eight tables. Most of them were a mistake, and the +reason is worth stating because it applies to every piece of state here: + +> **A database that describes the cluster can be wrong. The cluster cannot.** + +An engine that crashes during a restart leaves a row saying "running". A node +whose GPU developed an ECC fault leaves a row saying the card is fine. Every +such row is a trap: the scheduler believes it, places work accordingly, and the +failure surfaces somewhere else entirely. Rebuilding the same state from the +fleet on every reconnect cannot go stale, because the fleet is the thing being +described. + +So: + +| State | Persisted | Where it actually comes from | +|---|---|---| +| Nodes, GPUs, fabrics | no | Every agent sends its full inventory on registration (§5.6) | +| Replicas, shards | no | Reported by agents; see the restart hole below | +| Models, profiles | no | The YAML catalog is the source of truth | +| API keys and the tenancy tree | **yes** | Created at runtime, secret, rotated (§7.9) | +| Usage records and rollups | **yes** | Accounting must survive a restart (§7.9) | +| Capacity profiles from `ctl bench` | **yes** | Measured, not declared (§6.2.1) | +| Audit events | **yes** | An audit trail that forgets is not one | + +Everything in the "yes" column belongs to tenancy and benchmarking, which is +phase 5. **The store lands there, not in phase 3**, so its schema is shaped by +real requirements instead of guesses. + +#### The control plane restart hole + +Not persisting replica state has one consequence that must be closed before it +matters. If the control plane restarts while engines are running: + +1. The engines keep serving — correct, and the reason the router talks to them + directly. +2. Agents reconnect and re-register their hardware. +3. The control plane has no memory of any replica, because `Register` does not + carry one. +4. The reconciler sees zero replicas and places new ones. +5. The old engines keep holding GPUs that the inventory now believes are free, + so the new engines fail to allocate — with an error pointing at the new + process rather than the old one. + +The fix is on the agent side, not in a database: `Register` carries the shards +the agent is currently supervising, and the control plane rebuilds replica state +from what the fleet reports. The supervisor already knows its live shards and +already finds, through pid files, the ones that survived an agent restart +(`FindOrphans`). A shard nothing claims after one reconciliation pass is an +orphan and is killed. + +This is strictly better than a database would be: it cannot disagree with +reality, and it covers agent restarts and control plane restarts with one +mechanism. + +#### Store choice + +SQLite in WAL mode, via `modernc.org/sqlite` so the server stays cgo-free. +Embedded means no second service to operate, and WAL gives concurrent readers +alongside one writer, which matches the access pattern: many reads for auth, +one batched writer for usage. + +It stops being enough at the point where several router instances must write at +once. The store therefore sits behind an interface from the start, and the swap +to Postgres comes with multi-instance deployment rather than before it. --- @@ -797,6 +861,186 @@ profile** (§7.3). - v1 is a single instance with in-memory limiters. With multiple instances this becomes Redis or consistent-hash stickiness (v2). +#### 7.9.1 How keys are stored + +A key is a 256-bit random value, shown once at creation and never again: + +``` +sk-llmgw- +``` + +Only `sha256(key)` is stored, indexed. Authentication hashes what the client +presented and looks it up. Nothing can reconstruct the key from the database, +so a database leak does not hand over working credentials. + +**SHA-256 rather than bcrypt or argon2, deliberately.** Slow password hashing +exists because humans pick guessable passwords, and the cost buys time against +an offline dictionary attack. A 256-bit random value has no dictionary to +attack: at any hash rate, guessing it is infeasible. What a slow hash would buy +here is nothing, and what it would cost is tens of milliseconds on every single +request. Using the fast hash is the correct choice precisely *because* the input +is high entropy — and that is the assumption to re-examine if the key format +ever changes. + +Alongside the hash, a `key_prefix` (`sk-llmgw-4f2a…`) is stored in clear so the +UI and audit log can identify a key without revealing it. + +```sql +CREATE TABLE api_keys ( + id TEXT PRIMARY KEY, + key_hash BLOB NOT NULL UNIQUE, -- sha256 of the key; the only copy + key_prefix TEXT NOT NULL, -- for display, never for auth + name TEXT NOT NULL, + project_id TEXT NOT NULL REFERENCES projects(id), + band TEXT, -- §7.2.2; NULL inherits the project's + routing_profile TEXT, -- §7.3 + created_at INTEGER NOT NULL, + expires_at INTEGER, -- NULL never expires + revoked_at INTEGER, -- soft delete, so usage rows keep a valid FK + last_used_at INTEGER -- written lazily; see below +); +CREATE INDEX api_keys_by_project ON api_keys(project_id); +``` + +Revocation is a soft delete. Usage rows reference the key, and an audit trail +that loses its subject when the key is deleted is not an audit trail. + +`last_used_at` is written at most once a minute per key, from the same batch +writer as usage. Writing it per request would put a database write back on the +request path for a field nobody reads in real time. + +#### 7.9.2 The tenancy tree and how limits resolve + +```sql +CREATE TABLE organizations (id TEXT PRIMARY KEY, name TEXT NOT NULL); +CREATE TABLE teams (id TEXT PRIMARY KEY, org_id TEXT NOT NULL REFERENCES organizations(id), name TEXT NOT NULL); +CREATE TABLE projects (id TEXT PRIMARY KEY, team_id TEXT NOT NULL REFERENCES teams(id), name TEXT NOT NULL); + +-- Limits hang off any level rather than being duplicated onto each one. +CREATE TABLE limits ( + scope_type TEXT NOT NULL, -- 'org' | 'team' | 'project' | 'key' + scope_id TEXT NOT NULL, + rpm INTEGER, -- NULL means no limit at this level + tpm INTEGER, + budget_units REAL, -- per period; NULL means no budget + budget_period TEXT, -- 'day' | 'month' + allowed_models TEXT, -- JSON array; NULL means all + PRIMARY KEY (scope_type, scope_id) +); +``` + +A request's effective limit is the **most restrictive** value along +key → project → team → org. A team cannot raise a limit its organization set, +which is the only composition rule that makes a budget mean anything. + +Resolving four levels per request would be four lookups on the hot path, so the +resolved view is computed once and cached per key hash, invalidated whenever +anything on that key's chain changes. Authentication is then a single map +lookup, with the database consulted only on a cache miss. + +#### 7.9.3 How accounting is written + +The volume decides the design. At 1000 requests/sec, one row per request is +86 million rows a day, and SQLite has a single writer. Inserting per request +would serialize every request behind that writer. + +So the path is: + +``` +request completes + → usage record onto a buffered channel (nanoseconds, never blocks) + → batch writer: every 200ms or 500 records, one transaction + → hourly rollup job collapses raw rows + → raw rows older than the retention window are deleted +``` + +```sql +CREATE TABLE usage ( + ts INTEGER NOT NULL, -- unix millis + key_id TEXT NOT NULL, + project_id TEXT NOT NULL, + model TEXT NOT NULL, + served_model TEXT NOT NULL, -- differs from model when a fallback served it + pool TEXT, + status INTEGER NOT NULL, + prompt_tokens INTEGER NOT NULL, + completion_tokens INTEGER NOT NULL, + cached_tokens INTEGER NOT NULL, -- prefix cache hit, from the engine + ttft_ms INTEGER, + duration_ms INTEGER NOT NULL, + queue_ms INTEGER -- time spent in admission (§7.2) +); +CREATE INDEX usage_by_key_ts ON usage(key_id, ts); + +-- What survives retention, and what budgets are enforced against. +CREATE TABLE usage_rollup ( + hour_ts INTEGER NOT NULL, + key_id TEXT NOT NULL, + project_id TEXT NOT NULL, + model TEXT NOT NULL, + requests INTEGER NOT NULL, + prompt_tokens INTEGER NOT NULL, + completion_tokens INTEGER NOT NULL, + cached_tokens INTEGER NOT NULL, + units REAL NOT NULL, -- billable cost, per §7.9.4 + PRIMARY KEY (hour_ts, key_id, model) +); +``` + +**The honest tradeoff:** records sitting in the channel when the process is +killed are lost — at most a few hundred milliseconds of accounting. Making that +impossible means a durable write before each response, which puts a disk fsync +on every request. For usage that informs budgets and invoices, losing a fraction +of a second of records to a hard kill is the better trade. **If this ever has to +feed external billing, that trade has to be revisited**, and the answer then is +an append-only journal the batch writer drains, not a synchronous insert. + +#### 7.9.4 How budgets are enforced without reading the database + +Budgets are enforced per request, so they cannot involve a query. Instead: + +- On startup, the current period's spend is loaded per key from `usage_rollup` + plus the un-rolled tail of `usage`. **A restart must not reset a budget**, which + it would if the counter lived only in memory. +- From then on an in-memory counter is incremented as responses complete. The + database catches up through the batch writer. +- At the period boundary the counter resets and the new period starts empty. + +"Units" are deliberately not dollars. Self-hosted models have no per-token +price, so a unit is whatever the operator configures — GPU-seconds, a notional +rate per million tokens, or 1.0 for everything, which reduces budgets to token +quotas. This is also what makes the plan's budget fallback work: a model priced +at zero stays available to a key whose budget is spent (§7.7). + +#### 7.9.5 Token counting + +Two numbers exist and they are not the same: + +- **The estimate**, computed before dispatch as `bytes/4`, used to reserve TPM + quota and to score prefix affinity. It is a guess and is labelled as one. +- **The engine's report**, taken from the response's `usage` block. This is + authoritative and is what gets recorded. + +The router asks engines for usage even when the client did not, then drops that +chunk on the way back (§7.8), so accounting does not depend on client behaviour. +TPM is reserved at request start as `max_tokens + estimate` and reconciled +against the real figure on completion — reserving optimistically would let a +key exceed its limit by a whole response. + +The estimate stops being a guess only when a real tokenizer arrives, which the +plan defers to v2 along with tier 2 prefix routing (§7.5, §14). + +#### 7.9.6 Multiple router instances + +Everything above assumes one instance. Two would each hold their own rate limit +counters and each allow the configured limit, so the effective limit doubles. + +v1 is single-instance and this is a documented constraint rather than a bug. The +answer when it arrives is either shared counters in Redis, or consistent hashing +on the key so every request for one key reaches the same instance. The second is +cheaper and degrades better, but rebalances counters whenever the instance set +changes. + --- ## 8. Observability @@ -1015,7 +1259,10 @@ work_dir: /var/lib/llmgw-agent remaining item. ### Phase 3 — Control plane (2–3 weeks) -- [ ] SQLite store and migrations +- [ ] `Register` carries the agent's running shards, so a control plane restart + rebuilds replica state from the fleet instead of orphaning live engines (§6.5) +- [ ] Wire the supervisor's orphan scan: a shard nothing claims after one + reconciliation pass is killed - [x] Inventory: nodes, hardware, fabric attachments, allocations - [x] Placement: profile priority, bin packing, NVLink-aware GPU selection, `explain-placement` rejection reasons, property-based invariant tests @@ -1039,7 +1286,11 @@ work_dir: /var/lib/llmgw-agent - [ ] Replica-level failure handling (one dead shard kills the replica) - **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it to the other one. -### Phase 5 — Production features (2–3 weeks) +### Phase 5 — Production features (3 weeks) +- [ ] **SQLite store and migrations.** It arrives here rather than in phase 3 + because everything that genuinely needs persisting — keys, usage, capacity + profiles — is a phase 5 feature. Writing it earlier would have meant + designing a schema against guesses (§6.5). - [ ] Tenancy: org/team/project/key hierarchy, budgets, RPM/TPM, usage records (§7.9) - [ ] Budget fallback, model allowlists, key→band mapping - [ ] Routing profiles from configuration plus the `x-llmgw-profile` header @@ -1106,6 +1357,9 @@ work_dir: /var/lib/llmgw-agent | **Tuning the admission thresholds** | Too low a `target_waiting_per_replica` starves the GPU; too high brings queueing delay back. This single number governs the system's behaviour, so measure it with `ctl bench` rather than guessing. | | KV event loss | ZMQ is fire-and-forget; a lost event makes the router silently misroute. Sequence-gap detection and falling back to tier 1 are mandatory (§7.5). | | Metric staleness | At 1000 requests/sec with a 2s scrape interval, decisions rest entirely on stale data. That is why the `inflight` scorer is not optional. | +| **Usage lost on a hard kill** | Records buffered for the batch writer are lost if the process is killed — at most a few hundred milliseconds of accounting. Acceptable for budgets; **not acceptable if this ever feeds external billing**, and the answer then is an append-only journal rather than a synchronous insert (§7.9.3). | +| **Rate limits with several routers** | Counters are per instance, so N instances allow N times the configured limit. v1 is single-instance by design; Redis or consistent-hash stickiness is the fix (§7.9.6). | +| **Control plane restart** | Until `Register` carries running shards, restarting the control plane orphans live engines: they keep holding GPUs the inventory believes are free, and the next placement fails to allocate with an error pointing at the wrong process (§6.5). | ---