plan: correct the store design and specify how tenancy is actually stored

Two problems with what v0.1 said about persistence.

First, it listed eight tables, most of which were a mistake. A database that
describes the cluster can be wrong; the cluster cannot. An engine that dies
during a restart leaves a row saying "running", and the scheduler believes it.
Nodes, replicas, shards and the catalog are all rebuilt from the fleet or from
configuration on every reconnect, so they need no rows at all. What genuinely
needs persisting — API keys, usage, capacity profiles — is entirely phase 5
work, so the store moves there and gets a schema shaped by real requirements
instead of guesses.

Not persisting replica state has one consequence that must be closed first:
restarting the control plane currently orphans live engines. They keep holding
GPUs the inventory believes are free, and the next placement fails to allocate
with an error pointing at the new process rather than the old one. The fix
belongs on the agent, not in a database — Register carries the shards the agent
is supervising, and the supervisor's orphan scan (written, tested, never called)
kills whatever nothing claims. That cannot disagree with reality, and it covers
agent restarts and control plane restarts with one mechanism. Added to phase 3.

Second, §7.9 described tenancy behaviour but never said how any of it is stored,
which left the most security-sensitive part of the system unspecified. It now
covers the schema for keys and the tenancy tree, why a fast hash is the correct
choice for a 256-bit random key and what assumption that rests on, how limits
resolve to the most restrictive value along the chain, why authentication is a
map lookup rather than four queries, the batching pipeline that keeps 86 million
rows a day off the request path, what "units" mean when self-hosted models have
no per-token price, and how budgets survive a restart.

Two tradeoffs are now stated rather than left implicit: usage buffered for the
batch writer is lost on a hard kill, which is fine for budgets and not fine for
external billing; and rate limit counters are per instance, so N routers allow N
times the limit. Both are in the risk table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ilkerandClaude Opus 5 committed 2026-09-16 06:08:16 +02:00
1 parent 2373726efd
commit 40427b2705
1 file changed
+262 -8
+262 -8
View File
@@ -21,7 +21,8 @@
saturation) and **scheduling** (plugin-based filter/score/pick with queue, KV-cache and
prefix awareness), plus fallback, retry, cooldown and circuit breaking.
- Fabric awareness: a multi-node replica **never** spans two fabrics; a TP group **never** mixes GPU types.
- API keys, rate limits, token accounting, Prometheus metrics.
- API keys, rate limits, token accounting and budgets, Prometheus metrics. Persistence for these is
the only thing the store exists for (§6.5, §7.9).
### 1.2 Out of scope (at least for v1)
- Inference itself. Go only orchestrates and proxies; vLLM/SGLang run the model.
@@ -46,6 +47,7 @@
| **Routing Profile** | The scorer set and weights the router uses for a class of request (chat / batch / long-context). Not to be confused with the model **Profile** above. |
| **Band** | A priority class (platinum/gold/silver/bronze). The unit of admission decisions. |
| **Block** | The unit of KV cache accounting; a fixed number of tokens (16 in vLLM, 64 by default in the GIE indexer). |
| **Unit** | The billing unit budgets are counted in. Self-hosted models have no per-token price, so an operator defines what a unit is: GPU-seconds, a notional rate per million tokens, or 1.0 everywhere, which turns budgets into token quotas (§7.9.4). |
---
@@ -82,7 +84,7 @@ semantic routing / LLM-based model selection (out of scope), and external provid
│ │ (data) │◄─┤ - Inventory │ │
│ │ │ │ - Catalog │ │
│ └─────┬──────┘ │ - Scheduler │ │
│ │ │ - Store (SQLite) │ │
│ │ │ - Store (phase 5) │ │
│ │ └─────────▲────────────┘ │
└────────┼───────────────────┼──────────────┘
HTTP (eth) │ │ gRPC stream (mTLS, agent dials out)
@@ -474,10 +476,72 @@ the replicas it knows about, which mitigates the single-point-of-failure risk in
cache ages past `endpoint_cache_max_age` (say 5 min) it only raises a warning metric; it never
stops traffic.
### 6.5 Store
- SQLite in WAL mode. Tables: `nodes`, `models`, `profiles`, `replicas`, `shards`, `api_keys`, `usage_rollup`, `events`.
- Inventory and replica state can be rebuilt from the agents; what genuinely needs to persist is the catalog, the keys and the usage records.
- HA (v2): Raft-replicated state, or an external Postgres/etcd.
### 6.5 What is stored, and what is not
v0.1 of this plan listed eight tables. Most of them were a mistake, and the
reason is worth stating because it applies to every piece of state here:
> **A database that describes the cluster can be wrong. The cluster cannot.**
An engine that crashes during a restart leaves a row saying "running". A node
whose GPU developed an ECC fault leaves a row saying the card is fine. Every
such row is a trap: the scheduler believes it, places work accordingly, and the
failure surfaces somewhere else entirely. Rebuilding the same state from the
fleet on every reconnect cannot go stale, because the fleet is the thing being
described.
So:
| State | Persisted | Where it actually comes from |
|---|---|---|
| Nodes, GPUs, fabrics | no | Every agent sends its full inventory on registration (§5.6) |
| Replicas, shards | no | Reported by agents; see the restart hole below |
| Models, profiles | no | The YAML catalog is the source of truth |
| API keys and the tenancy tree | **yes** | Created at runtime, secret, rotated (§7.9) |
| Usage records and rollups | **yes** | Accounting must survive a restart (§7.9) |
| Capacity profiles from `ctl bench` | **yes** | Measured, not declared (§6.2.1) |
| Audit events | **yes** | An audit trail that forgets is not one |
Everything in the "yes" column belongs to tenancy and benchmarking, which is
phase 5. **The store lands there, not in phase 3**, so its schema is shaped by
real requirements instead of guesses.
#### The control plane restart hole
Not persisting replica state has one consequence that must be closed before it
matters. If the control plane restarts while engines are running:
1. The engines keep serving — correct, and the reason the router talks to them
directly.
2. Agents reconnect and re-register their hardware.
3. The control plane has no memory of any replica, because `Register` does not
carry one.
4. The reconciler sees zero replicas and places new ones.
5. The old engines keep holding GPUs that the inventory now believes are free,
so the new engines fail to allocate — with an error pointing at the new
process rather than the old one.
The fix is on the agent side, not in a database: `Register` carries the shards
the agent is currently supervising, and the control plane rebuilds replica state
from what the fleet reports. The supervisor already knows its live shards and
already finds, through pid files, the ones that survived an agent restart
(`FindOrphans`). A shard nothing claims after one reconciliation pass is an
orphan and is killed.
This is strictly better than a database would be: it cannot disagree with
reality, and it covers agent restarts and control plane restarts with one
mechanism.
#### Store choice
SQLite in WAL mode, via `modernc.org/sqlite` so the server stays cgo-free.
Embedded means no second service to operate, and WAL gives concurrent readers
alongside one writer, which matches the access pattern: many reads for auth,
one batched writer for usage.
It stops being enough at the point where several router instances must write at
once. The store therefore sits behind an interface from the start, and the swap
to Postgres comes with multi-instance deployment rather than before it.
---
@@ -797,6 +861,186 @@ profile** (§7.3).
- v1 is a single instance with in-memory limiters. With multiple instances this becomes Redis or
consistent-hash stickiness (v2).
#### 7.9.1 How keys are stored
A key is a 256-bit random value, shown once at creation and never again:
```
sk-llmgw-<base62(32 random bytes)>
```
Only `sha256(key)` is stored, indexed. Authentication hashes what the client
presented and looks it up. Nothing can reconstruct the key from the database,
so a database leak does not hand over working credentials.
**SHA-256 rather than bcrypt or argon2, deliberately.** Slow password hashing
exists because humans pick guessable passwords, and the cost buys time against
an offline dictionary attack. A 256-bit random value has no dictionary to
attack: at any hash rate, guessing it is infeasible. What a slow hash would buy
here is nothing, and what it would cost is tens of milliseconds on every single
request. Using the fast hash is the correct choice precisely *because* the input
is high entropy — and that is the assumption to re-examine if the key format
ever changes.
Alongside the hash, a `key_prefix` (`sk-llmgw-4f2a…`) is stored in clear so the
UI and audit log can identify a key without revealing it.
```sql
CREATE TABLE api_keys (
id TEXT PRIMARY KEY,
key_hash BLOB NOT NULL UNIQUE, -- sha256 of the key; the only copy
key_prefix TEXT NOT NULL, -- for display, never for auth
name TEXT NOT NULL,
project_id TEXT NOT NULL REFERENCES projects(id),
band TEXT, -- §7.2.2; NULL inherits the project's
routing_profile TEXT, -- §7.3
created_at INTEGER NOT NULL,
expires_at INTEGER, -- NULL never expires
revoked_at INTEGER, -- soft delete, so usage rows keep a valid FK
last_used_at INTEGER -- written lazily; see below
);
CREATE INDEX api_keys_by_project ON api_keys(project_id);
```
Revocation is a soft delete. Usage rows reference the key, and an audit trail
that loses its subject when the key is deleted is not an audit trail.
`last_used_at` is written at most once a minute per key, from the same batch
writer as usage. Writing it per request would put a database write back on the
request path for a field nobody reads in real time.
#### 7.9.2 The tenancy tree and how limits resolve
```sql
CREATE TABLE organizations (id TEXT PRIMARY KEY, name TEXT NOT NULL);
CREATE TABLE teams (id TEXT PRIMARY KEY, org_id TEXT NOT NULL REFERENCES organizations(id), name TEXT NOT NULL);
CREATE TABLE projects (id TEXT PRIMARY KEY, team_id TEXT NOT NULL REFERENCES teams(id), name TEXT NOT NULL);
-- Limits hang off any level rather than being duplicated onto each one.
CREATE TABLE limits (
scope_type TEXT NOT NULL, -- 'org' | 'team' | 'project' | 'key'
scope_id TEXT NOT NULL,
rpm INTEGER, -- NULL means no limit at this level
tpm INTEGER,
budget_units REAL, -- per period; NULL means no budget
budget_period TEXT, -- 'day' | 'month'
allowed_models TEXT, -- JSON array; NULL means all
PRIMARY KEY (scope_type, scope_id)
);
```
A request's effective limit is the **most restrictive** value along
key → project → team → org. A team cannot raise a limit its organization set,
which is the only composition rule that makes a budget mean anything.
Resolving four levels per request would be four lookups on the hot path, so the
resolved view is computed once and cached per key hash, invalidated whenever
anything on that key's chain changes. Authentication is then a single map
lookup, with the database consulted only on a cache miss.
#### 7.9.3 How accounting is written
The volume decides the design. At 1000 requests/sec, one row per request is
86 million rows a day, and SQLite has a single writer. Inserting per request
would serialize every request behind that writer.
So the path is:
```
request completes
→ usage record onto a buffered channel (nanoseconds, never blocks)
→ batch writer: every 200ms or 500 records, one transaction
→ hourly rollup job collapses raw rows
→ raw rows older than the retention window are deleted
```
```sql
CREATE TABLE usage (
ts INTEGER NOT NULL, -- unix millis
key_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
served_model TEXT NOT NULL, -- differs from model when a fallback served it
pool TEXT,
status INTEGER NOT NULL,
prompt_tokens INTEGER NOT NULL,
completion_tokens INTEGER NOT NULL,
cached_tokens INTEGER NOT NULL, -- prefix cache hit, from the engine
ttft_ms INTEGER,
duration_ms INTEGER NOT NULL,
queue_ms INTEGER -- time spent in admission (§7.2)
);
CREATE INDEX usage_by_key_ts ON usage(key_id, ts);
-- What survives retention, and what budgets are enforced against.
CREATE TABLE usage_rollup (
hour_ts INTEGER NOT NULL,
key_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
requests INTEGER NOT NULL,
prompt_tokens INTEGER NOT NULL,
completion_tokens INTEGER NOT NULL,
cached_tokens INTEGER NOT NULL,
units REAL NOT NULL, -- billable cost, per §7.9.4
PRIMARY KEY (hour_ts, key_id, model)
);
```
**The honest tradeoff:** records sitting in the channel when the process is
killed are lost — at most a few hundred milliseconds of accounting. Making that
impossible means a durable write before each response, which puts a disk fsync
on every request. For usage that informs budgets and invoices, losing a fraction
of a second of records to a hard kill is the better trade. **If this ever has to
feed external billing, that trade has to be revisited**, and the answer then is
an append-only journal the batch writer drains, not a synchronous insert.
#### 7.9.4 How budgets are enforced without reading the database
Budgets are enforced per request, so they cannot involve a query. Instead:
- On startup, the current period's spend is loaded per key from `usage_rollup`
plus the un-rolled tail of `usage`. **A restart must not reset a budget**, which
it would if the counter lived only in memory.
- From then on an in-memory counter is incremented as responses complete. The
database catches up through the batch writer.
- At the period boundary the counter resets and the new period starts empty.
"Units" are deliberately not dollars. Self-hosted models have no per-token
price, so a unit is whatever the operator configures — GPU-seconds, a notional
rate per million tokens, or 1.0 for everything, which reduces budgets to token
quotas. This is also what makes the plan's budget fallback work: a model priced
at zero stays available to a key whose budget is spent (§7.7).
#### 7.9.5 Token counting
Two numbers exist and they are not the same:
- **The estimate**, computed before dispatch as `bytes/4`, used to reserve TPM
quota and to score prefix affinity. It is a guess and is labelled as one.
- **The engine's report**, taken from the response's `usage` block. This is
authoritative and is what gets recorded.
The router asks engines for usage even when the client did not, then drops that
chunk on the way back (§7.8), so accounting does not depend on client behaviour.
TPM is reserved at request start as `max_tokens + estimate` and reconciled
against the real figure on completion — reserving optimistically would let a
key exceed its limit by a whole response.
The estimate stops being a guess only when a real tokenizer arrives, which the
plan defers to v2 along with tier 2 prefix routing (§7.5, §14).
#### 7.9.6 Multiple router instances
Everything above assumes one instance. Two would each hold their own rate limit
counters and each allow the configured limit, so the effective limit doubles.
v1 is single-instance and this is a documented constraint rather than a bug. The
answer when it arrives is either shared counters in Redis, or consistent hashing
on the key so every request for one key reaches the same instance. The second is
cheaper and degrades better, but rebalances counters whenever the instance set
changes.
---
## 8. Observability
@@ -1015,7 +1259,10 @@ work_dir: /var/lib/llmgw-agent
remaining item.
### Phase 3 — Control plane (2–3 weeks)
- [ ] SQLite store and migrations
- [ ] `Register` carries the agent's running shards, so a control plane restart
rebuilds replica state from the fleet instead of orphaning live engines (§6.5)
- [ ] Wire the supervisor's orphan scan: a shard nothing claims after one
reconciliation pass is killed
- [x] Inventory: nodes, hardware, fabric attachments, allocations
- [x] Placement: profile priority, bin packing, NVLink-aware GPU selection,
`explain-placement` rejection reasons, property-based invariant tests
@@ -1039,7 +1286,11 @@ work_dir: /var/lib/llmgw-agent
- [ ] Replica-level failure handling (one dead shard kills the replica)
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it to the other one.
### Phase 5 — Production features (2–3 weeks)
### Phase 5 — Production features (3 weeks)
- [ ] **SQLite store and migrations.** It arrives here rather than in phase 3
because everything that genuinely needs persisting — keys, usage, capacity
profiles — is a phase 5 feature. Writing it earlier would have meant
designing a schema against guesses (§6.5).
- [ ] Tenancy: org/team/project/key hierarchy, budgets, RPM/TPM, usage records (§7.9)
- [ ] Budget fallback, model allowlists, key→band mapping
- [ ] Routing profiles from configuration plus the `x-llmgw-profile` header
@@ -1106,6 +1357,9 @@ work_dir: /var/lib/llmgw-agent
| **Tuning the admission thresholds** | Too low a `target_waiting_per_replica` starves the GPU; too high brings queueing delay back. This single number governs the system's behaviour, so measure it with `ctl bench` rather than guessing. |
| KV event loss | ZMQ is fire-and-forget; a lost event makes the router silently misroute. Sequence-gap detection and falling back to tier 1 are mandatory (§7.5). |
| Metric staleness | At 1000 requests/sec with a 2s scrape interval, decisions rest entirely on stale data. That is why the `inflight` scorer is not optional. |
| **Usage lost on a hard kill** | Records buffered for the batch writer are lost if the process is killed — at most a few hundred milliseconds of accounting. Acceptable for budgets; **not acceptable if this ever feeds external billing**, and the answer then is an append-only journal rather than a synchronous insert (§7.9.3). |
| **Rate limits with several routers** | Counters are per instance, so N instances allow N times the configured limit. v1 is single-instance by design; Redis or consistent-hash stickiness is the fix (§7.9.6). |
| **Control plane restart** | Until `Register` carries running shards, restarting the control plane orphans live engines: they keep holding GPUs the inventory believes are free, and the next placement fails to allocate with an error pointing at the wrong process (§6.5). |
---