Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 19:45:37 +02:00
1 parent e59ff52932
commit 29a7347df2
19 files changed
+436 -17

No files matched your search

+6 -1
View File
@@ -32,7 +32,12 @@ the text, the value that matched, or the placeholder map of a redaction. See
**Usage records**, one per request: the key, the project, the model asked for,
the model that answered, prompt and completion token counts, cost, HTTP status
and latency. No message content. The row is in `internal/store/store.go` as
`Usage` and there is no field for text in it.
`Usage` and there is no field for text in it. What the client chose to say
about the request is kept with it: OpenAI's `user` field and its `metadata`
or `x-llmgw-tags` — short strings, set by the client, for attributing spend.
A client that puts a person's name or address there puts it in the usage
records; the field is the client's, and so is that choice. Tags are not sent
on to the engine.
**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
never stored — so a leaked database hands over no working credential, and
+35
View File
@@ -1062,6 +1062,41 @@ outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
`max_connections = 40` are enough for a store whose working set is measured in
megabytes.
## Tagging requests
A client can say what a request was for — which feature, which customer,
which environment — and the spend can then be found by it inside a project:
```
curl http://gateway:8080/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "x-llmgw-tags: feature=search, env=prod" \
-d '{"model": "qwen-1.5b", "user": "customer-42",
"metadata": {"feature": "search"},
"messages": [{"role": "user", "content": "hello"}]}'
```
- **`metadata`** is OpenAI's: an object of strings, up to 16 keys. **`x-llmgw-tags`**
is the same as a header, for clients that cannot change the body; for a
key both give, the header wins. A key is 1 to 64 letters, digits, `_`,
`.` or `-`, a value at most 512 bytes; anything else is refused with 400
rather than dropped, since a tag lost silently is spend nobody finds.
- **`user`** is OpenAI's end-user field: the client's own name for whoever
it serves, not an account here. It is passed on to the engine, as OpenAI
does; tags are not.
- **Where they show:** every usage record keeps them, and `llmgw ctl request
<id>` prints them. For Prometheus, name the tags worth a series:
```yaml
router:
tag_labels: [feature, env]
```
`llmgw_tagged_requests_total{tag,value,outcome}` and
`llmgw_tagged_tokens_total{tag,value,direction}` count those, the first
100 values of each apart and the rest as `other`: a tag is the client's
to set, and every value is a series.
## Conversations across models
Off everywhere until a project asks. llmgw otherwise stores no message content
+11
View File
@@ -190,6 +190,7 @@ rather than repeated.
| `session_header` | `string` | [Router](#router) |
| `inject_usage` | `bool` | [Router](#router) |
| `tenant_label` | `string` | [Router](#router) |
| `tag_labels` | `[]string` | [Router](#router) |
| `max_body_bytes` | `int64` | [Router](#router) |
| `trusted_proxies` | `[]string` | [Router](#router) |
| `timeouts` | `Timeouts` | [Router](#router) |
@@ -1557,6 +1558,16 @@ labelled by — llmgw_tenant_requests_total and its siblings: "org",
on a cluster of thousands is more series than Prometheus wants, and a
team is what an alert is usually about; "team" is the default.
### `tag_labels`
`[]string`
TagLabels are the request tags (metadata, x-llmgw-tags) counted in
llmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value.
Only these: a tag is the client's to set, and every value of one is
a series. The first 100 values of each are counted apart, the rest as
"other". Empty counts none; the usage records keep every tag anyway.
### `max_body_bytes`
`int64`