Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 19:45:37 +02:00
1 parent e59ff52932
commit 29a7347df2
19 files changed
+436 -17

No files matched your search

+35
View File
@@ -1062,6 +1062,41 @@ outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
`max_connections = 40` are enough for a store whose working set is measured in
megabytes.
## Tagging requests
A client can say what a request was for — which feature, which customer,
which environment — and the spend can then be found by it inside a project:
```
curl http://gateway:8080/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "x-llmgw-tags: feature=search, env=prod" \
-d '{"model": "qwen-1.5b", "user": "customer-42",
"metadata": {"feature": "search"},
"messages": [{"role": "user", "content": "hello"}]}'
```
- **`metadata`** is OpenAI's: an object of strings, up to 16 keys. **`x-llmgw-tags`**
is the same as a header, for clients that cannot change the body; for a
key both give, the header wins. A key is 1 to 64 letters, digits, `_`,
`.` or `-`, a value at most 512 bytes; anything else is refused with 400
rather than dropped, since a tag lost silently is spend nobody finds.
- **`user`** is OpenAI's end-user field: the client's own name for whoever
it serves, not an account here. It is passed on to the engine, as OpenAI
does; tags are not.
- **Where they show:** every usage record keeps them, and `llmgw ctl request
<id>` prints them. For Prometheus, name the tags worth a series:
```yaml
router:
tag_labels: [feature, env]
```
`llmgw_tagged_requests_total{tag,value,outcome}` and
`llmgw_tagged_tokens_total{tag,value,direction}` count those, the first
100 values of each apart and the rest as `other`: a tag is the client's
to set, and every value is a series.
## Conversations across models
Off everywhere until a project asks. llmgw otherwise stores no message content