Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e59ff52932
commit
29a7347df2
19 files changed
+436
-17
No files matched your search
@@ -1062,6 +1062,41 @@ outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
|
||||
`max_connections = 40` are enough for a store whose working set is measured in
|
||||
megabytes.
|
||||
|
||||
## Tagging requests
|
||||
|
||||
A client can say what a request was for — which feature, which customer,
|
||||
which environment — and the spend can then be found by it inside a project:
|
||||
|
||||
```
|
||||
curl http://gateway:8080/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "x-llmgw-tags: feature=search, env=prod" \
|
||||
-d '{"model": "qwen-1.5b", "user": "customer-42",
|
||||
"metadata": {"feature": "search"},
|
||||
"messages": [{"role": "user", "content": "hello"}]}'
|
||||
```
|
||||
|
||||
- **`metadata`** is OpenAI's: an object of strings, up to 16 keys. **`x-llmgw-tags`**
|
||||
is the same as a header, for clients that cannot change the body; for a
|
||||
key both give, the header wins. A key is 1 to 64 letters, digits, `_`,
|
||||
`.` or `-`, a value at most 512 bytes; anything else is refused with 400
|
||||
rather than dropped, since a tag lost silently is spend nobody finds.
|
||||
- **`user`** is OpenAI's end-user field: the client's own name for whoever
|
||||
it serves, not an account here. It is passed on to the engine, as OpenAI
|
||||
does; tags are not.
|
||||
- **Where they show:** every usage record keeps them, and `llmgw ctl request
|
||||
<id>` prints them. For Prometheus, name the tags worth a series:
|
||||
|
||||
```yaml
|
||||
router:
|
||||
tag_labels: [feature, env]
|
||||
```
|
||||
|
||||
`llmgw_tagged_requests_total{tag,value,outcome}` and
|
||||
`llmgw_tagged_tokens_total{tag,value,direction}` count those, the first
|
||||
100 values of each apart and the rest as `other`: a tag is the client's
|
||||
to set, and every value is a series.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
|
||||
Reference in new issue
Block a user