Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e59ff52932
commit
29a7347df2
19 files changed
+436
-17
No files matched your search
+6
-1
@@ -32,7 +32,12 @@ the text, the value that matched, or the placeholder map of a redaction. See
|
||||
**Usage records**, one per request: the key, the project, the model asked for,
|
||||
the model that answered, prompt and completion token counts, cost, HTTP status
|
||||
and latency. No message content. The row is in `internal/store/store.go` as
|
||||
`Usage` and there is no field for text in it.
|
||||
`Usage` and there is no field for text in it. What the client chose to say
|
||||
about the request is kept with it: OpenAI's `user` field and its `metadata`
|
||||
or `x-llmgw-tags` — short strings, set by the client, for attributing spend.
|
||||
A client that puts a person's name or address there puts it in the usage
|
||||
records; the field is the client's, and so is that choice. Tags are not sent
|
||||
on to the engine.
|
||||
|
||||
**API keys**, as SHA-256 hashes. The key itself is shown once, at creation, and
|
||||
never stored — so a leaked database hands over no working credential, and
|
||||
|
||||
@@ -1062,6 +1062,41 @@ outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
|
||||
`max_connections = 40` are enough for a store whose working set is measured in
|
||||
megabytes.
|
||||
|
||||
## Tagging requests
|
||||
|
||||
A client can say what a request was for — which feature, which customer,
|
||||
which environment — and the spend can then be found by it inside a project:
|
||||
|
||||
```
|
||||
curl http://gateway:8080/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "x-llmgw-tags: feature=search, env=prod" \
|
||||
-d '{"model": "qwen-1.5b", "user": "customer-42",
|
||||
"metadata": {"feature": "search"},
|
||||
"messages": [{"role": "user", "content": "hello"}]}'
|
||||
```
|
||||
|
||||
- **`metadata`** is OpenAI's: an object of strings, up to 16 keys. **`x-llmgw-tags`**
|
||||
is the same as a header, for clients that cannot change the body; for a
|
||||
key both give, the header wins. A key is 1 to 64 letters, digits, `_`,
|
||||
`.` or `-`, a value at most 512 bytes; anything else is refused with 400
|
||||
rather than dropped, since a tag lost silently is spend nobody finds.
|
||||
- **`user`** is OpenAI's end-user field: the client's own name for whoever
|
||||
it serves, not an account here. It is passed on to the engine, as OpenAI
|
||||
does; tags are not.
|
||||
- **Where they show:** every usage record keeps them, and `llmgw ctl request
|
||||
<id>` prints them. For Prometheus, name the tags worth a series:
|
||||
|
||||
```yaml
|
||||
router:
|
||||
tag_labels: [feature, env]
|
||||
```
|
||||
|
||||
`llmgw_tagged_requests_total{tag,value,outcome}` and
|
||||
`llmgw_tagged_tokens_total{tag,value,direction}` count those, the first
|
||||
100 values of each apart and the rest as `other`: a tag is the client's
|
||||
to set, and every value is a series.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
|
||||
@@ -190,6 +190,7 @@ rather than repeated.
|
||||
| `session_header` | `string` | [Router](#router) |
|
||||
| `inject_usage` | `bool` | [Router](#router) |
|
||||
| `tenant_label` | `string` | [Router](#router) |
|
||||
| `tag_labels` | `[]string` | [Router](#router) |
|
||||
| `max_body_bytes` | `int64` | [Router](#router) |
|
||||
| `trusted_proxies` | `[]string` | [Router](#router) |
|
||||
| `timeouts` | `Timeouts` | [Router](#router) |
|
||||
@@ -1557,6 +1558,16 @@ labelled by — llmgw_tenant_requests_total and its siblings: "org",
|
||||
on a cluster of thousands is more series than Prometheus wants, and a
|
||||
team is what an alert is usually about; "team" is the default.
|
||||
|
||||
### `tag_labels`
|
||||
|
||||
`[]string`
|
||||
|
||||
TagLabels are the request tags (metadata, x-llmgw-tags) counted in
|
||||
llmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value.
|
||||
Only these: a tag is the client's to set, and every value of one is
|
||||
a series. The first 100 values of each are counted apart, the rest as
|
||||
"other". Empty counts none; the usage records keep every tag anyway.
|
||||
|
||||
### `max_body_bytes`
|
||||
|
||||
`int64`
|
||||
|
||||
Reference in new issue
Block a user