Prices from what the cluster costs: ctl prices suggest
CI / test (push) Successful in 9m51s

auth.gpu_hour_cost (and core_hour_cost) is what an hour of a GPU (a core)
costs the operator. The leader keeps, by the hour, the GPU- and core-hours
each model's replicas hold (held_1h); llmgw ctl prices suggest spreads a
model's over the tokens it served, an output token counted as four input
ones and a cached prompt token as a tenth of one, and --apply writes the
prices into auth.pricing. Idle hours are in the cost, so a model held and
little used comes out dear; one held and billed nothing is shown with what
it held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-15 06:54:55 +01:00
1 parent b8d3c084c5
commit 8547528fb7
22 files changed
+834 -1

No files matched your search

+127
View File
@@ -95,6 +95,9 @@
{
"name": "presets"
},
{
"name": "prices"
},
{
"name": "projects"
},
@@ -4396,6 +4399,34 @@
}
}
},
"/api/v1/prices/suggest": {
"get": {
"operationId": "getPricesSuggest",
"summary": "Each model's price, from what it held of the cluster",
"description": "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as auth.pricing takes them. current is the price in force. 409 with no cost set.",
"tags": [
"prices"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PriceSuggestions"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/projects": {
"post": {
"operationId": "postProjects",
@@ -8846,6 +8877,73 @@
}
}
},
"PriceSuggestion": {
"type": "object",
"description": "PriceSuggestion is one model's.",
"properties": {
"model": {
"type": "string"
},
"gpu_hours": {
"type": "number"
},
"core_hours": {
"type": "number"
},
"cost": {
"type": "number",
"description": "Cost is what it held cost, in budget units."
},
"prompt_tokens": {
"type": "integer",
"format": "int64"
},
"completion_tokens": {
"type": "integer",
"format": "int64"
},
"price": {
"$ref": "#/components/schemas/accounting.Price",
"description": "Price is the suggestion; zero with a Note saying why there is none."
},
"note": {
"type": "string"
},
"current": {
"$ref": "#/components/schemas/accounting.Price"
}
}
},
"PriceSuggestions": {
"type": "object",
"description": "PriceSuggestions is a model's price from its cost, for each model.",
"properties": {
"days": {
"type": "integer",
"description": "Days is the window; RecordedFrom the first hour held is known for, later than the window's start on a cluster that began keeping it since, which makes the window shorter than asked."
},
"since": {
"type": "string",
"format": "date-time"
},
"recorded_from": {
"type": "string",
"format": "date-time"
},
"gpu_hour_cost": {
"type": "number"
},
"core_hour_cost": {
"type": "number"
},
"models": {
"type": "array",
"items": {
"$ref": "#/components/schemas/PriceSuggestion"
}
}
}
},
"ProfileReplicasRequest": {
"type": "object",
"description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
@@ -10007,6 +10105,35 @@
}
}
},
"accounting.Price": {
"type": "object",
"description": "Price is what one model costs, in micro-units per thousand tokens.\n\nMicro-units rather than a float because spend is accumulated by repeated addition, and repeated addition of floats drifts. What a unit means is the operator's: self-hosted models have no per-token price, so it may be GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into a token quota.",
"properties": {
"PromptPer1k": {
"type": "integer",
"format": "int64"
},
"CompletionPer1k": {
"type": "integer",
"format": "int64"
},
"CachedPer1k": {
"type": "integer",
"format": "int64",
"description": "CachedPer1k is what a prompt token costs that the engine or provider read from its cache instead of computing: the part of prompt_tokens the answer counts as cached_tokens. Zero charges those at PromptPer1k, as before there was a rate for them."
},
"PerImage": {
"type": "integer",
"format": "int64",
"description": "PerImage is what one picture costs, for a model that makes them: a picture's cost is the card's time, which its prompt's few tokens do not measure."
},
"PerMesh": {
"type": "integer",
"format": "int64",
"description": "PerMesh is what one 3D model costs: minutes of a card, which no token count measures."
}
}
},
"accounting.Rates": {
"type": "object",
"description": "Rates are an accountant's two limiters' counts.",
+13
View File
@@ -457,6 +457,19 @@ by itself. Three things are worth knowing before you edit it:
cache (the answer's `cached_tokens`) are charged at it, the rest at the
prompt rate. A provider that bills cached reads at a tenth is
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
- What to price a self-hosted model at follows from what the cluster costs.
Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
are in: the hardware written off over its years, its power and room, and the
people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
`auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
prices suggest` spreads them over the tokens the model served: an output
token counted as four input ones (making one is that much dearer), a cached
prompt token as a tenth of one. Idle hours are in the cost, so a model held
all month and seldom used comes out dear, which is the point; one held and
billed nothing (a guard's judge) is shown with what it held. `--apply`
writes the prices into `auth.pricing`.
- `control.enabled: true` is what makes this a control plane while its catalog
is empty. Without it, a file with no models is refused as having nothing to
serve.
+17
View File
@@ -43,6 +43,8 @@ rather than repeated.
| `personal` | `Personal` | [Auth](#auth) |
| `pricing` | `map[string]Price` | [Auth](#auth) |
| `default_price` | `Price` | [Auth](#auth) |
| `gpu_hour_cost` | `float64` | [Auth](#auth) |
| `core_hour_cost` | `float64` | [Auth](#auth) |
| `flush_interval` | `time.Duration` | [Auth](#auth) |
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
@@ -722,6 +724,21 @@ table costs whatever DefaultPrice says, which is nothing unless set.
Takes a [config.Price](#configprice) block.
### `gpu_hour_cost`
`float64`
GPUHourCost is what an hour of one GPU costs the operator, in the
units budgets are in: the hardware written off over its years, its
power and room, and the people who run it, over the GPUs and the
hours of a month. With CoreHourCost for a CPU core, it is what
`llmgw ctl prices suggest` prices each model from: the hours its
replicas held, idle ones too, over the tokens it served.
### `core_hour_cost`
`float64`
### `flush_interval`
`time.Duration`