auth.gpu_hour_cost (and core_hour_cost) is what an hour of a GPU (a core) costs the operator. The leader keeps, by the hour, the GPU- and core-hours each model's replicas hold (held_1h); llmgw ctl prices suggest spreads a model's over the tokens it served, an output token counted as four input ones and a cached prompt token as a tenth of one, and --apply writes the prices into auth.pricing. Idle hours are in the cost, so a model held and little used comes out dear; one held and billed nothing is shown with what it held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b8d3c084c5
commit
8547528fb7
22 files changed
+834
-1
No files matched your search
@@ -95,6 +95,9 @@
|
||||
{
|
||||
"name": "presets"
|
||||
},
|
||||
{
|
||||
"name": "prices"
|
||||
},
|
||||
{
|
||||
"name": "projects"
|
||||
},
|
||||
@@ -4396,6 +4399,34 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/prices/suggest": {
|
||||
"get": {
|
||||
"operationId": "getPricesSuggest",
|
||||
"summary": "Each model's price, from what it held of the cluster",
|
||||
"description": "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as auth.pricing takes them. current is the price in force. 409 with no cost set.",
|
||||
"tags": [
|
||||
"prices"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/PriceSuggestions"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/projects": {
|
||||
"post": {
|
||||
"operationId": "postProjects",
|
||||
@@ -8846,6 +8877,73 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"PriceSuggestion": {
|
||||
"type": "object",
|
||||
"description": "PriceSuggestion is one model's.",
|
||||
"properties": {
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"gpu_hours": {
|
||||
"type": "number"
|
||||
},
|
||||
"core_hours": {
|
||||
"type": "number"
|
||||
},
|
||||
"cost": {
|
||||
"type": "number",
|
||||
"description": "Cost is what it held cost, in budget units."
|
||||
},
|
||||
"prompt_tokens": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"completion_tokens": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"price": {
|
||||
"$ref": "#/components/schemas/accounting.Price",
|
||||
"description": "Price is the suggestion; zero with a Note saying why there is none."
|
||||
},
|
||||
"note": {
|
||||
"type": "string"
|
||||
},
|
||||
"current": {
|
||||
"$ref": "#/components/schemas/accounting.Price"
|
||||
}
|
||||
}
|
||||
},
|
||||
"PriceSuggestions": {
|
||||
"type": "object",
|
||||
"description": "PriceSuggestions is a model's price from its cost, for each model.",
|
||||
"properties": {
|
||||
"days": {
|
||||
"type": "integer",
|
||||
"description": "Days is the window; RecordedFrom the first hour held is known for, later than the window's start on a cluster that began keeping it since, which makes the window shorter than asked."
|
||||
},
|
||||
"since": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"recorded_from": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"gpu_hour_cost": {
|
||||
"type": "number"
|
||||
},
|
||||
"core_hour_cost": {
|
||||
"type": "number"
|
||||
},
|
||||
"models": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/PriceSuggestion"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ProfileReplicasRequest": {
|
||||
"type": "object",
|
||||
"description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
|
||||
@@ -10007,6 +10105,35 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"accounting.Price": {
|
||||
"type": "object",
|
||||
"description": "Price is what one model costs, in micro-units per thousand tokens.\n\nMicro-units rather than a float because spend is accumulated by repeated addition, and repeated addition of floats drifts. What a unit means is the operator's: self-hosted models have no per-token price, so it may be GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into a token quota.",
|
||||
"properties": {
|
||||
"PromptPer1k": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"CompletionPer1k": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"CachedPer1k": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "CachedPer1k is what a prompt token costs that the engine or provider read from its cache instead of computing: the part of prompt_tokens the answer counts as cached_tokens. Zero charges those at PromptPer1k, as before there was a rate for them."
|
||||
},
|
||||
"PerImage": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "PerImage is what one picture costs, for a model that makes them: a picture's cost is the card's time, which its prompt's few tokens do not measure."
|
||||
},
|
||||
"PerMesh": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "PerMesh is what one 3D model costs: minutes of a card, which no token count measures."
|
||||
}
|
||||
}
|
||||
},
|
||||
"accounting.Rates": {
|
||||
"type": "object",
|
||||
"description": "Rates are an accountant's two limiters' counts.",
|
||||
|
||||
@@ -457,6 +457,19 @@ by itself. Three things are worth knowing before you edit it:
|
||||
cache (the answer's `cached_tokens`) are charged at it, the rest at the
|
||||
prompt rate. A provider that bills cached reads at a tenth is
|
||||
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
|
||||
- What to price a self-hosted model at follows from what the cluster costs.
|
||||
Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
|
||||
are in: the hardware written off over its years, its power and room, and the
|
||||
people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
|
||||
server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
|
||||
`auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
|
||||
hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
|
||||
prices suggest` spreads them over the tokens the model served: an output
|
||||
token counted as four input ones (making one is that much dearer), a cached
|
||||
prompt token as a tenth of one. Idle hours are in the cost, so a model held
|
||||
all month and seldom used comes out dear, which is the point; one held and
|
||||
billed nothing (a guard's judge) is shown with what it held. `--apply`
|
||||
writes the prices into `auth.pricing`.
|
||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||
is empty. Without it, a file with no models is refused as having nothing to
|
||||
serve.
|
||||
|
||||
@@ -43,6 +43,8 @@ rather than repeated.
|
||||
| `personal` | `Personal` | [Auth](#auth) |
|
||||
| `pricing` | `map[string]Price` | [Auth](#auth) |
|
||||
| `default_price` | `Price` | [Auth](#auth) |
|
||||
| `gpu_hour_cost` | `float64` | [Auth](#auth) |
|
||||
| `core_hour_cost` | `float64` | [Auth](#auth) |
|
||||
| `flush_interval` | `time.Duration` | [Auth](#auth) |
|
||||
| `batch_size` | `int` | [Auth](#auth) |
|
||||
| `buffer_size` | `int` | [Auth](#auth) |
|
||||
@@ -722,6 +724,21 @@ table costs whatever DefaultPrice says, which is nothing unless set.
|
||||
|
||||
Takes a [config.Price](#configprice) block.
|
||||
|
||||
### `gpu_hour_cost`
|
||||
|
||||
`float64`
|
||||
|
||||
GPUHourCost is what an hour of one GPU costs the operator, in the
|
||||
units budgets are in: the hardware written off over its years, its
|
||||
power and room, and the people who run it, over the GPUs and the
|
||||
hours of a month. With CoreHourCost for a CPU core, it is what
|
||||
`llmgw ctl prices suggest` prices each model from: the hours its
|
||||
replicas held, idle ones too, over the tokens it served.
|
||||
|
||||
### `core_hour_cost`
|
||||
|
||||
`float64`
|
||||
|
||||
### `flush_interval`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Reference in new issue
Block a user