auth.gpu_hour_cost (and core_hour_cost) is what an hour of a GPU (a core) costs the operator. The leader keeps, by the hour, the GPU- and core-hours each model's replicas hold (held_1h); llmgw ctl prices suggest spreads a model's over the tokens it served, an output token counted as four input ones and a cached prompt token as a tenth of one, and --apply writes the prices into auth.pricing. Idle hours are in the cost, so a model held and little used comes out dear; one held and billed nothing is shown with what it held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b8d3c084c5
commit
8547528fb7
22 files changed
+834
-1
No files matched your search
@@ -457,6 +457,19 @@ by itself. Three things are worth knowing before you edit it:
|
||||
cache (the answer's `cached_tokens`) are charged at it, the rest at the
|
||||
prompt rate. A provider that bills cached reads at a tenth is
|
||||
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
|
||||
- What to price a self-hosted model at follows from what the cluster costs.
|
||||
Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
|
||||
are in: the hardware written off over its years, its power and room, and the
|
||||
people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
|
||||
server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
|
||||
`auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
|
||||
hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
|
||||
prices suggest` spreads them over the tokens the model served: an output
|
||||
token counted as four input ones (making one is that much dearer), a cached
|
||||
prompt token as a tenth of one. Idle hours are in the cost, so a model held
|
||||
all month and seldom used comes out dear, which is the point; one held and
|
||||
billed nothing (a guard's judge) is shown with what it held. `--apply`
|
||||
writes the prices into `auth.pricing`.
|
||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||
is empty. Without it, a file with no models is refused as having nothing to
|
||||
serve.
|
||||
|
||||
Reference in new issue
Block a user