llmgw server is real now. It fronts a static set of engine replicas with an OpenAI-compatible API, scoring endpoints through the pipeline added in the previous commit. - proxy: SSE relayed event by event with a flush after each, cancellation propagated upstream, and separate deadlines for dial, headers, first token, inter-token idle and total. The idle watchdog is what distinguishes a wedged engine from a slow one. - router: dispatch with retry and model fallback. Retry only happens while nothing has reached the client; after that, splicing two responses together would be worse than the short stream the client already sees. - config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides. Validation rejects duplicate endpoint ids and aliases that shadow a model, because both misroute silently at request time rather than failing loudly. - metrics: requests, duration, TTFT, tokens, plus per-scorer values and per-filter drops on a separate admin listener with health and pprof. Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across 3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV blocks on every engine; router overhead p99 is about 0.9 ms against the plan's 2 ms budget. The handlers live in internal/router rather than an internal/router/openai package as the plan's tree sketched: they need the dispatch loop, and splitting them apart would have meant an interface with exactly one implementation. Plan §4 updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f0c1ca87f1
commit
a3f71aa5ee
20 files changed
+3028
-26
No files matched your search
@@ -0,0 +1,39 @@
|
||||
package proxy
|
||||
|
||||
import "encoding/json"
|
||||
|
||||
// usageEnvelope mirrors just the usage block of an OpenAI response.
|
||||
// Everything else in the payload is passed through untouched, so only these
|
||||
// fields are ever decoded.
|
||||
type usageEnvelope struct {
|
||||
Usage *struct {
|
||||
PromptTokens int `json:"prompt_tokens"`
|
||||
CompletionTokens int `json:"completion_tokens"`
|
||||
TotalTokens int `json:"total_tokens"`
|
||||
PromptTokensDetails *struct {
|
||||
CachedTokens int `json:"cached_tokens"`
|
||||
} `json:"prompt_tokens_details"`
|
||||
} `json:"usage"`
|
||||
}
|
||||
|
||||
// usageFromJSON extracts token usage from a response or an SSE chunk.
|
||||
// It returns nil when the payload carries no usage block, which is the normal
|
||||
// case for every chunk but the last.
|
||||
func usageFromJSON(b []byte) *Usage {
|
||||
var env usageEnvelope
|
||||
if err := json.Unmarshal(b, &env); err != nil || env.Usage == nil {
|
||||
return nil
|
||||
}
|
||||
u := &Usage{
|
||||
PromptTokens: env.Usage.PromptTokens,
|
||||
CompletionTokens: env.Usage.CompletionTokens,
|
||||
TotalTokens: env.Usage.TotalTokens,
|
||||
}
|
||||
if d := env.Usage.PromptTokensDetails; d != nil {
|
||||
u.CachedTokens = d.CachedTokens
|
||||
}
|
||||
if u.TotalTokens == 0 {
|
||||
u.TotalTokens = u.PromptTokens + u.CompletionTokens
|
||||
}
|
||||
return u
|
||||
}
|
||||
Reference in new issue
Block a user