Give every engine a key only the router holds
Engines listen on every address their node has, with no key. Checked on the live cluster: from a neighbouring node, a request straight to an engine's port was answered — past the gateway's authentication, quotas and accounting, by anyone who could reach the node. Each replica's engine is now started with a key and refuses requests without it. The key is derived from the replica id with an HMAC over a secret kept in the store, rather than stored per replica: the router has to send the right key to replicas this process never started — adopted after a restart, or placed by the other control plane — and a function of a shared secret gives that without a table to keep in step. A key taken from one node opens that replica only. The secret is made on first use and copied with the rest of the store when moving to PostgreSQL; a new one would lock the router out of every running engine at the moment of the move. The key reaches llama.cpp and vLLM through their environment (LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every user on the node can read; SGLang only takes it as an argument. The agent's readiness check, the router's proxy and its metrics scrape send it, the client's own Authorization still never reaches the engine, and `ctl bench` fetches the key of the replica it is pointed at from the admin API. Engines started before this keep running without a key until their replica is next started. With no store there is nowhere to keep the secret, and engines start without one, with a warning. Also formats two files that went in unformatted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d167686cf1
commit
9853936e0b
34 files changed
+356
-29
No files matched your search
@@ -355,3 +355,22 @@ func TestAStreamingRequestKeepsTheShortLivenessTimeout(t *testing.T) {
|
||||
t.Errorf("took %s; the streaming liveness timeout is not being applied", elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheEngineGetsLlmgwsKeyNotTheClients(t *testing.T) {
|
||||
var got string
|
||||
up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got = r.Header.Get("Authorization")
|
||||
io.WriteString(w, `{}`)
|
||||
}))
|
||||
defer up.Close()
|
||||
|
||||
client := http.Header{"Authorization": {"Bearer sk-the-callers-own"}}
|
||||
_, err := testProxy().Do(t.Context(), httptest.NewRecorder(),
|
||||
Request{URL: up.URL, Body: []byte(`{}`), Header: client, APIKey: "llmgw-engine"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != "Bearer llmgw-engine" {
|
||||
t.Errorf("the engine was sent %q", got)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user