Give every engine a key only the router holds

Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.

Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.

The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.

Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.

Also formats two files that went in unformatted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 00:38:29 +02:00
1 parent d167686cf1
commit 9853936e0b
34 files changed
+356 -29

No files matched your search

+15 -3
View File
@@ -2525,7 +2525,11 @@ type StartShard struct {
Fabric string `protobuf:"bytes,14,opt,name=fabric,proto3" json:"fabric,omitempty"`
// profile_fp identifies the version of the profile, so the agent can say
// which one a running shard was started from. The agent never interprets it.
ProfileFp string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
ProfileFp string `protobuf:"bytes,15,opt,name=profile_fp,json=profileFp,proto3" json:"profile_fp,omitempty"`
// api_key is what the engine demands of every request, derived by the
// control plane from the replica id. The router sends it; nothing else
// that can reach the node's ports has it.
ApiKey string `protobuf:"bytes,16,opt,name=api_key,json=apiKey,proto3" json:"api_key,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -2665,6 +2669,13 @@ func (x *StartShard) GetProfileFp() string {
return ""
}
func (x *StartShard) GetApiKey() string {
if x != nil {
return x.ApiKey
}
return ""
}
type EngineProfile struct {
state protoimpl.MessageState `protogen:"open.v1"`
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
@@ -3693,7 +3704,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\n" +
"bytes_done\x18\x02 \x01(\x03R\tbytesDone\x12\x1f\n" +
"\vbytes_total\x18\x03 \x01(\x03R\n" +
"bytesTotal\"\xd9\x03\n" +
"bytesTotal\"\xf2\x03\n" +
"\n" +
"StartShard\x12\x1d\n" +
"\n" +
@@ -3716,7 +3727,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x04pool\x18\r \x01(\tR\x04pool\x12\x16\n" +
"\x06fabric\x18\x0e \x01(\tR\x06fabric\x12\x1d\n" +
"\n" +
"profile_fp\x18\x0f \x01(\tR\tprofileFp\"\xfd\x02\n" +
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\"\xfd\x02\n" +
"\rEngineProfile\x12\x12\n" +
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
+4
View File
@@ -524,6 +524,10 @@ message StartShard {
// profile_fp identifies the version of the profile, so the agent can say
// which one a running shard was started from. The agent never interprets it.
string profile_fp = 15;
// api_key is what the engine demands of every request, derived by the
// control plane from the replica id. The router sends it; nothing else
// that can reach the node's ports has it.
string api_key = 16;
}
message EngineProfile {