Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 bd12b0909a Serve a model from the registry with one command
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m50s
Release / build (push) Successful in 1m8s
Adding a model took four steps and a YAML file — plan it, fetch it into the
mirror, write its catalog entry with a memory figure somebody estimated,
scale it — and whether it would load was found out at the end.

`llmgw ctl model serve OWNER/REPO` does it as one job in the control plane,
where the registry, the mirror, the catalog and the fleet already are:

- the registry: the GGUF file set matching --quant (Q4_K_M by default) or
  --file, leaving out vision projectors and importance matrices, and naming
  what the repository does have when nothing matches; gated repositories are
  refused;
- the mirror: fetched at the repository's revision, with progress, and each
  file's sha256 compared with what the registry published — the mirror is
  the one place a corrupted download is caught once rather than loaded on
  every node;
- the check: whether the nodes' llama.cpp loads the architecture, and the
  memory it needs from its own header;
- the catalog, then a wait for a replica that stops early, with the reason,
  when the model is parked or fits on no node;
- a smoke test: one short request to the replica, with its engine key, and
  the answer and tokens per second.

ctl starts the job and prints each step as it changes; a download of hours
does not depend on the terminal. --on-demand adds the model without starting
a replica. Jobs are kept in memory until the control plane restarts.

The catalog name a repository suggests moved into the registry package, so
the mirror page and this agree on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:05:13 +02:00
Ilker ManapandClaude Opus 5.5 116e75f8cb Upgrade the cluster with one command, over the agents' own streams
Release / build (push) Successful in 1m5s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m42s
Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.

`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.

A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.

The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:54:36 +02:00
25 changed files with 2028 additions and 98 deletions

No files matched your search

+23
View File
@@ -77,6 +77,29 @@ more than 26 hours old or there has never been one. docs/install.md now says how
to restore, including why running engines survive it: the engine-key secret
comes back with the store.
### A model from the registry in one command
`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
mirror fetch it and checks what arrived against the registry's published
sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
header, adds it to the catalog, waits for a replica — stopping early with the
reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
is shown as it happens. It runs in the control plane as a job, so a download of
hours does not depend on a terminal staying open. `--on-demand` adds it without
starting a replica.
### Upgrading from one command
`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
machine and then every node in turn. Each agent is told the version over its
existing stream, fetches the release for its own platform, checks it against
the release's checksums, keeps the old binary beside the new and restarts,
while its engines keep serving. A node that does not come back on the new
version, ready, with its replicas, stops the upgrade before the next one.
Running it again resumes. Agents from before this have to be moved by hand
once.
### Following a request
Every response carries `X-Request-Id` — the client's own when it sends a plain
+241 -57
View File
@@ -104,6 +104,7 @@ type AgentMessage struct {
// *AgentMessage_Logs
// *AgentMessage_FabricScan
// *AgentMessage_FabricTest
// *AgentMessage_Upgrade
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -200,6 +201,15 @@ func (x *AgentMessage) GetFabricTest() *FabricTestReply {
return nil
}
func (x *AgentMessage) GetUpgrade() *UpgradeReply {
if x != nil {
if x, ok := x.Msg.(*AgentMessage_Upgrade); ok {
return x.Upgrade
}
}
return nil
}
type isAgentMessage_Msg interface {
isAgentMessage_Msg()
}
@@ -228,6 +238,10 @@ type AgentMessage_FabricTest struct {
FabricTest *FabricTestReply `protobuf:"bytes,6,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
}
type AgentMessage_Upgrade struct {
Upgrade *UpgradeReply `protobuf:"bytes,7,opt,name=upgrade,proto3,oneof"`
}
func (*AgentMessage_Register) isAgentMessage_Msg() {}
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
@@ -240,6 +254,8 @@ func (*AgentMessage_FabricScan) isAgentMessage_Msg() {}
func (*AgentMessage_FabricTest) isAgentMessage_Msg() {}
func (*AgentMessage_Upgrade) isAgentMessage_Msg() {}
// ControlMessage is anything the control plane tells a node.
type ControlMessage struct {
state protoimpl.MessageState `protogen:"open.v1"`
@@ -254,6 +270,7 @@ type ControlMessage struct {
// *ControlMessage_Logs
// *ControlMessage_FabricScan
// *ControlMessage_FabricTest
// *ControlMessage_Upgrade
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -377,6 +394,15 @@ func (x *ControlMessage) GetFabricTest() *FabricTestRequest {
return nil
}
func (x *ControlMessage) GetUpgrade() *UpgradeAgent {
if x != nil {
if x, ok := x.Msg.(*ControlMessage_Upgrade); ok {
return x.Upgrade
}
}
return nil
}
type isControlMessage_Msg interface {
isControlMessage_Msg()
}
@@ -417,6 +443,10 @@ type ControlMessage_FabricTest struct {
FabricTest *FabricTestRequest `protobuf:"bytes,9,opt,name=fabric_test,json=fabricTest,proto3,oneof"`
}
type ControlMessage_Upgrade struct {
Upgrade *UpgradeAgent `protobuf:"bytes,10,opt,name=upgrade,proto3,oneof"`
}
func (*ControlMessage_Start) isControlMessage_Msg() {}
func (*ControlMessage_Stop) isControlMessage_Msg() {}
@@ -435,6 +465,8 @@ func (*ControlMessage_FabricScan) isControlMessage_Msg() {}
func (*ControlMessage_FabricTest) isControlMessage_Msg() {}
func (*ControlMessage_Upgrade) isControlMessage_Msg() {}
// Register is the first message on every stream. It carries the node's full
// hardware inventory, so the control plane needs no separate discovery call and
// a reconnecting agent automatically refreshes what the scheduler knows.
@@ -3460,11 +3492,144 @@ func (x *FabricTestReply) GetTookMs() int64 {
return 0
}
// UpgradeAgent tells an agent to replace its own binary with a release and
// restart.
//
// The agent fetches the release for its own platform and checks it against
// the release's checksums before replacing anything, keeps the old binary
// beside the new, then exits so systemd starts the new one. Engines keep
// running throughout, as for any agent restart, and are adopted on the way
// back. Over the stream the agent already holds, so upgrading a fleet needs
// no ssh into it.
type UpgradeAgent struct {
state protoimpl.MessageState `protogen:"open.v1"`
RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
// version must be a release: v0.10.4 or v0.10.4-rc.2.
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
// releases is the base URL release files are downloaded from.
Releases string `protobuf:"bytes,3,opt,name=releases,proto3" json:"releases,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *UpgradeAgent) Reset() {
*x = UpgradeAgent{}
mi := &file_llmgw_v1_agent_proto_msgTypes[33]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *UpgradeAgent) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*UpgradeAgent) ProtoMessage() {}
func (x *UpgradeAgent) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[33]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use UpgradeAgent.ProtoReflect.Descriptor instead.
func (*UpgradeAgent) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{33}
}
func (x *UpgradeAgent) GetRequestId() string {
if x != nil {
return x.RequestId
}
return ""
}
func (x *UpgradeAgent) GetVersion() string {
if x != nil {
return x.Version
}
return ""
}
func (x *UpgradeAgent) GetReleases() string {
if x != nil {
return x.Releases
}
return ""
}
// UpgradeReply says whether the new binary is in place. Sent before the
// agent exits; an empty error means it will be back on the new version.
type UpgradeReply struct {
state protoimpl.MessageState `protogen:"open.v1"`
RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"`
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"`
Error string `protobuf:"bytes,3,opt,name=error,proto3" json:"error,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *UpgradeReply) Reset() {
*x = UpgradeReply{}
mi := &file_llmgw_v1_agent_proto_msgTypes[34]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *UpgradeReply) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*UpgradeReply) ProtoMessage() {}
func (x *UpgradeReply) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[34]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use UpgradeReply.ProtoReflect.Descriptor instead.
func (*UpgradeReply) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{34}
}
func (x *UpgradeReply) GetRequestId() string {
if x != nil {
return x.RequestId
}
return ""
}
func (x *UpgradeReply) GetVersion() string {
if x != nil {
return x.Version
}
return ""
}
func (x *UpgradeReply) GetError() string {
if x != nil {
return x.Error
}
return ""
}
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
const file_llmgw_v1_agent_proto_rawDesc = "" +
"\n" +
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xd2\x02\n" +
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\x86\x03\n" +
"\fAgentMessage\x120\n" +
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
@@ -3473,8 +3638,9 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\vfabric_scan\x18\x05 \x01(\v2\x19.llmgw.v1.FabricScanReplyH\x00R\n" +
"fabricScan\x12<\n" +
"\vfabric_test\x18\x06 \x01(\v2\x19.llmgw.v1.FabricTestReplyH\x00R\n" +
"fabricTestB\x05\n" +
"\x03msg\"\xe5\x03\n" +
"fabricTest\x122\n" +
"\aupgrade\x18\a \x01(\v2\x16.llmgw.v1.UpgradeReplyH\x00R\aupgradeB\x05\n" +
"\x03msg\"\x99\x04\n" +
"\x0eControlMessage\x12,\n" +
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
@@ -3486,7 +3652,9 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\vfabric_scan\x18\b \x01(\v2\x1b.llmgw.v1.FabricScanRequestH\x00R\n" +
"fabricScan\x12>\n" +
"\vfabric_test\x18\t \x01(\v2\x1b.llmgw.v1.FabricTestRequestH\x00R\n" +
"fabricTestB\x05\n" +
"fabricTest\x122\n" +
"\aupgrade\x18\n" +
" \x01(\v2\x16.llmgw.v1.UpgradeAgentH\x00R\aupgradeB\x05\n" +
"\x03msg\"\xe3\x06\n" +
"\bRegister\x12\x17\n" +
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12\x1a\n" +
@@ -3795,7 +3963,17 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x04gbps\x18\x02 \x01(\x01R\x04gbps\x12\x16\n" +
"\x06output\x18\x03 \x01(\tR\x06output\x12\x14\n" +
"\x05error\x18\x04 \x01(\tR\x05error\x12\x17\n" +
"\atook_ms\x18\x05 \x01(\x03R\x06tookMs2O\n" +
"\atook_ms\x18\x05 \x01(\x03R\x06tookMs\"c\n" +
"\fUpgradeAgent\x12\x1d\n" +
"\n" +
"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x1a\n" +
"\breleases\x18\x03 \x01(\tR\breleases\"]\n" +
"\fUpgradeReply\x12\x1d\n" +
"\n" +
"request_id\x18\x01 \x01(\tR\trequestId\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x14\n" +
"\x05error\x18\x03 \x01(\tR\x05error2O\n" +
"\fAgentService\x12?\n" +
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
"\fcom.llmgw.v1B\n" +
@@ -3814,7 +3992,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
}
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 39)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 41)
var file_llmgw_v1_agent_proto_goTypes = []any{
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
@@ -3850,12 +4028,14 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
(*FabricScanReply)(nil), // 31: llmgw.v1.FabricScanReply
(*FabricTestRequest)(nil), // 32: llmgw.v1.FabricTestRequest
(*FabricTestReply)(nil), // 33: llmgw.v1.FabricTestReply
nil, // 34: llmgw.v1.Register.LabelsEntry
nil, // 35: llmgw.v1.Register.EngineVersionsEntry
nil, // 36: llmgw.v1.GPU.NicPathsEntry
nil, // 37: llmgw.v1.IBPort.CountersEntry
nil, // 38: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 39: llmgw.v1.EngineProfile.EnvEntry
(*UpgradeAgent)(nil), // 34: llmgw.v1.UpgradeAgent
(*UpgradeReply)(nil), // 35: llmgw.v1.UpgradeReply
nil, // 36: llmgw.v1.Register.LabelsEntry
nil, // 37: llmgw.v1.Register.EngineVersionsEntry
nil, // 38: llmgw.v1.GPU.NicPathsEntry
nil, // 39: llmgw.v1.IBPort.CountersEntry
nil, // 40: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 41: llmgw.v1.EngineProfile.EnvEntry
}
var file_llmgw_v1_agent_proto_depIdxs = []int32{
3, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -3864,50 +4044,52 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
29, // 3: llmgw.v1.AgentMessage.logs:type_name -> llmgw.v1.LogsReply
31, // 4: llmgw.v1.AgentMessage.fabric_scan:type_name -> llmgw.v1.FabricScanReply
33, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
22, // 6: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
25, // 7: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
26, // 8: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
27, // 9: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
24, // 10: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
17, // 11: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
28, // 12: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
30, // 13: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
32, // 14: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
34, // 15: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
7, // 16: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
8, // 17: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
35, // 18: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
6, // 19: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
9, // 20: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
5, // 21: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
4, // 22: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
14, // 23: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 24: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
36, // 25: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
11, // 26: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
37, // 27: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
13, // 28: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
5, // 29: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
9, // 30: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
38, // 31: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
19, // 32: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
10, // 33: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
12, // 34: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
8, // 35: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
15, // 36: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
14, // 37: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
18, // 38: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
0, // 39: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
21, // 40: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
23, // 41: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
39, // 42: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
1, // 43: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
2, // 44: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
44, // [44:45] is the sub-list for method output_type
43, // [43:44] is the sub-list for method input_type
43, // [43:43] is the sub-list for extension type_name
43, // [43:43] is the sub-list for extension extendee
0, // [0:43] is the sub-list for field type_name
35, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
22, // 7: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
25, // 8: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
26, // 9: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
27, // 10: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
24, // 11: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
17, // 12: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
28, // 13: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
30, // 14: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
32, // 15: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
34, // 16: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
36, // 17: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
7, // 18: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
8, // 19: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
37, // 20: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
6, // 21: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
9, // 22: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
5, // 23: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
4, // 24: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
14, // 25: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 26: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
38, // 27: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
11, // 28: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
39, // 29: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
13, // 30: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
5, // 31: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
9, // 32: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
40, // 33: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
19, // 34: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
10, // 35: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
12, // 36: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
8, // 37: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
15, // 38: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
14, // 39: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
18, // 40: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
0, // 41: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
21, // 42: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
23, // 43: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
41, // 44: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
1, // 45: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
2, // 46: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
46, // [46:47] is the sub-list for method output_type
45, // [45:46] is the sub-list for method input_type
45, // [45:45] is the sub-list for extension type_name
45, // [45:45] is the sub-list for extension extendee
0, // [0:45] is the sub-list for field type_name
}
func init() { file_llmgw_v1_agent_proto_init() }
@@ -3922,6 +4104,7 @@ func file_llmgw_v1_agent_proto_init() {
(*AgentMessage_Logs)(nil),
(*AgentMessage_FabricScan)(nil),
(*AgentMessage_FabricTest)(nil),
(*AgentMessage_Upgrade)(nil),
}
file_llmgw_v1_agent_proto_msgTypes[1].OneofWrappers = []any{
(*ControlMessage_Start)(nil),
@@ -3933,6 +4116,7 @@ func file_llmgw_v1_agent_proto_init() {
(*ControlMessage_Logs)(nil),
(*ControlMessage_FabricScan)(nil),
(*ControlMessage_FabricTest)(nil),
(*ControlMessage_Upgrade)(nil),
}
type x struct{}
out := protoimpl.TypeBuilder{
@@ -3940,7 +4124,7 @@ func file_llmgw_v1_agent_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
NumEnums: 1,
NumMessages: 39,
NumMessages: 41,
NumExtensions: 0,
NumServices: 1,
},
+27
View File
@@ -26,6 +26,7 @@ message AgentMessage {
LogsReply logs = 4;
FabricScanReply fabric_scan = 5;
FabricTestReply fabric_test = 6;
UpgradeReply upgrade = 7;
}
}
@@ -41,6 +42,7 @@ message ControlMessage {
LogsRequest logs = 7;
FabricScanRequest fabric_scan = 8;
FabricTestRequest fabric_test = 9;
UpgradeAgent upgrade = 10;
}
}
@@ -649,3 +651,28 @@ message FabricTestReply {
string error = 4;
int64 took_ms = 5;
}
// UpgradeAgent tells an agent to replace its own binary with a release and
// restart.
//
// The agent fetches the release for its own platform and checks it against
// the release's checksums before replacing anything, keeps the old binary
// beside the new, then exits so systemd starts the new one. Engines keep
// running throughout, as for any agent restart, and are adopted on the way
// back. Over the stream the agent already holds, so upgrading a fleet needs
// no ssh into it.
message UpgradeAgent {
string request_id = 1;
// version must be a release: v0.10.4 or v0.10.4-rc.2.
string version = 2;
// releases is the base URL release files are downloaded from.
string releases = 3;
}
// UpgradeReply says whether the new binary is in place. Sent before the
// agent exits; an empty error means it will be back on the new version.
message UpgradeReply {
string request_id = 1;
string version = 2;
string error = 3;
}
+29 -32
View File
@@ -909,55 +909,52 @@ refuses every request.
## Upgrading
One binary on every machine, so an upgrade is a copy and a restart. The part
worth care is the copy, not the order.
On the control plane:
```
sudo llmgw ctl upgrade v0.10.4
```
It upgrades this machine first — fetching the release for its platform,
checking it against the release's checksums, keeping the old binary as
`/usr/local/bin/llmgw.bak-pre-<version>`, restarting `llmgw-server` and waiting
for every node to reconnect — and then each node in turn. A node's agent is told
the version over the stream it already holds, fetches and checks the release
itself, swaps it in and restarts; no ssh into the nodes is needed. Release
candidates (`v0.10.4-rc.2`) work the same way.
A node counts as done when it is back on the new version, ready, with as many
ready replicas as before. One that is not within `--timeout` (three minutes)
stops the upgrade there, so a bad release reaches one node rather than all of
them; `llmgw ctl logs <node>` says why, and the node keeps its previous binary
beside the new one. Nodes already on the version are skipped, so running the
command again resumes where it stopped. `--nodes a,b` limits it to some nodes,
and `--skip-control-plane` leaves this machine alone.
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
keep running when the agent restarts, and the agent reports them back on its
next connection. The control plane holds no replica state across its own
restart either — it rebuilds the fleet from what the agents report. A rolling
upgrade of the whole cluster interrupts no request that is not in flight at the
moment of the restart.
restart either — it rebuilds the fleet from what the agents report.
**A mixed fleet is fine while you roll.** The agent protocol only ever gains
fields, and both sides treat an absent one as unknown rather than as a default.
Order does not matter; finish the roll the same day anyway, because
`ctl nodes` reports each agent's version and a fleet that disagrees with itself
is one more thing to hold in your head.
```
VERSION=v0.9.0
# copy to every machine first, install nowhere yet
for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
done
```
**Check every size before installing anything.** This is not defensive writing:
on a host whose login shell prints anything, `scp` fails outright, and the
obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
and writes a *zero-byte file* while reporting success. Installing that on a
fleet takes the fleet down. Compare the number on each machine with the local
one, then:
```
sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
sudo systemctl restart llmgw-agent # or llmgw-server, or llmgw-mirror
llmgw --version
```
Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
installing the release binary and restarting `llmgw-agent`, and every upgrade
after that is the command above. The mirror, which is not an agent, is
upgraded the same way by hand.
Then confirm what you have, rather than assuming:
```
llmgw ctl nodes # every agent back, LAST SEEN in seconds
llmgw ctl nodes # every agent on the new version, LAST SEEN in seconds
llmgw ctl replicas # same replicas, adopted rather than replaced
```
A replica may come back marked **`ready (stale profile)`**. That means it is
running a profile that has since been edited — the reconciler leaves a serving
replica alone on purpose, because churning one costs more than the difference.
Restart it when you want the new profile to take effect; the flag is there so
the choice is yours rather than silent.
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
the new one before stopping the old.
## Backing up
+9
View File
@@ -170,6 +170,7 @@ rather than repeated.
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
| `models` | `[]catalog.Model` | [Server](#server) |
@@ -1274,6 +1275,14 @@ BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw-server.
### `releases`
`string`
Releases is where `llmgw ctl upgrade` fetches release binaries from,
for this machine and — through their agents — every node. Empty means
the project's own releases.
### `router`
`Router`
+50
View File
@@ -8,6 +8,7 @@ import (
"encoding/hex"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"io"
"log/slog"
"maps"
@@ -18,6 +19,7 @@ import (
"os"
"os/exec"
"path/filepath"
"runtime"
"slices"
"strconv"
"strings"
@@ -1119,6 +1121,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
a.removeWeights(m.RemoveWeights.GetPaths())
case *llmgwv1.ControlMessage_Provision:
a.setProvision(m.Provision)
case *llmgwv1.ControlMessage_Upgrade:
go a.upgrade(ctx, m.Upgrade, out)
case *llmgwv1.ControlMessage_Logs:
reply := a.logsReply(m.Logs)
// Off the receive loop: a full send queue must not stop commands being
@@ -1874,3 +1878,49 @@ func (a *Agent) sampleNVLink(hb *llmgwv1.Heartbeat) {
m := discoverIMEX(ctx)
hb.Imex = &llmgwv1.IMEX{State: m.State, Channels: int32(m.Channels)}
}
// upgradeExit is the exit status after an upgrade: non-zero, so systemd's
// Restart=on-failure starts the new binary, and distinct, so the journal says
// why the agent went.
const upgradeExit = 75
// exitForUpgrade ends the process; a variable so a test can see it called
// rather than have its own process end.
var exitForUpgrade = func() { os.Exit(upgradeExit) }
// upgradeTarget is the binary an upgrade replaces; a variable so a test
// replaces a file of its own rather than the test binary running it.
var upgradeTarget = os.Executable
// upgrade replaces this agent's binary with a release and restarts into it.
//
// Engines are not touched: an agent restart leaves them running and adopts
// them again, which is what makes upgrading a node something that can be done
// while it serves.
func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<- *llmgwv1.AgentMessage) {
reply := &llmgwv1.UpgradeReply{RequestId: m.GetRequestId(), Version: m.GetVersion()}
send := func() {
select {
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
case <-ctx.Done():
}
}
a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
self, err := upgradeTarget()
if err == nil {
dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
cancel()
}
if err != nil {
reply.Error = err.Error()
a.log.Error("upgrade failed; still running the old version", "to", m.GetVersion(), "err", err)
send()
return
}
send()
a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
// A moment for the reply to leave before the stream goes.
time.Sleep(500 * time.Millisecond)
exitForUpgrade()
}
+78
View File
@@ -0,0 +1,78 @@
package agent
import (
"crypto/sha256"
"encoding/hex"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"runtime"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
)
func TestAnAgentUpgradesItselfAndSaysSoBeforeRestarting(t *testing.T) {
body := []byte("the new agent")
sum := sha256.Sum256(body)
asset := release.AssetName("v9.9.9", runtime.GOOS, runtime.GOARCH)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v9.9.9/checksums.txt":
_, _ = w.Write([]byte(hex.EncodeToString(sum[:]) + " " + asset + "\n"))
case "/v9.9.9/" + asset:
_, _ = w.Write(body)
default:
http.NotFound(w, r)
}
}))
defer srv.Close()
bin := filepath.Join(t.TempDir(), "llmgw")
_ = os.WriteFile(bin, []byte("the old agent"), 0o755)
exited := make(chan struct{}, 1)
oldTarget, oldExit := upgradeTarget, exitForUpgrade
upgradeTarget = func() (string, error) { return bin, nil }
exitForUpgrade = func() { exited <- struct{}{} }
t.Cleanup(func() { upgradeTarget, exitForUpgrade = oldTarget, oldExit })
cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir()}
cfg.Superv.WorkDir = t.TempDir()
a, err := New(cfg, nil, nil)
if err != nil {
t.Fatal(err)
}
out := make(chan *llmgwv1.AgentMessage, 1)
go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u1", Version: "v9.9.9", Releases: srv.URL}, out)
select {
case m := <-out:
if r := m.GetUpgrade(); r.GetRequestId() != "u1" || r.GetError() != "" {
t.Fatalf("reply %+v", r)
}
case <-time.After(10 * time.Second):
t.Fatal("no reply")
}
select {
case <-exited:
case <-time.After(5 * time.Second):
t.Fatal("the agent did not restart into the new binary")
}
if b, _ := os.ReadFile(bin); string(b) != "the new agent" {
t.Errorf("binary holds %q", b)
}
// A version with no release answers with why, and does not restart.
go a.upgrade(t.Context(), &llmgwv1.UpgradeAgent{RequestId: "u2", Version: "v9.9.8", Releases: srv.URL}, out)
if r := (<-out).GetUpgrade(); r.GetError() == "" {
t.Error("an upgrade to a release that does not exist reported success")
}
select {
case <-exited:
t.Error("a failed upgrade restarted the agent")
case <-time.After(time.Second):
}
}
+1
View File
@@ -141,6 +141,7 @@ func newCtlCmd() *cobra.Command {
newScaleCmd(client),
newRestartCmd(client),
newRetryCmd(client),
newUpgradeCmd(client),
newEngineCheckCmd(),
newToolCheckCmd(),
newWeightsCmd(client, &asJSON),
+197
View File
@@ -0,0 +1,197 @@
package cli
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"runtime"
"slices"
"strings"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
)
func newUpgradeCmd(client func() *ctlClient) *cobra.Command {
var (
nodes []string
skipCP bool
releases string
perNode time.Duration
serverSvc string
)
c := &cobra.Command{
Use: "upgrade VERSION",
Short: "Upgrade this control plane, then every node, one at a time",
Long: "Moves the cluster to a release: v0.10.4, or a candidate like v0.10.4-rc.2.\n\n" +
"This machine first — run it with sudo on the control plane, since it\n" +
"replaces /usr/local/bin/llmgw and restarts llmgw-server. Then each node, in\n" +
"turn: its agent fetches the release for its own platform, checks it\n" +
"against the release's checksums, swaps it in and restarts. Engines keep\n" +
"serving through all of it and are adopted again; no ssh into the nodes is\n" +
"needed.\n\n" +
"A node counts as done when it is back on the new version, ready, with as\n" +
"many ready replicas as before. One that is not within --timeout stops the\n" +
"upgrade there, so a bad release reaches one node rather than all of them.\n" +
"Nodes already on the version are skipped, so running it again resumes.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
v := args[0]
if !release.Tag.MatchString(v) {
return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
}
out := cmd.OutOrStdout()
ctx := cmd.Context()
if !skipCP {
if err := upgradeControlPlane(ctx, out, client, v, releases, serverSvc); err != nil {
return err
}
}
return upgradeNodes(ctx, out, client, v, nodes, perNode)
},
}
c.Flags().StringSliceVar(&nodes, "nodes", nil, "only these nodes (default: all)")
c.Flags().BoolVar(&skipCP, "skip-control-plane", false, "leave this machine as it is and upgrade only nodes")
c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from, for this machine")
c.Flags().DurationVar(&perNode, "timeout", 3*time.Minute, "how long a node may take to come back")
c.Flags().StringVar(&serverSvc, "service", "llmgw-server", "this machine's systemd unit")
return c
}
func upgradeControlPlane(ctx context.Context, out io.Writer, client func() *ctlClient, v, releases, svc string) error {
if version.Version == v {
fmt.Fprintf(out, "this machine already runs %s\n", v)
return nil
}
if os.Geteuid() != 0 {
return errors.New("replacing this machine's binary needs root: run it with sudo, " +
"or pass --skip-control-plane to upgrade only the nodes")
}
self, err := os.Executable()
if err != nil {
return err
}
fmt.Fprintf(out, "== this machine: %s -> %s\n", version.Version, v)
backup, err := release.Install(ctx, nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
if err != nil {
return err
}
fmt.Fprintf(out, " installed, checksum verified; the old binary is %s\n", backup)
var before []adminapi.NodeView
_ = client().get(ctx, "/api/v1/nodes", &before)
if b, err := exec.CommandContext(ctx, "systemctl", "restart", svc).CombinedOutput(); err != nil { //nolint:gosec // fixed command
return fmt.Errorf("restarting %s: %v: %s", svc, err, strings.TrimSpace(string(b)))
}
fmt.Fprintf(out, " %s restarted; waiting for its nodes to reconnect\n", svc)
// Back when the admin API answers and every node that was ready is ready
// again: agents reconnect within seconds of the control plane returning.
want := 0
for _, n := range before {
if n.State == "ready" {
want++
}
}
deadline := time.Now().Add(2 * time.Minute)
for {
var now []adminapi.NodeView
if err := client().get(ctx, "/api/v1/nodes", &now); err == nil {
ready := 0
for _, n := range now {
if n.State == "ready" {
ready++
}
}
if ready >= want {
fmt.Fprintf(out, " back: %d of %d nodes ready\n", ready, len(now))
return nil
}
}
if time.Now().After(deadline) {
return fmt.Errorf("%s did not come back with its %d nodes within two minutes; "+
"see journalctl -u %s, and %s holds the previous binary", svc, want, svc, backup)
}
time.Sleep(2 * time.Second)
}
}
func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient, v string, only []string, perNode time.Duration) error {
var nodes []adminapi.NodeView
if err := client().get(ctx, "/api/v1/nodes", &nodes); err != nil {
return err
}
slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
done, skipped := 0, 0
for _, n := range nodes {
if len(only) > 0 && !slices.Contains(only, n.ID) {
continue
}
if n.AgentVersion == v {
skipped++
continue
}
start := time.Now()
before := readyReplicasOn(ctx, client, n.ID)
fmt.Fprintf(out, "== %s: %s -> %s\n", n.ID, n.AgentVersion, v)
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
if err := client().post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
return fmt.Errorf("%s: %w\nstopped here; `llmgw ctl logs %s` says more", n.ID, err, n.ID)
}
fmt.Fprintf(out, " new binary in place; waiting for it to come back with its %d replica(s)\n", before)
deadline := time.Now().Add(perNode)
for {
var now []adminapi.NodeView
_ = client().get(ctx, "/api/v1/nodes", &now)
var cur adminapi.NodeView
for _, x := range now {
if x.ID == n.ID {
cur = x
}
}
after := readyReplicasOn(ctx, client, n.ID)
if cur.AgentVersion == v && cur.State == "ready" && after >= before {
fmt.Fprintf(out, " done in %s: ready, replicas %d/%d\n",
time.Since(start).Round(time.Second), after, before)
done++
break
}
if time.Now().After(deadline) {
return fmt.Errorf("%s did not come back within %s (version %q, state %q, replicas %d/%d)\n"+
"stopped here, before the next node; `llmgw ctl logs %s` says why, and the node "+
"keeps its previous binary beside the new one", n.ID, perNode, cur.AgentVersion,
cur.State, after, before, n.ID)
}
time.Sleep(2 * time.Second)
}
}
fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
return nil
}
// readyReplicasOn counts the ready replicas with a shard on a node.
func readyReplicasOn(ctx context.Context, client func() *ctlClient, node string) int {
var reps []adminapi.ReplicaView
if err := client().get(ctx, "/api/v1/replicas", &reps); err != nil {
return 0
}
n := 0
for _, r := range reps {
if r.State != "ready" {
continue
}
for _, s := range r.Shards {
if s.Node == node {
n++
break
}
}
}
return n
}
+88
View File
@@ -0,0 +1,88 @@
package cli
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
)
// fakeFleet is an admin API whose nodes come back on the version they are told
// to upgrade to — except those listed in stuck.
type fakeFleet struct {
mu sync.Mutex
versions map[string]string
stuck map[string]bool
told []string
}
func (f *fakeFleet) server(t *testing.T) *httptest.Server {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
defer f.mu.Unlock()
switch {
case r.URL.Path == "/api/v1/nodes":
var out []adminapi.NodeView
for id, v := range f.versions {
out = append(out, adminapi.NodeView{ID: id, State: "ready", AgentVersion: v})
}
_ = json.NewEncoder(w).Encode(out)
case r.URL.Path == "/api/v1/replicas":
var out []adminapi.ReplicaView
for id := range f.versions {
out = append(out, adminapi.ReplicaView{ID: "r-" + id, State: "ready",
Shards: []adminapi.ShardView{{Node: id}}})
}
_ = json.NewEncoder(w).Encode(out)
case strings.HasSuffix(r.URL.Path, "/upgrade"):
id := strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/api/v1/nodes/"), "/upgrade")
var req adminapi.UpgradeRequest
_ = json.NewDecoder(r.Body).Decode(&req)
f.told = append(f.told, id)
if !f.stuck[id] {
f.versions[id] = req.Version
}
_, _ = w.Write([]byte(`{}`))
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
return srv
}
func TestAnUpgradeGoesNodeByNodeAndSkipsTheDone(t *testing.T) {
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.1", "c": "v1.0.0"}}
srv := f.server(t)
var out bytes.Buffer
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second)
if err != nil {
t.Fatal(err)
}
if strings.Join(f.told, ",") != "a,c" {
t.Errorf("told %v; b was already on the version", f.told)
}
if !strings.Contains(out.String(), "2 node(s) upgraded to v1.0.1, 1 already on it") {
t.Errorf("output:\n%s", out.String())
}
}
func TestAnUpgradeStopsAtTheFirstNodeThatDoesNotComeBack(t *testing.T) {
// A bad release reaches one node, not all of them.
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0"}, stuck: map[string]bool{"a": true}}
srv := f.server(t)
var out bytes.Buffer
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 3*time.Second)
if err == nil || !strings.Contains(err.Error(), "a did not come back") {
t.Fatalf("err %v", err)
}
if len(f.told) != 1 {
t.Errorf("went on to %v after a node failed to come back", f.told)
}
}
+2 -1
View File
@@ -28,7 +28,8 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
Use: "model",
Short: "Add or remove models while the cluster is running",
}
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client))
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
newModelServeCmd(client))
return c
}
+80
View File
@@ -0,0 +1,80 @@
package cli
import (
"encoding/json"
"errors"
"fmt"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
)
func newModelServeCmd(client func() *ctlClient) *cobra.Command {
var (
req modelserve.Request
onDemand bool
)
c := &cobra.Command{
Use: "serve OWNER/REPO",
Short: "Take a model from the registry to a replica that answers, in one command",
Long: "Picks the GGUF file (--quant, Q4_K_M by default, or --file), has the mirror\n" +
"fetch it and checks what arrived against the registry's checksum, checks\n" +
"that the nodes' llama.cpp can load it and sizes its memory from its own\n" +
"header, adds it to the catalog, waits for a replica, and asks it to say\n" +
"hello. Each step is shown as it happens; the first that fails stops it.\n\n" +
" llmgw ctl model serve bartowski/Qwen2.5-3B-Instruct-GGUF\n" +
" llmgw ctl model serve Qwen/Qwen2.5-7B-Instruct-GGUF --quant Q5_K_M --name qwen-7b\n" +
" llmgw ctl model serve owner/big-GGUF --on-demand",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
req.Repo = args[0]
if onDemand {
req.Replicas = 0
}
body, _ := json.Marshal(req)
var job modelserve.Job
if err := client().post(cmd.Context(), "/api/v1/models/serve", body, &job); err != nil {
return err
}
out := cmd.OutOrStdout()
fmt.Fprintf(out, "job %s\n", job.ID)
shown := map[string]string{}
for {
for _, s := range job.Steps {
line := s.State + " " + s.Detail
if shown[s.Name] == line {
continue
}
shown[s.Name] = line
mark := map[string]string{modelserve.Running: "..", modelserve.Done: "ok",
modelserve.Failed: "!!"}[s.State]
fmt.Fprintf(out, "%-3s %-11s %s\n", mark, s.Name, s.Detail)
}
switch job.State {
case modelserve.Done:
fmt.Fprintf(out, "\n%s is served.\n", job.Model)
return nil
case modelserve.Failed:
return errors.New(job.Error)
}
time.Sleep(2 * time.Second)
if err := client().get(cmd.Context(), "/api/v1/jobs/"+job.ID, &job); err != nil {
return err
}
}
},
}
f := c.Flags()
f.StringVar(&req.Quant, "quant", "", "the quantisation to serve (default Q4_K_M)")
f.StringVar(&req.File, "file", "", "the file to serve, instead of choosing by --quant")
f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
return c
}
+34 -1
View File
@@ -5,6 +5,7 @@ import (
"crypto/tls"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
"github.com/prometheus/client_golang/prometheus"
@@ -171,8 +172,17 @@ func newServerCmd() *cobra.Command {
if err != nil {
return err
}
// Kept past this block: serving a model from the registry needs
// both the admin API and the mirror, and the mirror is set up
// below.
var (
adminAPI *adminapi.API
catalogSvc *modelsvc.Service
)
if admin != nil {
api := adminapi.New(admin)
adminAPI = api
api.SetReleases(cfg.Releases)
// Catalog changes need somewhere to persist, so this is wired
// only when there is a store. Without one, the endpoints say so
// rather than accepting a model that vanishes on restart.
@@ -189,6 +199,7 @@ func newServerCmd() *cobra.Command {
admin.ReplaceCatalog(ctx, built)
}
api.SetModels(svc)
catalogSvc = svc
// And the mirror page, so a model that has just been
// fetched can be served without writing YAML by hand.
rt.SetCatalogAdmin(svc)
@@ -258,7 +269,16 @@ func newServerCmd() *cobra.Command {
// Browsing the registry is what turns "paste a resolve URL"
// into "find the model". It needs a way out to the internet
// from this machine, which the mirror's own fetching does not.
rt.SetRegistry(hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
registry := hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout)
rt.SetRegistry(registry)
// One command from a registry repository to a replica that
// answers: `llmgw ctl model serve`.
if adminAPI != nil && catalogSvc != nil {
adminAPI.SetServe(modelserve.New(modelserve.Deps{
Registry: registry, Mirror: mc, Catalog: catalogSvc, Fleet: admin,
LlamaVersion: cfg.WantedEngines()["llamacpp"], Log: log,
}))
}
log.Info("model mirror configured",
"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
}
@@ -546,6 +566,19 @@ func (r *dispatcherRef) FabricTest(ctx context.Context, nodeID string, req *llmg
return ft.FabricTest(ctx, nodeID, req)
}
// Upgrade forwards a node upgrade, as Logs does.
func (r *dispatcherRef) Upgrade(ctx context.Context, nodeID, version, releases string) error {
d, err := r.get()
if err != nil {
return err
}
u, ok := d.(control.Upgrader)
if !ok {
return errors.New("this dispatcher cannot upgrade nodes")
}
return u.Upgrade(ctx, nodeID, version, releases)
}
func (r *dispatcherRef) StartShard(ctx context.Context, cmd reconciler.ShardCommand) error {
d, err := r.get()
if err != nil {
+5
View File
@@ -86,6 +86,11 @@ type Server struct {
// Empty means /var/backups/llmgw-server.
BackupDir string `koanf:"backup_dir"`
// Releases is where `llmgw ctl upgrade` fetches release binaries from,
// for this machine and — through their agents — every node. Empty means
// the project's own releases.
Releases string `koanf:"releases"`
Router Router `koanf:"router"`
Control Control `koanf:"control"`
Models []catalog.Model `koanf:"models"`
+80
View File
@@ -10,6 +10,8 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"io"
"net/http"
"slices"
@@ -30,6 +32,11 @@ import (
// API serves the management endpoints.
type API struct {
// releases is where release binaries are fetched from, for upgrades.
releases string
// serve runs registry-to-replica jobs; nil without a mirror and a store.
serve *modelserve.Server
models Models
capacity Capacity
usage Usage
@@ -74,10 +81,13 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("POST /api/v1/weights/collect", a.collectWeights)
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
mux.HandleFunc("POST /api/v1/restart", a.restart)
mux.HandleFunc("POST /api/v1/models/serve", a.serveModel)
mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
}
@@ -1165,3 +1175,73 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
})
}
// SetReleases sets where upgrades fetch release binaries from.
func (a *API) SetReleases(base string) { a.releases = base }
// UpgradeRequest names the release a node should run.
type UpgradeRequest struct {
Version string `json:"version"`
}
func (a *API) upgradeNode(w http.ResponseWriter, r *http.Request) {
var req UpgradeRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the upgrade request is not valid JSON: "+err.Error())
return
}
if !release.Tag.MatchString(req.Version) {
writeError(w, http.StatusBadRequest, req.Version+" is not a release version")
return
}
u := a.ctl.Upgrader()
if u == nil {
writeError(w, http.StatusNotImplemented, "this control plane cannot upgrade nodes")
return
}
base := a.releases
if base == "" {
base = release.DefaultBase
}
if err := u.Upgrade(r.Context(), r.PathValue("id"), req.Version, base); err != nil {
writeError(w, http.StatusBadGateway, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"node": r.PathValue("id"), "version": req.Version})
}
// SetServe supplies what serves a model from the registry.
func (a *API) SetServe(s *modelserve.Server) { a.serve = s }
func (a *API) serveModel(w http.ResponseWriter, r *http.Request) {
if a.serve == nil {
writeError(w, http.StatusNotImplemented,
"serving from the registry needs a mirror and a store, and this control plane lacks one")
return
}
var req modelserve.Request
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
req.By = r.Header.Get("X-Llmgw-Actor")
job, err := a.serve.Start(req)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusAccepted, job)
}
func (a *API) job(w http.ResponseWriter, r *http.Request) {
if a.serve == nil {
writeError(w, http.StatusNotFound, "no such job")
return
}
job, ok := a.serve.Get(r.PathValue("id"))
if !ok {
writeError(w, http.StatusNotFound, "no such job; jobs are kept only until the control plane restarts")
return
}
writeJSON(w, http.StatusOK, job)
}
@@ -264,3 +264,22 @@ func TestAGPUFallingOutOfItsDomainTakesTheNodeOut(t *testing.T) {
t.Errorf("imex = %+v", n.IMEX)
}
}
func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
s := newTestService()
c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
s.conns["n1"] = c
done := make(chan error, 1)
go func() { done <- s.Upgrade(t.Context(), "n1", "v1.2.3", "https://r") }()
req := (<-c.send).GetUpgrade()
if req.GetVersion() != "v1.2.3" || req.GetReleases() != "https://r" {
t.Fatalf("sent %v", req)
}
s.handle("n1", &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: &llmgwv1.UpgradeReply{
RequestId: req.GetRequestId(), Version: "v1.2.3", Error: "checksum mismatch",
}}})
if err := <-done; err == nil || err.Error() != "checksum mismatch" {
t.Errorf("the agent's refusal was not passed on: %v", err)
}
}
+52
View File
@@ -60,6 +60,8 @@ type Service struct {
logMu sync.Mutex
logWaits map[string]chan *llmgwv1.LogsReply
logSeq uint64
// upgradeWaits are the upgrade requests awaiting a reply; guarded by logMu.
upgradeWaits map[string]chan *llmgwv1.UpgradeReply
// scanWaits are the fabric scans awaiting a reply, by request id.
scanWaits map[string]chan *llmgwv1.FabricScanReply
testWaits map[string]chan *llmgwv1.FabricTestReply
@@ -309,6 +311,15 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
ch <- m.FabricScan
}
case *llmgwv1.AgentMessage_Upgrade:
s.logMu.Lock()
ch, ok := s.upgradeWaits[m.Upgrade.GetRequestId()]
delete(s.upgradeWaits, m.Upgrade.GetRequestId())
s.logMu.Unlock()
if ok {
ch <- m.Upgrade
}
case *llmgwv1.AgentMessage_Logs:
s.logMu.Lock()
ch, ok := s.logWaits[m.Logs.GetRequestId()]
@@ -842,3 +853,44 @@ func installable(reg *llmgwv1.Register) []string {
// The admin API finds the bandwidth test by asserting this; a signature that
// drifted would make it quietly nil.
var _ fabrictest.Asker = (*Service)(nil)
// Upgrade tells a node's agent to replace its binary with a release and
// restart, and returns once the agent says the new binary is in place — or
// why it is not. The agent then exits and comes back on the new version; the
// caller watches for that in the inventory.
func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string) error {
s.logMu.Lock()
if s.upgradeWaits == nil {
s.upgradeWaits = map[string]chan *llmgwv1.UpgradeReply{}
}
s.logSeq++
id := fmt.Sprintf("upgrade-%d", s.logSeq)
ch := make(chan *llmgwv1.UpgradeReply, 1)
s.upgradeWaits[id] = ch
s.logMu.Unlock()
defer func() {
s.logMu.Lock()
delete(s.upgradeWaits, id)
s.logMu.Unlock()
}()
if err := s.send(nodeID, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Upgrade{
Upgrade: &llmgwv1.UpgradeAgent{RequestId: id, Version: version, Releases: releases},
}}); err != nil {
return err
}
s.log.Info("told a node to upgrade", "node", nodeID, "version", version)
// A download of some tens of megabytes on a slow link, and no longer: an
// agent too old to know this message never answers.
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
defer cancel()
select {
case r := <-ch:
if r.GetError() != "" {
return errors.New(r.GetError())
}
return nil
case <-ctx.Done():
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
}
}
+11
View File
@@ -211,6 +211,17 @@ func (c *Controller) WeightRemover() WeightRemover {
return r
}
// Upgrader can tell a node's agent to upgrade itself.
type Upgrader interface {
Upgrade(ctx context.Context, nodeID, version, releases string) error
}
// Upgrader returns the dispatcher's upgrade capability, or nil.
func (c *Controller) Upgrader() Upgrader {
u, _ := c.disp.(Upgrader)
return u
}
// LogReader can ask a node for the recent lines of one of its logs.
type LogReader interface {
Logs(ctx context.Context, nodeID, source string, lines int) (out, sources []string, err error)
+552
View File
@@ -0,0 +1,552 @@
// Package modelserve takes a model from a registry repository to a replica
// answering requests, as one job.
//
// Adding a model was four commands and a hand-written YAML file: plan it,
// fetch it into the mirror, write its catalog entry with a memory figure
// somebody guessed, scale it — and then find out whether it loads. This runs
// those steps in the control plane, where the mirror, the registry, the
// catalog and the fleet already are, and reports each one as it goes, so
// whatever asks — ctl, a page — only has to watch.
package modelserve
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"regexp"
"slices"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
)
// Registry reads a repository.
type Registry interface {
Repo(ctx context.Context, id string) (hf.Repo, error)
}
// Mirror fetches files and reports on them.
type Mirror interface {
Fetch(ctx context.Context, req mirror.FetchRequest) (mirror.FetchResponse, error)
List(ctx context.Context) (mirror.ModelsResponse, error)
}
// Catalog adds a model.
type Catalog interface {
Add(ctx context.Context, m catalog.Model, by string) error
}
// Fleet is what the job watches once the model is in the catalog.
type Fleet interface {
Replicas() []reconciler.Replica
ModelHealth() []reconciler.ModelHealth
EngineKey(replicaID string) string
}
// Deps are what a job needs.
type Deps struct {
Registry Registry
Mirror Mirror
Catalog Catalog
Fleet Fleet
// LlamaVersion is the llama.cpp build the nodes install, which decides
// which architectures load.
LlamaVersion string
HTTP *http.Client
Log *slog.Logger
// Poll is how often the job looks at the mirror and the fleet.
Poll time.Duration
// ReadyTimeout bounds the wait for a replica: the node fetching the
// weights from the mirror and the engine loading them.
ReadyTimeout time.Duration
}
// Request is what to serve.
type Request struct {
// Repo is the registry repository, owner/name.
Repo string `json:"repo"`
// Quant picks the file by its quantisation, Q4_K_M unless given.
Quant string `json:"quant,omitempty"`
// File picks it by name instead.
File string `json:"file,omitempty"`
// Name is the catalog name; the repository suggests one otherwise.
Name string `json:"name,omitempty"`
Aliases []string `json:"aliases,omitempty"`
Description string `json:"description,omitempty"`
// Replicas to run; 0 makes it on demand, loaded when first asked for.
Replicas int `json:"replicas"`
// CtxPool is llama.cpp's --ctx-size; ContextLength what one request may
// use. They default to 16384 and 4096, what four parallel slots divide.
CtxPool int `json:"ctx_pool,omitempty"`
ContextLength int `json:"context_length,omitempty"`
// By is who asked, for the catalog's record.
By string `json:"-"`
}
// The states a job and its steps can be in.
const (
Running = "running"
Done = "done"
Failed = "failed"
Skipped = "skipped"
)
// Step is one stage of a job.
type Step struct {
Name string `json:"name"`
State string `json:"state"`
Detail string `json:"detail,omitempty"`
At time.Time `json:"at"`
}
// Job is one model being served, and where it stands.
type Job struct {
ID string `json:"id"`
Request Request `json:"request"`
State string `json:"state"`
Error string `json:"error,omitempty"`
Model string `json:"model,omitempty"`
Steps []Step `json:"steps"`
Started time.Time `json:"started"`
// Reply is what the model said to the smoke test, and TokensPerSecond
// how fast it said it.
Reply string `json:"reply,omitempty"`
TokensPerSecond float64 `json:"tokens_per_second,omitempty"`
}
// Server runs jobs and remembers them.
type Server struct {
deps Deps
mu sync.Mutex
jobs map[string]*Job
seq int
}
// New builds a server.
func New(d Deps) *Server {
if d.Poll <= 0 {
d.Poll = 2 * time.Second
}
if d.ReadyTimeout <= 0 {
d.ReadyTimeout = 30 * time.Minute
}
if d.HTTP == nil {
d.HTTP = &http.Client{Timeout: 5 * time.Minute}
}
if d.Log == nil {
d.Log = slog.Default()
}
return &Server{deps: d, jobs: map[string]*Job{}}
}
var repoPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$`)
// Start begins a job and returns at once; the job runs on its own.
func (s *Server) Start(req Request) (Job, error) {
req.Repo = strings.Trim(strings.TrimSpace(req.Repo), "/")
if !repoPattern.MatchString(req.Repo) {
return Job{}, fmt.Errorf("%q is not a repository; it looks like owner/name", req.Repo)
}
if req.Replicas < 0 {
return Job{}, errors.New("replicas cannot be negative")
}
if req.Quant == "" && req.File == "" {
req.Quant = "Q4_K_M"
}
if req.CtxPool <= 0 {
req.CtxPool = 16384
}
if req.ContextLength <= 0 {
req.ContextLength = 4096
}
s.mu.Lock()
s.seq++
j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
State: Running, Started: time.Now()}
s.jobs[j.ID] = j
s.mu.Unlock()
go s.run(j)
return s.snapshot(j), nil
}
// Get returns a job as it stands.
func (s *Server) Get(id string) (Job, bool) {
s.mu.Lock()
j, ok := s.jobs[id]
s.mu.Unlock()
if !ok {
return Job{}, false
}
return s.snapshot(j), true
}
func (s *Server) snapshot(j *Job) Job {
s.mu.Lock()
defer s.mu.Unlock()
out := *j
out.Steps = slices.Clone(j.Steps)
return out
}
// step starts a new step, or updates the detail of the current one.
func (s *Server) step(j *Job, name, state, detail string) {
s.mu.Lock()
defer s.mu.Unlock()
if n := len(j.Steps); n > 0 && j.Steps[n-1].Name == name {
j.Steps[n-1].State, j.Steps[n-1].Detail, j.Steps[n-1].At = state, detail, time.Now()
return
}
j.Steps = append(j.Steps, Step{Name: name, State: state, Detail: detail, At: time.Now()})
}
func (s *Server) fail(j *Job, name string, err error) {
s.step(j, name, Failed, err.Error())
s.mu.Lock()
j.State, j.Error = Failed, err.Error()
s.mu.Unlock()
s.deps.Log.Warn("serving a model failed", "job", j.ID, "repo", j.Request.Repo, "step", name, "err", err)
}
func (s *Server) run(j *Job) {
ctx := context.Background()
req := j.Request
// 1. The registry: which files.
s.step(j, "registry", Running, "reading "+req.Repo)
repo, err := s.deps.Registry.Repo(ctx, req.Repo)
if err != nil {
s.fail(j, "registry", err)
return
}
if repo.Gated.Is() {
s.fail(j, "registry", fmt.Errorf("%s is gated: it needs an account's approval the mirror does not have", req.Repo))
return
}
set, err := pick(repo, req)
if err != nil {
s.fail(j, "registry", err)
return
}
name := req.Name
if name == "" {
name = hf.ModelName(repo.ID)
}
s.mu.Lock()
j.Model = name
s.mu.Unlock()
s.step(j, "registry", Done, fmt.Sprintf("%s, %.1f GB in %d file(s), revision %s",
set.Name, gb(set.Bytes), len(set.Files), repo.ShortRevision()))
// 2. The mirror: fetch, and check what arrived against what was published.
source, err := s.mirror(ctx, j, repo, set, name)
if err != nil {
s.fail(j, "mirror", err)
return
}
// 3. Whether the nodes' llama.cpp loads it, and how much memory it needs.
s.step(j, "check", Running, "reading its header from the mirror")
var header gguf.Header
if !strings.HasSuffix(source, "/") {
v := gguf.Check(ctx, source, s.deps.LlamaVersion)
if !v.Servable {
s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run this: %s", s.deps.LlamaVersion, v.Reason))
return
}
header = v.Header
}
memGB := int((header.MemoryBytes(set.Bytes, req.CtxPool) + 1<<30 - 1) >> 30)
basis := fmt.Sprintf("weights %.1f GB + KV cache %.1f GB for %d tokens + engine",
gb(set.Bytes), gb(int64(req.CtxPool)*header.KVBytesPerToken()), req.CtxPool)
if memGB == 0 {
// A set of shards, or a header that did not say: weights plus half
// again plus two, the rough figure from before shapes were read.
memGB = int(gb(set.Bytes)*1.5) + 2
basis = "weights plus half again plus two; the header gave no shape"
}
s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
// 4. The catalog.
m := catalog.Model{
Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
Profiles: []catalog.Profile{{
Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB,
}},
Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
}
s.step(j, "catalog", Running, "adding "+name)
if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil {
s.fail(j, "catalog", err)
return
}
if req.Replicas == 0 {
s.step(j, "catalog", Done, name+" added on demand: it loads when first asked for")
s.finish(j)
return
}
s.step(j, "catalog", Done, fmt.Sprintf("%s added, %d replica(s)", name, req.Replicas))
// 5. A replica, ready.
rep, err := s.waitReady(ctx, j, name)
if err != nil {
s.fail(j, "placement", err)
return
}
// 6. Does it answer.
s.step(j, "smoke test", Running, "asking "+rep.ID+" to say hello")
reply, tps, err := s.smoke(ctx, rep, name)
if err != nil {
s.fail(j, "smoke test", err)
return
}
s.mu.Lock()
j.Reply, j.TokensPerSecond = reply, tps
s.mu.Unlock()
s.step(j, "smoke test", Done, fmt.Sprintf("answered at %.1f tokens/s: %q", tps, reply))
s.finish(j)
}
func (s *Server) finish(j *Job) {
s.mu.Lock()
j.State = Done
s.mu.Unlock()
s.deps.Log.Info("model served", "job", j.ID, "repo", j.Request.Repo, "model", j.Model)
}
// pick chooses the file set to serve.
func pick(repo hf.Repo, req Request) (hf.FileSet, error) {
var weights []hf.File
for _, f := range repo.Files {
lower := strings.ToLower(f.Path)
if !strings.HasSuffix(lower, ".gguf") || strings.Contains(lower, "mmproj") ||
strings.Contains(lower, "imatrix") {
continue
}
weights = append(weights, f)
}
sets := hf.Group(weights)
if len(sets) == 0 {
return hf.FileSet{}, fmt.Errorf("%s has no GGUF files; the nodes run llama.cpp, which needs one", repo.ID)
}
var names []string
for _, st := range sets {
names = append(names, st.Name)
if req.File != "" && (st.Name == req.File || slices.ContainsFunc(st.Files, func(f hf.File) bool { return f.Path == req.File })) {
return st, nil
}
}
if req.File == "" {
want := strings.ToLower(req.Quant)
var match []hf.FileSet
for _, st := range sets {
if strings.Contains(strings.ToLower(st.Name), want) {
match = append(match, st)
}
}
if len(match) == 1 {
return match[0], nil
}
if len(match) > 1 {
// Both a split and a joined copy of one quantisation: the joined
// one, which is one download to verify rather than several.
slices.SortFunc(match, func(a, b hf.FileSet) int { return a.Shards - b.Shards })
return match[0], nil
}
}
return hf.FileSet{}, fmt.Errorf("%s has no file matching %q; it has: %s",
repo.ID, or(req.File, req.Quant), strings.Join(names, ", "))
}
// mirror fetches the set into the mirror and returns the source a catalog
// entry names: the file, or the directory for a set of shards.
func (s *Server) mirror(ctx context.Context, j *Job, repo hf.Repo, set hf.FileSet, name string) (string, error) {
version := repo.ShortRevision()
s.step(j, "mirror", Running, "asking the mirror for "+set.Name)
var base string
for _, f := range set.Files {
file := mirror.LastPathElement(f.Path)
resp, err := s.deps.Mirror.Fetch(ctx, mirror.FetchRequest{
Source: repo.DownloadURL("", f.Path), Model: name, Version: version, File: file,
})
// Already there is not a failure: the same commit fetched before.
if err != nil && !strings.Contains(err.Error(), "already") {
return "", err
}
base = resp.URL
}
deadline := time.Now().Add(6 * time.Hour)
for {
got, err := s.deps.Mirror.List(ctx)
if err != nil {
return "", err
}
var done, total int64
present, failed := 0, ""
sums := map[string]string{}
for _, st := range got.Models {
if st.Model != name || st.Version != version {
continue
}
done += st.BytesDone
total += max(st.BytesTotal, st.BytesDone)
switch st.State {
case mirror.Present:
present++
sums[st.File] = st.SHA256
case mirror.Failed:
failed = st.Err
}
}
if failed != "" {
return "", fmt.Errorf("the mirror could not fetch it: %s", failed)
}
if present >= len(set.Files) {
// What arrived against what the registry published: a mirror
// copies one file to every node, and is the one place a corrupted
// download is caught once instead of loaded everywhere.
for _, f := range set.Files {
file := mirror.LastPathElement(f.Path)
if f.SHA256 != "" && sums[file] != "" && !strings.EqualFold(f.SHA256, sums[file]) {
return "", fmt.Errorf("%s arrived with sha256 %s, and the registry published %s",
file, sums[file][:12], f.SHA256[:12])
}
}
source := base
if len(set.Files) > 1 {
source = strings.TrimSuffix(source, mirror.LastPathElement(source))
}
s.step(j, "mirror", Done, fmt.Sprintf("%.1f GB on the mirror, checksums match the registry", gb(total)))
return source, nil
}
if total > 0 {
s.step(j, "mirror", Running, fmt.Sprintf("fetching: %.1f of %.1f GB (%d%%)",
gb(done), gb(total), done*100/total))
}
if time.Now().After(deadline) {
return "", errors.New("the mirror had not finished after six hours")
}
time.Sleep(s.deps.Poll)
}
}
// waitReady waits for a replica of the model to serve, reporting what it is
// doing and giving up early when it cannot be placed or keeps failing.
func (s *Server) waitReady(ctx context.Context, j *Job, name string) (reconciler.Replica, error) {
s.step(j, "placement", Running, "waiting for the control plane to place it")
started := time.Now()
deadline := started.Add(s.deps.ReadyTimeout)
for {
for _, h := range s.deps.Fleet.ModelHealth() {
if h.Model != name {
continue
}
if h.Parked {
return reconciler.Replica{}, fmt.Errorf("it failed to start %d times and is parked: %s", h.Failures, h.Reason)
}
// A minute's grace from when this step began, not from the job:
// the download before it can take hours, and the first pass may
// not have run yet.
if h.Unplaceable && time.Since(started) > time.Minute {
return reconciler.Replica{}, fmt.Errorf("no node can host it:\n%s", h.Placement)
}
}
for _, r := range s.deps.Fleet.Replicas() {
if r.Model != name {
continue
}
if r.State == reconciler.Ready {
s.step(j, "placement", Done, fmt.Sprintf("%s ready on %s at %s", r.ID,
strings.Join(r.NodeIDs(), ","), r.Endpoint))
return r, nil
}
s.step(j, "placement", Running, fmt.Sprintf("%s on %s is %s", r.ID,
strings.Join(r.NodeIDs(), ","), r.State))
}
if time.Now().After(deadline) {
return reconciler.Replica{}, fmt.Errorf("no replica was ready after %s", s.deps.ReadyTimeout)
}
select {
case <-ctx.Done():
return reconciler.Replica{}, ctx.Err()
case <-time.After(s.deps.Poll):
}
}
}
// smoke sends one short request straight to the replica, with its engine key.
func (s *Server) smoke(ctx context.Context, r reconciler.Replica, name string) (string, float64, error) {
body, _ := json.Marshal(map[string]any{
"model": name,
"max_tokens": 32,
"messages": []map[string]string{{"role": "user", "content": "Say hello in one short sentence."}},
})
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
strings.TrimRight(r.Endpoint, "/")+"/v1/chat/completions", bytes.NewReader(body))
if err != nil {
return "", 0, err
}
req.Header.Set("Content-Type", "application/json")
if k := s.deps.Fleet.EngineKey(r.ID); k != "" {
req.Header.Set("Authorization", "Bearer "+k)
}
start := time.Now()
resp, err := s.deps.HTTP.Do(req)
if err != nil {
return "", 0, err
}
defer resp.Body.Close()
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if resp.StatusCode != http.StatusOK {
return "", 0, fmt.Errorf("the replica answered %s: %s", resp.Status, strings.TrimSpace(string(b)))
}
var out struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
Usage struct {
CompletionTokens int `json:"completion_tokens"`
} `json:"usage"`
}
if err := json.Unmarshal(b, &out); err != nil || len(out.Choices) == 0 {
return "", 0, fmt.Errorf("the replica's answer was not a completion: %s", strings.TrimSpace(string(b)))
}
secs := time.Since(start).Seconds()
tps := 0.0
if secs > 0 {
tps = float64(out.Usage.CompletionTokens) / secs
}
return strings.TrimSpace(out.Choices[0].Message.Content), tps, nil
}
func gb(b int64) float64 { return float64(b) / (1 << 30) }
func or(a, b string) string {
if a != "" {
return a
}
return b
}
func orUnknown(arch string) string {
if arch == "" {
return "architecture not read"
}
return arch
}
@@ -0,0 +1,225 @@
package modelserve
import (
"bytes"
"context"
"encoding/binary"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
)
// ggufHeader is enough of a GGUF file for its architecture and shape.
func ggufHeader() []byte {
var b bytes.Buffer
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
u32 := func(k string, v uint32) { str(k); w(uint32(4)); w(v) }
b.WriteString("GGUF")
w(uint32(3))
w(uint64(0))
w(uint64(5))
str("general.architecture")
w(uint32(8))
str("qwen2")
u32("qwen2.block_count", 28)
u32("qwen2.attention.head_count", 12)
u32("qwen2.attention.head_count_kv", 2)
u32("qwen2.embedding_length", 1536)
return b.Bytes()
}
type fakeRegistry struct{ repo hf.Repo }
func (f fakeRegistry) Repo(context.Context, string) (hf.Repo, error) { return f.repo, nil }
// fakeMirror serves the header at every file URL and reports each fetched
// file present, with the checksum it was told to.
type fakeMirror struct {
mu sync.Mutex
srv *httptest.Server
got []mirror.FetchRequest
sums map[string]string
}
func (m *fakeMirror) Fetch(_ context.Context, r mirror.FetchRequest) (mirror.FetchResponse, error) {
m.mu.Lock()
defer m.mu.Unlock()
m.got = append(m.got, r)
return mirror.FetchResponse{URL: m.srv.URL + "/" + r.Model + "/" + r.Version + "/" + r.File}, nil
}
func (m *fakeMirror) List(context.Context) (mirror.ModelsResponse, error) {
m.mu.Lock()
defer m.mu.Unlock()
var out mirror.ModelsResponse
for _, r := range m.got {
out.Models = append(out.Models, mirror.Status{Model: r.Model, Version: r.Version, File: r.File,
State: mirror.Present, BytesDone: 1 << 30, BytesTotal: 1 << 30, SHA256: m.sums[r.File]})
}
return out, nil
}
type fakeCatalog struct {
mu sync.Mutex
added []catalog.Model
}
func (c *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
c.mu.Lock()
defer c.mu.Unlock()
c.added = append(c.added, m)
return nil
}
// fakeFleet has a replica ready at engine once the catalog holds the model.
type fakeFleet struct {
cat *fakeCatalog
engine string
health []reconciler.ModelHealth
}
func (f *fakeFleet) Replicas() []reconciler.Replica {
f.cat.mu.Lock()
defer f.cat.mu.Unlock()
var out []reconciler.Replica
for _, m := range f.cat.added {
out = append(out, reconciler.Replica{ID: m.Name + "-r1", Model: m.Name,
State: reconciler.Ready, Endpoint: f.engine,
Shards: []reconciler.Shard{{NodeID: "node01"}}})
}
return out
}
func (f *fakeFleet) ModelHealth() []reconciler.ModelHealth { return f.health }
func (f *fakeFleet) EngineKey(string) string { return "k" }
func setup(t *testing.T, files []hf.File, sums map[string]string) (*Server, *fakeMirror, *fakeCatalog, *fakeFleet) {
t.Helper()
header := ggufHeader()
files0 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write(header)
}))
t.Cleanup(files0.Close)
engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer k" {
w.WriteHeader(http.StatusUnauthorized)
return
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"Hello!"}}],"usage":{"completion_tokens":3}}`))
}))
t.Cleanup(engine.Close)
m := &fakeMirror{srv: files0, sums: sums}
c := &fakeCatalog{}
f := &fakeFleet{cat: c, engine: engine.URL}
s := New(Deps{
Registry: fakeRegistry{repo: hf.Repo{ID: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Revision: "abcdef123456", Files: files}},
Mirror: m, Catalog: c, Fleet: f, Poll: 10 * time.Millisecond, ReadyTimeout: 5 * time.Second,
})
return s, m, c, f
}
func wait(t *testing.T, s *Server, id string) Job {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
for time.Now().Before(deadline) {
if j, _ := s.Get(id); j.State != Running {
return j
}
time.Sleep(10 * time.Millisecond)
}
t.Fatal("the job never finished")
return Job{}
}
func TestAModelGoesFromTheRegistryToAnAnsweringReplica(t *testing.T) {
files := []hf.File{
{Path: "Qwen2.5-1.5B-Instruct-Q8_0.gguf", Bytes: 2 << 30, SHA256: "aa"},
{Path: "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "bb"},
{Path: "mmproj-Q4_K_M.gguf", Bytes: 1 << 20},
}
s, m, c, _ := setup(t, files, map[string]string{"Qwen2.5-1.5B-Instruct-Q4_K_M.gguf": "BB"})
job, err := s.Start(Request{Repo: "bartowski/Qwen2.5-1.5B-Instruct-GGUF", Replicas: 1})
if err != nil {
t.Fatal(err)
}
j := wait(t, s, job.ID)
if j.State != Done {
t.Fatalf("job %s: %s\n%+v", j.State, j.Error, j.Steps)
}
if len(m.got) != 1 || m.got[0].File != "Qwen2.5-1.5B-Instruct-Q4_K_M.gguf" || m.got[0].Version != "abcdef12" {
t.Errorf("fetched %+v; expected the Q4_K_M file at its revision", m.got)
}
mod := c.added[0]
if mod.Name != "qwen2.5-1.5b-instruct" || mod.Profiles[0].MinMemoryGB == 0 || mod.Scaling.Min != 1 {
t.Errorf("catalog entry %+v", mod)
}
if j.Reply != "Hello!" {
t.Errorf("the smoke test's answer was %q", j.Reply)
}
var names []string
for _, st := range j.Steps {
names = append(names, st.Name)
}
if strings.Join(names, ",") != "registry,mirror,check,catalog,placement,smoke test" {
t.Errorf("steps %v", names)
}
}
func TestAQuantTheRepositoryDoesNotHaveIsNamedWithWhatItHas(t *testing.T) {
s, _, c, _ := setup(t, []hf.File{{Path: "m-Q8_0.gguf", Bytes: 1}}, nil)
job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
j := wait(t, s, job.ID)
if j.State != Failed || !strings.Contains(j.Error, "m-Q8_0.gguf") {
t.Errorf("%s: %s", j.State, j.Error)
}
if len(c.added) != 0 {
t.Error("something was added to the catalog")
}
}
func TestADownloadThatDoesNotMatchTheRegistryIsNotServed(t *testing.T) {
files := []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30, SHA256: "aaaaaaaaaaaaaaaa"}}
s, _, c, _ := setup(t, files, map[string]string{"m-Q4_K_M.gguf": "bbbbbbbbbbbbbbbb"})
job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
j := wait(t, s, job.ID)
if j.State != Failed || !strings.Contains(j.Error, "registry published") {
t.Errorf("%s: %s", j.State, j.Error)
}
if len(c.added) != 0 {
t.Error("a corrupted download reached the catalog")
}
}
func TestAParkedModelStopsTheWait(t *testing.T) {
s, _, _, f := setup(t, []hf.File{{Path: "m-Q4_K_M.gguf", Bytes: 1 << 30}}, nil)
f.health = []reconciler.ModelHealth{{Model: "qwen2.5-1.5b-instruct", Parked: true, Failures: 5, Reason: "unknown model architecture"}}
// A fleet where it never becomes ready, only parked.
s.deps.Fleet = parkedFleet{f}
job, _ := s.Start(Request{Repo: "a/b", Replicas: 1})
j := wait(t, s, job.ID)
if j.State != Failed || !strings.Contains(j.Error, "parked") {
t.Errorf("%s: %s", j.State, j.Error)
}
}
type parkedFleet struct{ *fakeFleet }
func (parkedFleet) Replicas() []reconciler.Replica { return nil }
func TestARepositoryThatIsNotOneIsRefused(t *testing.T) {
s := New(Deps{})
for _, r := range []string{"", "noslash", "../etc/passwd", "a/b/c"} {
if _, err := s.Start(Request{Repo: r}); err == nil {
t.Errorf("%q accepted", r)
}
}
}
+13
View File
@@ -558,3 +558,16 @@ func Group(files []File) []FileSet {
}
return out
}
// ModelName is the catalog name a repository suggests: its last element,
// lowercased, with the -GGUF marker dropped. The owner is not part of what
// anybody calls the model, and "-GGUF" describes the file format.
func ModelName(repoID string) string {
name := repoID
if i := strings.LastIndex(name, "/"); i >= 0 {
name = name[i+1:]
}
name = strings.TrimSuffix(name, "-GGUF")
name = strings.TrimSuffix(name, "-gguf")
return strings.ToLower(name)
}
+2 -1
View File
@@ -18,6 +18,7 @@ import (
"encoding/pem"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"net"
"os"
"path/filepath"
@@ -335,7 +336,7 @@ func certNames(path string) ([]string, error) {
// releaseTag is a version CI published: v0.10.4, or a release candidate of
// it, v0.10.4-rc.2. Anything else — a build with commits past its tag — has no
// release to download.
var releaseTag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)
var releaseTag = release.Tag
func (r *Runner) binary(ctx context.Context, res *Result) error {
v := r.Opts.Version
+148
View File
@@ -0,0 +1,148 @@
// Package release fetches and installs llmgw's own release binaries.
//
// One implementation for everywhere a binary is replaced — the control plane
// upgrading itself, and an agent told to upgrade by it — so the checks are
// the same on both: the release's own checksum is verified before anything is
// installed, the old binary is kept beside the new one, and the swap is a
// rename, so a machine is never left without a binary that runs.
package release
import (
"bufio"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"regexp"
"strings"
"time"
)
// DefaultBase is where CI publishes releases.
const DefaultBase = "https://gitea.mynodes.xyz/ilker/llmgw/releases/download"
// Tag is a version CI published: v0.10.4, or a release candidate v0.10.4-rc.2.
var Tag = regexp.MustCompile(`^v\d+\.\d+\.\d+(-rc\.\d+)?$`)
// AssetName is the release file for a platform.
func AssetName(version, goos, goarch string) string {
return "llmgw-" + version + "-" + goos + "-" + goarch
}
// Install fetches version for this platform from base, checks it against the
// release's checksums, and puts it at target, keeping the old one as
// target.bak-pre-<version>. It returns the backup's path.
func Install(ctx context.Context, client *http.Client, base, version, goos, goarch, target string) (string, error) {
if !Tag.MatchString(version) {
return "", fmt.Errorf("%q is not a release version", version)
}
if client == nil {
client = &http.Client{Timeout: 10 * time.Minute}
}
target, err := filepath.EvalSymlinks(target)
if err != nil {
return "", err
}
dir := filepath.Dir(target)
asset := AssetName(version, goos, goarch)
root := strings.TrimSuffix(base, "/") + "/" + version + "/"
want, err := checksum(ctx, client, root+"checksums.txt", asset)
if err != nil {
return "", err
}
// Beside the target, so the final rename is on one filesystem and atomic.
tmp, err := os.CreateTemp(dir, ".llmgw-"+version+"-*")
if err != nil {
return "", err
}
defer os.Remove(tmp.Name())
got, err := download(ctx, client, root+asset, tmp)
if cerr := tmp.Close(); err == nil {
err = cerr
}
if err != nil {
return "", err
}
if got != want {
return "", fmt.Errorf("%s does not match the release's checksum (got %s, expected %s)",
asset, got[:12], want[:12])
}
if err := os.Chmod(tmp.Name(), 0o755); err != nil {
return "", err
}
backup := target + ".bak-pre-" + version
if err := copyFile(target, backup); err != nil {
return "", fmt.Errorf("keeping the old binary: %w", err)
}
if err := os.Rename(tmp.Name(), target); err != nil {
return "", err
}
return backup, nil
}
func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
body, err := get(ctx, client, url)
if err != nil {
return "", err
}
defer body.Close()
sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
for sc.Scan() {
f := strings.Fields(sc.Text())
if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
return f[0], nil
}
}
return "", fmt.Errorf("the release's checksums list no %s", asset)
}
func download(ctx context.Context, client *http.Client, url string, dst io.Writer) (string, error) {
body, err := get(ctx, client, url)
if err != nil {
return "", err
}
defer body.Close()
h := sha256.New()
if _, err := io.Copy(io.MultiWriter(dst, h), body); err != nil {
return "", fmt.Errorf("downloading %s: %w", url, err)
}
return hex.EncodeToString(h.Sum(nil)), nil
}
func get(ctx context.Context, client *http.Client, url string) (io.ReadCloser, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("fetching %s: %w", url, err)
}
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
return nil, fmt.Errorf("fetching %s: %s", url, resp.Status)
}
return resp.Body, nil
}
func copyFile(src, dst string) error {
in, err := os.Open(src) //nolint:gosec // our own binary
if err != nil {
return err
}
defer in.Close()
out, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) //nolint:gosec // beside it
if err != nil {
return err
}
if _, err := io.Copy(out, in); err != nil {
out.Close()
return err
}
return out.Close()
}
+61
View File
@@ -0,0 +1,61 @@
package release
import (
"crypto/sha256"
"encoding/hex"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
)
func releaseServer(t *testing.T, version, asset string, body []byte, sum string) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/" + version + "/checksums.txt":
_, _ = w.Write([]byte(sum + " " + asset + "\n"))
case "/" + version + "/" + asset:
_, _ = w.Write(body)
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
return srv
}
func TestABinaryIsReplacedOnlyWithWhatTheReleasePublished(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "llmgw")
_ = os.WriteFile(target, []byte("old"), 0o755)
body := []byte("new binary")
sum := sha256.Sum256(body)
asset := AssetName("v1.2.3", "linux", "amd64")
srv := releaseServer(t, "v1.2.3", asset, body, hex.EncodeToString(sum[:]))
backup, err := Install(t.Context(), srv.Client(), srv.URL, "v1.2.3", "linux", "amd64", target)
if err != nil {
t.Fatal(err)
}
if b, _ := os.ReadFile(target); string(b) != "new binary" {
t.Errorf("target holds %q", b)
}
if b, _ := os.ReadFile(backup); string(b) != "old" {
t.Errorf("the old binary was not kept: %q", b)
}
// A download that does not match its checksum changes nothing.
bad := releaseServer(t, "v1.2.4", AssetName("v1.2.4", "linux", "amd64"), []byte("tampered"),
hex.EncodeToString(sum[:]))
if _, err := Install(t.Context(), bad.Client(), bad.URL, "v1.2.4", "linux", "amd64", target); err == nil {
t.Fatal("a binary not matching its checksum was installed")
}
if b, _ := os.ReadFile(target); string(b) != "new binary" {
t.Errorf("a failed install touched the binary: %q", b)
}
if _, err := Install(t.Context(), nil, srv.URL, "../../x", "linux", "amd64", target); err == nil {
t.Error("a version that is not one was fetched")
}
}
+1 -6
View File
@@ -621,12 +621,7 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
// is not part of what anybody calls the model, and "-GGUF" describes the file
// format rather than the model, which is already clear from the file being
// fetched.
func defaultModelName(repoID string) string {
name := mirror.LastPathElement(repoID)
name = strings.TrimSuffix(name, "-GGUF")
name = strings.TrimSuffix(name, "-gguf")
return strings.ToLower(name)
}
func defaultModelName(repoID string) string { return hf.ModelName(repoID) }
// verdictFor judges a registry file set by the repository's architecture. A
// vision projector is its own architecture whatever the repository says.