Ilker ManapandClaude Opus 5.5 141a1f46c5 Slurm: a new leader keeps the nodes it has not heard from yet ...

Tried on the PoC: with a job on a lent node, the leader restarted and its
first slurm.conf had no node in it (none had reported to it yet), so
slurmctld dropped the node and the job ended in NODE_FAIL.

The leader keeps slurm.conf's nodes and the controller's address in the
store's cluster state (slurm-nodes); the next one starts from them and
writes the same files until the nodes report again. With the store
unread, the controller gets no files that leave a node out. A node is
told nothing, rather than that it has no part, until its controller's
address is known: no part stops slurmd, and its jobs with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-11 22:33:53 +01:00

..

admintoken_test.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

admintoken.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

agent.go

The nodes' agent settings kept with the cluster's, agent.yaml down to what an agent needs to start

2026-10-08 18:06:36 +02:00

answercache_test.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

answercache.go

Answers to similar questions, not only the same one

2026-10-25 17:47:11 +01:00

audit.go

Audit log: every change through the admin API and the administration pages, with ctl audit

2026-10-04 16:25:56 +02:00

backup_test.go

Backups reach a replicated store: pg_dump is given the cluster's authority with verify-full

2026-10-08 21:04:46 +02:00

backup.go

Backups reach a replicated store: pg_dump is given the cluster's authority with verify-full

2026-10-08 21:04:46 +02:00

backupcopy.go

server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine

2026-10-04 23:09:59 +02:00

bench_overhead_test.go

What llmgw adds to a request, measured: ctl bench overhead

2026-10-17 00:23:26 +02:00

bench_overhead.go

What llmgw adds to a request, measured: ctl bench overhead

2026-10-17 00:23:26 +02:00

bench_test.go

ctl bench reads a guarded engine's metrics with the engine's key, or says why it cannot

2026-10-05 00:20:25 +02:00

bench.go

What llmgw adds to a request, measured: ctl bench overhead

2026-10-17 00:23:26 +02:00

bundle.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

capacity.go

Keep what a benchmark found out, instead of leaving it in a terminal

2026-09-27 11:51:58 +02:00

cert.go

Add a private CA and mutual TLS for the agent stream

2026-09-16 07:28:56 +02:00

clustersettings_test.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

clustersettings.go

Slurm beside llmgw, P3 and P4: the leader hands nodes over; ctl slurm

2026-11-11 12:33:14 +01:00

comfyuigate.go

ComfyUI as an engine of the cluster's own

2026-10-10 00:39:10 +02:00

controlplanes_test.go

The probe reads the floating address from its file each time

2026-10-05 02:56:23 +02:00

controlplanes.go

The probe reads the floating address from its file each time

2026-10-05 02:56:23 +02:00

ctl_3d.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

ctl_config_test.go

The guardrails' judge has its own command, and its trouble is shown

2026-10-06 10:47:37 +02:00

ctl_config.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

ctl_eval_test.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

ctl_eval.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

ctl_fabrictest.go

Measure two nodes' fabric rail by rail: llmgw ctl fabric-test

2026-09-29 21:29:23 +02:00

ctl_logs.go

Show a node's logs on the admin page

2026-09-28 12:27:47 +02:00

ctl_mcp.go

The company's MCP tools under the guardrails, a rate and a breaker

2026-10-20 23:00:22 +02:00

ctl_request.go

A ComfyUI model may make meshes: from a picture, with its licence inside

2026-10-10 18:30:52 +02:00

ctl_restart.go

ctl restart --max-surge and --max-unavailable

2026-10-07 13:09:10 +02:00

ctl_slurm_test.go

Slurm beside llmgw, P3 and P4: the leader hands nodes over; ctl slurm

2026-11-11 12:33:14 +01:00

ctl_slurm.go

Slurm in the first install: server init --slurm, and slurm.all_nodes

2026-11-11 14:50:35 +01:00

ctl_status_test.go

Draw the console's overview in its panel, not over the screen

2026-10-03 23:09:40 +02:00

ctl_upgrade_test.go

Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream

2026-10-30 11:14:02 +01:00

ctl_upgrade.go

Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream

2026-10-30 11:14:02 +01:00

ctl.go

Slurm beside llmgw, P3 and P4: the leader hands nodes over; ctl slurm

2026-11-11 12:33:14 +01:00

ctlweights.go

Deleting unused model files says what went, and sets go whole

2026-11-03 10:29:16 +01:00

dispatcherref_test.go

Measure two nodes' fabric rail by rail: llmgw ctl fabric-test

2026-09-29 21:29:23 +02:00

doctor.go

A node's own readiness and its multi-node readiness are kept apart

2026-10-07 13:11:16 +02:00

enginebuild.go

Choose each engine's build from the node's hardware, for every GPU generation

2026-10-01 14:35:58 +02:00

enginecheck_gpu_test.go

An adopted engine's state is written without its command; engine-check finds the installed build

2026-10-07 14:26:28 +02:00

enginecheck_test.go

SGLang read in its own metric names, started serving them, and asked for its own help

2026-10-05 00:20:25 +02:00

enginecheck.go

FreeToken, an engine for models larger than the card

2026-10-31 22:10:05 +01:00

facility_test.go

Facility-aware placement: racks, PDUs, holds and power caps

2026-10-05 01:17:49 +02:00

facility.go

Facility-aware placement: racks, PDUs, holds and power caps

2026-10-05 01:17:49 +02:00

federation.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

filter_test.go

Let ctl ask a question instead of printing the whole fleet

2026-09-27 07:30:51 +02:00

filter.go

Facility-aware placement: racks, PDUs, holds and power caps

2026-10-05 01:17:49 +02:00

firewallcmd_test.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

firewallcmd.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

firewallpub_test.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

firewallpub.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

guard_judge.go

How fast the judge is, measured, and said at install

2026-10-20 22:35:19 +02:00

guard.go

Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask

2026-10-06 12:41:08 +02:00

join.go

Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane

2026-10-05 00:13:39 +02:00

judgekeeper_test.go

The judge init installs is no longer replaced by the leader's own

2026-10-19 20:37:10 +02:00

judgekeeper.go

The judge init installs is no longer replaced by the leader's own

2026-10-19 20:37:10 +02:00

judges_known.go

A judge the leader serves by itself is sized as init's is

2026-10-19 16:32:04 +02:00

kb.go

Training data from a collection: llmgw ctl kb dataset

2026-10-08 20:04:37 +02:00

keys_lifecycle_test.go

Key lifecycle: lifetime limits by scope, rotation with overlap, kinds and expiry in the listing

2026-10-04 16:45:05 +02:00

keys.go

keys limits changes only the limits it is given

2026-10-20 17:32:43 +02:00

leaderforward_test.go

Standbys that answer the client API themselves, behind router.active_active

2026-11-01 13:47:47 +01:00

leaderforward.go

Standbys that answer the client API themselves, behind router.active_active

2026-11-01 13:47:47 +01:00

memory.go

Manage people and the tenancy tree, from the web and the CLI

2026-09-28 16:26:39 +02:00

mig_test.go

MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>

2026-10-05 00:01:14 +02:00

mirror_gate_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

mirror_import_test.go

TensorRT-LLM as an engine, and models from Ollama into the mirror

2026-10-05 00:02:01 +02:00

mirror_import.go

TensorRT-LLM as an engine, and models from Ollama into the mirror

2026-10-05 00:02:01 +02:00

mirror_peer_test.go

mirror add-peer on a mirror machine of its own says what to run on a control plane

2026-10-29 18:30:30 +01:00

mirror_peer.go

mirror add-peer on a mirror machine of its own says what to run on a control plane

2026-10-29 18:30:30 +01:00

mirror_test.go

Hide the mirror's bookkeeping, bound GGUF array nesting, and forget old serve jobs and downloads

2026-10-04 10:56:16 +02:00

mirror.go

Weights from Hugging Face pinned to a commit and checked against the sha256 it published

2026-10-30 07:33:15 +01:00

mirrorinit.go

A second mirror: llmgw mirror add-peer, and on a store machine at setup

2026-10-29 13:43:27 +01:00

model_canary.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

model_presets_test.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

model_presets.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

model_serve.go

CHANGELOG: one list

2026-10-30 10:39:30 +01:00

model_update.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

model.go

Replicas counted per profile: seven on cards and four on CPUs under one name

2026-11-03 09:21:50 +01:00

modelplan.go

How fast a model answers on the CPU, before it is served

2026-11-03 08:52:15 +01:00

movestore.go

Move a store to another one without reissuing every key

2026-09-25 16:28:38 +02:00

node.go

The control plane uses whichever mirror answers

2026-10-29 13:43:27 +01:00

notify.go

Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates

2026-10-04 16:58:11 +02:00

org_test.go

keys limits changes only the limits it is given

2026-10-20 17:32:43 +02:00

org.go

Let a person belong to many places, or to none

2026-09-28 19:03:46 +02:00

quota.go

Set capacity quotas, class, owner and reservation from the admin pages

2026-09-30 22:40:45 +02:00

release.go

Sign releases, and install only signed ones once a key is trusted

2026-09-30 23:03:48 +02:00

requestlog.go

Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster

2026-10-04 23:27:31 +02:00

resolve.go

Manage people and the tenancy tree, from the web and the CLI

2026-09-28 16:26:39 +02:00

restore_test.go

server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine

2026-10-04 23:09:59 +02:00

restore.go

server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine

2026-10-04 23:09:59 +02:00

root.go

Strata runs as an engine: a 125B MoE on one card

2026-10-29 15:09:13 +01:00

selfupgrade.go

An upgrade that leaves nodes behind says which

2026-10-14 09:01:09 +02:00

servable_test.go

Refuse weights the nodes' llama.cpp cannot load, before fetching them

2026-09-28 20:44:36 +02:00

server_agent_test.go

Take agent_tls.insecure as consent to an agent port without TLS too

2026-10-04 10:11:40 +02:00

server_args_test.go

A broken directory setting, or a second server, no longer takes the cluster down

2026-10-19 11:10:43 +02:00

server.go

Slurm: a new leader keeps the nodes it has not heard from yet

2026-11-11 22:33:53 +01:00

serveradd.go

A floating address keeps to its own network, and Caddy follows the leader

2026-10-15 17:35:10 +02:00

serverinit_bundle_test.go

Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane

2026-10-05 00:13:39 +02:00

serverinit_bundle.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

serverinit_choices_test.go

Slurm in the first install: server init --slurm, and slurm.all_nodes

2026-11-11 14:50:35 +01:00

serverinit_choices.go

Slurm in the first install: server init --slurm, and slurm.all_nodes

2026-11-11 14:50:35 +01:00

serverinit_cluster_test.go

A three-machine install can give the judges machines of their own

2026-10-19 17:45:00 +02:00

serverinit_cluster.go

Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream

2026-10-30 11:14:02 +01:00

serverinit_firewall.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

serverinit_judges_test.go

A three-machine install can give the judges machines of their own

2026-10-19 17:45:00 +02:00

serverinit_judges.go

How fast the judge is, measured, and said at install

2026-10-20 22:35:19 +02:00

serverinit_memory_test.go

Fixes from the second review: grants, flags, updates, setup

2026-10-29 12:31:07 +01:00

serverinit_notify_test.go

Setup asks where alerts go, and sends each a test

2026-10-14 13:55:48 +02:00

serverinit_notify.go

Setup asks where alerts go, and sends each a test

2026-10-14 13:55:48 +02:00

serverinit_patroni.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

serverinit_postgres_test.go

Find the PostgreSQL a machine has, rather than assume its names

2026-09-28 14:47:55 +02:00

serverinit_postgres.go

The store's PostgreSQL sized to its machine's memory

2026-10-08 21:48:53 +02:00

serverinit_standby.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

serverinit_test.go

The mirror's address is one the nodes reach, and node add checks it

2026-10-06 09:36:09 +02:00

serverinit.go

Slurm in the first install: server init --slurm, and slurm.all_nodes

2026-11-11 14:50:35 +01:00

service_test.go

llmgw start --wait waits for the models, not for the setting

2026-10-07 21:43:45 +02:00

service.go

llmgw stop, start and restart act on the whole system

2026-10-07 18:01:45 +02:00

simengine.go

Onboard a node with a script, and find two examples that never worked

2026-09-16 12:28:36 +02:00

sourcenote_test.go

Say when a replica is running an edited profile, and check a source from the web too

2026-09-27 22:42:26 +02:00

sourcenote.go

Say when a replica is running an edited profile, and check a source from the web too

2026-09-27 22:42:26 +02:00

standbyroutes_test.go

Rate limits counted across the control planes with router.active_active

2026-11-01 17:30:08 +01:00

standbyroutes.go

Rate limits counted across the control planes with router.active_active

2026-11-01 17:30:08 +01:00

status_test.go

An on-demand model that is up reads as ready, not as not placed

2026-10-10 01:01:55 +02:00

status.go

llmgw stop, start and restart act on the whole system

2026-10-07 18:01:45 +02:00

storeflag_test.go

Stop calling the PostgreSQL store SQLite, and stop guessing where it is

2026-09-27 12:20:40 +02:00

storeflag.go

Stop calling the PostgreSQL store SQLite, and stop guessing where it is

2026-09-27 12:20:40 +02:00

strataserve.go

Strata runs as an engine: a 125B MoE on one card

2026-10-29 15:09:13 +01:00

system.go

llmgw start --wait waits for the models, not for the setting

2026-10-07 21:43:45 +02:00

threads.go

Manage people and the tenancy tree, from the web and the CLI

2026-09-28 16:26:39 +02:00

toolcheck_test.go

llmgw ctl tool-check asks with the engine's key

2026-10-08 11:24:37 +02:00

toolcheck.go

llmgw ctl tool-check asks with the engine's key

2026-10-08 11:24:37 +02:00

train.go

Trained adapters published, private to their owners like the knowledge bases they come from

2026-10-08 21:30:50 +02:00

tui_actions_test.go

Canary on the console's model menu; docs

2026-10-04 20:33:59 +02:00

tui_actions.go

Deleting unused model files says what went, and sets go whole

2026-11-03 10:29:16 +01:00

tui_audit.go

Usage export as CSV: by period and by place, person, key and model, from the rollups

2026-10-04 16:49:04 +02:00

tui_chart.go

Place chart points by their time, so a window's samples fill the chart's width

2026-10-04 08:55:19 +02:00

tui_choices.go

Slurm in the first install: server init --slurm, and slurm.all_nodes

2026-11-11 14:50:35 +01:00

tui_console.go

How much of each GPU works, beside whether anything does

2026-10-31 20:11:26 +01:00

tui_docs.go

A docs portal at /docs and a client developer's guide

2026-10-04 23:46:42 +02:00

tui_federation_test.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

tui_federation.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

tui_form.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_frame.go

Show the version top right on every screen

2026-10-04 01:18:45 +02:00

tui_guard.go

Guardrails in the console

2026-10-05 03:36:17 +02:00

tui_hf.go

The serve forms start at the longest context that fits

2026-11-08 14:38:41 +01:00

tui_home.go

Watch the cluster and edit every setting from the console

2026-10-03 22:54:09 +02:00

tui_init_test.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_init.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_jobs.go

Serve and update models in the background from the console, and follow them under Jobs

2026-10-04 01:27:02 +02:00

tui_kb_test.go

The console's Knowledge screen, and the manual's page on knowledge bases

2026-10-08 13:58:12 +02:00

tui_kb.go

Collections a part of the tree answers from when a request names none

2026-10-08 15:52:43 +02:00

tui_mcp_test.go

The company's MCP servers in the console

2026-10-28 13:23:25 +01:00

tui_mcp.go

Fixes from the second review: grants, flags, updates, setup

2026-10-29 12:31:07 +01:00

tui_mirror.go

The control plane uses whichever mirror answers

2026-10-29 13:43:27 +01:00

tui_monitor.go

How much of each GPU works, beside whether anything does

2026-10-31 20:11:26 +01:00

tui_node_overview_test.go

A node's console opens on what the machine is doing

2026-10-19 12:55:27 +02:00

tui_node_overview.go

A node's console opens on what the machine is doing

2026-10-19 12:55:27 +02:00

tui_node_test.go

Let server init ask for what it needs, and node add take several nodes

2026-09-29 21:54:18 +02:00

tui_node.go

Add a standby control plane with server add, and draw the screens as windows

2026-10-03 22:29:12 +02:00

tui_people.go

An organization's admin, and forms only for those who may send them

2026-11-08 11:38:06 +01:00

tui_presets_test.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

tui_presets.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

tui_quality_test.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

tui_quality.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

tui_setup_test.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_setup.go

A machine too small for the models it would serve is said so first

2026-10-25 16:19:00 +01:00

tui_signin_test.go

Sign-in is set up from the console, and a directory tried before it is kept

2026-10-19 12:55:27 +02:00

tui_signin.go

Sign-in is set up from the console, and a directory tried before it is kept

2026-10-19 12:55:27 +02:00

tui_widgets.go

Open setup screens or a console when llmgw runs bare in a terminal

2026-10-03 21:33:59 +02:00

usage.go

Usage export as CSV: by period and by place, person, key and model, from the rollups

2026-10-04 16:49:04 +02:00

users.go

An organization's admin, and forms only for those who may send them

2026-11-08 11:38:06 +01:00

wipe_test.go

The firewall turns off and on with the cluster's settings; llmgw wipe

2026-10-06 08:00:06 +02:00

wipe.go

The firewall turns off and on with the cluster's settings; llmgw wipe

2026-10-06 08:00:06 +02:00