Commit Graph
2 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 c97b671ef9 Prove the multi-node path on a simulated cluster
Phase 4 had been written and never run. The hardware is not coming, so the
cluster is simulated instead — four nodes across two fabrics, eight GPUs each,
built with the onboarding scripts from the previous commit.

`sim` is now an engine adapter, not just a standalone command. Its leader binds
a rendezvous port and does not serve until every worker has joined; its workers
serve nothing at all. That is the part stub processes could never exercise, and
it is why this path survived four phases untested: the command line was
checked, the sequence was not.

Measured on the cluster:

- a 2-node replica on each fabric, node01+node02 on ib-a and node03+node04 on
  ib-b, four GPUs from each
- a request served through one of them
- only the leader listening — node02 had no port open at all, which is
  ServesHTTP working end to end rather than in a unit test
- node02 killed; at t+60s it was declared lost, the ib-a replica failed and was
  reaped; at t+80s a replacement was ready on ib-b, the fabric that could still
  host it, while node01 sat idle rather than being paired across fabrics

The invariant held under the one condition that tests it: when a replica could
not be rebuilt where it was, it moved to a fabric that could hold all of it
rather than spanning two.

What this does not prove is written into the plan next to what it does. The
GPUs are an inventory, the fabric is a sysfs-shaped directory, and the engine
speaks no NCCL — so the launch sequence is now exercised while the launch
arguments for vLLM and SGLang are still read from documentation rather than
from a running engine.

docs/node-onboarding.md is the procedure, including how to run all of this on
machines that have neither GPUs nor InfiniBand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:34:23 +02:00
Ilker ManapandClaude Opus 5 c1ac2f022f Onboard a node with a script, and find two examples that never worked
New nodes had been joined by hand, four times, by me. There was no procedure,
so there was nothing to be wrong — and nothing to be right either. Two scripts
now: provision-node.sh makes the VM on Proxmox, onboard-node.sh gives it an
identity. They are separate because they are run by different people with
different access, and the second must run where the certificate authority is,
which is the one part that cannot be delegated.

Running them turned up why phase 4 was never verified. FabricAttachment
carried no koanf tags, so the `name:` key the shipped example documents bound
to nothing and `rate_gbps` was never read at all. An agent configured exactly
as instructed refused to start with "fabrics[0]: name is required" — naming
the key the operator had just written. Every fabric in this project had been
configured in a Go test, where the tags do not matter.

The server example did not load either: its models had no source, so it failed
validation the moment anything tried to parse it.

Both examples are now parsed by a test. A published example that has never
been loaded is a guess, and these two had been guesses since they were
written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:28:36 +02:00