Files
ilkerandClaude Opus 5.5 772579648c hpcod: one binary installs, checks and tests the control plane
An installation used to be this one: the deploy scripts and the Ansible
defaults carried one site's domain, hypervisor and login, the Proxmox side
and the database were set up by hand, and deploying meant copying scripts
and a tree of roles beside the binary and keeping them in step with it.

The binary now carries the roles, the Terraform module, the schema and the
catalogue (embed.go), and installs itself:

- hpcod setup: a full-screen installer for administrators. It asks where
  clusters are built, signs in to Proxmox before asking anything else
  (root's password once, or tokens), offers nodes and datastores from what
  Proxmox has, checks every field as it is typed, runs every step with its
  output live, checks the result and offers a test cluster. -yes runs the
  same steps from a site file; make deploy uses that.
- hpcod verify: each fact checked by using it.
- hpcod selftest: the smallest real cluster, by the portal's own path, a
  job run on it, then taken down.
- Closed networks: hpcod serves HTTPS itself from a CA name-constrained to
  the site's domains, issuing and renewing its own certificate, and
  answers DNS for the portal and every cluster name.
- A simulated Proxmox: Proxmox VE in QEMU on the control plane host, on a
  private bridge with NAT, for trying it without one.

Every step reads back what it changed and is safe to run again. The
site's values live in one file; group_vars/all/site.yml is generated from
it, and the playbooks refuse to run without it. ci_user and -ssh-user no
longer default to a personal account. scripts/B10, B20 and C20 are gone:
their work is the setup steps.

Proven end to end on a simulated Proxmox (Proxmox VE 9 in a VM on a real
host): the install, the Proxmox step by the password path, the head
template, verify READY, and a test cluster that ran srun on its compute
node and was destroyed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:20:04 +01:00

867 lines
31 KiB
Go

package provision
import (
"archive/tar"
"compress/gzip"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
"github.com/ilkermanap/hpcod/internal/store"
)
// Building a cluster takes minutes, so it cannot happen inside a request.
//
// The Runner starts the work, records that it is happening, and gets out
// of the way. What makes that safe is that the record is the only thing
// anyone reads afterwards: the state moves requested -> provisioning ->
// ready or failed, and the output goes to a file the portal can show. A
// job whose progress lived only in memory would be invisible the moment
// the portal restarted, and a cluster stuck at "provisioning" with nothing
// to read is worse than one that plainly failed.
// Store is the slice of the database the runner needs. An interface so a
// test can watch the transitions without a PostgreSQL.
type Store interface {
VMs(ctx context.Context, serialID int64) ([]store.VM, error)
// Apps is what the catalogue resolved when the cluster was allocated.
// Read here rather than resolved here: the provisioner has no
// catalogue and should not acquire one, or a cluster rebuilt after an
// edit to it would come back as different software.
Apps(ctx context.Context, serialID int64) ([]store.ClusterApp, error)
SetState(ctx context.Context, serialID int64, state string) error
// Live is read once at startup, to find work that was in flight when
// the process before this one stopped.
Live(ctx context.Context) ([]store.Cluster, error)
// UsageFor is what the cluster held and for how long, written beside
// the archive so the tarball can say what it was without a database.
UsageFor(ctx context.Context, serialID int64) (*store.Usage, error)
// Release gives the address and the hardware back. It is here rather
// than at the caller because it has to happen after the machines are
// gone, and only then.
Release(ctx context.Context, serialID int64) error
}
// Options configure where and how the tool runs.
type Options struct {
// EnvRoot holds one directory per cluster, each a Terraform working
// directory seeded from Template.
EnvRoot string
// Template is a prepared environment: the module wiring and the
// site's own variables, without anything cluster-specific.
Template string
// Binary is tofu or terraform.
Binary string
// MgmtFirst is the first address of the window reserved for head
// nodes, and MgmtCount is how many addresses follow it.
//
// A cluster's head takes MgmtFirst + (cluster id - 1). Deriving it
// from the id means two live clusters cannot be handed the same
// address and a rebuild gets the one it had, without a second thing
// to allocate and release. But the id counts from 1 and the
// management network already has other guests on it, so the window
// has to be stated: the first attempt used base + id and would have
// given cluster 10 the hypervisor's own address.
MgmtFirst string
MgmtCount int
// MgmtPrefix is the mask length written into the address.
MgmtPrefix int
// Timeout bounds one build. A run that hangs has to end, or the
// cluster sits at "provisioning" for ever.
Timeout time.Duration
// Registry, when set, is where a finished cluster is published so the
// proxy can route its name. Without it a cluster is built and
// unreachable, which is what happened when building moved into the
// portal and the publishing script stopped being called.
Registry *Registry
// PortalWait bounds how long to wait for the head to answer before
// giving up on publishing. The machines exist by then; it is the
// first boot that is still finishing.
PortalWait time.Duration
// headReady reports when a head node's portal is answering, which is
// how the build knows its first-boot playbook finished. Unexported
// and injected so the tests, which have no head node and no network
// to dial, can answer for it - dialling a real address from a unit
// test is its own kind of wrong.
headReady func(ctx context.Context, addr string, progress io.Writer) error
// ControlPlaneCert is the certificate this portal presents to cluster
// portals, in PEM. The cluster pins it and accepts nothing else, which
// is the whole of the single sign-on.
//
// It is carried here rather than left in the site's Terraform
// variables because the two have to be the same certificate. An
// operator pasting it into a tfvars by hand is how they come to differ
// - and they differ silently: the cluster builds, the portal answers,
// and every request through the proxy is refused at the handshake.
// Empty means the cluster keeps whatever its own playbook defaults to.
ControlPlaneCert string
// SSHKey and SSHUser reach a head node to take a copy of /home before
// the cluster is destroyed. Without a key nothing is archived, and a
// release says so rather than quietly throwing the work away.
SSHKey string
SSHUser string
}
func (o Options) withDefaults() Options {
if o.Binary == "" {
o.Binary = "tofu"
}
if o.MgmtCount == 0 {
o.MgmtCount = 40
}
if o.MgmtPrefix == 0 {
o.MgmtPrefix = 24
}
if o.Timeout == 0 {
o.Timeout = 30 * time.Minute
}
if o.PortalWait == 0 {
o.PortalWait = 10 * time.Minute
}
if o.SSHUser == "" {
o.SSHUser = "hpcod"
}
if o.headReady == nil {
wait := o.PortalWait
o.headReady = func(ctx context.Context, addr string, progress io.Writer) error {
return waitForPortal(ctx, addr, 10*time.Second, wait, progress)
}
}
return o
}
type Runner struct {
opt Options
store Store
// running guards against two builds of the same cluster at once, which
// would have two Terraform processes writing one state file.
mu sync.Mutex
running map[int64]bool
}
func NewRunner(st Store, opt Options) *Runner {
return &Runner{opt: opt.withDefaults(), store: st, running: map[int64]bool{}}
}
// ErrAlreadyRunning means a build for this cluster is in flight.
var ErrAlreadyRunning = errors.New("this cluster is already being built")
// Dir is where a cluster's Terraform state lives.
func (r *Runner) Dir(serialID int64) string {
return filepath.Join(r.opt.EnvRoot, fmt.Sprintf("c%d", serialID))
}
// LogPath is where its output goes. Kept beside the state so a cluster's
// directory holds everything about it.
func (r *Runner) LogPath(serialID int64) string {
return filepath.Join(r.Dir(serialID), "provision.log")
}
// Start begins a build and returns immediately. The caller learns how it
// went by reading the cluster's state, which is the point: an HTTP request
// must not be the only thing that knows a build is happening.
func (r *Runner) Start(c store.Cluster) error {
r.mu.Lock()
if r.running[c.SerialID] {
r.mu.Unlock()
return ErrAlreadyRunning
}
r.running[c.SerialID] = true
r.mu.Unlock()
// Marked before the goroutine starts, so a page loaded immediately
// afterwards does not still say "requested" and invite a second build.
ctx, cancel := context.WithTimeout(context.Background(), r.opt.Timeout)
if err := r.store.SetState(ctx, c.SerialID, "provisioning"); err != nil {
cancel()
r.finish(c.SerialID)
return err
}
go func() {
defer cancel()
defer r.finish(c.SerialID)
err := r.build(ctx, c)
state := "ready"
if err != nil {
state = "failed"
// Also to the service's own log. The cluster's log file is
// the detailed account, but it is a file that has to exist
// and be readable, and the one failure that leaves nothing
// to read is the one worth having a second copy of.
fmt.Fprintf(os.Stderr, "provision: cluster %s failed: %v\n", c.AccountingName, err)
}
// A separate context: the one above may be the thing that just
// expired, and the outcome has to be recorded either way.
rec, recCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer recCancel()
if serr := r.store.SetState(rec, c.SerialID, state); serr != nil {
// Nothing else can fix this; say it loudly enough to be found.
fmt.Fprintf(os.Stderr, "provision: cluster %d finished %s but the state could not be recorded: %v\n",
c.SerialID, state, serr)
}
}()
return nil
}
// Destroy takes a cluster's machines down and only then gives the record
// back.
//
// Releasing used to be a database write on its own: the record said the
// hardware was free while the virtual machines were still running on it,
// holding the cores, the memory and any passed-through device. The next
// tenant to ask for that GPU would be told it was theirs.
//
// The order is the whole point, and it is the opposite of building. The
// record is surrendered last, because a failed destroy must leave the
// cluster held: machines that still exist and a record that says nobody
// owns them is the one state nothing can recover from automatically.
func (r *Runner) Destroy(c store.Cluster, keepData bool) error {
r.mu.Lock()
if r.running[c.SerialID] {
r.mu.Unlock()
return ErrAlreadyRunning
}
r.running[c.SerialID] = true
r.mu.Unlock()
// A cluster that was never built has nothing to tear down. Asking
// Terraform to destroy an empty directory works, but it takes a
// provider download and a minute to say so, and it turns releasing a
// mistyped request into a wait.
if !r.hasState(c.SerialID) {
defer r.finish(c.SerialID)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
return r.store.Release(ctx, c.SerialID)
}
ctx, cancel := context.WithTimeout(context.Background(), r.opt.Timeout)
if err := r.store.SetState(ctx, c.SerialID, "deleting"); err != nil {
cancel()
r.finish(c.SerialID)
return err
}
go func() {
defer cancel()
defer r.finish(c.SerialID)
err := r.destroy(ctx, c, keepData)
rec, recCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer recCancel()
if err != nil {
if serr := r.store.SetState(rec, c.SerialID, "failed"); serr != nil {
fmt.Fprintf(os.Stderr, "provision: cluster %d failed to destroy and the state could not be recorded: %v\n",
c.SerialID, serr)
}
return
}
if serr := r.store.Release(rec, c.SerialID); serr != nil {
// The machines are gone and the record still holds their
// hardware. Nothing here can fix that; say it loudly.
fmt.Fprintf(os.Stderr, "provision: cluster %d was destroyed but the record could not be released: %v\n",
c.SerialID, serr)
}
}()
return nil
}
// hasState reports whether Terraform has ever built anything here. An empty
// state file counts as nothing: that is what init leaves behind when apply
// never ran.
func (r *Runner) hasState(serialID int64) bool {
fi, err := os.Stat(filepath.Join(r.Dir(serialID), "terraform.tfstate"))
return err == nil && fi.Size() > 0
}
func (r *Runner) destroy(ctx context.Context, c store.Cluster, keepData bool) error {
dir := r.Dir(c.SerialID)
log, err := os.OpenFile(r.LogPath(c.SerialID), os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o640)
if err != nil {
return err
}
defer log.Close()
fmt.Fprintf(log, "\n=== %s: destroying %s (cluster %d)\n",
time.Now().Format(time.RFC3339), c.AccountingName, c.ClusterID)
// The tenant's work lives on the head node's own disk, so it goes
// when the head does. Taking a copy first is the only moment it can
// be taken at all.
//
// A failed archive stops the release. Destroying anyway would be
// irreversible in the one direction that matters, and "the cluster is
// still here and here is why" is a recoverable position. The caller
// can ask again with keepData false when the data is genuinely not
// wanted, or when the head is too broken to read.
if keepData {
if err := r.archiveHome(ctx, c, dir, log); err != nil {
fmt.Fprintf(log, "\n=== not destroyed: the archive failed, so the data is still only on the head\n")
return fmt.Errorf("archiving /home: %w", err)
}
} else {
fmt.Fprintf(log, "\n=== releasing without keeping /home, as asked\n")
}
for _, args := range [][]string{
{"init", "-input=false", "-no-color"},
{"destroy", "-auto-approve", "-input=false", "-no-color"},
} {
if err := r.tofu(ctx, dir, log, args...); err != nil {
return err
}
}
fmt.Fprintf(log, "\n=== %s: destroyed\n", time.Now().Format(time.RFC3339))
return nil
}
// Recover marks work that was in flight when this process started.
//
// A build lives in a goroutine. Restart the portal - a deploy does
// exactly that, and systemd kills the whole control group - and the
// goroutine is gone while the record still says "provisioning". Nothing
// will ever move it again: the state is not a lie anyone notices, it just
// stops changing, and the page goes on showing a build that is not
// happening.
//
// So anything found moving at startup was moving in a process that no
// longer exists. It is marked failed, which is the honest description and
// the one that puts a Build button back on the page. The machines it left
// behind are still there and Terraform knows about them, so the retry is
// usually a no-op apply followed by the step that was interrupted.
//
// Not resumed automatically. A build that died halfway through creating
// virtual machines is not something to restart without someone looking,
// and "failed, here is the log, press the button" is a better offer than
// a second Terraform run nobody asked for.
func (r *Runner) Recover(ctx context.Context) (int, error) {
live, err := r.store.Live(ctx)
if err != nil {
return 0, err
}
n := 0
for _, c := range live {
if !moving(c.State) {
continue
}
// Not r.running: this is startup, so nothing is running here yet.
// The check is about the previous process, and the only evidence
// of it is the record.
if err := r.store.SetState(ctx, c.SerialID, "failed"); err != nil {
return n, fmt.Errorf("cluster %d was %s and could not be marked failed: %w",
c.SerialID, c.State, err)
}
if f, ferr := os.OpenFile(r.LogPath(c.SerialID),
os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o640); ferr == nil {
fmt.Fprintf(f, "\n=== %s: the portal restarted while this was %s; "+
"the machines are untouched, the step it was on did not finish\n",
time.Now().Format(time.RFC3339), c.State)
f.Close()
}
n++
}
return n, nil
}
// moving reports whether a state means something should be writing.
func moving(state string) bool {
return state == "provisioning" || state == "deleting"
}
func (r *Runner) finish(id int64) {
r.mu.Lock()
delete(r.running, id)
r.mu.Unlock()
}
// Running reports whether a build is in flight here. It is not the source
// of truth - the state column is - only a guard against starting a second
// one in this process.
func (r *Runner) Running(id int64) bool {
r.mu.Lock()
defer r.mu.Unlock()
return r.running[id]
}
func (r *Runner) build(ctx context.Context, c store.Cluster) error {
dir := r.Dir(c.SerialID)
// The log is opened before anything can fail, because the first thing
// that can fail is preparing the directory - and that used to happen
// with the log not yet created and the error dropped on the floor by
// the goroutine that called this. A cluster went to "failed" with
// nothing whatsoever to read: no log, no line in the journal, and the
// page offering a Retry button and no reason. It took a reproduction
// through the command line to find out the database was missing a
// table.
if err := os.MkdirAll(dir, 0o750); err != nil {
return err
}
log, err := os.OpenFile(r.LogPath(c.SerialID), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o640)
if err != nil {
return err
}
defer log.Close()
fmt.Fprintf(log, "=== %s: building %s (cluster %d) in %s\n",
time.Now().Format(time.RFC3339), c.AccountingName, c.ClusterID, dir)
if err := r.prepare(ctx, c, dir); err != nil {
fmt.Fprintf(log, "\n=== could not prepare the working directory: %v\n", err)
return err
}
for _, args := range [][]string{
{"init", "-input=false", "-no-color"},
{"apply", "-auto-approve", "-input=false", "-no-color"},
} {
if err := r.tofu(ctx, dir, log, args...); err != nil {
return err
}
}
// The compute nodes were created switched off, because they PXE boot
// from the head and there is nothing to boot from until Warewulf on
// it is configured. The head's portal answering is the signal that
// its playbook finished, so that is when they can be switched on.
if err := r.waitForHead(ctx, c, log); err != nil {
fmt.Fprintf(log, "\n=== built, but the head never came up: %v\n", err)
return err
}
// A second apply, with started true. Without it the cluster reports
// ready, publishes, serves a desktop request and queues it for ever
// behind nodes that are switched off - which is exactly what it did.
fmt.Fprintf(log, "\n=== powering on the compute nodes\n")
if err := r.tofu(ctx, dir, log,
"apply", "-auto-approve", "-input=false", "-no-color", "-var", "started=true"); err != nil {
fmt.Fprintf(log, "\n=== built, but the compute nodes are still off: %v\n", err)
return err
}
if err := r.publish(ctx, c, log); err != nil {
// The machines are built and working; they are just not reachable
// by name yet. Saying so and failing is better than reporting
// ready for a cluster whose link leads nowhere.
fmt.Fprintf(log, "\n=== built, but not published: %v\n", err)
return err
}
fmt.Fprintf(log, "\n=== %s: done\n", time.Now().Format(time.RFC3339))
return nil
}
// publish waits for the head's portal and records how to reach it.
func (r *Runner) publish(ctx context.Context, c store.Cluster, log io.Writer) error {
if r.opt.Registry == nil {
fmt.Fprintf(log, "\n=== no registry configured; not publishing\n")
return nil
}
cidr, err := r.MgmtCIDR(c.ClusterID)
if err != nil {
return err
}
addr := strings.Split(cidr, "/")[0] + ":443"
if err := r.opt.Registry.Publish(ctx, c.ClusterID, addr); err != nil {
return err
}
fmt.Fprintf(log, "=== published c%03d -> %s\n", c.ClusterID, addr)
return nil
}
// archiveHome copies the tenant's home directory off the head node.
//
// Compressed on the head and streamed here, rather than pulled raw and
// compressed centrally: the head's CPU is about to be thrown away, and the
// bytes crossing the management network should be the small ones.
//
// Caches are excluded by name. They are rebuildable by definition, and a
// Spack or conda build tree under a home directory is larger than
// everything worth keeping in it put together.
//
// The result is checked by reading it back as an archive. tar over ssh
// reports success from the local end of a pipe: a truncated stream, a
// refused connection after the first block, a sudo that asked for a
// password - all of them can leave a file that exists and is not an
// archive, and the moment to find that out is before the machines go.
func (r *Runner) archiveHome(ctx context.Context, c store.Cluster, dir string, log io.Writer) error {
if r.opt.SSHKey == "" {
return errors.New("no ssh key configured, so /home cannot be read from the head")
}
cidr, err := r.MgmtCIDR(c.ClusterID)
if err != nil {
return err
}
head := strings.Split(cidr, "/")[0]
out := filepath.Join(dir, "home.tgz")
fmt.Fprintf(log, "\n=== archiving /home from %s\n", head)
f, err := os.Create(out)
if err != nil {
return err
}
// Session state, not the tenant's work. recently-used.xbel is the one
// that stopped a release: a desktop was open, XFCE held the file, and
// tar could not read it - so a cluster could not be destroyed because
// somebody's list of recently opened files was busy.
remote := "sudo tar czf - --warning=no-file-changed " +
"--exclude=./*/.cache --exclude=./*/.conda/pkgs --exclude=./*/.spack " +
"--exclude=./*/spack-stage --exclude=./*/.singularity --exclude=./*/.apptainer " +
"--exclude=./*/.local/share/recently-used.xbel " +
"--exclude=./*/.dbus --exclude=./*/.gvfs " +
"--exclude=./*/.Xauthority --exclude=./*/.ICEauthority --exclude=./*/.esd_auth " +
"-C /home ."
cmd := exec.CommandContext(ctx, "ssh",
"-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null", "-o", "ConnectTimeout=15",
"-i", r.opt.SSHKey, r.opt.SSHUser+"@"+head, remote)
cmd.Stdout = f
cmd.Stderr = log
runErr := cmd.Run()
if closeErr := f.Close(); closeErr != nil {
return closeErr
}
// Deliberately not "runErr != nil, give up". tar exits non-zero when it
// could not read a single file, and it exits non-zero when it died in
// the middle - and those are opposite situations. The first cost a
// release: one desktop file was busy, and a complete archive of
// everybody's work was thrown away unread.
//
// So the exit code is a reason to look, not a verdict. What decides is
// the archive: it has to read back to the end, and it has to contain
// every home directory that is on the head. A tar that died in the
// middle fails the first of those, which is the case this guard exists
// for; a tar that skipped a lock file fails neither.
homes, n, readErr := archiveHomes(out)
if readErr != nil {
if runErr != nil {
return fmt.Errorf("%v, and the archive does not read back: %w", runErr, readErr)
}
return readErr
}
want, err := r.homesOn(ctx, head, log)
if err != nil {
// Not fatal on its own: without the list there is nothing to
// compare against, so the archive reading cleanly has to be
// enough. Said out loud, because it is a weaker check than the
// one this was supposed to make.
fmt.Fprintf(log, "=== could not list /home to check the archive against: %v\n", err)
} else {
var missing []string
for _, h := range want {
if !homes[h] {
missing = append(missing, h)
}
}
if len(missing) > 0 {
return fmt.Errorf("the archive is missing %s", strings.Join(missing, ", "))
}
fmt.Fprintf(log, "=== every home on the head is in the archive (%d)\n", len(want))
}
if runErr != nil {
// Everything that matters is in it. Loud, because somebody should
// know which files were skipped, but not a reason to keep a
// cluster nobody wants.
fmt.Fprintf(log, "=== tar reported errors (%v), but the archive reads back complete;\n", runErr)
fmt.Fprintf(log, " see its messages above for what it could not read\n")
}
fi, err := os.Stat(out)
if err != nil {
return err
}
fmt.Fprintf(log, "=== kept %s (%d entries, %d bytes)\n", out, n, fi.Size())
return nil
}
// homesOn lists the home directories the head actually has, which is what
// the archive has to be complete with respect to. Counting entries says the
// file is an archive; this says it is the right one.
func (r *Runner) homesOn(ctx context.Context, head string, log io.Writer) ([]string, error) {
cmd := exec.CommandContext(ctx, "ssh",
"-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null", "-o", "ConnectTimeout=15",
"-i", r.opt.SSHKey, r.opt.SSHUser+"@"+head,
"sudo find /home -mindepth 1 -maxdepth 1 -type d -printf '%f\\n'")
cmd.Stderr = log
out, err := cmd.Output()
if err != nil {
return nil, err
}
var homes []string
for _, line := range strings.Split(string(out), "\n") {
if line = strings.TrimSpace(line); line != "" {
homes = append(homes, line)
}
}
return homes, nil
}
// writeManifest records what the cluster was, next to what it produced.
//
// An archive that cannot say how many machines it came from or how long
// they ran is a directory of files somebody will be guessing about in six
// months. The database knows all of it and outlives the cluster, but the
// tarball can be moved off this host and the manifest travels with it.
func (r *Runner) writeManifest(ctx context.Context, c store.Cluster, dir string, log io.Writer) error {
u, err := r.store.UsageFor(ctx, c.SerialID)
if err != nil {
return err
}
vms, err := r.store.VMs(ctx, c.SerialID)
if err != nil {
return err
}
type node struct {
Role string `json:"role"`
Index int `json:"index"`
Host string `json:"host"`
Cores int `json:"cores"`
MemoryGB int `json:"memory_gb"`
Devices []string `json:"devices,omitempty"`
}
nodes := make([]node, 0, len(vms))
for _, vm := range vms {
n := node{Role: vm.Role, Index: vm.NodeIndex, Host: vm.HostNode,
Cores: vm.Cores, MemoryGB: vm.MemoryGB}
for _, d := range vm.Devices {
n.Devices = append(n.Devices, string(d.Capability)+" "+d.NativeID)
}
nodes = append(nodes, n)
}
body, err := json.MarshalIndent(struct {
Cluster string `json:"cluster"`
Tenant string `json:"tenant"`
Project string `json:"project,omitempty"`
Site string `json:"site"`
CreatedAt time.Time `json:"created_at"`
ReleasedAt time.Time `json:"released_at"`
WallSeconds int64 `json:"wall_seconds"`
Wall string `json:"wall"`
Nodes []node `json:"nodes"`
TotalCores int `json:"total_cores"`
TotalMemGB int `json:"total_memory_gb"`
CoreHours float64 `json:"core_hours"`
GBHours float64 `json:"gb_hours"`
Note string `json:"note"`
}{
Cluster: c.AccountingName, Tenant: u.TenantName, Project: u.ProjectName,
Site: u.Site, CreatedAt: u.CreatedAt, ReleasedAt: time.Now().UTC(),
WallSeconds: int64(u.Wall.Seconds()), Wall: u.Wall.Round(time.Second).String(),
Nodes: nodes, TotalCores: u.Cores, TotalMemGB: u.MemoryGB,
CoreHours: u.CoreHours, GBHours: u.GBHours,
Note: "Core-hours are reserved, not executed: the hardware was held " +
"whether or not a job ran on it.",
}, "", " ")
if err != nil {
return err
}
out := filepath.Join(dir, "manifest.json")
if err := os.WriteFile(out, append(body, '\n'), 0o640); err != nil {
return err
}
fmt.Fprintf(log, "=== described in %s: %d node(s), %s, %.2f core-hours\n",
out, len(nodes), u.Wall.Round(time.Second), u.CoreHours)
return nil
}
// archiveHomes reads the archive back to the end and returns which home
// directories are in it, and how many entries it holds.
//
// Reading to the end is the point. A tar that was cut off - the process
// killed, the connection dropped, the disk full - leaves a stream that stops
// mid-entry, and that is the one failure that must never be mistaken for a
// complete copy. Individual files it could not open do not show up here at
// all, which is correct: they are not in the archive, and whether that
// matters is decided by whose home is missing, not by how many files are.
func archiveHomes(path string) (map[string]bool, int, error) {
f, err := os.Open(path)
if err != nil {
return nil, 0, err
}
defer f.Close()
gz, err := gzip.NewReader(f)
if err != nil {
return nil, 0, fmt.Errorf("not a gzip stream: %w", err)
}
defer gz.Close()
tr := tar.NewReader(gz)
homes := map[string]bool{}
n := 0
for {
h, err := tr.Next()
if err == io.EOF {
return homes, n, nil
}
if err != nil {
return homes, n, fmt.Errorf("truncated archive after %d entries: %w", n, err)
}
// "./alice/..." - the first component after the leading ./ is
// whose work it is.
if p := strings.TrimPrefix(h.Name, "./"); p != "" {
if i := strings.Index(p, "/"); i > 0 {
homes[p[:i]] = true
} else if h.Typeflag == tar.TypeDir {
homes[strings.TrimSuffix(p, "/")] = true
}
}
n++
}
}
// waitForHead blocks until the head node's portal answers.
//
// That is the one signal available here that its first-boot playbook
// finished: the portal is the last thing the playbook starts. Everything
// after this - powering on the compute nodes, publishing the name -
// depends on the head being configured, and both used to be behind the
// registry, so a portal with no registry configured did neither.
func (r *Runner) waitForHead(ctx context.Context, c store.Cluster, log io.Writer) error {
cidr, err := r.MgmtCIDR(c.ClusterID)
if err != nil {
return err
}
addr := strings.Split(cidr, "/")[0] + ":443"
fmt.Fprintf(log, "\n=== waiting for the portal at %s\n", addr)
return r.opt.headReady(ctx, addr, log)
}
// tofu runs the tool once, with the environment every run needs.
func (r *Runner) tofu(ctx context.Context, dir string, log io.Writer, args ...string) error {
fmt.Fprintf(log, "\n=== %s %s\n", r.opt.Binary, strings.Join(args, " "))
cache := filepath.Join(r.opt.EnvRoot, ".plugin-cache")
if err := os.MkdirAll(cache, 0o750); err != nil {
return err
}
cmd := exec.CommandContext(ctx, r.opt.Binary, args...)
cmd.Dir = dir
cmd.Stdout, cmd.Stderr = log, log
// Terraform wants somewhere to put its plugin cache, and the service's
// own HOME is not guaranteed to be set. A shared cache, or every
// cluster's first init downloads the provider again - a hundred
// megabytes and a minute each time, for a file that is identical.
cmd.Env = append(os.Environ(),
"TF_IN_AUTOMATION=1", "HOME="+r.opt.EnvRoot, "TF_PLUGIN_CACHE_DIR="+cache)
if err := cmd.Run(); err != nil {
fmt.Fprintf(log, "\n=== failed: %v\n", err)
if ctx.Err() != nil {
return fmt.Errorf("%s %s: %w", r.opt.Binary, args[0], ctx.Err())
}
return fmt.Errorf("%s %s: %w", r.opt.Binary, args[0], err)
}
return nil
}
// prepare seeds the working directory and writes the cluster's variables.
func (r *Runner) prepare(ctx context.Context, c store.Cluster, dir string) error {
if r.opt.Template == "" {
return errors.New("no environment template configured, so there is nothing to build from")
}
if err := os.MkdirAll(dir, 0o750); err != nil {
return err
}
// Copy the template's files, but never over the state: a rebuild has
// to keep what Terraform already knows it created.
if err := copyTemplate(r.opt.Template, dir); err != nil {
return err
}
vms, err := r.store.VMs(ctx, c.SerialID)
if err != nil {
return err
}
cidr, err := r.MgmtCIDR(c.ClusterID)
if err != nil {
return err
}
apps, err := r.store.Apps(ctx, c.SerialID)
if err != nil {
return err
}
tf, err := TFVars(c, vms, apps, c.TenantName, 0, cidr)
if err != nil {
return err
}
if r.opt.ControlPlaneCert != "" {
tf += "\n# The certificate this cluster will trust, from the portal that is\n" +
"# building it - so the pin and the client are the same thing.\n" +
"control_plane_cert = " + quote(r.opt.ControlPlaneCert) + "\n"
}
return os.WriteFile(filepath.Join(dir, "cluster.auto.tfvars"), []byte(tf), 0o640)
}
// MgmtCIDR is the head node's address on the management network, or an
// error when the cluster's id falls outside the window.
//
// Refusing is the point. A cluster built with an address belonging to
// something else does not fail: it works until the other machine is
// restarted, and then two hosts answer to one address on a network with
// other people's guests on it. Better to build nothing.
func (r *Runner) MgmtCIDR(clusterID int16) (string, error) {
if r.opt.MgmtFirst == "" {
return "", errors.New("no management address window configured (-mgmt-first)")
}
first := net.ParseIP(r.opt.MgmtFirst).To4()
if first == nil {
return "", fmt.Errorf("management window start %q is not an IPv4 address", r.opt.MgmtFirst)
}
if int(clusterID) > r.opt.MgmtCount {
return "", fmt.Errorf("cluster id %d is past the %d addresses from %s; free some clusters or widen the window",
clusterID, r.opt.MgmtCount, r.opt.MgmtFirst)
}
last := int(first[3]) + int(clusterID) - 1
if last > 254 {
return "", fmt.Errorf("cluster id %d would need 10.%d.%d.%d, which is past the end of the network",
clusterID, first[1], first[2], last)
}
return fmt.Sprintf("%d.%d.%d.%d/%d", first[0], first[1], first[2], last, r.opt.MgmtPrefix), nil
}
func copyTemplate(from, to string) error {
entries, err := os.ReadDir(from)
if err != nil {
return err
}
for _, e := range entries {
if e.IsDir() {
continue // .terraform and the like belong to the working copy
}
name := e.Name()
switch {
case strings.HasPrefix(name, "terraform.tfstate"):
continue // never overwrite what Terraform knows it built
case name == "cluster.auto.tfvars":
continue // written per cluster, below
}
b, err := os.ReadFile(filepath.Join(from, name))
if err != nil {
return err
}
if err := os.WriteFile(filepath.Join(to, name), b, 0o640); err != nil {
return err
}
}
return nil
}