Machine tags (MAAS-style), changelog-driven release notes
- tags table + machines.tags/auto_tags; match rules as Go templates over .Node
(GPUs, IB, CPUs, memory, disks, NVMe, PCI vendors, manufacturer/product, OS)
re-evaluated at commissioning and on tag changes
- deployment: tag cloud-config fragments appended as MIME parts with additive merge;
deploy/tags/gpu-nvidia.yaml installs the NVIDIA driver per distro family
- stateless: tag overlays appended to the profile's overlays
- CLI: tag, tags, list -tag, deploy/power tag:NAME fan-out; UI: tags column, per-machine
editor, tag definitions panel; API: /api/v1/tags, /machines/{id}/tags, ?tag=
- CHANGELOG.md; release workflow publishes the tag's section as release notes and
refuses tags without one
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
cdd0ad3215
commit
0f798d8797
15 files changed
+844
-36
No files matched your search
@@ -10,6 +10,8 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: cmd/rollcall/go.mod
|
||||
@@ -25,6 +27,25 @@ jobs:
|
||||
done
|
||||
sha256sum rollcall-linux-* > SHA256SUMS
|
||||
|
||||
- name: Release notes from CHANGELOG.md
|
||||
env:
|
||||
TAG: ${{ github.ref_name }}
|
||||
run: |
|
||||
set -eu
|
||||
ver=${TAG#v}
|
||||
# the section "## [X.Y.Z]" up to the next "## " heading
|
||||
awk -v v="$ver" '
|
||||
/^## \[/ { inside = ($0 ~ "^## \\[" v "\\]") }
|
||||
inside && !/^## \[/ { print }
|
||||
' CHANGELOG.md > notes.md
|
||||
if ! grep -q '[^[:space:]]' notes.md; then
|
||||
echo "::error::CHANGELOG.md has no section '## [$ver]' for tag $TAG; add it before tagging" >&2
|
||||
exit 1
|
||||
fi
|
||||
prev=$(git describe --tags --abbrev=0 "$TAG^" 2>/dev/null || true)
|
||||
{ echo; echo "---"; if [ -n "$prev" ]; then echo "Commits: $prev...$TAG"; git log --no-merges --format='- %s' "$prev..$TAG"; else git log --no-merges --format='- %s' "$TAG"; fi; } >> notes.md
|
||||
echo "== release notes"; cat notes.md
|
||||
|
||||
- name: Create release and upload assets
|
||||
env:
|
||||
API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
|
||||
@@ -33,7 +54,7 @@ jobs:
|
||||
working-directory: cmd/rollcall
|
||||
run: |
|
||||
set -eu
|
||||
body=$(printf '{"tag_name":"%s","name":"%s","draft":false,"prerelease":false}' "$TAG" "$TAG")
|
||||
body=$(python3 -c 'import json,sys; print(json.dumps({"tag_name": sys.argv[1], "name": sys.argv[1], "body": open("../../notes.md").read(), "draft": False, "prerelease": "-" in sys.argv[1]}))' "$TAG")
|
||||
id=$(curl -sfS -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
|
||||
-d "$body" "$API/releases" | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])')
|
||||
for f in rollcall-linux-amd64 rollcall-linux-arm64 SHA256SUMS; do
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to Rollcall. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
|
||||
the release workflow uses the section of the pushed tag as the release notes and refuses to
|
||||
publish a tag that has no section here.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- Machine tags (MAAS-style): manual tags per machine, auto tags from match rules over the
|
||||
inventory (`{{ gt .Node.GPUs 0 }}`), re-evaluated at commissioning and when a tag changes.
|
||||
- Tags carry a cloud-init fragment merged into deployment user-data (`deploy/tags/gpu-nvidia.yaml`
|
||||
installs the NVIDIA driver on GPU nodes) and extra overlays for stateless nodes.
|
||||
- `rollcall tag`, `rollcall tags`, `rollcall list -tag`, `deploy tag:NAME` / `power tag:NAME`
|
||||
fan-out; tags column, editor and definitions panel in the web UI; `?tag=` filter on the API.
|
||||
- `rollcall version`; CD-built binary carries `git describe`.
|
||||
- Release notes from `CHANGELOG.md`.
|
||||
|
||||
### Fixed
|
||||
- Virtual BMC addresses are configured in netplan on the lab PXE server (they were lost on a
|
||||
network reconfigure, which made every BMC call fail).
|
||||
- Commissioning console no longer shows a failing autologin (`noautologin`).
|
||||
|
||||
## [0.1.0] - 2026-10-27
|
||||
|
||||
First tagged release.
|
||||
|
||||
### Added
|
||||
- Commissioning: Debian 13 live image with firmware, IB and NVMe tooling; hardware inventory
|
||||
(DMI, DIMMs, CPUs, disks, NVMe, NICs, InfiniBand, GPUs, PCI, LLDP, TPM); in-band BMC account
|
||||
creation with out-of-band hints for VMs; secrets sealed with AES-256-GCM; ipmitool server-side.
|
||||
- Single static Go binary: HTTP API, embedded web UI, CLI, PostgreSQL schema.
|
||||
- Stateless (Warewulf-style) nodes: profiles, node names and IP pools, rendered overlays, dnsmasq
|
||||
export, Slurm compute image and controller sync.
|
||||
- OS deployment: image catalog with nightly upstream sync and vendor checksum verification
|
||||
(Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9), streamed `zstd | dd` writes
|
||||
with sha256 and live progress, cloud-init NoCloud seed, SELinux relabel, UEFI boot entry,
|
||||
phone-home, release/wipe jobs.
|
||||
- Gitea Actions: CI, deploy to the PXE server, website publish, release on tags.
|
||||
- Lab: Proxmox SDN subnet, `pve-vbmc` IPMI emulator for VMs.
|
||||
@@ -225,6 +225,31 @@ up for ssh). A deployed machine's PXE entry defaults to local disk, so boot orde
|
||||
can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry
|
||||
for the installed loader. `rollcall recommission` or `release` take it back.
|
||||
|
||||
## Tags
|
||||
|
||||
Tags single out machines for special treatment, as in MAAS. A tag has a name
|
||||
and optionally a **match rule**, a **cloud-config fragment** and **overlays**.
|
||||
|
||||
```sh
|
||||
rollcall tags add gpu -match '{{ gt .Node.GPUs 0 }}' -cloud-config deploy/tags/gpu-nvidia.yaml -comment "NVIDIA nodes"
|
||||
rollcall tags add rack-3 -comment "rack 3, row B" # manual only
|
||||
rollcall tag m-2ef1dab828c8 +rack-3 +burn-in -staging # manual tags on one machine
|
||||
rollcall list -tag gpu
|
||||
rollcall deploy tag:gpu rocky-9 -boot # every gpu machine
|
||||
rollcall power tag:rack-3 off
|
||||
```
|
||||
|
||||
* **Auto tags** come from the match rule, a Go template over `.Node` (GPUs,
|
||||
IB GUIDs, CPUs, memory, disks, NVMe, PCI vendor ids, manufacturer, product,
|
||||
OS, existing tags). They are recomputed at every commissioning and whenever a
|
||||
tag definition is saved; manual tags are never touched by rules.
|
||||
* **At deployment** the fragments of the machine's tags are appended to the
|
||||
generated user-data as MIME parts (`packages`, `runcmd`, `write_files` merge
|
||||
additively; a `#!` fragment runs as a script). `deploy/tags/gpu-nvidia.yaml`
|
||||
installs the NVIDIA driver on Debian, Ubuntu and the RHEL family and reboots.
|
||||
* **For stateless nodes** a tag's overlays are applied after the profile's, so a
|
||||
`gpu` tag can add `gres.conf` lines or driver modules without a second profile.
|
||||
|
||||
## Lab
|
||||
|
||||
`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
|
||||
|
||||
+181
-17
@@ -95,12 +95,15 @@ func runClient(cmd string, args []string) error {
|
||||
}
|
||||
switch cmd {
|
||||
case "list":
|
||||
var ms []Summary
|
||||
if err := c.do("GET", "/api/v1/machines", nil, &ms); err != nil {
|
||||
fs := newFlags("list")
|
||||
tag := fs.String("tag", "", "only machines with this tag")
|
||||
_ = fs.Parse(args)
|
||||
ms, err := c.machines(*tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
f := "%-16s %-14s %-10s %-16s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %s\n"
|
||||
fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE/OS", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC")
|
||||
f := "%-16s %-14s %-10s %-16s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %-18s %s\n"
|
||||
fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE/OS", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC", "TAGS")
|
||||
for _, m := range ms {
|
||||
state := m.State
|
||||
if m.Job != nil {
|
||||
@@ -123,7 +126,7 @@ func runClient(cmd string, args []string) error {
|
||||
}
|
||||
model := strings.TrimSpace(deref(m.Manufacturer) + " " + deref(m.Product))
|
||||
fmt.Printf(f, m.ID, state, deref(m.NodeName), trunc(what, 16), deref(m.MgmtIP), trunc(model, 24), m.CPUs, human(m.MemoryBytes),
|
||||
fmt.Sprint(int(m.Disks)), human(m.StorageBytes), fmt.Sprint(m.NICs), fmt.Sprint(m.IBHCAs), fmt.Sprint(m.GPUs), bmc)
|
||||
fmt.Sprint(int(m.Disks)), human(m.StorageBytes), fmt.Sprint(m.NICs), fmt.Sprint(m.IBHCAs), fmt.Sprint(m.GPUs), trunc(bmc, 18), strings.Join(m.Tags, ","))
|
||||
}
|
||||
case "show":
|
||||
_ = need(1)
|
||||
@@ -151,13 +154,26 @@ func runClient(cmd string, args []string) error {
|
||||
}
|
||||
case "power":
|
||||
_ = need(2)
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+args[0]+"/power", obj{"action": args[1]}, &d); err != nil {
|
||||
ids, err := c.targets(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Println(str(d["output"]))
|
||||
if ok, _ := d["ok"].(bool); !ok {
|
||||
return fmt.Errorf("ipmitool failed")
|
||||
failed := 0
|
||||
for _, id := range ids {
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+id+"/power", obj{"action": args[1]}, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(ids) > 1 {
|
||||
fmt.Printf("%s: ", id)
|
||||
}
|
||||
fmt.Println(str(d["output"]))
|
||||
if ok, _ := d["ok"].(bool); !ok {
|
||||
failed++
|
||||
}
|
||||
}
|
||||
if failed > 0 {
|
||||
return fmt.Errorf("ipmitool failed on %d machine(s)", failed)
|
||||
}
|
||||
case "bootdev":
|
||||
_ = need(2)
|
||||
@@ -189,12 +205,147 @@ func runClient(cmd string, args []string) error {
|
||||
fmt.Println("deleted", str(d["deleted"]))
|
||||
case "deploy", "release", "jobs", "job":
|
||||
return c.deployCmd(cmd, args)
|
||||
case "tag", "tags":
|
||||
return c.tagCmd(cmd, args)
|
||||
default:
|
||||
return c.statelessCmd(cmd, args)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *client) machines(tag string) ([]Summary, error) {
|
||||
q := ""
|
||||
if tag != "" {
|
||||
q = "?tag=" + tag
|
||||
}
|
||||
var ms []Summary
|
||||
err := c.do("GET", "/api/v1/machines"+q, nil, &ms)
|
||||
return ms, err
|
||||
}
|
||||
|
||||
// targets expands "-tag NAME" (passed as a single "tag:NAME" or "-tag=NAME" token) or returns the id.
|
||||
func (c *client) targets(arg string) ([]string, error) {
|
||||
tag := ""
|
||||
switch {
|
||||
case strings.HasPrefix(arg, "tag:"):
|
||||
tag = strings.TrimPrefix(arg, "tag:")
|
||||
case strings.HasPrefix(arg, "-tag="):
|
||||
tag = strings.TrimPrefix(arg, "-tag=")
|
||||
default:
|
||||
return []string{arg}, nil
|
||||
}
|
||||
ms, err := c.machines(tag)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(ms) == 0 {
|
||||
return nil, fmt.Errorf("no machine has tag %s", tag)
|
||||
}
|
||||
var ids []string
|
||||
for _, m := range ms {
|
||||
ids = append(ids, m.ID)
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
func (c *client) tagCmd(cmd string, args []string) error {
|
||||
if cmd == "tag" {
|
||||
if len(args) < 2 {
|
||||
usage()
|
||||
}
|
||||
req := obj{"add": []string{}, "remove": []string{}}
|
||||
var add, rm []string
|
||||
for _, t := range args[1:] {
|
||||
switch {
|
||||
case strings.HasPrefix(t, "+"):
|
||||
add = append(add, t[1:])
|
||||
case strings.HasPrefix(t, "-"):
|
||||
rm = append(rm, t[1:])
|
||||
default:
|
||||
add = append(add, t)
|
||||
}
|
||||
}
|
||||
req["add"], req["remove"] = add, rm
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+args[0]+"/tags", req, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s: manual=%v auto=%v\n", str(d["id"]), d["tags"], d["auto_tags"])
|
||||
return nil
|
||||
}
|
||||
if len(args) < 1 {
|
||||
usage()
|
||||
}
|
||||
switch args[0] {
|
||||
case "list":
|
||||
var defs []TagDef
|
||||
if err := c.do("GET", "/api/v1/tags", nil, &defs); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%-16s %5s %-32s %-16s %-8s %s\n", "TAG", "MACH", "MATCH", "OVERLAYS", "CLOUDCFG", "COMMENT")
|
||||
for _, t := range defs {
|
||||
cc := "-"
|
||||
if t.CloudConfig != "" {
|
||||
cc = fmt.Sprintf("%d B", len(t.CloudConfig))
|
||||
}
|
||||
fmt.Printf("%-16s %5d %-32s %-16s %-8s %s\n", t.Name, t.Machines, trunc(strings.ReplaceAll(t.Match, "\n", " "), 32), strings.Join(t.Overlays, ","), cc, t.Comment)
|
||||
}
|
||||
case "show":
|
||||
if len(args) < 2 {
|
||||
usage()
|
||||
}
|
||||
var defs []TagDef
|
||||
if err := c.do("GET", "/api/v1/tags", nil, &defs); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, t := range defs {
|
||||
if t.Name == args[1] {
|
||||
fmt.Printf("tag %s (%d machines)\ncomment: %s\nmatch: %s\noverlays: %v\n--- cloud_config ---\n%s\n", t.Name, t.Machines, t.Comment, t.Match, t.Overlays, t.CloudConfig)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return errNotFound
|
||||
case "add":
|
||||
if len(args) < 2 {
|
||||
usage()
|
||||
}
|
||||
fs := newFlags("tags add")
|
||||
match := fs.String("match", "", "Go template over .Node that renders true, e.g. '{{ gt .Node.GPUs 0 }}'")
|
||||
ov := fs.String("overlays", "", "comma-separated overlays for stateless nodes with this tag")
|
||||
cc := fs.String("cloud-config", "", "file with a #cloud-config or #!/bin/sh fragment applied at deployment")
|
||||
comment := fs.String("comment", "", "comment")
|
||||
_ = fs.Parse(args[2:])
|
||||
t := TagDef{Name: args[1], Match: *match, Comment: *comment}
|
||||
if *ov != "" {
|
||||
t.Overlays = strings.Split(*ov, ",")
|
||||
}
|
||||
if *cc != "" {
|
||||
b, err := os.ReadFile(*cc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
t.CloudConfig = string(b)
|
||||
}
|
||||
var out TagDef
|
||||
if err := c.do("POST", "/api/v1/tags", t, &out); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("tag %s saved\n", out.Name)
|
||||
case "rm":
|
||||
if len(args) < 2 {
|
||||
usage()
|
||||
}
|
||||
var d obj
|
||||
if err := c.do("DELETE", "/api/v1/tags/"+args[1], nil, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Println("deleted", str(d["deleted"]))
|
||||
default:
|
||||
usage()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *client) deployCmd(cmd string, args []string) error {
|
||||
switch cmd {
|
||||
case "deploy":
|
||||
@@ -222,16 +373,29 @@ func (c *client) deployCmd(cmd string, args []string) error {
|
||||
}
|
||||
req["user_data"] = string(b)
|
||||
}
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+args[0]+"/deploy", req, &d); err != nil {
|
||||
ids, err := c.targets(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s: job %s %s/%s -> %s hostname=%s mgmt=%s\n", str(d["id"]), str(d["job"]), str(d["os"]), str(d["version"]), str(d["disk"]), str(d["hostname"]), str(d["mgmt_ip"]))
|
||||
if d["ipmi"] != nil {
|
||||
fmt.Println(str(d["ipmi"]))
|
||||
if len(ids) > 1 && (*host != "" || *disk != "") {
|
||||
return fmt.Errorf("-hostname/-disk cannot be used with a tag target")
|
||||
}
|
||||
if *watch {
|
||||
return c.watchJob(args[0])
|
||||
for _, id := range ids {
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+id+"/deploy", req, &d); err != nil {
|
||||
if len(ids) == 1 {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s: error: %v\n", id, err)
|
||||
continue
|
||||
}
|
||||
fmt.Printf("%s: job %s %s/%s -> %s hostname=%s mgmt=%s\n", str(d["id"]), str(d["job"]), str(d["os"]), str(d["version"]), str(d["disk"]), str(d["hostname"]), str(d["mgmt_ip"]))
|
||||
if d["ipmi"] != nil {
|
||||
fmt.Println(str(d["ipmi"]))
|
||||
}
|
||||
}
|
||||
if *watch && len(ids) == 1 {
|
||||
return c.watchJob(ids[0])
|
||||
}
|
||||
case "release":
|
||||
if len(args) < 1 {
|
||||
|
||||
@@ -260,6 +260,11 @@ func (a *App) renderSeed(j *Job, sp DeploySpec, token string) []seedFile {
|
||||
b.WriteString("final_message: \"rollcall: deployed $hostname up after $UPTIME s\"\n")
|
||||
ud = b.String()
|
||||
}
|
||||
if m, err := a.store.Get(context.Background(), j.MachineID); err == nil {
|
||||
if frags, err := a.tagFragments(context.Background(), m); err == nil && len(frags) > 0 {
|
||||
ud = multipartUserData(ud, frags)
|
||||
}
|
||||
}
|
||||
return []seedFile{
|
||||
{Path: "var/lib/cloud/seed/nocloud/meta-data", Mode: "0600", Content: meta},
|
||||
{Path: "var/lib/cloud/seed/nocloud/user-data", Mode: "0600", Content: ud},
|
||||
|
||||
@@ -44,6 +44,9 @@ func usage() {
|
||||
rollcall deploy ID OS [-version V] [-disk /dev/sdX] [-hostname H] [-upgrade] [-user-data FILE] [-boot] [-watch]
|
||||
rollcall release ID [-boot] [-watch]
|
||||
rollcall jobs ID [-watch] | job cancel ID
|
||||
rollcall tag ID +NAME... -NAME... manual tags on a machine
|
||||
rollcall tags list | add NAME [-match TMPL] [-overlays a,b] [-cloud-config FILE] [-comment C] | show NAME | rm NAME
|
||||
(deploy and power also take -tag NAME instead of ID to act on every machine with that tag)
|
||||
env: ROLLCALL_URL (default from config listen), ROLLCALL_TOKEN, ROLLCALL_CONFIG
|
||||
`)
|
||||
os.Exit(2)
|
||||
@@ -71,7 +74,7 @@ func main() {
|
||||
fmt.Println(randomToken(24))
|
||||
case "list", "show", "events", "power", "bootdev", "recommission", "delete",
|
||||
"images", "profile", "assign", "unassign", "overlay", "render",
|
||||
"deploy", "release", "jobs", "job":
|
||||
"deploy", "release", "jobs", "job", "tag", "tags":
|
||||
if err := runClient(cmd, args); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
|
||||
+15
-1
@@ -129,6 +129,11 @@ func (a *App) routes() http.Handler {
|
||||
mux.HandleFunc("GET /api/v1/jobs/self", a.handleJobSelf) // job token auth inside
|
||||
mux.HandleFunc("POST /api/v1/jobs/self/progress", a.handleJobProgress) // job token auth inside
|
||||
mux.HandleFunc("POST /api/v1/phone-home/{token}", a.handlePhoneHome)
|
||||
// tags
|
||||
mux.HandleFunc("GET /api/v1/tags", a.adminAuth(a.handleTags))
|
||||
mux.HandleFunc("POST /api/v1/tags", a.adminAuth(a.handleTags))
|
||||
mux.HandleFunc("DELETE /api/v1/tags/{name}", a.adminAuth(a.handleTagDelete))
|
||||
mux.HandleFunc("POST /api/v1/machines/{id}/tags", a.adminAuth(a.handleMachineTags))
|
||||
// image files for the live boot (no auth: the kernel/initrd/squashfs must be fetchable)
|
||||
imgDir := a.cfg.ImageDir
|
||||
mux.Handle("GET /rollcall/", http.StripPrefix("/rollcall/", http.FileServer(http.Dir(imgDir))))
|
||||
@@ -239,6 +244,9 @@ type Summary struct {
|
||||
OSVersion *string `json:"os_version"`
|
||||
DeployedAt *time.Time `json:"deployed_at"`
|
||||
Job *Job `json:"job,omitempty"` // active deployment job, if any
|
||||
Tags []string `json:"tags"` // effective (manual + auto)
|
||||
ManualTags []string `json:"manual_tags"`
|
||||
AutoTags []string `json:"auto_tags"`
|
||||
}
|
||||
|
||||
func num(v any) float64 {
|
||||
@@ -256,7 +264,8 @@ func summarize(m *Machine) Summary {
|
||||
StorageBytes: num(dig(inv, "storage", "total_bytes")), NICs: listLen(inv["network"]),
|
||||
IBHCAs: listLen(dig(inv, "infiniband", "hcas")), GPUs: listLen(inv["gpus"]),
|
||||
Profile: m.Profile, NodeName: m.NodeName, MgmtIP: m.MgmtIP, IBIP: m.IBIP, BootedAt: m.BootedAt,
|
||||
OS: m.OS, OSVersion: m.OSVersion, DeployedAt: m.DeployedAt}
|
||||
OS: m.OS, OSVersion: m.OSVersion, DeployedAt: m.DeployedAt,
|
||||
Tags: effectiveTags(m), ManualTags: m.Tags, AutoTags: m.AutoTags}
|
||||
if m.BMC != nil {
|
||||
var b obj
|
||||
if json.Unmarshal(m.BMC, &b) == nil {
|
||||
@@ -397,6 +406,7 @@ func (a *App) handleCommission(w http.ResponseWriter, r *http.Request) {
|
||||
if full, err := a.store.Get(r.Context(), m.ID); err == nil {
|
||||
m = full // carries profile / node token when this is a re-commission of a stateless node
|
||||
}
|
||||
a.retag(r.Context(), []*Machine{m})
|
||||
if err := a.writePXEAny(r.Context(), m); err != nil {
|
||||
log.Printf("pxe write failed for %s: %v", m.ID, err)
|
||||
_ = a.store.AddEvent(r.Context(), m.ID, "pxe-write-failed", err.Error())
|
||||
@@ -431,8 +441,12 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
active, _ := a.store.ActiveJobs(r.Context())
|
||||
wantTag := r.URL.Query().Get("tag")
|
||||
out := make([]Summary, 0, len(ms))
|
||||
for _, m := range ms {
|
||||
if wantTag != "" && !hasTag(effectiveTags(m), wantTag) {
|
||||
continue
|
||||
}
|
||||
s := summarize(m)
|
||||
s.Job = active[m.ID]
|
||||
out = append(out, s)
|
||||
|
||||
+50
-10
@@ -23,6 +23,7 @@ import (
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -255,15 +256,22 @@ type NodeCtx struct {
|
||||
GPUs int
|
||||
IBGUIDs []string
|
||||
HostKey, HostKeyPub string // only filled for the node being rendered
|
||||
Tags []string
|
||||
Manufacturer, Product string
|
||||
Disks int
|
||||
NVMe bool
|
||||
PCIVendors []string // vendor ids seen on the PCI bus (e.g. 10de = NVIDIA, 15b3 = Mellanox)
|
||||
OS string
|
||||
}
|
||||
|
||||
type overlayCtx struct {
|
||||
Node NodeCtx
|
||||
Nodes []NodeCtx // all machines with a profile, sorted by name
|
||||
Profile Profile
|
||||
Vars map[string]any // cluster_vars overlaid with profile vars
|
||||
Server string
|
||||
Now string
|
||||
Node NodeCtx
|
||||
Nodes []NodeCtx // all machines with a profile, sorted by name
|
||||
Profile Profile
|
||||
Overlays []string // profile overlays plus those of the node's tags
|
||||
Vars map[string]any // cluster_vars overlaid with profile vars
|
||||
Server string
|
||||
Now string
|
||||
}
|
||||
|
||||
func atoi(v any) int {
|
||||
@@ -308,6 +316,21 @@ func nodeCtx(m *Machine) NodeCtx {
|
||||
n.IBGUIDs = append(n.IBGUIDs, g)
|
||||
}
|
||||
}
|
||||
n.Tags = effectiveTags(m)
|
||||
n.Manufacturer, n.Product, n.OS = deref(m.Manufacturer), deref(m.Product), deref(m.OS)
|
||||
n.Disks = int(num(dig(inv, "storage", "disk_count")))
|
||||
for _, d := range toList(dig(inv, "storage", "disks")) {
|
||||
if strings.HasPrefix(str(dig(d, "name")), "nvme") || str(dig(d, "tran")) == "nvme" {
|
||||
n.NVMe = true
|
||||
}
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, p := range toList(inv["pci"]) {
|
||||
if v := str(dig(p, "vendor_id")); v != "" && !seen[v] {
|
||||
seen[v] = true
|
||||
n.PCIVendors = append(n.PCIVendors, v)
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
@@ -332,7 +355,12 @@ var tmplFuncs = template.FuncMap{
|
||||
}
|
||||
return s
|
||||
},
|
||||
"json": func(v any) string { b, _ := json.Marshal(v); return string(b) },
|
||||
"json": func(v any) string { b, _ := json.Marshal(v); return string(b) },
|
||||
"has": hasTag,
|
||||
"contains": func(s, sub string) bool { return strings.Contains(s, sub) },
|
||||
"hasPrefix": strings.HasPrefix,
|
||||
"matches": func(pattern, s string) bool { ok, _ := regexp.MatchString(pattern, s); return ok },
|
||||
"gte": func(a, b int) bool { return a >= b },
|
||||
}
|
||||
|
||||
func (a *App) hostKeyFor(ctx context.Context, m *Machine) (priv, pub string, err error) {
|
||||
@@ -397,7 +425,19 @@ func (a *App) buildCtx(ctx context.Context, m *Machine) (*overlayCtx, error) {
|
||||
for k, v := range p.Vars {
|
||||
vars[k] = v
|
||||
}
|
||||
return &overlayCtx{Node: me, Nodes: nodes, Profile: *p, Vars: vars, Server: a.cfg.ServerURL,
|
||||
overlays := append([]string{}, p.Overlays...)
|
||||
if defs, err := a.store.Tags(ctx); err == nil {
|
||||
for _, t := range defs {
|
||||
if hasTag(me.Tags, t.Name) {
|
||||
for _, o := range t.Overlays {
|
||||
if !hasTag(overlays, o) {
|
||||
overlays = append(overlays, o)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return &overlayCtx{Node: me, Nodes: nodes, Profile: *p, Overlays: overlays, Vars: vars, Server: a.cfg.ServerURL,
|
||||
Now: time.Now().Format(time.RFC3339)}, nil
|
||||
}
|
||||
|
||||
@@ -413,7 +453,7 @@ func (a *App) renderOverlay(ctx context.Context, m *Machine, stage string) ([]by
|
||||
var buf bytes.Buffer
|
||||
tw := tar.NewWriter(&buf)
|
||||
seen := map[string]bool{}
|
||||
for _, name := range oc.Profile.Overlays {
|
||||
for _, name := range oc.Overlays {
|
||||
root := filepath.Join(a.cfg.OverlaysDir, name, stage)
|
||||
if st, err := os.Stat(root); err != nil || !st.IsDir() {
|
||||
continue
|
||||
@@ -518,7 +558,7 @@ func (a *App) renderFile(ctx context.Context, profile, stage, rel string) ([]byt
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, name := range p.Overlays {
|
||||
for _, name := range oc.Overlays {
|
||||
for _, cand := range []string{rel + ".tmpl", rel} {
|
||||
fp := filepath.Join(a.cfg.OverlaysDir, name, stage, cand)
|
||||
data, err := os.ReadFile(fp)
|
||||
|
||||
+25
-3
@@ -75,6 +75,18 @@ CREATE TABLE IF NOT EXISTS jobs (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS jobs_machine ON jobs(machine_id, created);
|
||||
CREATE INDEX IF NOT EXISTS jobs_token ON jobs(token_hash);
|
||||
CREATE TABLE IF NOT EXISTS tags (
|
||||
name text PRIMARY KEY,
|
||||
comment text,
|
||||
match text,
|
||||
cloud_config text,
|
||||
overlays jsonb NOT NULL DEFAULT '[]',
|
||||
created timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
ALTER TABLE machines ADD COLUMN IF NOT EXISTS tags jsonb NOT NULL DEFAULT '[]';
|
||||
ALTER TABLE machines ADD COLUMN IF NOT EXISTS auto_tags jsonb NOT NULL DEFAULT '[]';
|
||||
CREATE INDEX IF NOT EXISTS machines_tags ON machines USING gin(tags);
|
||||
CREATE INDEX IF NOT EXISTS machines_auto_tags ON machines USING gin(auto_tags);
|
||||
`
|
||||
|
||||
var errNotFound = errors.New("not found")
|
||||
@@ -103,6 +115,8 @@ type Machine struct {
|
||||
OS *string `json:"os"`
|
||||
OSVersion *string `json:"os_version"`
|
||||
DeployedAt *time.Time `json:"deployed_at"`
|
||||
Tags []string `json:"tags"` // manual
|
||||
AutoTags []string `json:"auto_tags"` // from tag match rules, recomputed at commissioning
|
||||
bmcSecret []byte
|
||||
nodeToken []byte
|
||||
hostKey []byte
|
||||
@@ -127,15 +141,15 @@ func openStore(ctx context.Context, url string) (*Store, error) {
|
||||
|
||||
const machineCols = `id, uuid, serial, hostname, manufacturer, product, state, first_seen, last_seen,
|
||||
commissioned_at, macs, bmc, bmc_secret, inventory, profile, node_name, mgmt_ip, ib_ip, node_token, host_key, booted_at,
|
||||
os, os_version, deployed_at`
|
||||
os, os_version, deployed_at, tags, auto_tags`
|
||||
|
||||
func scanMachine(row pgx.Row, full bool) (*Machine, error) {
|
||||
m := &Machine{}
|
||||
var macs []byte
|
||||
var macs, tags, auto []byte
|
||||
err := row.Scan(&m.ID, &m.UUID, &m.Serial, &m.Hostname, &m.Manufacturer, &m.Product, &m.State,
|
||||
&m.FirstSeen, &m.LastSeen, &m.CommissionedAt, &macs, &m.BMC, &m.bmcSecret, &m.Inventory,
|
||||
&m.Profile, &m.NodeName, &m.MgmtIP, &m.IBIP, &m.nodeToken, &m.hostKey, &m.BootedAt,
|
||||
&m.OS, &m.OSVersion, &m.DeployedAt)
|
||||
&m.OS, &m.OSVersion, &m.DeployedAt, &tags, &auto)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, errNotFound
|
||||
}
|
||||
@@ -143,6 +157,14 @@ func scanMachine(row pgx.Row, full bool) (*Machine, error) {
|
||||
return nil, err
|
||||
}
|
||||
_ = json.Unmarshal(macs, &m.MACs)
|
||||
_ = json.Unmarshal(tags, &m.Tags)
|
||||
_ = json.Unmarshal(auto, &m.AutoTags)
|
||||
if m.Tags == nil {
|
||||
m.Tags = []string{}
|
||||
}
|
||||
if m.AutoTags == nil {
|
||||
m.AutoTags = []string{}
|
||||
}
|
||||
if !full {
|
||||
m.Inventory = nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
package main
|
||||
|
||||
// Machine tags, MAAS-style. A tag is a name plus optional
|
||||
// match Go template over the node context that renders "true" for machines
|
||||
// that should carry the tag automatically (re-evaluated at commissioning
|
||||
// and whenever a tag definition changes);
|
||||
// cloud_config a #cloud-config (or #!/bin/sh) fragment merged into the user-data of
|
||||
// every deployment of a tagged machine (e.g. gpu -> NVIDIA packages);
|
||||
// overlays extra overlay directories applied to tagged stateless nodes.
|
||||
// Manual tags are set per machine; effective tags = manual ∪ auto.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/textproto"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"text/template"
|
||||
"time"
|
||||
)
|
||||
|
||||
type TagDef struct {
|
||||
Name string `json:"name"`
|
||||
Comment string `json:"comment"`
|
||||
Match string `json:"match"`
|
||||
CloudConfig string `json:"cloud_config"`
|
||||
Overlays []string `json:"overlays"`
|
||||
Created time.Time `json:"created"`
|
||||
Machines int `json:"machines,omitempty"` // filled by the list handler
|
||||
}
|
||||
|
||||
var tagNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,63}$`)
|
||||
|
||||
// ---------------------------------------------------------------- store
|
||||
|
||||
func (s *Store) Tags(ctx context.Context) ([]TagDef, error) {
|
||||
rows, err := s.pool.Query(ctx, `SELECT name, coalesce(comment,''), coalesce(match,''), coalesce(cloud_config,''), overlays, created FROM tags ORDER BY name`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := []TagDef{}
|
||||
for rows.Next() {
|
||||
var t TagDef
|
||||
var ov []byte
|
||||
if err := rows.Scan(&t.Name, &t.Comment, &t.Match, &t.CloudConfig, &ov, &t.Created); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
_ = json.Unmarshal(ov, &t.Overlays)
|
||||
if t.Overlays == nil {
|
||||
t.Overlays = []string{}
|
||||
}
|
||||
out = append(out, t)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) PutTag(ctx context.Context, t TagDef) error {
|
||||
if t.Overlays == nil {
|
||||
t.Overlays = []string{}
|
||||
}
|
||||
ov, _ := json.Marshal(t.Overlays)
|
||||
_, err := s.pool.Exec(ctx, `INSERT INTO tags(name, comment, match, cloud_config, overlays) VALUES ($1,$2,$3,$4,$5)
|
||||
ON CONFLICT (name) DO UPDATE SET comment=$2, match=$3, cloud_config=$4, overlays=$5`,
|
||||
t.Name, t.Comment, t.Match, t.CloudConfig, ov)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) DeleteTag(ctx context.Context, name string) error {
|
||||
tag, err := s.pool.Exec(ctx, `DELETE FROM tags WHERE name=$1`, name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return errNotFound
|
||||
}
|
||||
// drop it from every machine (manual and auto)
|
||||
_, err = s.pool.Exec(ctx, `UPDATE machines SET tags = tags - $1, auto_tags = auto_tags - $1`, name)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) SetMachineTags(ctx context.Context, id string, manual, auto []string) error {
|
||||
mb, _ := json.Marshal(sortedUnique(manual))
|
||||
ab, _ := json.Marshal(sortedUnique(auto))
|
||||
tag, err := s.pool.Exec(ctx, `UPDATE machines SET tags=$2, auto_tags=$3 WHERE id=$1`, id, mb, ab)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return errNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func sortedUnique(l []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := []string{}
|
||||
for _, s := range l {
|
||||
s = strings.TrimSpace(s)
|
||||
if s != "" && !seen[s] {
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func hasTag(list []string, t string) bool {
|
||||
for _, x := range list {
|
||||
if x == t {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// effectiveTags is manual ∪ auto, sorted.
|
||||
func effectiveTags(m *Machine) []string {
|
||||
return sortedUnique(append(append([]string{}, m.Tags...), m.AutoTags...))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- matching
|
||||
|
||||
// matchTag renders the tag's match template against the node context; "true" (after
|
||||
// trimming) means the tag applies.
|
||||
func matchTag(t TagDef, n NodeCtx) (bool, error) {
|
||||
if strings.TrimSpace(t.Match) == "" {
|
||||
return false, nil
|
||||
}
|
||||
tmpl, err := template.New(t.Name).Funcs(tmplFuncs).Option("missingkey=zero").Parse(t.Match)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("tag %s: %w", t.Name, err)
|
||||
}
|
||||
var out bytes.Buffer
|
||||
if err := tmpl.Execute(&out, map[string]any{"Node": n}); err != nil {
|
||||
return false, fmt.Errorf("tag %s: %w", t.Name, err)
|
||||
}
|
||||
return strings.TrimSpace(out.String()) == "true", nil
|
||||
}
|
||||
|
||||
// autoTagsFor evaluates every tag with a match rule against a machine.
|
||||
func autoTagsFor(defs []TagDef, m *Machine) ([]string, error) {
|
||||
n := nodeCtx(m)
|
||||
var out []string
|
||||
var firstErr error
|
||||
for _, t := range defs {
|
||||
ok, err := matchTag(t, n)
|
||||
if err != nil && firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
if ok {
|
||||
out = append(out, t.Name)
|
||||
}
|
||||
}
|
||||
return sortedUnique(out), firstErr
|
||||
}
|
||||
|
||||
// retag recomputes auto tags for the given machines (all when nil).
|
||||
func (a *App) retag(ctx context.Context, only []*Machine) {
|
||||
defs, err := a.store.Tags(ctx)
|
||||
if err != nil {
|
||||
log.Printf("retag: %v", err)
|
||||
return
|
||||
}
|
||||
ms := only
|
||||
if ms == nil {
|
||||
if ms, err = a.store.List(ctx); err != nil {
|
||||
log.Printf("retag: %v", err)
|
||||
return
|
||||
}
|
||||
}
|
||||
for _, m := range ms {
|
||||
auto, err := autoTagsFor(defs, m)
|
||||
if err != nil {
|
||||
log.Printf("retag %s: %v", m.ID, err)
|
||||
}
|
||||
if strings.Join(auto, ",") == strings.Join(m.AutoTags, ",") {
|
||||
continue
|
||||
}
|
||||
if err := a.store.SetMachineTags(ctx, m.ID, m.Tags, auto); err != nil {
|
||||
log.Printf("retag %s: %v", m.ID, err)
|
||||
continue
|
||||
}
|
||||
_ = a.store.AddEvent(ctx, m.ID, "auto-tags", strings.Join(auto, " "))
|
||||
m.AutoTags = auto
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- cloud-init fragments
|
||||
|
||||
// tagFragments returns the cloud_config fragments of the machine's effective tags.
|
||||
func (a *App) tagFragments(ctx context.Context, m *Machine) ([]TagDef, error) {
|
||||
defs, err := a.store.Tags(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
eff := effectiveTags(m)
|
||||
var out []TagDef
|
||||
for _, t := range defs {
|
||||
if hasTag(eff, t.Name) && strings.TrimSpace(t.CloudConfig) != "" {
|
||||
out = append(out, t)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// multipartUserData packs the base cloud-config and tag fragments into the
|
||||
// multipart/mixed user-data cloud-init expects. Cloud-config parts get a merge
|
||||
// directive so lists (packages, runcmd, write_files) append instead of replace.
|
||||
func multipartUserData(base string, frags []TagDef) string {
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
add := func(name, body string) {
|
||||
ctype := "text/cloud-config"
|
||||
if strings.HasPrefix(body, "#!") {
|
||||
ctype = "text/x-shellscript"
|
||||
} else if strings.HasPrefix(body, "#cloud-boothook") {
|
||||
ctype = "text/cloud-boothook"
|
||||
} else if !strings.HasPrefix(body, "#cloud-config") {
|
||||
body = "#cloud-config\n" + body
|
||||
}
|
||||
if ctype == "text/cloud-config" && !strings.Contains(body, "merge_how") && !strings.Contains(body, "merge_type") {
|
||||
body = strings.Replace(body, "#cloud-config", "#cloud-config\nmerge_how: \"list(append)+dict(recurse_array,no_replace)+str()\"", 1)
|
||||
}
|
||||
h := textproto.MIMEHeader{}
|
||||
h.Set("Content-Type", ctype+`; charset="us-ascii"`)
|
||||
h.Set("Content-Disposition", `attachment; filename="`+name+`"`)
|
||||
h.Set("MIME-Version", "1.0")
|
||||
p, _ := w.CreatePart(h)
|
||||
_, _ = p.Write([]byte(body))
|
||||
if !strings.HasSuffix(body, "\n") {
|
||||
_, _ = p.Write([]byte("\n"))
|
||||
}
|
||||
}
|
||||
add("rollcall-base.cfg", base)
|
||||
for _, t := range frags {
|
||||
add("tag-"+t.Name+".cfg", t.CloudConfig)
|
||||
}
|
||||
_ = w.Close()
|
||||
head := "Content-Type: multipart/mixed; boundary=\"" + w.Boundary() + "\"\nMIME-Version: 1.0\nNumber-Attachments: " + fmt.Sprint(len(frags)+1) + "\n\n"
|
||||
return head + buf.String()
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- handlers
|
||||
|
||||
func (a *App) handleTags(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.Method {
|
||||
case http.MethodGet:
|
||||
defs, err := a.store.Tags(r.Context())
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
ms, _ := a.store.List(r.Context())
|
||||
for i := range defs {
|
||||
for _, m := range ms {
|
||||
if hasTag(effectiveTags(m), defs[i].Name) {
|
||||
defs[i].Machines++
|
||||
}
|
||||
}
|
||||
}
|
||||
writeJSON(w, 200, defs)
|
||||
case http.MethodPost:
|
||||
var t TagDef
|
||||
if err := json.NewDecoder(r.Body).Decode(&t); err != nil || !tagNameRe.MatchString(t.Name) {
|
||||
fail(w, 400, fmt.Errorf("name is required: lowercase letters, digits, . _ -"))
|
||||
return
|
||||
}
|
||||
if _, err := template.New(t.Name).Funcs(tmplFuncs).Parse(t.Match); err != nil {
|
||||
fail(w, 400, fmt.Errorf("match: %w", err))
|
||||
return
|
||||
}
|
||||
for _, o := range t.Overlays {
|
||||
if strings.ContainsAny(o, "/ \t") {
|
||||
fail(w, 400, fmt.Errorf("overlay names must be plain identifiers"))
|
||||
return
|
||||
}
|
||||
}
|
||||
if err := a.store.PutTag(r.Context(), t); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
a.retag(r.Context(), nil)
|
||||
a.rewriteTaggedPXE(r.Context(), t.Name)
|
||||
writeJSON(w, 200, t)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) handleTagDelete(w http.ResponseWriter, r *http.Request) {
|
||||
if err := a.store.DeleteTag(r.Context(), r.PathValue("name")); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, obj{"deleted": r.PathValue("name")})
|
||||
}
|
||||
|
||||
// handleMachineTags: {"add":["gpu"],"remove":["staging"]} or {"set":[...]} for manual tags.
|
||||
func (a *App) handleMachineTags(w http.ResponseWriter, r *http.Request) {
|
||||
m, err := a.store.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Add, Remove, Set []string
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
manual := m.Tags
|
||||
if req.Set != nil {
|
||||
manual = req.Set
|
||||
}
|
||||
manual = append(append([]string{}, manual...), req.Add...)
|
||||
for _, name := range append(req.Add, req.Set...) {
|
||||
if !tagNameRe.MatchString(name) {
|
||||
fail(w, 400, fmt.Errorf("bad tag name %q", name))
|
||||
return
|
||||
}
|
||||
}
|
||||
var kept []string
|
||||
for _, t := range manual {
|
||||
if !hasTag(req.Remove, t) {
|
||||
kept = append(kept, t)
|
||||
}
|
||||
}
|
||||
if err := a.store.SetMachineTags(r.Context(), m.ID, kept, m.AutoTags); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
// tags without a definition are fine (free-form), but create the definition so they show up in lists
|
||||
defs, _ := a.store.Tags(r.Context())
|
||||
for _, t := range kept {
|
||||
known := false
|
||||
for _, d := range defs {
|
||||
if d.Name == t {
|
||||
known = true
|
||||
}
|
||||
}
|
||||
if !known {
|
||||
_ = a.store.PutTag(r.Context(), TagDef{Name: t, Comment: "created from machine " + m.ID})
|
||||
}
|
||||
}
|
||||
m, _ = a.store.Get(r.Context(), m.ID)
|
||||
_ = a.store.AddEvent(r.Context(), m.ID, "tags", strings.Join(effectiveTags(m), " "))
|
||||
if m.Profile != nil { // overlays may have changed
|
||||
_ = a.writePXEAny(r.Context(), m)
|
||||
}
|
||||
writeJSON(w, 200, obj{"id": m.ID, "tags": m.Tags, "auto_tags": m.AutoTags, "effective": effectiveTags(m)})
|
||||
}
|
||||
|
||||
// rewriteTaggedPXE is a no-op placeholder for PXE entries (tags do not change the kernel
|
||||
// line today); kept so overlay changes can hook in later.
|
||||
func (a *App) rewriteTaggedPXE(ctx context.Context, tag string) {}
|
||||
+23
-2
@@ -37,6 +37,8 @@ function setStatus(msg, bad) { const el = $('#status'); el.textContent = msg; el
|
||||
|
||||
let profiles = [];
|
||||
let osImages = [];
|
||||
let tagDefs = [];
|
||||
const tagPills = (m, editable) => (m.tags || []).map(t => { const auto = (m.auto_tags || []).includes(t) && !(m.manual_tags || []).includes(t); return `<span class="pill tag${auto ? ' auto' : ''}" title="${auto ? 'auto (match rule)' : 'manual'}">${esc(t)}${editable && !auto ? ` <a href="#" data-rmmtag="${esc(t)}" title="remove">×</a>` : ''}</span>`; }).join(' ');
|
||||
async function loadList() {
|
||||
if (!token) return askToken();
|
||||
try {
|
||||
@@ -57,6 +59,11 @@ async function loadProfiles() {
|
||||
const sel = $('#assign-profile'); const cur = sel.value;
|
||||
sel.innerHTML = '<option value="">profile…</option>' + profiles.map(p => `<option value="${esc(p.name)}">${esc(p.name)} (${esc(p.image)})</option>`).join('');
|
||||
sel.value = cur;
|
||||
tagDefs = await api('GET', '/api/v1/tags');
|
||||
$('#tags').innerHTML = table(['Tag', 'Machines', 'Match', 'Overlays', 'Cloud-config', 'Comment', ''], tagDefs.map(t => [`<span class="pill">${esc(t.name)}</span>`, t.machines || 0, `<span class="mono">${esc(t.match || '')}</span>`, esc((t.overlays || []).join(', ')), t.cloud_config ? `${t.cloud_config.length} B` : '', esc(t.comment || ''), `<button class="btn" data-edittag="${esc(t.name)}">edit</button> <button class="btn danger" data-rmtag="${esc(t.name)}">rm</button>`]));
|
||||
document.querySelectorAll('[data-rmtag]').forEach(b => b.onclick = async () => { if (confirm(`Delete tag ${b.dataset.rmtag}? It is removed from every machine.`)) { try { await api('DELETE', '/api/v1/tags/' + b.dataset.rmtag); } catch (e) { alert(e.message); } loadProfiles(); } });
|
||||
document.querySelectorAll('[data-edittag]').forEach(b => b.onclick = () => { const t = tagDefs.find(x => x.name === b.dataset.edittag); const f = $('#tag-form'); f.name.value = t.name; f.comment.value = t.comment || ''; f.match.value = t.match || ''; f.overlays.value = (t.overlays || []).join(','); f.cloud_config.value = t.cloud_config || ''; f.scrollIntoView(); });
|
||||
$('#tag-names').innerHTML = tagDefs.map(t => `<option value="${esc(t.name)}">`).join('');
|
||||
osImages = await api('GET', '/api/v1/os');
|
||||
$('#osimages').innerHTML = table(['OS', 'Current version', 'Family', 'Compressed', 'Disk needed', 'Imported', 'Track', 'Versions', 'Status'],
|
||||
osImages.map(i => { const c = i.current || {}; const s = i.source || {}; return [esc(i.name), `<span class="mono">${esc(c.version || '-')}</span>`, esc(s.family || c.family), c.size ? human(c.size) : '', c.raw_size ? human(c.raw_size) : '', c.imported ? esc(new Date(c.imported).toLocaleString()) : '', esc(s.track || ''), (i.versions || []).length, i.warning ? `<span class="bad">${esc(i.warning)}</span>` : '<span class="ok">ok</span>']; }));
|
||||
@@ -76,7 +83,7 @@ $('#profile-form').onsubmit = async ev => {
|
||||
|
||||
function renderList() {
|
||||
const q = $('#filter').value.toLowerCase();
|
||||
const rows = machines.filter(m => !q || JSON.stringify([m.id, m.state, m.manufacturer, m.product, m.serial, m.macs, m.hostname]).toLowerCase().includes(q));
|
||||
const rows = machines.filter(m => !q || JSON.stringify([m.id, m.state, m.manufacturer, m.product, m.serial, m.macs, m.hostname, m.node_name, m.os, m.profile, m.tags]).toLowerCase().includes(q));
|
||||
$('#count').textContent = `${rows.length} / ${machines.length}`;
|
||||
$('#empty').hidden = machines.length > 0;
|
||||
$('#machines tbody').innerHTML = rows.map(m => {
|
||||
@@ -87,7 +94,7 @@ function renderList() {
|
||||
const what = m.profile ? esc(m.profile) : m.os ? `${esc(m.os)}<br><span class="muted mono">${esc(m.os_version || '')}</span>` : '';
|
||||
return `<tr data-id="${esc(m.id)}">
|
||||
<td class="mono">${esc(m.id)}</td><td><span class="pill ${esc(m.state)}">${state}</span></td>
|
||||
<td>${esc(m.node_name || '')}${m.mgmt_ip ? `<br><span class="muted mono">${esc(m.mgmt_ip)}</span>` : ''}</td><td>${what}</td>
|
||||
<td>${esc(m.node_name || '')}${m.mgmt_ip ? `<br><span class="muted mono">${esc(m.mgmt_ip)}</span>` : ''}</td><td>${what}</td><td>${tagPills(m, false)}</td>
|
||||
<td>${esc((m.manufacturer || '') + ' ' + (m.product || ''))}</td><td class="mono">${esc(m.serial || '')}</td>
|
||||
<td class="num">${esc(m.cpus)}</td><td class="num">${human(m.memory_bytes)}</td><td class="num">${m.disks | 0}</td>
|
||||
<td class="num">${human(m.storage_bytes)}</td><td class="num">${m.nics}</td><td class="num">${m.ib_hcas}</td><td class="num">${m.gpus}</td>
|
||||
@@ -120,6 +127,8 @@ function renderDetail(d) {
|
||||
diskSel.innerHTML = '<option value="">disk: auto (smallest that fits)</option>' + ((st.disks || []).map(x => `<option value="${esc(x.path || '/dev/' + x.name)}">${esc(x.name)} ${human(x.size)} ${esc(x.model || '')}</option>`).join(''));
|
||||
diskSel.value = curDisk;
|
||||
$('#job-cancel').hidden = !s.job;
|
||||
$('#d-tags').innerHTML = tagPills(s, true) || '<span class="muted">none</span>';
|
||||
document.querySelectorAll('[data-rmmtag]').forEach(a => a.onclick = ev => { ev.preventDefault(); action(async () => { const r = await api('POST', `/api/v1/machines/${current}/tags`, { remove: [a.dataset.rmmtag] }); await loadDetail(current); return 'tags: ' + r.effective.join(' '); }, 'untag'); });
|
||||
const cards = [];
|
||||
const job = s.job || (d.jobs || [])[0];
|
||||
if (s.os || job) {
|
||||
@@ -218,6 +227,18 @@ const doAssign = boot => action(async () => {
|
||||
$('#assign').onclick = () => doAssign(false);
|
||||
$('#assign-boot').onclick = () => { if (confirm('Assign and power-cycle this machine into the stateless image?')) doAssign(true); };
|
||||
$('#unassign').onclick = () => { if (confirm('Remove the profile? The node name and IP are released.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/unassign`); await loadDetail(current); return 'state ' + r.state; }, 'unassign'); };
|
||||
$('#tag-add').onkeydown = ev => {
|
||||
if (ev.key !== 'Enter') return;
|
||||
ev.preventDefault();
|
||||
const t = $('#tag-add').value.trim().replace(/^\+/, ''); if (!t) return;
|
||||
action(async () => { const r = await api('POST', `/api/v1/machines/${current}/tags`, { add: [t] }); $('#tag-add').value = ''; await loadDetail(current); loadProfiles(); return 'tags: ' + r.effective.join(' '); }, 'tag');
|
||||
};
|
||||
$('#tag-form').onsubmit = async ev => {
|
||||
ev.preventDefault();
|
||||
const f = new FormData(ev.target);
|
||||
const body = { name: f.get('name'), comment: f.get('comment'), match: f.get('match'), cloud_config: f.get('cloud_config'), overlays: (f.get('overlays') || '').split(',').map(s => s.trim()).filter(Boolean) };
|
||||
try { await api('POST', '/api/v1/tags', body); $('#tag-out').textContent = 'saved (auto tags re-evaluated)'; loadList(); } catch (e) { $('#tag-out').textContent = e.message; }
|
||||
};
|
||||
$('#deploy').onclick = () => {
|
||||
const os = $('#deploy-os').value; if (!os) { $('#action-out').textContent = 'pick an OS image'; return; }
|
||||
const disk = $('#deploy-disk').value;
|
||||
|
||||
@@ -33,7 +33,7 @@
|
||||
</div>
|
||||
<table id="machines">
|
||||
<thead><tr>
|
||||
<th>ID</th><th>State</th><th>Node</th><th>Profile / OS</th><th>Model</th><th>Serial</th><th class="num">CPU</th><th class="num">RAM</th>
|
||||
<th>ID</th><th>State</th><th>Node</th><th>Profile / OS</th><th>Tags</th><th>Model</th><th>Serial</th><th class="num">CPU</th><th class="num">RAM</th>
|
||||
<th class="num">Disks</th><th class="num">Storage</th><th class="num">NIC</th><th class="num">IB</th><th class="num">GPU</th>
|
||||
<th>BMC</th><th>Last seen</th>
|
||||
</tr></thead>
|
||||
@@ -45,6 +45,16 @@
|
||||
<summary>Provisioning: OS images (deployment), profiles & root images (stateless)</summary>
|
||||
<div class="grid" style="margin-top:10px">
|
||||
<div class="card wide"><h3>OS images for deployment <span class="muted" style="text-transform:none">· refreshed nightly by <code>rollcall os sync</code></span></h3><div id="osimages"></div></div>
|
||||
<div class="card wide"><h3>Tags <span class="muted" style="text-transform:none">· auto rules are re-evaluated at commissioning; cloud-config fragments apply at deployment; overlays to stateless nodes</span></h3><div id="tags"></div>
|
||||
<form id="tag-form" class="kv" style="grid-template-columns:max-content 1fr;align-items:start;margin-top:10px">
|
||||
<label>Name</label><input name="name" required placeholder="gpu">
|
||||
<label>Comment</label><input name="comment" placeholder="NVIDIA GPU nodes">
|
||||
<label>Match</label><input name="match" placeholder="{{ gt .Node.GPUs 0 }} (empty = manual only)">
|
||||
<label>Overlays</label><input name="overlays" placeholder="gpu-slurm">
|
||||
<label>Cloud-config</label><textarea name="cloud_config" rows="6" placeholder="#cloud-config packages: [nvidia-driver]" style="font-family:var(--mono);font-size:12px;width:100%;border:1px solid var(--line);border-radius:6px;background:var(--panel);color:var(--fg);padding:6px"></textarea>
|
||||
<span></span><span><button class="btn primary" type="submit">Save tag</button> <span id="tag-out" class="muted"></span></span>
|
||||
</form>
|
||||
</div>
|
||||
<div class="card"><h3>Profiles</h3><div id="profiles"></div></div>
|
||||
<div class="card"><h3>Stateless root images</h3><div id="images"></div></div>
|
||||
<div class="card wide"><h3>Add / update profile</h3>
|
||||
@@ -97,6 +107,11 @@
|
||||
<button id="release" class="btn warn" title="wipe the disk signatures and return the machine to ready">Release</button>
|
||||
<button id="job-cancel" class="btn" hidden>Cancel job</button>
|
||||
<span class="sep"></span>
|
||||
<span class="muted">Tags:</span>
|
||||
<span id="d-tags"></span>
|
||||
<input id="tag-add" type="text" placeholder="+tag" style="width:90px" list="tag-names">
|
||||
<datalist id="tag-names"></datalist>
|
||||
<span class="sep"></span>
|
||||
<button id="recommission" class="btn warn">Re-commission</button>
|
||||
<button id="recommission-now" class="btn warn" title="remove per-MAC PXE config, set bootdev pxe and power-cycle via BMC">Re-commission now</button>
|
||||
<button id="delete" class="btn danger">Delete</button>
|
||||
|
||||
@@ -35,6 +35,9 @@ tbody tr { cursor: pointer; } tbody tr:hover { background: var(--hover); }
|
||||
.pill { display: inline-block; padding: 2px 8px; border-radius: 999px; background: var(--pill); font-size: 12px; }
|
||||
.pill.ready, .pill.booted, .pill.deployed { color: var(--ok); } .pill.recommission, .pill.new, .pill.stateless, .pill.deploying, .pill.releasing { color: var(--warn); }
|
||||
.pill.deploy-failed, .pill.release-failed { color: var(--danger); }
|
||||
.pill.tag { background: var(--hover); border: 1px solid var(--line); font-family: var(--mono); font-size: 11px; }
|
||||
.pill.tag.auto { border-style: dashed; color: var(--muted); }
|
||||
.pill.tag a { color: var(--danger); text-decoration: none; }
|
||||
.bar { position: relative; height: 10px; background: var(--pill); border-radius: 999px; overflow: hidden; margin: 6px 0; }
|
||||
.bar > span { position: absolute; inset: 0 auto 0 0; background: var(--accent); transition: width .5s; }
|
||||
.actions select, .actions input[type=text] { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; }
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Tag fragments
|
||||
|
||||
Each file here is a cloud-init fragment that a tag can carry. When a machine
|
||||
with that tag is deployed, the fragment is added to the generated user-data as
|
||||
a separate MIME part (cloud-config parts merge with `list(append)`, so
|
||||
`packages`, `runcmd` and `write_files` accumulate; a part starting with `#!`
|
||||
runs as a script).
|
||||
|
||||
| File | Tag | Use |
|
||||
|---|---|---|
|
||||
| `gpu-nvidia.yaml` | `gpu` (auto: `{{ gt .Node.GPUs 0 }}`) | NVIDIA driver from the distro's preferred source, reboot when done |
|
||||
|
||||
Register: `rollcall tags add gpu -match '{{ gt .Node.GPUs 0 }}' -cloud-config deploy/tags/gpu-nvidia.yaml -comment "NVIDIA GPUs"`.
|
||||
|
||||
Match templates see `.Node`: `GPUs`, `IBGUIDs`, `CPUs`, `Sockets`, `MemMB`,
|
||||
`Disks`, `NVMe`, `PCIVendors` (e.g. `10de` NVIDIA, `15b3` Mellanox), `Manufacturer`,
|
||||
`Product`, `MACs`, `Tags`, `OS`. Helpers: `has`, `contains`, `hasPrefix`, `matches`,
|
||||
`gte`, plus Go's `eq/gt/lt/and/or/not`. Examples:
|
||||
|
||||
```
|
||||
{{ gt .Node.GPUs 0 }}
|
||||
{{ has .Node.PCIVendors "15b3" }}
|
||||
{{ and (gte .Node.MemMB 500000) (gt .Node.Sockets 1) }}
|
||||
{{ matches "(?i)PowerEdge R7" .Node.Product }}
|
||||
```
|
||||
@@ -0,0 +1,48 @@
|
||||
#cloud-config
|
||||
# Rollcall tag fragment: NVIDIA driver + container toolkit on GPU nodes.
|
||||
# Register with: rollcall tags add gpu -match '{{ gt .Node.GPUs 0 }}' -cloud-config deploy/tags/gpu-nvidia.yaml
|
||||
# Merged into the deployment user-data of every machine tagged "gpu"; distro is detected at runtime.
|
||||
packages:
|
||||
- pciutils
|
||||
- curl
|
||||
- gnupg
|
||||
write_files:
|
||||
- path: /usr/local/sbin/rollcall-nvidia-install.sh
|
||||
permissions: "0755"
|
||||
content: |
|
||||
#!/bin/sh
|
||||
# Installs the NVIDIA driver from the distribution's preferred source. Logs to /var/log/rollcall-nvidia.log
|
||||
set -eu
|
||||
exec >>/var/log/rollcall-nvidia.log 2>&1
|
||||
. /etc/os-release
|
||||
echo "== $(date -Is) nvidia install on $PRETTY_NAME"
|
||||
lspci -nn | grep -i 10de || { echo "no NVIDIA device"; exit 0; }
|
||||
case "$ID" in
|
||||
debian)
|
||||
# contrib + non-free for nvidia-driver (kernel headers for DKMS)
|
||||
sed -i 's/^\(deb .* main\)$/\1 contrib non-free non-free-firmware/' /etc/apt/sources.list 2>/dev/null || true
|
||||
if [ -f /etc/apt/sources.list.d/debian.sources ]; then sed -i 's/^Components: main$/Components: main contrib non-free non-free-firmware/' /etc/apt/sources.list.d/debian.sources; fi
|
||||
apt-get update
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y "linux-headers-$(uname -r)" nvidia-driver nvidia-smi firmware-misc-nonfree
|
||||
;;
|
||||
ubuntu)
|
||||
apt-get update
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-drivers-common
|
||||
ubuntu-drivers install --gpgpu || ubuntu-drivers autoinstall
|
||||
;;
|
||||
rocky|almalinux|centos|rhel)
|
||||
rel=$(echo "$VERSION_ID" | cut -d. -f1)
|
||||
dnf -y install epel-release || dnf -y install "https://dl.fedoraproject.org/pub/epel/epel-release-latest-$rel.noarch.rpm"
|
||||
dnf config-manager --add-repo "https://developer.download.nvidia.com/compute/cuda/repos/rhel$rel/x86_64/cuda-rhel$rel.repo"
|
||||
dnf -y install kernel-devel-"$(uname -r)" kernel-headers-"$(uname -r)" dkms
|
||||
dnf -y module install nvidia-driver:latest-dkms || dnf -y install nvidia-driver nvidia-driver-cuda
|
||||
;;
|
||||
*) echo "unsupported distro $ID"; exit 1 ;;
|
||||
esac
|
||||
echo "== done; nvidia-smi after reboot"
|
||||
runcmd:
|
||||
- [ /usr/local/sbin/rollcall-nvidia-install.sh ]
|
||||
power_state:
|
||||
mode: reboot
|
||||
condition: test -e /var/log/rollcall-nvidia.log && grep -q "== done" /var/log/rollcall-nvidia.log
|
||||
message: "rollcall: rebooting to load the NVIDIA driver"
|
||||
Reference in new issue
Block a user