Region as certificate authority, BMC power drivers, IPv6
CA (llmgw-style) - region creates its CA (ECDSA P-256) and serves API/UI/images over HTTPS on tls_listen with a certificate for every name/address it has (reissued on change or a month before expiry); admin_http policy (allow|loopback|deny) for the admin API over plain HTTP; /ca.pem - rack controllers enroll with one-time join codes (CA fingerprint + secret + expiry + address): rollcall controller join-code, rollcall rack join; mutual TLS from then on, identity from the certificate (CN rack:NAME, serial on record), removal revokes, renewal a month before expiry; the shared rack_secret stays as a legacy fallback - clients: CA from the server config, ROLLCALL_CA, or trusted on first connect in the console after showing the fingerprint (kept in client.json); deployed machines trust it via cloud-init ca_certs (allowed on RHEL-family images), overlays get .CA; rollcall ca show|issue Power drivers - ipmi (cipher/privilege/interface), redfish (system discovery, ResetType from AllowableValues or ActionInfo, boot override with If-Match), idrac, ilo, supermicro, xcc, openbmc, proxmox (API token), manual; executed on the rack controller; Redfish auto-detection after commissioning; operator settings survive recommissioning; rollcall bmc, BMC editing in web UI and console - lab: pve_redfish.py Redfish emulator per Proxmox VM (iDRAC, iLO, generic profiles) IPv6 - IPv6 subnets and IPAM (netip), dual-stack default layout (all subnets of the boot VLAN), per-family default routes, accept-ra off where addresses are planned, dnsmasq RA/DHCPv6 with router lifetime 0 unless the rack is the gateway, AAAA host records from IPAM, IPv6 listen addresses and URLs, rack interfaces and region certificate with IPv6, agent reports IPv6 Tests for CA/join/pinning/CSR signing, Redfish (iDRAC, iLO fallbacks, probe), Proxmox, manual, IPv6 candidates, dual-stack plans and dnsmasq IPv6. Verified in the lab: rack2 enrolled and running on mutual TLS only, revocation and single-use codes; iLO auto-detected through rack2, iDRAC and Proxmox drivers, wrong password not saved; Debian 13 (bond+VLAN), Rocky 9 and AlmaLinux 9 dual-stack with Redfish-driven deployments; CA trusted on Debian and AlmaLinux; admin CLI over HTTPS on IPv4 and IPv6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
736a7e38a3
commit
c8465c235a
29 files changed
+3156
-241
No files matched your search
@@ -38,6 +38,16 @@ publish a tag that has no section here.
|
||||
if dnsmasq does not come back.
|
||||
- Web UI: Network page (rack controllers, subnets with ranges and usage, layouts), network tab per
|
||||
machine, layout action; console: Network page, network section, layout action.
|
||||
- Certificate authority on the region (llmgw-style): HTTPS listener with an auto-renewed region
|
||||
certificate, `admin_http` policy, `/ca.pem`; rack controllers enroll with one-time join codes
|
||||
(`rollcall controller join-code`, `rollcall rack join`) and use mutual TLS, renewing their
|
||||
certificate themselves; removal revokes; deployed machines and overlays trust the CA;
|
||||
`rollcall ca show|issue`; the console trusts a CA on first connect after showing its fingerprint.
|
||||
- Power drivers: ipmi, redfish, idrac, ilo, supermicro, xcc, openbmc, proxmox, manual, run on the
|
||||
rack controller; Redfish auto-detection after commissioning; `rollcall bmc`, BMC editing in the
|
||||
web UI and console. Lab Redfish emulator with iDRAC/iLO/generic profiles.
|
||||
- IPv6: subnets, IPAM, dual-stack default layout, per-family default routes, dnsmasq RA/DHCPv6,
|
||||
AAAA host records, IPv6 listen addresses and URLs, IPv6 in the commissioning inventory.
|
||||
- `rollcall version`; CD-built binary carries `git describe`.
|
||||
- Release notes from `CHANGELOG.md`.
|
||||
|
||||
|
||||
@@ -380,14 +380,11 @@ controller (`rack_name`, default its hostname), which is all a single-server
|
||||
installation needs. Remote ones:
|
||||
|
||||
```sh
|
||||
# region: config.json gets "rack_secret": "<rollcall keygen>"
|
||||
# rack host (Debian/Ubuntu), from deploy/ with the binary next to it:
|
||||
cat > /etc/rollcall/rack.json <<EOF
|
||||
{"name": "hall-b", "region": "http://10.0.0.234", "secret": "<rack_secret>",
|
||||
"url": "http://10.213.78.1", "listen": "10.213.78.1:80"}
|
||||
EOF
|
||||
sh install-rack.sh # dnsmasq, syslinux, cache dirs, rollcall-rack.service
|
||||
rollcall controllers # on the region: it shows up online
|
||||
rollcall controller join-code -name hall-b -addr 10.0.0.234:443 # on the region: one word, one use, 60 min
|
||||
# on the rack host (Debian/Ubuntu), from deploy/ with the binary next to it:
|
||||
./rollcall rack join rollcall-join1-... -url http://10.213.78.1 # writes /etc/rollcall/rack.json + certificate
|
||||
sh install-rack.sh # dnsmasq, syslinux, cache dirs, rollcall-rack.service
|
||||
rollcall controllers # on the region: it shows up online
|
||||
rollcall vlan add hall-b 0 -dhcp -rack hall-b
|
||||
```
|
||||
|
||||
@@ -399,9 +396,89 @@ hand-written DHCP configuration for the same interface is the usual cause:
|
||||
|
||||
The lab runs both: `pxe-server` (embedded, 10.213.77.0/24) and `pxe-rack2`
|
||||
(VM 213, remote, 10.213.78.0/24 on SDN vnet `pxenet2`), with node002 behind
|
||||
pxe-rack2 and its virtual BMC on pxe-rack2. Rack ↔ region traffic is plain
|
||||
HTTP in the lab; BMC passwords travel on it for queued commands, so put the
|
||||
region behind TLS on real networks.
|
||||
pxe-rack2 and its virtual BMC on pxe-rack2. Rack ↔ region traffic is mutual
|
||||
TLS (next section), so the BMC passwords of queued commands are encrypted.
|
||||
|
||||
## Certificate authority and TLS
|
||||
|
||||
The region is the installation's certificate authority, as the llmgw control
|
||||
plane is: `ca_dir` (default `/etc/rollcall/ca`) holds `ca.pem` and
|
||||
`ca-key.pem` (ECDSA P-256, ten years), created on first start.
|
||||
|
||||
* **HTTPS.** The region serves API, UI and images on `tls_listen` (default
|
||||
`:443`) with a certificate for every name and address it has plus
|
||||
`tls_names`; it is reissued when they change or a month before it ends.
|
||||
`admin_http` decides what plain HTTP may still do with the admin API:
|
||||
`allow` (default, for compatibility), `loopback` (the CLI on the region
|
||||
host) or `deny`. Nodes keep booting over HTTP: firmware and the live image
|
||||
cannot check a certificate before they have one.
|
||||
* **Rack controllers** join with a code from `rollcall controller join-code`:
|
||||
one word carrying the first half of the CA's SHA-256, a one-time secret, an
|
||||
expiry and the region's address. `rollcall rack join` checks the region
|
||||
against the fingerprint, sends a CSR with the secret and gets a
|
||||
client+server certificate (`rack:<name>`, a year). From then on every rack
|
||||
call is mutual TLS and the region identifies the rack by its certificate;
|
||||
`rollcall controller rm` revokes it at once. Racks renew a month before
|
||||
expiry. The old `rack_secret` is still accepted when configured.
|
||||
* **Clients.** On the region host the CLI reads the CA from `ca_dir`;
|
||||
elsewhere `ROLLCALL_CA=ca.pem`, or the console shows the CA fingerprint on
|
||||
first connect and keeps the CA in `~/.config/rollcall/client.json` once you
|
||||
confirm it (`rollcall ca show` prints it on the region). `/ca.pem` serves
|
||||
the certificate; the web login links to it.
|
||||
* **Machines** trust it: deployments add it with cloud-init `ca_certs`
|
||||
(allowed on RHEL-family images, where cloud-init marks the module
|
||||
unverified), stateless overlays get it as `.CA`. `rollcall ca issue NAME
|
||||
-host h1,h2 [-client]` issues leaves for anything else (Slurm REST, a
|
||||
registry, a web front end).
|
||||
|
||||
## Power drivers
|
||||
|
||||
Every machine's BMC record names a driver; commands run on the rack controller
|
||||
that reaches the BMC network.
|
||||
|
||||
| Driver | For |
|
||||
|---|---|
|
||||
| `ipmi` | IPMI 2.0 over LAN (ipmitool lanplus), any vendor; settings `cipher` (17 on newer BMCs), `privilege`, `interface` |
|
||||
| `redfish` | DMTF Redfish, any vendor: system discovered, ResetType chosen from what the BMC allows, boot override once (with If-Match where the BMC wants it) |
|
||||
| `idrac` | Dell iDRAC 8/9 (Redfish, `System.Embedded.1`) |
|
||||
| `ilo` | HPE iLO 4 (2.30+), 5, 6 (Redfish; no PowerCycle, so ForceRestart; PushPowerButton for soft off on older firmware) |
|
||||
| `supermicro`, `xcc`, `openbmc` | Supermicro, Lenovo XClarity, OpenBMC (Redfish) |
|
||||
| `proxmox` | Proxmox VE VMs: API token (user = token id, password = secret), settings `node`, `vmid` |
|
||||
| `manual` | no power control; actions say so |
|
||||
|
||||
After commissioning, a BMC still on IPMI is asked whether it speaks Redfish
|
||||
with the same account; if it does, the driver for its vendor takes over
|
||||
(event `bmc driver`). An operator's choice is never overridden.
|
||||
|
||||
```sh
|
||||
rollcall bmc drivers
|
||||
rollcall bmc m-1234 -driver idrac -address 10.0.5.21 -user root -password - -verify < pw
|
||||
rollcall bmc m-1234 -driver redfish -set system=/redfish/v1/Systems/1 -set verify_tls=true
|
||||
rollcall bmc m-1234 -probe # detect Redfish again
|
||||
```
|
||||
|
||||
The lab has `lab/pve-vbmc/pve_redfish.py`, a Redfish service per Proxmox VM
|
||||
with iDRAC, iLO and generic profiles, next to the IPMI emulator.
|
||||
|
||||
## IPv6
|
||||
|
||||
Subnets can be IPv6 (`rollcall subnet add fd00:50::/64 ...`). IPAM hands out
|
||||
addresses from static ranges (walked from their start), deployments get them
|
||||
in `network-config` with a separate default route per family, and the default
|
||||
layout links every subnet of the boot NIC's VLAN, so an IPv6 prefix next to
|
||||
the IPv4 one makes machines dual-stack. Interfaces with planned addresses
|
||||
ignore router advertisements. dnsmasq serves router advertisements and
|
||||
DHCPv6 on IPv6 subnets with dhcp on; the rack offers itself as default router
|
||||
only when it is the subnet's gateway. Host files carry AAAA records, rack
|
||||
controllers report their IPv6 addresses, the region certificate covers them,
|
||||
and region, racks and clients work over IPv6 (`https://[fd00::1]`).
|
||||
|
||||
PXE stays IPv4: syslinux cannot netboot over IPv6, so the provisioning VLAN
|
||||
needs an IPv4 subnet for commissioning and deployment.
|
||||
|
||||
Verified in the lab: Debian 13 (with a bond and VLAN), Rocky 9 and AlmaLinux 9
|
||||
dual-stack on two segments, AAAA lookups through dnsmasq, the admin CLI over
|
||||
`https://[fdc3:...]`.
|
||||
|
||||
## Lab
|
||||
|
||||
|
||||
@@ -262,6 +262,8 @@ def network():
|
||||
"numa_node": readf(f"{sysd}/device/numa_node") or None,
|
||||
"ipv4": [a["local"] + "/" + str(a["prefixlen"]) for a in addrs.get(ifn, {}).get("addr_info", [])
|
||||
if a.get("family") == "inet"],
|
||||
"ipv6": [a["local"] + "/" + str(a["prefixlen"]) for a in addrs.get(ifn, {}).get("addr_info", [])
|
||||
if a.get("family") == "inet6" and a.get("scope") == "global"],
|
||||
}
|
||||
if l.get("link_type") == "ether":
|
||||
nic.update(ethtool_info(ifn))
|
||||
|
||||
@@ -0,0 +1,606 @@
|
||||
package main
|
||||
|
||||
// The region is the certificate authority of its installation, the way the
|
||||
// llmgw control plane is: one root every part trusts, leaves issued from it,
|
||||
// identity in the subject.
|
||||
//
|
||||
// - the region serves its API and UI over HTTPS with a certificate from it;
|
||||
// - rack controllers join with a one-time code (CA fingerprint, secret,
|
||||
// expiry, address in one word) and get a client+server certificate whose
|
||||
// common name is their name; from then on they talk to the region over
|
||||
// mutual TLS and the shared rack_secret is not needed;
|
||||
// - deployed machines trust it (cloud-init ca_certs), stateless overlays get
|
||||
// it as .CA, and `rollcall ca issue` hands out leaves for anything else.
|
||||
//
|
||||
// Nodes still boot over plain HTTP: PXE firmware and the live image have no
|
||||
// way to check a certificate before they have one.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"math/big"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
caValidity = 10 * 365 * 24 * time.Hour
|
||||
leafValidity = 365 * 24 * time.Hour
|
||||
backdate = time.Hour
|
||||
renewBefore = 30 * 24 * time.Hour
|
||||
joinPrefix = "rollcall-join1-"
|
||||
)
|
||||
|
||||
type CA struct {
|
||||
Cert *x509.Certificate
|
||||
Key *ecdsa.PrivateKey
|
||||
PEM []byte
|
||||
}
|
||||
|
||||
func randomSerial() (*big.Int, error) {
|
||||
return rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128))
|
||||
}
|
||||
|
||||
func pemEncode(kind string, der []byte) []byte {
|
||||
return pem.EncodeToMemory(&pem.Block{Type: kind, Bytes: der})
|
||||
}
|
||||
|
||||
func newCA(cn string) (*CA, error) {
|
||||
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
serial, err := randomSerial()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tmpl := &x509.Certificate{SerialNumber: serial, Subject: pkix.Name{CommonName: cn, Organization: []string{"rollcall"}},
|
||||
NotBefore: time.Now().Add(-backdate), NotAfter: time.Now().Add(caValidity),
|
||||
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign, BasicConstraintsValid: true, IsCA: true, MaxPathLenZero: true}
|
||||
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cert, err := x509.ParseCertificate(der)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &CA{Cert: cert, Key: key, PEM: pemEncode("CERTIFICATE", der)}, nil
|
||||
}
|
||||
|
||||
func parseCertPEM(b []byte) (*x509.Certificate, error) {
|
||||
blk, _ := pem.Decode(b)
|
||||
if blk == nil {
|
||||
return nil, errors.New("not PEM")
|
||||
}
|
||||
return x509.ParseCertificate(blk.Bytes)
|
||||
}
|
||||
|
||||
func parseKeyPEM(b []byte) (*ecdsa.PrivateKey, error) {
|
||||
blk, _ := pem.Decode(b)
|
||||
if blk == nil {
|
||||
return nil, errors.New("not PEM")
|
||||
}
|
||||
return x509.ParseECPrivateKey(blk.Bytes)
|
||||
}
|
||||
|
||||
// loadOrCreateCA keeps the authority in dir as ca.pem / ca-key.pem.
|
||||
func loadOrCreateCA(dir, cn string) (*CA, error) {
|
||||
cp, kp := filepath.Join(dir, "ca.pem"), filepath.Join(dir, "ca-key.pem")
|
||||
if cb, err := os.ReadFile(cp); err == nil {
|
||||
kb, err := os.ReadFile(kp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cert, err := parseCertPEM(cb)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", cp, err)
|
||||
}
|
||||
key, err := parseKeyPEM(kb)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", kp, err)
|
||||
}
|
||||
return &CA{Cert: cert, Key: key, PEM: cb}, nil
|
||||
}
|
||||
ca, err := newCA(cn)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
kder, _ := x509.MarshalECPrivateKey(ca.Key)
|
||||
if err := os.WriteFile(kp, pemEncode("EC PRIVATE KEY", kder), 0o600); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.WriteFile(cp, ca.PEM, 0o644); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
log.Printf("created certificate authority %q in %s (fingerprint %s)", cn, dir, ca.Fingerprint())
|
||||
return ca, nil
|
||||
}
|
||||
|
||||
func certFingerprint(c *x509.Certificate) [32]byte { return sha256.Sum256(c.Raw) }
|
||||
|
||||
// Fingerprint is the SHA-256 of the CA certificate, as people compare it.
|
||||
func (ca *CA) Fingerprint() string { return fpString(certFingerprint(ca.Cert)) }
|
||||
|
||||
func fpString(sum [32]byte) string {
|
||||
h := hex.EncodeToString(sum[:])
|
||||
var p []string
|
||||
for i := 0; i < len(h); i += 2 {
|
||||
p = append(p, strings.ToUpper(h[i:i+2]))
|
||||
}
|
||||
return "sha256:" + strings.Join(p, ":")
|
||||
}
|
||||
|
||||
func splitHosts(hosts []string) (ips []net.IP, dns []string) {
|
||||
seen := map[string]bool{}
|
||||
for _, h := range hosts {
|
||||
h = strings.Trim(strings.TrimSpace(h), "[]")
|
||||
if h == "" || seen[h] {
|
||||
continue
|
||||
}
|
||||
seen[h] = true
|
||||
if ip := net.ParseIP(h); ip != nil {
|
||||
ips = append(ips, ip)
|
||||
} else {
|
||||
dns = append(dns, h)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// sign issues a certificate for pub with the given name, hosts and usages.
|
||||
func (ca *CA) sign(pub crypto.PublicKey, cn string, hosts []string, usage ...x509.ExtKeyUsage) ([]byte, *x509.Certificate, error) {
|
||||
serial, err := randomSerial()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
tmpl := &x509.Certificate{SerialNumber: serial, Subject: pkix.Name{CommonName: cn, Organization: []string{"rollcall"}},
|
||||
NotBefore: time.Now().Add(-backdate), NotAfter: time.Now().Add(leafValidity),
|
||||
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment, ExtKeyUsage: usage}
|
||||
tmpl.IPAddresses, tmpl.DNSNames = splitHosts(hosts)
|
||||
der, err := x509.CreateCertificate(rand.Reader, tmpl, ca.Cert, pub, ca.Key)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
cert, err := x509.ParseCertificate(der)
|
||||
return pemEncode("CERTIFICATE", der), cert, err
|
||||
}
|
||||
|
||||
// issue makes a key and a certificate for it.
|
||||
func (ca *CA) issue(cn string, hosts []string, usage ...x509.ExtKeyUsage) (certPEM, keyPEM []byte, err error) {
|
||||
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
certPEM, _, err = ca.sign(&key.PublicKey, cn, hosts, usage...)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
kder, _ := x509.MarshalECPrivateKey(key)
|
||||
return certPEM, pemEncode("EC PRIVATE KEY", kder), nil
|
||||
}
|
||||
|
||||
// hostAddresses: names and addresses this machine answers on (no link-local).
|
||||
func hostAddresses() []string {
|
||||
h, _ := os.Hostname()
|
||||
out := []string{"localhost", "127.0.0.1", "::1"}
|
||||
if h != "" {
|
||||
out = append(out, h)
|
||||
}
|
||||
ifs, _ := net.Interfaces()
|
||||
for _, i := range ifs {
|
||||
addrs, _ := i.Addrs()
|
||||
for _, a := range addrs {
|
||||
if ipn, ok := a.(*net.IPNet); ok && !ipn.IP.IsLinkLocalUnicast() && !ipn.IP.IsLoopback() {
|
||||
out = append(out, ipn.IP.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// regionCert keeps dir/region.pem valid for every name and address the region has.
|
||||
func regionCert(ca *CA, dir string, extra []string) (tls.Certificate, error) {
|
||||
cp, kp := filepath.Join(dir, "region.pem"), filepath.Join(dir, "region-key.pem")
|
||||
want := append(hostAddresses(), extra...)
|
||||
if cb, err := os.ReadFile(cp); err == nil {
|
||||
if cert, err := parseCertPEM(cb); err == nil && time.Until(cert.NotAfter) > renewBefore && cert.CheckSignatureFrom(ca.Cert) == nil && coversHosts(cert, want) {
|
||||
if pair, err := tls.LoadX509KeyPair(cp, kp); err == nil {
|
||||
pair.Certificate = append(pair.Certificate, ca.Cert.Raw)
|
||||
return pair, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
h, _ := os.Hostname()
|
||||
certPEM, keyPEM, err := ca.issue("rollcall region "+h, want, x509.ExtKeyUsageServerAuth)
|
||||
if err != nil {
|
||||
return tls.Certificate{}, err
|
||||
}
|
||||
if err := os.WriteFile(kp, keyPEM, 0o600); err != nil {
|
||||
return tls.Certificate{}, err
|
||||
}
|
||||
if err := os.WriteFile(cp, certPEM, 0o644); err != nil {
|
||||
return tls.Certificate{}, err
|
||||
}
|
||||
log.Printf("issued the region's TLS certificate for %s", strings.Join(want, " "))
|
||||
pair, err := tls.X509KeyPair(certPEM, keyPEM)
|
||||
if err != nil {
|
||||
return pair, err
|
||||
}
|
||||
pair.Certificate = append(pair.Certificate, ca.Cert.Raw) // clients that pin the CA find it in the chain
|
||||
return pair, nil
|
||||
}
|
||||
|
||||
func coversHosts(c *x509.Certificate, hosts []string) bool {
|
||||
for _, h := range hosts {
|
||||
if c.VerifyHostname(strings.Trim(h, "[]")) != nil {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- join codes
|
||||
|
||||
type joinCode struct {
|
||||
Fingerprint [16]byte // first half of the CA certificate's SHA-256
|
||||
Secret [16]byte
|
||||
Expires int64
|
||||
Addr string // host:port of the region's HTTPS listener
|
||||
}
|
||||
|
||||
func (c joinCode) String() string {
|
||||
b := make([]byte, 0, 36+len(c.Addr))
|
||||
b = append(b, c.Fingerprint[:]...)
|
||||
b = append(b, c.Secret[:]...)
|
||||
b = binary.BigEndian.AppendUint32(b, uint32(c.Expires))
|
||||
b = append(b, c.Addr...)
|
||||
return joinPrefix + base64.RawURLEncoding.EncodeToString(b)
|
||||
}
|
||||
|
||||
func parseJoinCode(s string) (joinCode, error) {
|
||||
raw, ok := strings.CutPrefix(strings.TrimSpace(s), joinPrefix)
|
||||
if !ok {
|
||||
return joinCode{}, errors.New("not a join code: it starts " + joinPrefix)
|
||||
}
|
||||
b, err := base64.RawURLEncoding.DecodeString(raw)
|
||||
if err != nil || len(b) < 36+3 {
|
||||
return joinCode{}, errors.New("the join code is damaged: copy it whole")
|
||||
}
|
||||
var c joinCode
|
||||
copy(c.Fingerprint[:], b[:16])
|
||||
copy(c.Secret[:], b[16:32])
|
||||
c.Expires = int64(binary.BigEndian.Uint32(b[32:36]))
|
||||
c.Addr = string(b[36:])
|
||||
if _, _, err := net.SplitHostPort(c.Addr); err != nil {
|
||||
return joinCode{}, errors.New("the join code is damaged: copy it whole")
|
||||
}
|
||||
if time.Now().Unix() > c.Expires {
|
||||
return joinCode{}, fmt.Errorf("the join code expired at %s; ask for a new one", time.Unix(c.Expires, 0).Format(time.RFC3339))
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// pinnedTLS trusts a server whose chain carries the CA with this fingerprint.
|
||||
func pinnedTLS(fp [16]byte) *tls.Config {
|
||||
return &tls.Config{MinVersion: tls.VersionTLS12, InsecureSkipVerify: true, //nolint:gosec // verified below against the pinned CA
|
||||
VerifyPeerCertificate: func(raw [][]byte, _ [][]*x509.Certificate) error {
|
||||
var leaf *x509.Certificate
|
||||
for i, r := range raw {
|
||||
c, err := x509.ParseCertificate(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if i == 0 {
|
||||
leaf = c
|
||||
}
|
||||
sum := sha256.Sum256(r)
|
||||
if c.IsCA && string(sum[:16]) == string(fp[:]) {
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(c)
|
||||
_, err := leaf.Verify(x509.VerifyOptions{Roots: pool})
|
||||
return err
|
||||
}
|
||||
}
|
||||
return errors.New("the server's certificate is not from the authority this code is for")
|
||||
}}
|
||||
}
|
||||
|
||||
func (s *Store) PutJoinToken(ctx context.Context, hash, name string, expires time.Time) error {
|
||||
_, err := s.pool.Exec(ctx, `INSERT INTO join_tokens(hash, name, expires) VALUES ($1,nullif($2,''),$3)`, hash, name, expires)
|
||||
return err
|
||||
}
|
||||
|
||||
// UseJoinToken consumes a token; it returns the name it was bound to.
|
||||
func (s *Store) UseJoinToken(ctx context.Context, hash string) (string, error) {
|
||||
var name string
|
||||
err := s.pool.QueryRow(ctx, `UPDATE join_tokens SET used=now() WHERE hash=$1 AND used IS NULL AND expires > now() RETURNING coalesce(name,'')`, hash).Scan(&name)
|
||||
if err != nil {
|
||||
return "", errors.New("unknown, used or expired join code")
|
||||
}
|
||||
return name, nil
|
||||
}
|
||||
|
||||
func (s *Store) SetRackCert(ctx context.Context, name, serial string, expires time.Time) error {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE rack_controllers SET cert_serial=$2, cert_expires=$3 WHERE name=$1`, name, serial, expires)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) RackCertSerial(ctx context.Context, name string) string {
|
||||
var serial string
|
||||
_ = s.pool.QueryRow(ctx, `SELECT coalesce(cert_serial,'') FROM rack_controllers WHERE name=$1`, name).Scan(&serial)
|
||||
return serial
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- HTTP
|
||||
|
||||
// peerRack is the rack controller a mutual-TLS request comes from, if any:
|
||||
// a verified client certificate whose serial is the one on record.
|
||||
func (a *App) peerRack(r *http.Request) string {
|
||||
if r.TLS == nil || len(r.TLS.VerifiedChains) == 0 || len(r.TLS.PeerCertificates) == 0 {
|
||||
return ""
|
||||
}
|
||||
c := r.TLS.PeerCertificates[0]
|
||||
name := strings.TrimPrefix(c.Subject.CommonName, "rack:")
|
||||
if name == c.Subject.CommonName { // only rack certificates
|
||||
return ""
|
||||
}
|
||||
if a.store.RackCertSerial(r.Context(), name) != c.SerialNumber.Text(16) {
|
||||
return ""
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
func (a *App) handleCAPEM(w http.ResponseWriter, r *http.Request) {
|
||||
if a.ca == nil {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/x-pem-file")
|
||||
w.Header().Set("Content-Disposition", `attachment; filename="rollcall-ca.pem"`)
|
||||
w.Header().Set("X-Rollcall-CA-Fingerprint", a.ca.Fingerprint())
|
||||
_, _ = w.Write(a.ca.PEM)
|
||||
}
|
||||
|
||||
// handleJoinCode makes a one-time code for a rack controller (admin).
|
||||
func (a *App) handleJoinCode(w http.ResponseWriter, r *http.Request) {
|
||||
if a.ca == nil || a.cfg.TLSListen == "" {
|
||||
fail(w, 409, errors.New("the region has no TLS listener (tls_listen)"))
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Name string `json:"name"`
|
||||
TTL int `json:"ttl_minutes"`
|
||||
Addr string `json:"addr"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&req)
|
||||
if req.TTL <= 0 {
|
||||
req.TTL = 60
|
||||
}
|
||||
addr := req.Addr
|
||||
if addr == "" {
|
||||
host := r.Host
|
||||
if h, _, err := net.SplitHostPort(host); err == nil {
|
||||
host = h
|
||||
}
|
||||
if ip := net.ParseIP(strings.Trim(host, "[]")); (ip != nil && ip.IsLoopback()) || host == "localhost" {
|
||||
fail(w, 400, errors.New("say where rack controllers reach the region's HTTPS: addr (CLI -addr host:443)"))
|
||||
return
|
||||
}
|
||||
_, port, _ := net.SplitHostPort(a.cfg.TLSListen)
|
||||
addr = net.JoinHostPort(strings.Trim(host, "[]"), port)
|
||||
}
|
||||
var c joinCode
|
||||
sum := certFingerprint(a.ca.Cert)
|
||||
copy(c.Fingerprint[:], sum[:16])
|
||||
_, _ = rand.Read(c.Secret[:])
|
||||
c.Expires = time.Now().Add(time.Duration(req.TTL) * time.Minute).Unix()
|
||||
c.Addr = addr
|
||||
h := sha256.Sum256(c.Secret[:])
|
||||
if err := a.store.PutJoinToken(r.Context(), hex.EncodeToString(h[:]), req.Name, time.Unix(c.Expires, 0)); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, obj{"code": c.String(), "expires": time.Unix(c.Expires, 0), "addr": addr, "name": req.Name, "ca_fingerprint": a.ca.Fingerprint()})
|
||||
}
|
||||
|
||||
type enrollRequest struct {
|
||||
Secret string `json:"secret"` // base64 of the code's secret
|
||||
Name string `json:"name"`
|
||||
CSR string `json:"csr"`
|
||||
Rack RackController `json:"rack"`
|
||||
}
|
||||
|
||||
func (a *App) signRackCSR(csrPEM, name string, rc RackController) ([]byte, *x509.Certificate, error) {
|
||||
blk, _ := pem.Decode([]byte(csrPEM))
|
||||
if blk == nil {
|
||||
return nil, nil, errors.New("csr is not PEM")
|
||||
}
|
||||
csr, err := x509.ParseCertificateRequest(blk.Bytes)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if err := csr.CheckSignature(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
hosts := []string{name}
|
||||
if u := strings.TrimPrefix(strings.TrimPrefix(rc.URL, "http://"), "https://"); u != "" {
|
||||
if h, _, err := net.SplitHostPort(u); err == nil {
|
||||
hosts = append(hosts, h)
|
||||
} else {
|
||||
hosts = append(hosts, strings.Trim(u, "/[]"))
|
||||
}
|
||||
}
|
||||
for _, i := range rc.Interfaces {
|
||||
for _, ad := range i.Addrs {
|
||||
hosts = append(hosts, stripPrefix(ad))
|
||||
}
|
||||
}
|
||||
return a.ca.sign(csr.PublicKey, "rack:"+name, hosts, x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth)
|
||||
}
|
||||
|
||||
// handleRackEnroll: a rack controller presents a join code's secret and a CSR
|
||||
// over TLS; it gets a certificate and the CA. No admin token, no client cert.
|
||||
func (a *App) handleRackEnroll(w http.ResponseWriter, r *http.Request) {
|
||||
if r.TLS == nil || a.ca == nil {
|
||||
fail(w, 400, errors.New("enrollment needs HTTPS"))
|
||||
return
|
||||
}
|
||||
var req enrollRequest
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&req); err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
sec, err := base64.RawURLEncoding.DecodeString(req.Secret)
|
||||
if err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
h := sha256.Sum256(sec)
|
||||
bound, err := a.store.UseJoinToken(r.Context(), hex.EncodeToString(h[:]))
|
||||
if err != nil {
|
||||
fail(w, 401, err)
|
||||
return
|
||||
}
|
||||
name := req.Name
|
||||
if bound != "" {
|
||||
name = bound
|
||||
}
|
||||
if !tagNameRe.MatchString(name) || name == a.cfg.RackName {
|
||||
fail(w, 400, fmt.Errorf("bad rack controller name %q", name))
|
||||
return
|
||||
}
|
||||
certPEM, cert, err := a.signRackCSR(req.CSR, name, req.Rack)
|
||||
if err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
rc := req.Rack
|
||||
rc.Name, rc.Kind = name, "remote"
|
||||
if err := a.store.PutRackController(r.Context(), rc); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
if err := a.store.SetRackCert(r.Context(), name, cert.SerialNumber.Text(16), cert.NotAfter); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
a.reloadRacks(r.Context())
|
||||
a.hub.bump()
|
||||
log.Printf("rack controller %s enrolled from %s, certificate until %s", name, r.RemoteAddr, cert.NotAfter.Format("2006-01-02"))
|
||||
writeJSON(w, 200, obj{"name": name, "cert": string(certPEM), "ca": string(a.ca.PEM)})
|
||||
}
|
||||
|
||||
// handleRackRenew issues a new certificate to a rack controller that still has a valid one.
|
||||
func (a *App) handleRackRenew(w http.ResponseWriter, r *http.Request, rack string) {
|
||||
if a.peerRack(r) != rack {
|
||||
fail(w, 401, errors.New("renewal needs the current certificate"))
|
||||
return
|
||||
}
|
||||
var req enrollRequest
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&req); err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
rc := a.racksSnapshot()[rack]
|
||||
certPEM, cert, err := a.signRackCSR(req.CSR, rack, rc)
|
||||
if err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
if err := a.store.SetRackCert(r.Context(), rack, cert.SerialNumber.Text(16), cert.NotAfter); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
log.Printf("rack controller %s renewed its certificate, until %s", rack, cert.NotAfter.Format("2006-01-02"))
|
||||
writeJSON(w, 200, obj{"name": rack, "cert": string(certPEM), "ca": string(a.ca.PEM)})
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- CLI: rollcall ca (on the region host)
|
||||
|
||||
func caCmd(args []string) error {
|
||||
cfg, err := loadConfig(envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(args) == 0 || args[0] == "show" {
|
||||
cb, err := os.ReadFile(filepath.Join(cfg.CADir, "ca.pem"))
|
||||
if err != nil {
|
||||
return fmt.Errorf("no CA yet (it is created when rollcall serve starts): %w", err)
|
||||
}
|
||||
c, err := parseCertPEM(cb)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s\n subject %s\n valid until %s\n fingerprint %s\n", filepath.Join(cfg.CADir, "ca.pem"), c.Subject.CommonName,
|
||||
c.NotAfter.Format("2006-01-02"), fpString(certFingerprint(c)))
|
||||
if rb, err := os.ReadFile(filepath.Join(cfg.CADir, "region.pem")); err == nil {
|
||||
if rc, err := parseCertPEM(rb); err == nil {
|
||||
names := append([]string{}, rc.DNSNames...)
|
||||
for _, ip := range rc.IPAddresses {
|
||||
names = append(names, ip.String())
|
||||
}
|
||||
sort.Strings(names)
|
||||
fmt.Printf("region certificate until %s for %s\n", rc.NotAfter.Format("2006-01-02"), strings.Join(names, " "))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if args[0] != "issue" || len(args) < 2 {
|
||||
usage()
|
||||
}
|
||||
fs := newFlags("ca issue")
|
||||
hosts := fs.String("host", "", "comma-separated DNS names and addresses")
|
||||
client := fs.Bool("client", false, "client certificate (identifies its holder) instead of a server one")
|
||||
out := fs.String("out", ".", "directory for NAME.pem and NAME-key.pem")
|
||||
_ = fs.Parse(args[2:])
|
||||
ca, err := loadOrCreateCA(cfg.CADir, "rollcall CA")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var hs []string
|
||||
if *hosts != "" {
|
||||
hs = strings.Split(*hosts, ",")
|
||||
}
|
||||
usage := x509.ExtKeyUsageServerAuth
|
||||
if *client {
|
||||
usage = x509.ExtKeyUsageClientAuth
|
||||
} else if len(hs) == 0 {
|
||||
return errors.New("a server certificate needs -host")
|
||||
}
|
||||
certPEM, keyPEM, err := ca.issue(args[1], hs, usage)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(*out, args[1]+"-key.pem"), keyPEM, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(*out, args[1]+".pem"), append(certPEM, ca.PEM...), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("wrote %s.pem (with the CA) and %s-key.pem in %s, valid until %s\n", args[1], args[1], *out, time.Now().Add(leafValidity).Format("2006-01-02"))
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestCAAndRegionCert(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
ca, err := loadOrCreateCA(dir, "test CA")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, err := loadOrCreateCA(dir, "test CA")
|
||||
if err != nil || again.Fingerprint() != ca.Fingerprint() {
|
||||
t.Fatalf("reload gives another CA: %v", err)
|
||||
}
|
||||
pair, err := regionCert(ca, dir, []string{"rollcall.example", "fd00:213:77::1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
leaf, _ := x509.ParseCertificate(pair.Certificate[0])
|
||||
for _, h := range []string{"rollcall.example", "fd00:213:77::1", "127.0.0.1", "localhost"} {
|
||||
if err := leaf.VerifyHostname(h); err != nil {
|
||||
t.Errorf("region certificate does not cover %s: %v", h, err)
|
||||
}
|
||||
}
|
||||
if len(pair.Certificate) != 2 {
|
||||
t.Errorf("the chain should carry the CA for pinning clients")
|
||||
}
|
||||
// unchanged names: the same certificate is reused
|
||||
p2, _ := regionCert(ca, dir, []string{"rollcall.example", "fd00:213:77::1"})
|
||||
if string(p2.Certificate[0]) != string(pair.Certificate[0]) {
|
||||
t.Errorf("certificate reissued without a reason")
|
||||
}
|
||||
}
|
||||
|
||||
func TestJoinCodeAndPinnedTLS(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
ca, _ := loadOrCreateCA(dir, "test CA")
|
||||
pair, _ := regionCert(ca, dir, nil)
|
||||
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("ok")) }))
|
||||
srv.TLS = &tls.Config{Certificates: []tls.Certificate{pair}}
|
||||
srv.StartTLS()
|
||||
defer srv.Close()
|
||||
|
||||
var c joinCode
|
||||
sum := certFingerprint(ca.Cert)
|
||||
copy(c.Fingerprint[:], sum[:16])
|
||||
c.Secret[0] = 7
|
||||
c.Expires = time.Now().Add(time.Hour).Unix()
|
||||
c.Addr = strings.TrimPrefix(srv.URL, "https://")
|
||||
got, err := parseJoinCode(c.String())
|
||||
if err != nil || got != c {
|
||||
t.Fatalf("round trip: %+v %v", got, err)
|
||||
}
|
||||
ok := &http.Client{Transport: &http.Transport{TLSClientConfig: pinnedTLS(c.Fingerprint)}}
|
||||
if resp, err := ok.Get(srv.URL); err != nil {
|
||||
t.Fatalf("pinned CA rejected: %v", err)
|
||||
} else {
|
||||
resp.Body.Close()
|
||||
}
|
||||
other, _ := newCA("another")
|
||||
osum := sha256.Sum256(other.Cert.Raw)
|
||||
var wrong [16]byte
|
||||
copy(wrong[:], osum[:16])
|
||||
bad := &http.Client{Transport: &http.Transport{TLSClientConfig: pinnedTLS(wrong)}}
|
||||
if _, err := bad.Get(srv.URL); err == nil {
|
||||
t.Errorf("a server from another authority was accepted")
|
||||
}
|
||||
expired := c
|
||||
expired.Expires = time.Now().Add(-time.Minute).Unix()
|
||||
if _, err := parseJoinCode(expired.String()); err == nil {
|
||||
t.Errorf("an expired code was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSignRackCSR(t *testing.T) {
|
||||
ca, _ := newCA("test CA")
|
||||
a := &App{ca: ca, cfg: &Config{}}
|
||||
_, csr, err := newCSR("rack:hall-b")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, cert, err := a.signRackCSR(csr, "hall-b", RackController{URL: "http://10.213.78.1", Interfaces: []RackIface{{Name: "eth1", Addrs: []string{"fd00:213:78::1/64"}}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(ca.Cert)
|
||||
if _, err := cert.Verify(x509.VerifyOptions{Roots: pool, KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}}); err != nil {
|
||||
t.Errorf("rack certificate is no client certificate: %v", err)
|
||||
}
|
||||
if cert.Subject.CommonName != "rack:hall-b" || cert.VerifyHostname("10.213.78.1") != nil || cert.VerifyHostname("fd00:213:78::1") != nil {
|
||||
t.Errorf("subject %s, SANs %v %v", cert.Subject.CommonName, cert.IPAddresses, cert.DNSNames)
|
||||
}
|
||||
}
|
||||
+156
-2
@@ -3,13 +3,17 @@ package main
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -17,14 +21,46 @@ import (
|
||||
type client struct {
|
||||
base, token string
|
||||
http *http.Client
|
||||
ca string
|
||||
}
|
||||
|
||||
// clientTLS: the region's CA from the server config (root on the region), the
|
||||
// client config (trusted on first connect), or ROLLCALL_CA; else the system roots.
|
||||
func clientTLS(caPEM string) *tls.Config {
|
||||
pem := []byte(caPEM)
|
||||
if f := os.Getenv("ROLLCALL_CA"); f != "" {
|
||||
if b, err := os.ReadFile(f); err == nil {
|
||||
pem = b
|
||||
}
|
||||
}
|
||||
if len(pem) == 0 {
|
||||
if cfg, err := loadConfig(envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json")); err == nil {
|
||||
pem, _ = os.ReadFile(filepath.Join(cfg.CADir, "ca.pem"))
|
||||
}
|
||||
}
|
||||
tc := &tls.Config{MinVersion: tls.VersionTLS12}
|
||||
if len(pem) > 0 {
|
||||
pool, err := x509.SystemCertPool()
|
||||
if err != nil || pool == nil {
|
||||
pool = x509.NewCertPool()
|
||||
}
|
||||
pool.AppendCertsFromPEM(pem)
|
||||
tc.RootCAs = pool
|
||||
}
|
||||
return tc
|
||||
}
|
||||
|
||||
func newHTTPClient(caPEM string) *http.Client {
|
||||
return &http.Client{Timeout: 90 * time.Second, Transport: &http.Transport{TLSClientConfig: clientTLS(caPEM), Proxy: http.ProxyFromEnvironment}}
|
||||
}
|
||||
|
||||
func newClient() (*client, error) {
|
||||
base, token := os.Getenv("ROLLCALL_URL"), os.Getenv("ROLLCALL_TOKEN")
|
||||
caPEM := ""
|
||||
if base == "" || token == "" {
|
||||
if cfg, err := loadConfig(envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json")); err == nil {
|
||||
if base == "" {
|
||||
base = "http://" + strings.Replace(cfg.Listen, "0.0.0.0", "127.0.0.1", 1)
|
||||
base = "http://" + localListen(cfg.Listen)
|
||||
}
|
||||
if token == "" {
|
||||
token = cfg.AdminToken
|
||||
@@ -39,12 +75,18 @@ func newClient() (*client, error) {
|
||||
if token == "" {
|
||||
token = cc.Token
|
||||
}
|
||||
caPEM = cc.CA
|
||||
}
|
||||
}
|
||||
if base == "" || token == "" {
|
||||
return nil, fmt.Errorf("set ROLLCALL_URL and ROLLCALL_TOKEN, make the server config readable, or run rollcall in a terminal to connect")
|
||||
}
|
||||
return &client{base: strings.TrimRight(base, "/"), token: token, http: &http.Client{Timeout: 90 * time.Second}}, nil
|
||||
if caPEM == "" {
|
||||
if cc, ok := loadClientConf(); ok && strings.TrimRight(cc.URL, "/") == strings.TrimRight(base, "/") {
|
||||
caPEM = cc.CA
|
||||
}
|
||||
}
|
||||
return &client{base: strings.TrimRight(base, "/"), token: token, http: newHTTPClient(caPEM), ca: caPEM}, nil
|
||||
}
|
||||
|
||||
func (c *client) do(method, path string, body any, out any) error {
|
||||
@@ -225,6 +267,8 @@ func runClient(cmd string, args []string) error {
|
||||
return c.tagCmd(cmd, args)
|
||||
case "location":
|
||||
return c.locationCmd(args)
|
||||
case "bmc":
|
||||
return c.bmcCmd(args)
|
||||
case "fabric", "vlan", "subnet", "range", "ip", "layout", "net", "controllers", "controller", "dhcp":
|
||||
return c.netCmd(cmd, args)
|
||||
default:
|
||||
@@ -813,3 +857,113 @@ func asList(v any) []any {
|
||||
}
|
||||
return []any{v}
|
||||
}
|
||||
|
||||
func (c *client) bmcCmd(args []string) error {
|
||||
if len(args) < 1 {
|
||||
usage()
|
||||
}
|
||||
if args[0] == "drivers" {
|
||||
var ds []powerDriver
|
||||
if err := c.do("GET", "/api/v1/power-drivers", nil, &ds); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, d := range ds {
|
||||
fmt.Printf("%-11s %-8s %s\n", d.Name, d.Family, d.Description)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
fs := newFlags("bmc")
|
||||
driver := fs.String("driver", "", "power driver (rollcall bmc drivers)")
|
||||
address := fs.String("address", "", "BMC address (host, IP or https://host:port)")
|
||||
port := fs.Int("port", 0, "port (0 = the driver's default)")
|
||||
user := fs.String("user", "", "account (proxmox: token id user@realm!name)")
|
||||
password := fs.String("password", "", "password (proxmox: token secret); - reads it from stdin")
|
||||
verify := fs.Bool("verify", false, "check with a status call before saving")
|
||||
probe := fs.Bool("probe", false, "re-run Redfish detection with the stored account")
|
||||
var sets multiFlag
|
||||
fs.Var(&sets, "set", "driver setting KEY=VALUE, repeatable (system, verify_tls, cipher, privilege, interface, node, vmid); KEY= removes it")
|
||||
_ = fs.Parse(args[1:])
|
||||
id := args[0]
|
||||
req := obj{}
|
||||
set := func(k, v string) {
|
||||
if v != "" {
|
||||
req[k] = v
|
||||
}
|
||||
}
|
||||
set("driver", *driver)
|
||||
set("address", *address)
|
||||
set("user", *user)
|
||||
if *port != 0 {
|
||||
req["port"] = *port
|
||||
}
|
||||
if *password == "-" {
|
||||
b, err := io.ReadAll(os.Stdin)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req["password"] = strings.TrimRight(string(b), "\r\n")
|
||||
} else if *password != "" {
|
||||
req["password"] = *password
|
||||
}
|
||||
if len(sets) > 0 {
|
||||
ex := map[string]string{}
|
||||
for _, s := range sets {
|
||||
k, v, _ := strings.Cut(s, "=")
|
||||
ex[k] = v
|
||||
}
|
||||
req["extra"] = ex
|
||||
}
|
||||
var d obj
|
||||
switch {
|
||||
case *probe:
|
||||
if err := c.do("POST", "/api/v1/machines/"+id+"/bmc", obj{"probe": true}, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
case len(req) == 0:
|
||||
if err := c.do("GET", "/api/v1/machines/"+id+"/bmc", nil, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
req["verify"] = *verify
|
||||
if err := c.do("POST", "/api/v1/machines/"+id+"/bmc", req, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
b, _ := d["bmc"].(obj)
|
||||
drv := str(b["driver"])
|
||||
if drv == "" {
|
||||
drv = "ipmi"
|
||||
}
|
||||
fmt.Printf("%s driver %s address %s port %s user %s source %s%s\n", id, drv, str(b["address"]), orDash(str(b["port"])), str(b["user"]), str(b["source"]),
|
||||
map[bool]string{true: " (set by operator)", false: ""}[b["driver_set"] == true])
|
||||
if ex, ok := b["extra"].(obj); ok && len(ex) > 0 {
|
||||
var kv []string
|
||||
for k, v := range ex {
|
||||
kv = append(kv, k+"="+str(v))
|
||||
}
|
||||
sort.Strings(kv)
|
||||
fmt.Printf(" settings: %s\n", strings.Join(kv, " "))
|
||||
}
|
||||
if rf, ok := b["redfish"].(obj); ok {
|
||||
fmt.Printf(" redfish %s · %s %s · %s\n", str(rf["redfish"]), str(rf["manufacturer"]), str(rf["model"]), str(rf["system"]))
|
||||
}
|
||||
if out := str(d["output"]); out != "" {
|
||||
fmt.Println(" " + out)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// localListen turns a listen address into one this host can dial.
|
||||
func localListen(listen string) string {
|
||||
host, port, err := net.SplitHostPort(listen)
|
||||
if err != nil {
|
||||
return listen
|
||||
}
|
||||
switch host {
|
||||
case "", "0.0.0.0":
|
||||
host = "127.0.0.1"
|
||||
case "::":
|
||||
host = "::1"
|
||||
}
|
||||
return net.JoinHostPort(host, port)
|
||||
}
|
||||
+23
-4
@@ -3,6 +3,7 @@ package main
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -36,7 +37,13 @@ type Config struct {
|
||||
// region/rack: this server runs an embedded rack controller named RackName;
|
||||
// remote rack controllers (rollcall rack) authenticate with RackSecret
|
||||
RackName string `json:"rack_name"`
|
||||
RackSecret string `json:"rack_secret"`
|
||||
RackSecret string `json:"rack_secret"` // legacy shared secret; rack controllers join with a code and mutual TLS now
|
||||
|
||||
// certificate authority and HTTPS
|
||||
CADir string `json:"ca_dir"` // ca.pem, ca-key.pem, region.pem, region-key.pem (default /etc/rollcall/ca)
|
||||
TLSListen string `json:"tls_listen"` // HTTPS listener, default ":443"; "off" disables
|
||||
TLSNames []string `json:"tls_names"` // extra names/addresses for the region certificate
|
||||
AdminHTTP string `json:"admin_http"` // admin API over plain HTTP: allow (default) | loopback | deny
|
||||
}
|
||||
|
||||
// BMCHint maps a machine MAC to a known BMC (virtual lab nodes).
|
||||
@@ -64,11 +71,11 @@ func loadConfig(path string) (*Config, error) {
|
||||
}
|
||||
}
|
||||
if c.ServerURL == "" {
|
||||
host, _, _ := strings.Cut(c.Listen, ":")
|
||||
if host == "" || host == "0.0.0.0" {
|
||||
host, _, err := net.SplitHostPort(c.Listen)
|
||||
if err != nil || host == "" || host == "0.0.0.0" || host == "::" {
|
||||
return nil, fmt.Errorf("%s: server_url is required when listen is not a specific address", path)
|
||||
}
|
||||
c.ServerURL = "http://" + c.Listen
|
||||
c.ServerURL = "http://" + c.Listen // host:port, IPv6 already bracketed ([::1]:80)
|
||||
}
|
||||
c.ServerURL = strings.TrimRight(c.ServerURL, "/")
|
||||
if c.ImagesDir == "" {
|
||||
@@ -86,5 +93,17 @@ func loadConfig(path string) (*Config, error) {
|
||||
if c.RackName == "" {
|
||||
c.RackName, _ = os.Hostname()
|
||||
}
|
||||
if c.CADir == "" {
|
||||
c.CADir = "/etc/rollcall/ca"
|
||||
}
|
||||
switch c.TLSListen {
|
||||
case "":
|
||||
c.TLSListen = ":443"
|
||||
case "off", "none":
|
||||
c.TLSListen = ""
|
||||
}
|
||||
if c.AdminHTTP == "" {
|
||||
c.AdminHTTP = "allow"
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
+17
-4
@@ -289,15 +289,28 @@ func (a *App) renderSeed(j *Job, sp DeploySpec, token string) []seedFile {
|
||||
b.WriteString("final_message: \"rollcall: deployed $hostname up after $UPTIME s\"\n")
|
||||
ud = b.String()
|
||||
}
|
||||
if merr == nil {
|
||||
if frags, err := a.tagFragments(context.Background(), m); err == nil && len(frags) > 0 {
|
||||
ud = multipartUserData(ud, frags)
|
||||
var frags []TagDef
|
||||
if a.ca != nil {
|
||||
var cc strings.Builder
|
||||
cc.WriteString("#cloud-config\n# rollcall: trust the installation's certificate authority\nca_certs:\n trusted:\n - |\n")
|
||||
for _, l := range strings.Split(strings.TrimSpace(string(a.ca.PEM)), "\n") {
|
||||
cc.WriteString(" " + l + "\n")
|
||||
}
|
||||
frags = append(frags, TagDef{Name: "rollcall-ca", CloudConfig: cc.String()})
|
||||
}
|
||||
if merr == nil {
|
||||
if tf, err := a.tagFragments(context.Background(), m); err == nil {
|
||||
frags = append(frags, tf...)
|
||||
}
|
||||
}
|
||||
if len(frags) > 0 {
|
||||
ud = multipartUserData(ud, frags)
|
||||
}
|
||||
files := []seedFile{
|
||||
{Path: "var/lib/cloud/seed/nocloud/meta-data", Mode: "0600", Content: meta},
|
||||
{Path: "var/lib/cloud/seed/nocloud/user-data", Mode: "0600", Content: ud},
|
||||
{Path: "etc/cloud/cloud.cfg.d/99-rollcall.cfg", Mode: "0644", Content: "# written by rollcall at deploy time\ndatasource_list: [ NoCloud, None ]\n"},
|
||||
// ca_certs is "not verified" on some distros (Rocky, Alma) and skipped unless allowed
|
||||
{Path: "etc/cloud/cloud.cfg.d/99-rollcall.cfg", Mode: "0644", Content: "# written by rollcall at deploy time\ndatasource_list: [ NoCloud, None ]\nunverified_modules: [ ca_certs ]\n"},
|
||||
}
|
||||
if sp.NetworkConfig != "" {
|
||||
files = append(files, seedFile{Path: "var/lib/cloud/seed/nocloud/network-config", Mode: "0600", Content: sp.NetworkConfig})
|
||||
|
||||
+218
-24
@@ -1,50 +1,244 @@
|
||||
package main
|
||||
|
||||
// A machine's BMC: the record commissioning (or an operator) leaves in
|
||||
// machines.bmc, the sealed password, and the API that sets and probes it.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// bmcCreds is what the server needs to talk to a machine's BMC.
|
||||
// bmcCreds is what a power driver needs.
|
||||
type bmcCreds struct {
|
||||
Address string
|
||||
Port int
|
||||
User string
|
||||
Password string
|
||||
}
|
||||
|
||||
var powerActions = map[string][]string{
|
||||
"status": {"chassis", "power", "status"},
|
||||
"on": {"chassis", "power", "on"},
|
||||
"off": {"chassis", "power", "off"},
|
||||
"soft": {"chassis", "power", "soft"},
|
||||
"cycle": {"chassis", "power", "cycle"},
|
||||
"reset": {"chassis", "power", "reset"},
|
||||
Driver string `json:"driver"`
|
||||
Address string `json:"address"`
|
||||
Port int `json:"port"`
|
||||
User string `json:"user"`
|
||||
Password string `json:"password"`
|
||||
Extra map[string]string `json:"extra"`
|
||||
}
|
||||
|
||||
var bootDevices = map[string]bool{"pxe": true, "disk": true, "cdrom": true, "bios": true, "none": true, "safe": true}
|
||||
|
||||
type bmcRecord struct {
|
||||
Driver string `json:"driver"`
|
||||
Address string `json:"address"`
|
||||
Port int `json:"port"`
|
||||
User string `json:"user"`
|
||||
Extra map[string]string `json:"extra"`
|
||||
}
|
||||
|
||||
func (a *App) credsFor(m *Machine) (*bmcCreds, error) {
|
||||
if m.BMC == nil || m.bmcSecret == nil {
|
||||
return nil, fmt.Errorf("no BMC credentials stored")
|
||||
}
|
||||
var b struct {
|
||||
Address string `json:"address"`
|
||||
Port int `json:"port"`
|
||||
User string `json:"user"`
|
||||
}
|
||||
var b bmcRecord
|
||||
if err := json.Unmarshal(m.BMC, &b); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if b.Address == "" || b.User == "" {
|
||||
return nil, fmt.Errorf("BMC record has no address/user")
|
||||
if b.Driver == "manual" {
|
||||
return &bmcCreds{Driver: "manual"}, nil
|
||||
}
|
||||
if b.Address == "" {
|
||||
return nil, fmt.Errorf("BMC record has no address")
|
||||
}
|
||||
pw, err := a.sealer.Open(m.bmcSecret, m.ID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("decrypt BMC secret: %w", err)
|
||||
}
|
||||
if b.Port == 0 {
|
||||
b.Port = 623
|
||||
if b.Driver == "" {
|
||||
b.Driver = "ipmi"
|
||||
}
|
||||
return &bmcCreds{Address: b.Address, Port: b.Port, User: b.User, Password: pw}, nil
|
||||
if b.Extra == nil {
|
||||
b.Extra = map[string]string{}
|
||||
}
|
||||
return &bmcCreds{Driver: b.Driver, Address: b.Address, Port: b.Port, User: b.User, Password: pw, Extra: b.Extra}, nil
|
||||
}
|
||||
|
||||
var errNoBMC = errors.New("no BMC credentials stored")
|
||||
|
||||
// power runs a power operation for a machine on the rack controller that can reach its BMC.
|
||||
func (a *App) power(ctx context.Context, m *Machine, op, arg string) (string, error) {
|
||||
c, err := a.credsFor(m)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%w: %v", errNoBMC, err)
|
||||
}
|
||||
return a.powerVia(ctx, a.rackOf(m), c, op, arg)
|
||||
}
|
||||
|
||||
func (a *App) powerVia(ctx context.Context, rack string, c *bmcCreds, op, arg string) (string, error) {
|
||||
if rack == a.cfg.RackName || rack == "" {
|
||||
return powerRun(ctx, a.cfg.IPMITool, c, op, arg)
|
||||
}
|
||||
r, ok := a.racksSnapshot()[rack]
|
||||
if !ok || !rackOnline(r, a.cfg.RackName) {
|
||||
return "", fmt.Errorf("rack controller %s is offline", rack)
|
||||
}
|
||||
return a.hub.dispatch(ctx, rack, &rackJob{Kind: "power", Driver: c.Driver, Address: c.Address, Port: c.Port, User: c.User,
|
||||
Password: c.Password, Extra: c.Extra, Op: op, Arg: arg})
|
||||
}
|
||||
|
||||
func (s *Store) SetBMC(ctx context.Context, id string, bmc []byte, secret []byte) error {
|
||||
if secret != nil {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE machines SET bmc=$2, bmc_secret=$3 WHERE id=$1`, id, bmc, secret)
|
||||
return err
|
||||
}
|
||||
_, err := s.pool.Exec(ctx, `UPDATE machines SET bmc=$2 WHERE id=$1`, id, bmc)
|
||||
return err
|
||||
}
|
||||
|
||||
// probeBMC: after commissioning, a BMC still on plain IPMI is asked whether it
|
||||
// speaks Redfish with the same account; if it does, the vendor's driver takes over.
|
||||
// An operator's explicit choice (driver_set) is left alone.
|
||||
func (a *App) probeBMC(m *Machine) {
|
||||
if m.BMC == nil || m.bmcSecret == nil {
|
||||
return
|
||||
}
|
||||
var b obj
|
||||
if json.Unmarshal(m.BMC, &b) != nil || b["driver_set"] == true || str(b["address"]) == "" {
|
||||
return
|
||||
}
|
||||
if d := str(b["driver"]); d != "" && d != "ipmi" {
|
||||
return
|
||||
}
|
||||
c, err := a.credsFor(m)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
probe := *c
|
||||
probe.Driver, probe.Port = "redfish", 0
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90e9)
|
||||
defer cancel()
|
||||
out, err := a.powerVia(ctx, a.rackOf(m), &probe, "probe", "")
|
||||
if err != nil {
|
||||
_ = a.store.AddEvent(ctx, m.ID, "bmc probe", "staying on ipmi: "+oneLine(err.Error()))
|
||||
return
|
||||
}
|
||||
var res obj
|
||||
if json.Unmarshal([]byte(out), &res) != nil || str(res["driver"]) == "" {
|
||||
return
|
||||
}
|
||||
extra, _ := b["extra"].(map[string]any)
|
||||
if extra == nil {
|
||||
extra = map[string]any{}
|
||||
}
|
||||
extra["system"] = str(res["system"])
|
||||
b["driver"], b["extra"], b["redfish"] = str(res["driver"]), extra, res
|
||||
nb, _ := json.Marshal(b)
|
||||
if err := a.store.SetBMC(ctx, m.ID, nb, nil); err != nil {
|
||||
log.Printf("probe %s: %v", m.ID, err)
|
||||
return
|
||||
}
|
||||
_ = a.store.AddEvent(ctx, m.ID, "bmc driver", fmt.Sprintf("%s (Redfish %s, %s %s)", str(res["driver"]), str(res["redfish"]), str(res["manufacturer"]), str(res["model"])))
|
||||
log.Printf("%s: BMC speaks Redfish %s (%s) -> driver %s", m.ID, str(res["redfish"]), str(res["vendor"]), str(res["driver"]))
|
||||
}
|
||||
|
||||
// handleBMC: GET shows the record (no secret); POST sets driver, address,
|
||||
// account and extra settings, checked with a status call first when verify is set.
|
||||
func (a *App) handleBMC(w http.ResponseWriter, r *http.Request) {
|
||||
m, err := a.store.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
var b obj
|
||||
_ = json.Unmarshal(m.BMC, &b)
|
||||
if b == nil {
|
||||
b = obj{}
|
||||
}
|
||||
if r.Method == http.MethodGet {
|
||||
writeJSON(w, 200, obj{"id": m.ID, "bmc": b, "has_secret": m.bmcSecret != nil, "drivers": powerDrivers})
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Driver *string `json:"driver"`
|
||||
Address *string `json:"address"`
|
||||
Port *int `json:"port"`
|
||||
User *string `json:"user"`
|
||||
Password *string `json:"password"`
|
||||
Extra map[string]string `json:"extra"`
|
||||
Verify bool `json:"verify"`
|
||||
Probe bool `json:"probe"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
if req.Probe { // re-run Redfish detection
|
||||
b["driver_set"] = false
|
||||
b["driver"] = "ipmi"
|
||||
nb, _ := json.Marshal(b)
|
||||
_ = a.store.SetBMC(r.Context(), m.ID, nb, nil)
|
||||
m, _ = a.store.Get(r.Context(), m.ID)
|
||||
a.probeBMC(m)
|
||||
m, _ = a.store.Get(r.Context(), m.ID)
|
||||
_ = json.Unmarshal(m.BMC, &b)
|
||||
writeJSON(w, 200, obj{"id": m.ID, "bmc": b})
|
||||
return
|
||||
}
|
||||
if req.Driver != nil {
|
||||
if _, ok := driverByName(*req.Driver); !ok {
|
||||
fail(w, 400, fmt.Errorf("unknown driver %q", *req.Driver))
|
||||
return
|
||||
}
|
||||
b["driver"] = *req.Driver
|
||||
}
|
||||
if req.Address != nil {
|
||||
b["address"] = strings.TrimSpace(*req.Address)
|
||||
}
|
||||
if req.Port != nil {
|
||||
b["port"] = *req.Port
|
||||
}
|
||||
if req.User != nil {
|
||||
b["user"] = *req.User
|
||||
}
|
||||
if req.Extra != nil {
|
||||
ex, _ := b["extra"].(map[string]any)
|
||||
if ex == nil {
|
||||
ex = map[string]any{}
|
||||
}
|
||||
for k, v := range req.Extra {
|
||||
if v == "" {
|
||||
delete(ex, k)
|
||||
} else {
|
||||
ex[k] = v
|
||||
}
|
||||
}
|
||||
b["extra"] = ex
|
||||
}
|
||||
b["driver_set"] = true
|
||||
if str(b["source"]) == "" {
|
||||
b["source"] = "operator"
|
||||
}
|
||||
secret := m.bmcSecret
|
||||
if req.Password != nil {
|
||||
if secret, err = a.sealer.Seal(*req.Password, m.ID); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
nb, _ := json.Marshal(b)
|
||||
trial := *m
|
||||
trial.BMC, trial.bmcSecret = nb, secret
|
||||
out := ""
|
||||
if req.Verify {
|
||||
out, err = a.power(r.Context(), &trial, "status", "")
|
||||
if err != nil {
|
||||
fail(w, 409, fmt.Errorf("not saved, the BMC did not answer: %v %s", err, out))
|
||||
return
|
||||
}
|
||||
b["configured"], b["verified_oob"] = true, true
|
||||
nb, _ = json.Marshal(b)
|
||||
}
|
||||
if err := a.store.SetBMC(r.Context(), m.ID, nb, secret); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
_ = a.store.AddEvent(r.Context(), m.ID, "bmc set", fmt.Sprintf("driver=%s address=%s user=%s", str(b["driver"]), str(b["address"]), str(b["user"])))
|
||||
writeJSON(w, 200, obj{"id": m.ID, "bmc": b, "output": out})
|
||||
}
|
||||
+17
-2
@@ -28,12 +28,15 @@ func usage() {
|
||||
rollcall tui [-dump WxH -page P -machine ID -plain]
|
||||
rollcall serve [-config PATH] region: API, database, web UI, embedded rack controller
|
||||
rollcall rack [-config /etc/rollcall/rack.json] a rack controller next to the machines
|
||||
rollcall rack join CODE -url http://ADDR [-name N] [-listen ADDR:80] enroll a rack controller (writes rack.json + certificate)
|
||||
rollcall ca show | issue NAME -host h1,h2 [-client] [-out DIR] the installation's certificate authority (region host)
|
||||
rollcall version
|
||||
rollcall keygen
|
||||
rollcall list
|
||||
rollcall show ID [-json]
|
||||
rollcall events ID
|
||||
rollcall power ID status|on|off|soft|cycle|reset
|
||||
rollcall bmc ID [-driver D] [-address A] [-user U] [-password P|-] [-set K=V] [-verify] [-probe] | bmc drivers
|
||||
rollcall bootdev ID pxe|disk|cdrom|bios|none
|
||||
rollcall recommission ID [-now]
|
||||
rollcall delete ID
|
||||
@@ -60,7 +63,7 @@ func usage() {
|
||||
rollcall ip list [-subnet CIDR] | reserve IP [-name N] | release IP
|
||||
rollcall layout list | show NAME | add NAME FILE.json | rm NAME
|
||||
rollcall net show ID | set ID [-layout L] [-rack R] [-static IFACE=IP] [-unstatic IP]
|
||||
rollcall controllers | controller rm NAME | dhcp CONTROLLER
|
||||
rollcall controllers | controller rm NAME | controller join-code [-name N] [-ttl 60] [-addr HOST:443] | dhcp CONTROLLER
|
||||
(deploy, power and location take tag:NAME instead of ID to act on every machine with that tag)
|
||||
env: ROLLCALL_URL (default from config listen), ROLLCALL_TOKEN, ROLLCALL_CONFIG;
|
||||
away from the server the console saves URL and token in ~/.config/rollcall/client.json
|
||||
@@ -86,9 +89,21 @@ func main() {
|
||||
cmd, args := os.Args[1], os.Args[2:]
|
||||
switch cmd {
|
||||
case "rack":
|
||||
if len(args) > 0 && args[0] == "join" {
|
||||
if err := rackJoin(args[1:]); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err := runRack(args); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
case "ca":
|
||||
if err := caCmd(args); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
case "tui", "console":
|
||||
if err := tuiDump(args); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
@@ -108,7 +123,7 @@ func main() {
|
||||
case "list", "show", "events", "power", "bootdev", "recommission", "delete",
|
||||
"images", "profile", "assign", "unassign", "overlay", "render",
|
||||
"deploy", "release", "jobs", "job", "tag", "tags", "location",
|
||||
"fabric", "vlan", "subnet", "range", "ip", "layout", "net", "controllers", "controller", "dhcp":
|
||||
"bmc", "fabric", "vlan", "subnet", "range", "ip", "layout", "net", "controllers", "controller", "dhcp":
|
||||
if err := runClient(cmd, args); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
|
||||
+40
-11
@@ -116,12 +116,18 @@ func (a *App) handleSubnets(w http.ResponseWriter, r *http.Request) {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
ip, n, err := net.ParseCIDR(sn.CIDR)
|
||||
if err != nil || ip.To4() == nil {
|
||||
fail(w, 400, errors.New("cidr must be an IPv4 network, e.g. 10.0.0.0/24"))
|
||||
_, n, err := net.ParseCIDR(sn.CIDR)
|
||||
if err != nil {
|
||||
fail(w, 400, errors.New("cidr must be a network, e.g. 10.0.0.0/24 or fd00:1::/64"))
|
||||
return
|
||||
}
|
||||
sn.CIDR = n.String()
|
||||
if familyOf(sn.CIDR) == 6 {
|
||||
if ones, _ := n.Mask.Size(); ones > 126 {
|
||||
fail(w, 400, errors.New("an IPv6 subnet needs room: /126 or shorter (/64 is usual)"))
|
||||
return
|
||||
}
|
||||
}
|
||||
if sn.Gateway != "" && !sn.contains(sn.Gateway) {
|
||||
fail(w, 400, fmt.Errorf("gateway %s is outside %s", sn.Gateway, sn.CIDR))
|
||||
return
|
||||
@@ -202,16 +208,17 @@ func (a *App) handleRanges(w http.ResponseWriter, r *http.Request) {
|
||||
fail(w, 404, fmt.Errorf("subnet %s: %w", rg.Subnet, errSubnetUnknown))
|
||||
return
|
||||
}
|
||||
lo, ok1 := ip4(rg.Start)
|
||||
hi, ok2 := ip4(rg.End)
|
||||
if !ok1 || !ok2 || hi < lo || !sn.contains(rg.Start) || !sn.contains(rg.End) {
|
||||
lo, ok1 := parseAddr(rg.Start)
|
||||
hi, ok2 := parseAddr(rg.End)
|
||||
if !ok1 || !ok2 || hi.Less(lo) || !sn.contains(rg.Start) || !sn.contains(rg.End) {
|
||||
fail(w, 400, fmt.Errorf("range %s-%s is not inside %s", rg.Start, rg.End, sn.CIDR))
|
||||
return
|
||||
}
|
||||
rg.Start, rg.End = lo.String(), hi.String()
|
||||
for _, o := range sn.Ranges {
|
||||
ol, _ := ip4(o.Start)
|
||||
oh, _ := ip4(o.End)
|
||||
if lo <= oh && ol <= hi {
|
||||
ol, _ := parseAddr(o.Start)
|
||||
oh, _ := parseAddr(o.End)
|
||||
if !oh.Less(lo) && !hi.Less(ol) {
|
||||
fail(w, 409, fmt.Errorf("overlaps the %s range %s-%s", o.Kind, o.Start, o.End))
|
||||
return
|
||||
}
|
||||
@@ -534,7 +541,7 @@ func (c *client) netCmd(cmd string, args []string) error {
|
||||
if err := c.do("GET", "/api/v1/subnets", nil, &ss); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%-18s %-12s %-14s %-12s %-15s %-12s %9s %s\n", "SUBNET", "PURPOSE", "FABRIC/VLAN", "DHCP RACK", "GATEWAY", "DOMAIN", "USED", "RANGES")
|
||||
fmt.Printf("%-24s %-12s %-14s %-12s %-15s %-12s %9s %s\n", "SUBNET", "PURPOSE", "FABRIC/VLAN", "DHCP RACK", "GATEWAY", "DOMAIN", "USED", "RANGES")
|
||||
for _, s := range ss {
|
||||
dhcp := "-"
|
||||
if s.DHCP {
|
||||
@@ -547,7 +554,11 @@ func (c *client) netCmd(cmd string, args []string) error {
|
||||
for _, r := range s.Ranges {
|
||||
rs = append(rs, fmt.Sprintf("%s:%s-%s", r.Kind, r.Start, lastOctets(r.End)))
|
||||
}
|
||||
fmt.Printf("%-18s %-12s %-14s %-12s %-15s %-12s %4d/%-4d %s\n", s.CIDR, s.Purpose, fmt.Sprintf("%s/%d", orDash(s.Fabric), s.VID), dhcp, orDash(s.Gateway), orDash(s.Domain), s.Used, s.Size, strings.Join(rs, " "))
|
||||
size := strconv.Itoa(s.Size)
|
||||
if s.Size == 0 {
|
||||
size = "∞"
|
||||
}
|
||||
fmt.Printf("%-24s %-12s %-14s %-12s %-15s %-12s %4d/%-4s %s\n", s.CIDR, s.Purpose, fmt.Sprintf("%s/%d", orDash(s.Fabric), s.VID), dhcp, orDash(s.Gateway), orDash(s.Domain), s.Used, size, strings.Join(rs, " "))
|
||||
}
|
||||
case "show":
|
||||
need(2)
|
||||
@@ -743,6 +754,19 @@ func (c *client) netCmd(cmd string, args []string) error {
|
||||
case cmd == "controller" && sub == "rm":
|
||||
need(2)
|
||||
return c.do("DELETE", "/api/v1/controllers/"+args[1], nil, nil)
|
||||
case cmd == "controller" && sub == "join-code":
|
||||
fs := newFlags("controller join-code")
|
||||
name := fs.String("name", "", "bind the code to this rack controller name")
|
||||
ttl := fs.Int("ttl", 60, "minutes the code is good for")
|
||||
addr := fs.String("addr", "", "host:port the rack reaches the region's HTTPS at (default: how you reached it)")
|
||||
_ = fs.Parse(args[1:])
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/controllers/join", obj{"name": *name, "ttl_minutes": *ttl, "addr": *addr}, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s\n\none use, until %s; region %s, CA %s\non the rack host: rollcall rack join <code> -url http://<its address on the segment>\n",
|
||||
str(d["code"]), str(d["expires"]), str(d["addr"]), str(d["ca_fingerprint"]))
|
||||
return nil
|
||||
default:
|
||||
var rs []RackController
|
||||
if err := c.do("GET", "/api/v1/controllers", nil, &rs); err != nil {
|
||||
@@ -774,6 +798,11 @@ func (m *multiFlag) String() string { return strings.Join(*m, ",") }
|
||||
func (m *multiFlag) Set(v string) error { *m = append(*m, v); return nil }
|
||||
|
||||
func lastOctets(ip string) string {
|
||||
if strings.Contains(ip, ":") {
|
||||
if i := strings.LastIndex(ip, ":"); i >= 0 {
|
||||
return ip[i:]
|
||||
}
|
||||
}
|
||||
if i := strings.LastIndex(ip, "."); i >= 0 {
|
||||
return ip[i+1:]
|
||||
}
|
||||
|
||||
+213
-66
@@ -13,6 +13,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/netip"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
@@ -63,7 +64,8 @@ type Subnet struct {
|
||||
Comment string `json:"comment"`
|
||||
Ranges []IPRange `json:"ranges"`
|
||||
Used int `json:"used"`
|
||||
Size int `json:"size"`
|
||||
Size int `json:"size"` // host addresses; 0 when too many to count (IPv6)
|
||||
Family int `json:"family"` // 4 or 6
|
||||
}
|
||||
|
||||
type IPAddr struct {
|
||||
@@ -192,9 +194,17 @@ func (s *Store) Subnets(ctx context.Context) ([]Subnet, error) {
|
||||
if sn.DNS == nil {
|
||||
sn.DNS = []string{}
|
||||
}
|
||||
sn.Family = familyOf(sn.CIDR)
|
||||
if _, n, err := net.ParseCIDR(sn.CIDR); err == nil {
|
||||
ones, bits := n.Mask.Size()
|
||||
sn.Size = max(0, 1<<(bits-ones)-2)
|
||||
switch {
|
||||
case bits-ones > 30:
|
||||
sn.Size = 0
|
||||
case sn.Family == 6:
|
||||
sn.Size = 1<<(bits-ones) - 1
|
||||
default:
|
||||
sn.Size = max(0, 1<<(bits-ones)-2)
|
||||
}
|
||||
}
|
||||
sn.Ranges = []IPRange{}
|
||||
out = append(out, sn)
|
||||
@@ -374,15 +384,31 @@ func u2ip(u uint32) string {
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// parseAddr reads an IPv4 or IPv6 address (brackets and a /prefix are ignored).
|
||||
func parseAddr(s string) (netip.Addr, bool) {
|
||||
a, err := netip.ParseAddr(strings.Trim(stripPrefix(strings.TrimSpace(s)), "[]"))
|
||||
if err != nil {
|
||||
return netip.Addr{}, false
|
||||
}
|
||||
return a.Unmap(), true
|
||||
}
|
||||
|
||||
func ipLess(a, b string) bool {
|
||||
x, ok1 := ip4(a)
|
||||
y, ok2 := ip4(b)
|
||||
x, ok1 := parseAddr(a)
|
||||
y, ok2 := parseAddr(b)
|
||||
if ok1 && ok2 {
|
||||
return x < y
|
||||
return x.Less(y)
|
||||
}
|
||||
return a < b
|
||||
}
|
||||
|
||||
func familyOf(cidr string) int {
|
||||
if p, err := netip.ParsePrefix(cidr); err == nil && p.Addr().Is6() && !p.Addr().Is4In6() {
|
||||
return 6
|
||||
}
|
||||
return 4
|
||||
}
|
||||
|
||||
func stripPrefix(s string) string { ip, _, _ := strings.Cut(s, "/"); return ip }
|
||||
|
||||
func (sn Subnet) contains(ip string) bool {
|
||||
@@ -426,24 +452,22 @@ func subnetByRef(subnets []Subnet, ref string) *Subnet {
|
||||
|
||||
// freeCandidates lists the addresses IPAM may hand out in a subnet, in order:
|
||||
// inside static ranges when the subnet has any, never in dynamic or reserved
|
||||
// ranges, never the network, broadcast, gateway, a rack controller or a used one.
|
||||
// ranges, never the network (or IPv4 broadcast), the gateway, a rack
|
||||
// controller or a used one. IPv6 subnets are walked from the start of the
|
||||
// static ranges (or the subnet) for a bounded number of steps.
|
||||
func freeCandidates(sn Subnet, used map[string]bool, avoid []string, limit int) []string {
|
||||
_, n, err := net.ParseCIDR(sn.CIDR)
|
||||
if err != nil || n.IP.To4() == nil {
|
||||
pfx, err := netip.ParsePrefix(sn.CIDR)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
ones, bits := n.Mask.Size()
|
||||
if bits-ones > 16 { // a /15 or larger: refuse to walk it
|
||||
ones = bits - 16
|
||||
}
|
||||
base := binary.BigEndian.Uint32(n.IP.To4())
|
||||
last := base | (1<<(bits-ones) - 1)
|
||||
type span struct{ lo, hi uint32 }
|
||||
pfx = pfx.Masked()
|
||||
base := pfx.Addr()
|
||||
type span struct{ lo, hi netip.Addr }
|
||||
var static, blocked []span
|
||||
for _, r := range sn.Ranges {
|
||||
lo, ok1 := ip4(r.Start)
|
||||
hi, ok2 := ip4(r.End)
|
||||
if !ok1 || !ok2 {
|
||||
lo, ok1 := parseAddr(r.Start)
|
||||
hi, ok2 := parseAddr(r.End)
|
||||
if !ok1 || !ok2 || hi.Less(lo) {
|
||||
continue
|
||||
}
|
||||
if r.Kind == "static" {
|
||||
@@ -452,35 +476,58 @@ func freeCandidates(sn Subnet, used map[string]bool, avoid []string, limit int)
|
||||
blocked = append(blocked, span{lo, hi})
|
||||
}
|
||||
}
|
||||
skip := map[uint32]bool{base: true, last: true}
|
||||
if g, ok := ip4(sn.Gateway); ok {
|
||||
skip := map[netip.Addr]bool{base: true}
|
||||
if base.Is4() {
|
||||
last := base.As4()
|
||||
bits := pfx.Bits()
|
||||
v := binary.BigEndian.Uint32(last[:]) | (1<<(32-bits) - 1)
|
||||
var b [4]byte
|
||||
binary.BigEndian.PutUint32(b[:], v)
|
||||
skip[netip.AddrFrom4(b)] = true
|
||||
}
|
||||
if g, ok := parseAddr(sn.Gateway); ok {
|
||||
skip[g] = true
|
||||
}
|
||||
for _, a := range avoid {
|
||||
if u, ok := ip4(stripPrefix(a)); ok {
|
||||
if u, ok := parseAddr(a); ok {
|
||||
skip[u] = true
|
||||
}
|
||||
}
|
||||
in := func(x uint32, ss []span) bool {
|
||||
in := func(x netip.Addr, ss []span) bool {
|
||||
for _, s := range ss {
|
||||
if x >= s.lo && x <= s.hi {
|
||||
if !x.Less(s.lo) && !s.hi.Less(x) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
var out []string
|
||||
for x := base + 1; x < last; x++ {
|
||||
if skip[x] || in(x, blocked) || (len(static) > 0 && !in(x, static)) {
|
||||
continue
|
||||
try := func(x netip.Addr) bool {
|
||||
if skip[x] || in(x, blocked) || used[x.String()] {
|
||||
return false
|
||||
}
|
||||
ip := u2ip(x)
|
||||
if used[ip] {
|
||||
continue
|
||||
out = append(out, x.String())
|
||||
return limit > 0 && len(out) >= limit
|
||||
}
|
||||
const maxSteps = 1 << 16
|
||||
if len(static) > 0 {
|
||||
sort.Slice(static, func(i, j int) bool { return static[i].lo.Less(static[j].lo) })
|
||||
for _, s := range static {
|
||||
steps := 0
|
||||
for x := s.lo; pfx.Contains(x) && !s.hi.Less(x) && steps < maxSteps; x = x.Next() {
|
||||
steps++
|
||||
if try(x) {
|
||||
return out
|
||||
}
|
||||
}
|
||||
}
|
||||
out = append(out, ip)
|
||||
if limit > 0 && len(out) >= limit {
|
||||
break
|
||||
return out
|
||||
}
|
||||
steps := 0
|
||||
for x := base.Next(); x.IsValid() && pfx.Contains(x) && steps < maxSteps; x = x.Next() {
|
||||
steps++
|
||||
if try(x) {
|
||||
return out
|
||||
}
|
||||
}
|
||||
return out
|
||||
@@ -636,6 +683,7 @@ func selectNICs(nics []invNIC, boot *invNIC, sel string) []invNIC {
|
||||
// ---------------------------------------------------------------- plan
|
||||
|
||||
type PlanLink struct {
|
||||
Family int `json:"family"`
|
||||
Subnet string `json:"subnet"`
|
||||
Mode string `json:"mode"`
|
||||
IP string `json:"ip"`
|
||||
@@ -667,12 +715,20 @@ type NetPlan struct {
|
||||
member map[string]invNIC
|
||||
}
|
||||
|
||||
func defaultLayout(bootSubnet *Subnet) *NetLayout {
|
||||
link := LayoutLink{Subnet: "boot", Mode: "auto", Gateway: true}
|
||||
// defaultLayout: the boot NIC on its subnet, plus every other subnet of the same
|
||||
// VLAN (an IPv6 prefix next to the IPv4 one makes the machine dual-stack).
|
||||
func defaultLayout(bootSubnet *Subnet, subnets []Subnet) *NetLayout {
|
||||
links := []LayoutLink{{Subnet: "boot", Mode: "auto", Gateway: true}}
|
||||
if bootSubnet == nil {
|
||||
link = LayoutLink{Subnet: "boot", Mode: "dhcp"}
|
||||
links = []LayoutLink{{Subnet: "boot", Mode: "dhcp"}}
|
||||
} else if bootSubnet.VLANID != 0 {
|
||||
for _, s := range subnets {
|
||||
if s.VLANID == bootSubnet.VLANID && s.CIDR != bootSubnet.CIDR {
|
||||
links = append(links, LayoutLink{Subnet: s.CIDR, Mode: "auto"})
|
||||
}
|
||||
}
|
||||
}
|
||||
return &NetLayout{Name: "(default)", Interfaces: []LayoutIface{{Type: "physical", Select: "boot", Links: []LayoutLink{link}}}}
|
||||
return &NetLayout{Name: "(default)", Interfaces: []LayoutIface{{Type: "physical", Select: "boot", Links: links}}}
|
||||
}
|
||||
|
||||
// resolvePlan turns a layout into concrete interfaces for one machine. alloc is
|
||||
@@ -689,7 +745,7 @@ func resolvePlan(l *NetLayout, nics []invNIC, boot *invNIC, subnets []Subnet, ha
|
||||
}
|
||||
}
|
||||
if l == nil {
|
||||
l = defaultLayout(bootSubnet)
|
||||
l = defaultLayout(bootSubnet, subnets)
|
||||
}
|
||||
p := &NetPlan{Layout: l.Name, Warnings: []string{}, member: map[string]invNIC{}}
|
||||
used := map[string]bool{}
|
||||
@@ -796,13 +852,13 @@ func resolvePlan(l *NetLayout, nics []invNIC, boot *invNIC, subnets []Subnet, ha
|
||||
}
|
||||
if sn == nil {
|
||||
if mode == "dhcp" || mode == "link_up" {
|
||||
pi.Links = append(pi.Links, PlanLink{Mode: mode})
|
||||
pi.Links = append(pi.Links, PlanLink{Mode: mode, Family: 4})
|
||||
continue
|
||||
}
|
||||
p.Warnings = append(p.Warnings, fmt.Sprintf("%s: subnet %q is not defined", pi.Name, lk.Subnet))
|
||||
continue
|
||||
}
|
||||
pl := PlanLink{Subnet: sn.CIDR, Mode: mode, Prefix: sn.bits(), Gateway: sn.Gateway, Default: lk.Gateway, DNS: sn.DNS, Domain: sn.Domain}
|
||||
pl := PlanLink{Family: familyOf(sn.CIDR), Subnet: sn.CIDR, Mode: mode, Prefix: sn.bits(), Gateway: sn.Gateway, Default: lk.Gateway, DNS: sn.DNS, Domain: sn.Domain}
|
||||
if pi.MTU == 0 && sn.MTU > 0 && sn.MTU != 1500 {
|
||||
pi.MTU = sn.MTU
|
||||
}
|
||||
@@ -830,14 +886,18 @@ func resolvePlan(l *NetLayout, nics []invNIC, boot *invNIC, subnets []Subnet, ha
|
||||
}
|
||||
p.Ifaces = append(p.Ifaces, pi)
|
||||
}
|
||||
// one default route: the flagged link, else the first addressed link with a gateway on the boot NIC, else anywhere
|
||||
hasDefault := false
|
||||
for _, pi := range p.Ifaces {
|
||||
for _, pl := range pi.Links {
|
||||
hasDefault = hasDefault || pl.Default
|
||||
// one default route per family: the flagged link, else the first addressed
|
||||
// link with a gateway on the boot NIC, else anywhere
|
||||
for _, fam := range []int{4, 6} {
|
||||
hasDefault := false
|
||||
for _, pi := range p.Ifaces {
|
||||
for _, pl := range pi.Links {
|
||||
hasDefault = hasDefault || (pl.Default && pl.Family == fam)
|
||||
}
|
||||
}
|
||||
if hasDefault {
|
||||
continue
|
||||
}
|
||||
}
|
||||
if !hasDefault {
|
||||
pickDefault := func(onBoot bool) bool {
|
||||
for i := range p.Ifaces {
|
||||
carriesBoot := false
|
||||
@@ -849,7 +909,7 @@ func resolvePlan(l *NetLayout, nics []invNIC, boot *invNIC, subnets []Subnet, ha
|
||||
}
|
||||
for j := range p.Ifaces[i].Links {
|
||||
pl := &p.Ifaces[i].Links[j]
|
||||
if pl.Gateway != "" && (pl.Mode == "auto" || pl.Mode == "static") {
|
||||
if pl.Family == fam && pl.Gateway != "" && (pl.Mode == "auto" || pl.Mode == "static") {
|
||||
pl.Default = true
|
||||
return true
|
||||
}
|
||||
@@ -864,13 +924,19 @@ func resolvePlan(l *NetLayout, nics []invNIC, boot *invNIC, subnets []Subnet, ha
|
||||
return p
|
||||
}
|
||||
|
||||
// primaryIPs returns the address carrying the default route and the first IB address.
|
||||
// primaryIPs returns the address carrying the default route (IPv4 first) and the first IB address.
|
||||
func (p *NetPlan) primaryIPs() (mgmt, ib string) {
|
||||
for _, fam := range []int{4, 6} {
|
||||
for _, pi := range p.Ifaces {
|
||||
for _, pl := range pi.Links {
|
||||
if pl.Default && pl.IP != "" && mgmt == "" && pl.Family == fam {
|
||||
mgmt = pl.IP
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, pi := range p.Ifaces {
|
||||
for _, pl := range pi.Links {
|
||||
if pl.Default && pl.IP != "" && mgmt == "" {
|
||||
mgmt = pl.IP
|
||||
}
|
||||
if pi.IB && pl.IP != "" && ib == "" {
|
||||
ib = pl.IP
|
||||
}
|
||||
@@ -897,18 +963,27 @@ func networkConfig(p *NetPlan) map[string]any {
|
||||
e["mtu"] = pi.MTU
|
||||
}
|
||||
var addrs, dns, search []string
|
||||
dhcp := false
|
||||
var routes []map[string]any
|
||||
dhcp4, dhcp6 := false, false
|
||||
for _, pl := range pi.Links {
|
||||
switch pl.Mode {
|
||||
case "dhcp":
|
||||
dhcp = true
|
||||
if pl.Family == 6 {
|
||||
dhcp6 = true
|
||||
} else {
|
||||
dhcp4 = true
|
||||
}
|
||||
case "auto", "static":
|
||||
if pl.IP == "" {
|
||||
continue
|
||||
}
|
||||
addrs = append(addrs, fmt.Sprintf("%s/%d", pl.IP, pl.Prefix))
|
||||
if pl.Default && pl.Gateway != "" {
|
||||
e["routes"] = []map[string]any{{"to": "0.0.0.0/0", "via": pl.Gateway}}
|
||||
to := "0.0.0.0/0"
|
||||
if pl.Family == 6 {
|
||||
to = "::/0"
|
||||
}
|
||||
routes = append(routes, map[string]any{"to": to, "via": pl.Gateway})
|
||||
}
|
||||
for _, d := range pl.DNS {
|
||||
if !hasTag(dns, d) {
|
||||
@@ -920,10 +995,20 @@ func networkConfig(p *NetPlan) map[string]any {
|
||||
}
|
||||
}
|
||||
}
|
||||
e["dhcp4"] = dhcp
|
||||
e["dhcp4"] = dhcp4
|
||||
if dhcp6 {
|
||||
e["dhcp6"] = true
|
||||
} else {
|
||||
// addresses come from the plan (or none at all): router advertisements must not add
|
||||
// SLAAC/DHCPv6 addresses or a second default route
|
||||
e["accept-ra"] = false
|
||||
}
|
||||
if len(addrs) > 0 {
|
||||
e["addresses"] = addrs
|
||||
}
|
||||
if len(routes) > 0 {
|
||||
e["routes"] = routes
|
||||
}
|
||||
if len(dns) > 0 || len(search) > 0 {
|
||||
ns := map[string]any{}
|
||||
if len(dns) > 0 {
|
||||
@@ -961,6 +1046,7 @@ func networkConfig(p *NetPlan) map[string]any {
|
||||
for _, n := range pi.nics {
|
||||
e := physical(n, nil)
|
||||
e["dhcp4"] = false
|
||||
e["accept-ra"] = false
|
||||
if pi.MTU > 0 {
|
||||
e["mtu"] = pi.MTU
|
||||
}
|
||||
@@ -1162,47 +1248,108 @@ func dnsmasqConfig(rc RackController, subnets []Subnet, dnsDir string) (string,
|
||||
if !shared {
|
||||
b.WriteString("dhcp-authoritative\n")
|
||||
}
|
||||
// IPv6: router advertisements carry the prefix; the rack only offers
|
||||
// itself as the default router when it is the subnet's gateway
|
||||
v6ifaces := map[string]bool{}
|
||||
routerOn := map[string]bool{}
|
||||
for _, sn := range served {
|
||||
if familyOf(sn.CIDR) != 6 {
|
||||
continue
|
||||
}
|
||||
for _, ri := range rc.Interfaces {
|
||||
for _, ad := range ri.Addrs {
|
||||
if sn.contains(ad) {
|
||||
v6ifaces[ri.Name] = true
|
||||
if stripPrefix(ad) == sn.Gateway {
|
||||
routerOn[ri.Name] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(v6ifaces) > 0 {
|
||||
b.WriteString("enable-ra\n")
|
||||
for _, i := range sortedKeys(v6ifaces) {
|
||||
if !routerOn[i] {
|
||||
fmt.Fprintf(&b, "ra-param=%s,10,0\n", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, d := range domains {
|
||||
fmt.Fprintf(&b, "local=/%s/\n", d)
|
||||
}
|
||||
for i, sn := range served {
|
||||
tag := fmt.Sprintf("s%d", i)
|
||||
v6 := familyOf(sn.CIDR) == 6
|
||||
fmt.Fprintf(&b, "\n# %s · %s vlan %d · %s\n", sn.CIDR, orDash(sn.Fabric), sn.VID, sn.Purpose)
|
||||
if sn.Domain != "" {
|
||||
fmt.Fprintf(&b, "domain=%s,%s\n", sn.Domain, sn.CIDR)
|
||||
}
|
||||
mask := sn.netmask()
|
||||
if v6 {
|
||||
mask = strconv.Itoa(sn.bits())
|
||||
}
|
||||
dyn := 0
|
||||
for _, r := range sn.Ranges {
|
||||
if r.Kind != "dynamic" {
|
||||
continue
|
||||
}
|
||||
lo, ok1 := ip4(r.Start)
|
||||
hi, ok2 := ip4(r.End)
|
||||
if !ok1 || !ok2 || hi < lo {
|
||||
lo, ok1 := parseAddr(r.Start)
|
||||
hi, ok2 := parseAddr(r.End)
|
||||
if !ok1 || !ok2 || hi.Less(lo) {
|
||||
continue
|
||||
}
|
||||
// two racks on one VLAN: each serves its own half of the pool
|
||||
if sn.SecondaryRack != "" {
|
||||
mid := lo + (hi-lo)/2
|
||||
if sn.SecondaryRack != "" && lo.Is4() {
|
||||
l4, h4 := lo.As4(), hi.As4()
|
||||
a, z := binary.BigEndian.Uint32(l4[:]), binary.BigEndian.Uint32(h4[:])
|
||||
mid := a + (z-a)/2
|
||||
if rc.Name == sn.PrimaryRack {
|
||||
hi = mid
|
||||
z = mid
|
||||
} else {
|
||||
lo = mid + 1
|
||||
a = mid + 1
|
||||
}
|
||||
var bl, bh [4]byte
|
||||
binary.BigEndian.PutUint32(bl[:], a)
|
||||
binary.BigEndian.PutUint32(bh[:], z)
|
||||
lo, hi = netip.AddrFrom4(bl), netip.AddrFrom4(bh)
|
||||
}
|
||||
fmt.Fprintf(&b, "dhcp-range=set:%s,%s,%s,%s,12h\n", tag, u2ip(lo), u2ip(hi), sn.netmask())
|
||||
fmt.Fprintf(&b, "dhcp-range=set:%s,%s,%s,%s,12h\n", tag, lo, hi, mask)
|
||||
dyn++
|
||||
}
|
||||
if dyn == 0 {
|
||||
// static leases only (known machines), no pool for strangers
|
||||
_, n, _ := net.ParseCIDR(sn.CIDR)
|
||||
fmt.Fprintf(&b, "dhcp-range=set:%s,%s,static,%s,12h\n", tag, n.IP.String(), sn.netmask())
|
||||
fmt.Fprintf(&b, "dhcp-range=set:%s,%s,static,%s,12h\n", tag, n.IP.String(), mask)
|
||||
}
|
||||
if v6 {
|
||||
if len(sn.DNS) > 0 {
|
||||
var ds []string
|
||||
for _, d := range sn.DNS {
|
||||
if a, ok := parseAddr(d); ok && a.Is6() {
|
||||
ds = append(ds, "["+a.String()+"]")
|
||||
}
|
||||
}
|
||||
if len(ds) > 0 {
|
||||
fmt.Fprintf(&b, "dhcp-option=tag:%s,option6:dns-server,%s\n", tag, strings.Join(ds, ","))
|
||||
}
|
||||
}
|
||||
if sn.Domain != "" {
|
||||
fmt.Fprintf(&b, "dhcp-option=tag:%s,option6:domain-search,%s\n", tag, sn.Domain)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if sn.Gateway != "" {
|
||||
fmt.Fprintf(&b, "dhcp-option=tag:%s,option:router,%s\n", tag, sn.Gateway)
|
||||
}
|
||||
if len(sn.DNS) > 0 {
|
||||
fmt.Fprintf(&b, "dhcp-option=tag:%s,option:dns-server,%s\n", tag, strings.Join(sn.DNS, ","))
|
||||
var ds []string
|
||||
for _, d := range sn.DNS {
|
||||
if a, ok := parseAddr(d); ok && a.Is4() {
|
||||
ds = append(ds, a.String())
|
||||
}
|
||||
}
|
||||
if len(ds) > 0 {
|
||||
fmt.Fprintf(&b, "dhcp-option=tag:%s,option:dns-server,%s\n", tag, strings.Join(ds, ","))
|
||||
}
|
||||
if sn.MTU > 0 && sn.MTU != 1500 {
|
||||
fmt.Fprintf(&b, "dhcp-option=tag:%s,option:mtu,%d\n", tag, sn.MTU)
|
||||
|
||||
@@ -49,7 +49,7 @@ func TestDefaultPlanAndNetworkConfig(t *testing.T) {
|
||||
b, _ := json.Marshal(networkConfig(p))
|
||||
s := string(b)
|
||||
for _, want := range []string{`"macaddress":"aa:bb:cc:00:00:01"`, `"set-name":"eno1"`, `"addresses":["10.0.0.21/24"]`,
|
||||
`"routes":[{"to":"0.0.0.0/0","via":"10.0.0.1"}]`, `"search":["lab.local"]`, `"dhcp4":false`} {
|
||||
`"routes":[{"to":"0.0.0.0/0","via":"10.0.0.1"}]`, `"search":["lab.local"]`, `"dhcp4":false`, `"accept-ra":false`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("network-config lacks %s:\n%s", want, s)
|
||||
}
|
||||
@@ -97,7 +97,7 @@ func TestBondVLANLayout(t *testing.T) {
|
||||
b, _ := json.Marshal(networkConfig(p))
|
||||
s := string(b)
|
||||
for _, want := range []string{`"bonds":{"bond0":`, `"interfaces":["ens1f0","ens1f1"]`, `"mode":"802.3ad"`, `"vlans":{"bond0.200":`,
|
||||
`"link":"bond0"`, `"id":200`, `"10.20.0.5/24"`, `"mtu":9000`, `"ibp1s0":{"addresses":["10.30.0.5/24"]`} {
|
||||
`"link":"bond0"`, `"id":200`, `"10.20.0.5/24"`, `"mtu":9000`, `"ibp1s0":{`, `"addresses":["10.30.0.5/24"]`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("network-config lacks %s:\n%s", want, s)
|
||||
}
|
||||
@@ -163,3 +163,64 @@ func TestDnsmasqConfig(t *testing.T) {
|
||||
t.Errorf("a rack that serves nothing must not get a config")
|
||||
}
|
||||
}
|
||||
|
||||
func v6Subnet() Subnet {
|
||||
return Subnet{CIDR: "fd00:213:77::/64", Gateway: "fd00:213:77::1", DNS: []string{"fd00:213:77::1"}, Domain: "lab.local", Fabric: "f0", VLANID: 1,
|
||||
DHCP: true, PrimaryRack: "r1", Purpose: "provisioning",
|
||||
Ranges: []IPRange{{Kind: "static", Start: "fd00:213:77::50", End: "fd00:213:77::99"}, {Kind: "dynamic", Start: "fd00:213:77::100", End: "fd00:213:77::1ff"}}}
|
||||
}
|
||||
|
||||
func TestIPv6Candidates(t *testing.T) {
|
||||
got := freeCandidates(v6Subnet(), map[string]bool{"fd00:213:77::50": true}, []string{"fd00:213:77::51/64"}, 2)
|
||||
if strings.Join(got, " ") != "fd00:213:77::52 fd00:213:77::53" {
|
||||
t.Errorf("v6 candidates %v", got)
|
||||
}
|
||||
open := Subnet{CIDR: "fd00:1::/64", Gateway: "fd00:1::1", Ranges: []IPRange{{Kind: "reserved", Start: "fd00:1::2", End: "fd00:1::ff"}}}
|
||||
if got := freeCandidates(open, nil, nil, 1); len(got) != 1 || got[0] != "fd00:1::100" {
|
||||
t.Errorf("v6 without static range %v", got)
|
||||
}
|
||||
if familyOf("fd00:1::/64") != 6 || familyOf("10.0.0.0/24") != 4 {
|
||||
t.Error("familyOf")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDualStackDefaultPlan(t *testing.T) {
|
||||
v4 := labSubnet()
|
||||
v4.VLANID = 1
|
||||
nics := []invNIC{{Name: "eno1", MAC: "aa:bb:cc:00:00:01", Link: true, IPv4: []string{"10.0.0.150/24"}}}
|
||||
p := resolvePlan(nil, nics, &nics[0], []Subnet{v4, v6Subnet()}, nil, allocFrom("10.0.0.21", "fd00:213:77::50"))
|
||||
if len(p.Ifaces) != 1 || len(p.Ifaces[0].Links) != 2 {
|
||||
t.Fatalf("plan %+v", p)
|
||||
}
|
||||
b, _ := json.Marshal(networkConfig(p))
|
||||
s := string(b)
|
||||
for _, want := range []string{`"10.0.0.21/24"`, `"fd00:213:77::50/64"`, `{"to":"0.0.0.0/0","via":"10.0.0.1"}`, `{"to":"::/0","via":"fd00:213:77::1"}`,
|
||||
`"addresses":["10.0.0.1","fd00:213:77::1"]`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("network-config lacks %s:\n%s", want, s)
|
||||
}
|
||||
}
|
||||
if mgmt, _ := p.primaryIPs(); mgmt != "10.0.0.21" {
|
||||
t.Errorf("mgmt prefers IPv4, got %s", mgmt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDnsmasqIPv6(t *testing.T) {
|
||||
rc := RackController{Name: "r1", Interfaces: []RackIface{{Name: "eth1", Addrs: []string{"10.0.0.1/24", "fd00:213:77::2/64"}}}}
|
||||
conf, _ := dnsmasqConfig(rc, []Subnet{labSubnet(), v6Subnet()}, "/d")
|
||||
for _, want := range []string{"enable-ra", "ra-param=eth1,10,0", "dhcp-range=set:s1,fd00:213:77::100,fd00:213:77::1ff,64,12h",
|
||||
"dhcp-option=tag:s1,option6:dns-server,[fd00:213:77::1]", "dhcp-option=tag:s1,option6:domain-search,lab.local", "dhcp-option=tag:s0,option:dns-server,10.0.0.1\n"} {
|
||||
if !strings.Contains(conf, want) {
|
||||
t.Errorf("dnsmasq conf lacks %q:\n%s", want, conf)
|
||||
}
|
||||
}
|
||||
if strings.Contains(conf, "tag:s1,option:router") {
|
||||
t.Errorf("no IPv4 router option on an IPv6 subnet:\n%s", conf)
|
||||
}
|
||||
// the rack is the IPv6 gateway: it advertises itself as router (no lifetime 0)
|
||||
rc.Interfaces[0].Addrs[1] = "fd00:213:77::1/64"
|
||||
conf, _ = dnsmasqConfig(rc, []Subnet{labSubnet(), v6Subnet()}, "/d")
|
||||
if strings.Contains(conf, "ra-param=eth1,10,0") {
|
||||
t.Errorf("router lifetime 0 although the rack is the gateway:\n%s", conf)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,535 @@
|
||||
package main
|
||||
|
||||
// Power drivers. A machine's BMC record names a driver; the command runs on
|
||||
// the rack controller that reaches the BMC network (the embedded one runs it
|
||||
// in-process). Every driver answers the same operations:
|
||||
//
|
||||
// status on off soft cycle reset bootdev(pxe|disk|bios|cdrom|none) probe
|
||||
//
|
||||
// and reports power state the way ipmitool does ("Chassis Power is on"), so
|
||||
// callers do not care which one ran.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os/exec"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type powerDriver struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
Family string `json:"family"` // ipmi | redfish | api | none
|
||||
DefaultPort int `json:"default_port"`
|
||||
}
|
||||
|
||||
var powerDrivers = []powerDriver{
|
||||
{"ipmi", "IPMI 2.0 over LAN (ipmitool lanplus); any vendor", "ipmi", 623},
|
||||
{"redfish", "DMTF Redfish, any vendor", "redfish", 443},
|
||||
{"idrac", "Dell iDRAC 8/9 (Redfish)", "redfish", 443},
|
||||
{"ilo", "HPE iLO 4 (2.30+), 5, 6 (Redfish)", "redfish", 443},
|
||||
{"supermicro", "Supermicro X11 and later (Redfish)", "redfish", 443},
|
||||
{"xcc", "Lenovo XClarity Controller (Redfish)", "redfish", 443},
|
||||
{"openbmc", "OpenBMC (Redfish)", "redfish", 443},
|
||||
{"proxmox", "Proxmox VE virtual machine (API token: user = token id, password = secret; extra node, vmid)", "api", 8006},
|
||||
{"manual", "no power control: someone presses the button", "none", 0},
|
||||
}
|
||||
|
||||
func driverByName(n string) (powerDriver, bool) {
|
||||
if n == "" {
|
||||
n = "ipmi"
|
||||
}
|
||||
for _, d := range powerDrivers {
|
||||
if d.Name == n {
|
||||
return d, true
|
||||
}
|
||||
}
|
||||
return powerDriver{}, false
|
||||
}
|
||||
|
||||
var powerOps = map[string]bool{"status": true, "on": true, "off": true, "soft": true, "cycle": true, "reset": true, "bootdev": true, "probe": true}
|
||||
|
||||
var errManualPower = errors.New("this machine has manual power control: do it by hand")
|
||||
|
||||
// powerRun executes one operation with a driver.
|
||||
func powerRun(ctx context.Context, ipmitool string, c *bmcCreds, op, arg string) (string, error) {
|
||||
if !powerOps[op] {
|
||||
return "", fmt.Errorf("unknown power operation %q", op)
|
||||
}
|
||||
if op == "bootdev" && !bootDevices[arg] {
|
||||
return "", fmt.Errorf("boot device must be one of pxe, disk, cdrom, bios, none")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 90*time.Second)
|
||||
defer cancel()
|
||||
d, ok := driverByName(c.Driver)
|
||||
if !ok {
|
||||
return "", fmt.Errorf("unknown power driver %q", c.Driver)
|
||||
}
|
||||
switch d.Family {
|
||||
case "ipmi":
|
||||
return ipmiOp(ctx, ipmitool, c, op, arg)
|
||||
case "redfish":
|
||||
if op == "probe" {
|
||||
return redfishProbe(ctx, c)
|
||||
}
|
||||
return redfishOp(ctx, c, op, arg)
|
||||
case "api":
|
||||
return proxmoxOp(ctx, c, op, arg)
|
||||
default:
|
||||
if op == "status" {
|
||||
return "Chassis Power is unknown (manual power control)", nil
|
||||
}
|
||||
return "", errManualPower
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- IPMI
|
||||
|
||||
func ipmiOp(ctx context.Context, tool string, c *bmcCreds, op, arg string) (string, error) {
|
||||
var args []string
|
||||
switch op {
|
||||
case "bootdev":
|
||||
args = []string{"chassis", "bootdev", arg}
|
||||
case "probe":
|
||||
args = []string{"mc", "info"}
|
||||
default:
|
||||
args = []string{"chassis", "power", op}
|
||||
}
|
||||
return ipmiExec(ctx, tool, c, args...)
|
||||
}
|
||||
|
||||
func ipmiExec(ctx context.Context, tool string, c *bmcCreds, args ...string) (string, error) {
|
||||
if tool == "" {
|
||||
tool = "ipmitool"
|
||||
}
|
||||
iface := c.Extra["interface"]
|
||||
if iface == "" {
|
||||
iface = "lanplus"
|
||||
}
|
||||
port := c.Port
|
||||
if port == 0 || port == 443 {
|
||||
port = 623
|
||||
}
|
||||
full := []string{"-I", iface, "-H", c.Address, "-p", strconv.Itoa(port), "-U", c.User, "-E"}
|
||||
if v := c.Extra["cipher"]; v != "" { // 17 on newer BMCs that dropped suite 3
|
||||
full = append(full, "-C", v)
|
||||
}
|
||||
if v := c.Extra["privilege"]; v != "" {
|
||||
full = append(full, "-L", v)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, tool, append(full, args...)...)
|
||||
cmd.Env = []string{"IPMI_PASSWORD=" + c.Password, "PATH=/usr/sbin:/usr/bin:/sbin:/bin"}
|
||||
var out bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &out
|
||||
err := cmd.Run()
|
||||
return strings.TrimSpace(out.String()), err
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Redfish
|
||||
|
||||
type rfClient struct {
|
||||
base, user, pass string
|
||||
hc *http.Client
|
||||
etag string
|
||||
}
|
||||
|
||||
func bmcHTTPClient(c *bmcCreds) *http.Client {
|
||||
// BMCs ship self-signed certificates; checking them is opt-in (extra verify_tls=true)
|
||||
tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: c.Extra["verify_tls"] != "true"}, //nolint:gosec
|
||||
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext}
|
||||
return &http.Client{Timeout: 45 * time.Second, Transport: tr}
|
||||
}
|
||||
|
||||
// bmcBase builds https://host[:port] from the record; IPMI's port is not Redfish's.
|
||||
func bmcBase(c *bmcCreds, scheme string, defPort int) string {
|
||||
a := strings.TrimRight(c.Address, "/")
|
||||
if strings.Contains(a, "://") {
|
||||
return a
|
||||
}
|
||||
port := c.Port
|
||||
if port == 623 || port == 0 {
|
||||
port = defPort
|
||||
}
|
||||
host := a
|
||||
if strings.Contains(host, ":") && !strings.HasPrefix(host, "[") {
|
||||
host = "[" + host + "]"
|
||||
}
|
||||
if (scheme == "https" && port == 443) || (scheme == "http" && port == 80) {
|
||||
return scheme + "://" + host
|
||||
}
|
||||
return fmt.Sprintf("%s://%s:%d", scheme, host, port)
|
||||
}
|
||||
|
||||
func newRF(c *bmcCreds) *rfClient {
|
||||
return &rfClient{base: bmcBase(c, "https", 443), user: c.User, pass: c.Password, hc: bmcHTTPClient(c)}
|
||||
}
|
||||
|
||||
func (r *rfClient) do(ctx context.Context, method, path string, body, out any) (int, error) {
|
||||
var rd io.Reader
|
||||
if body != nil {
|
||||
b, _ := json.Marshal(body)
|
||||
rd = bytes.NewReader(b)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, r.base+path, rd)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if r.user != "" {
|
||||
req.SetBasicAuth(r.user, r.pass)
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if method == "PATCH" && r.etag != "" {
|
||||
req.Header.Set("If-Match", r.etag) // some services (Supermicro, newer iDRAC) insist
|
||||
}
|
||||
resp, err := r.hc.Do(req)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
data, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
||||
if method == "GET" {
|
||||
if et := resp.Header.Get("ETag"); et != "" {
|
||||
r.etag = et
|
||||
}
|
||||
}
|
||||
if resp.StatusCode >= 300 {
|
||||
return resp.StatusCode, fmt.Errorf("%s %s: %s %s", method, path, resp.Status, redfishMessage(data))
|
||||
}
|
||||
if out != nil && len(data) > 0 {
|
||||
if err := json.Unmarshal(data, out); err != nil {
|
||||
return resp.StatusCode, fmt.Errorf("%s %s: %w", method, path, err)
|
||||
}
|
||||
}
|
||||
return resp.StatusCode, nil
|
||||
}
|
||||
|
||||
// redfishMessage pulls the human message out of a Redfish error body.
|
||||
func redfishMessage(data []byte) string {
|
||||
var e struct {
|
||||
Error struct {
|
||||
Message string `json:"message"`
|
||||
Ext []struct {
|
||||
Message string `json:"Message"`
|
||||
} `json:"@Message.ExtendedInfo"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if json.Unmarshal(data, &e) == nil {
|
||||
if len(e.Error.Ext) > 0 && e.Error.Ext[0].Message != "" {
|
||||
return e.Error.Ext[0].Message
|
||||
}
|
||||
if e.Error.Message != "" {
|
||||
return e.Error.Message
|
||||
}
|
||||
}
|
||||
return oneLine(string(data))
|
||||
}
|
||||
|
||||
// redfishSystemPath: extra system=…, else the vendor's usual one if it exists, else the first member.
|
||||
func (r *rfClient) system(ctx context.Context, c *bmcCreds) (string, obj, error) {
|
||||
var cands []string
|
||||
if s := c.Extra["system"]; s != "" {
|
||||
cands = append(cands, s)
|
||||
}
|
||||
switch c.Driver {
|
||||
case "idrac":
|
||||
cands = append(cands, "/redfish/v1/Systems/System.Embedded.1")
|
||||
case "ilo", "xcc", "supermicro":
|
||||
cands = append(cands, "/redfish/v1/Systems/1")
|
||||
}
|
||||
for _, p := range cands {
|
||||
var sys obj
|
||||
if _, err := r.do(ctx, "GET", p, nil, &sys); err == nil {
|
||||
return p, sys, nil
|
||||
} else if c.Extra["system"] != "" {
|
||||
return "", nil, err
|
||||
}
|
||||
}
|
||||
var coll struct {
|
||||
Members []struct {
|
||||
ID string `json:"@odata.id"`
|
||||
} `json:"Members"`
|
||||
}
|
||||
if _, err := r.do(ctx, "GET", "/redfish/v1/Systems", nil, &coll); err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
if len(coll.Members) == 0 {
|
||||
return "", nil, errors.New("the BMC lists no systems")
|
||||
}
|
||||
p := coll.Members[0].ID
|
||||
var sys obj
|
||||
_, err := r.do(ctx, "GET", p, nil, &sys)
|
||||
return p, sys, err
|
||||
}
|
||||
|
||||
// resetTypes are the ResetType values the system accepts.
|
||||
func (r *rfClient) resetTypes(ctx context.Context, sys obj) []string {
|
||||
act, _ := dig(sys, "Actions", "#ComputerSystem.Reset").(obj)
|
||||
var out []string
|
||||
for _, v := range toList(act["ResetType@Redfish.AllowableValues"]) {
|
||||
out = append(out, str(v))
|
||||
}
|
||||
if len(out) == 0 {
|
||||
if info := str(act["@Redfish.ActionInfo"]); info != "" {
|
||||
var ai struct {
|
||||
Parameters []struct {
|
||||
Name string `json:"Name"`
|
||||
AllowableValues []string `json:"AllowableValues"`
|
||||
} `json:"Parameters"`
|
||||
}
|
||||
if _, err := r.do(ctx, "GET", info, nil, &ai); err == nil {
|
||||
for _, p := range ai.Parameters {
|
||||
if p.Name == "ResetType" {
|
||||
out = p.AllowableValues
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func pickReset(allowed []string, prefs ...string) string {
|
||||
if len(allowed) == 0 {
|
||||
return prefs[0]
|
||||
}
|
||||
for _, p := range prefs {
|
||||
if hasTag(allowed, p) {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
var redfishBootTargets = map[string]string{"pxe": "Pxe", "disk": "Hdd", "bios": "BiosSetup", "cdrom": "Cd", "none": "None", "safe": "None"}
|
||||
|
||||
func redfishOp(ctx context.Context, c *bmcCreds, op, arg string) (string, error) {
|
||||
r := newRF(c)
|
||||
path, sys, err := r.system(ctx, c)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
state := strings.ToLower(str(sys["PowerState"])) // On | Off | PoweringOn | PoweringOff
|
||||
on := state == "on" || state == "poweringon"
|
||||
reset := func(prefs ...string) (string, error) {
|
||||
t := pickReset(r.resetTypes(ctx, sys), prefs...)
|
||||
if t == "" {
|
||||
return "", fmt.Errorf("the BMC accepts none of %v", prefs)
|
||||
}
|
||||
target := str(dig(sys, "Actions", "#ComputerSystem.Reset", "target"))
|
||||
if target == "" {
|
||||
target = path + "/Actions/ComputerSystem.Reset"
|
||||
}
|
||||
if _, err := r.do(ctx, "POST", target, obj{"ResetType": t}, nil); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "Chassis Power Control: " + t + " (redfish)", nil
|
||||
}
|
||||
switch op {
|
||||
case "status":
|
||||
if on {
|
||||
return "Chassis Power is on", nil
|
||||
}
|
||||
return "Chassis Power is off", nil
|
||||
case "on":
|
||||
if on {
|
||||
return "Chassis Power is on already", nil
|
||||
}
|
||||
return reset("On", "PushPowerButton")
|
||||
case "off":
|
||||
if !on {
|
||||
return "Chassis Power is off already", nil
|
||||
}
|
||||
return reset("ForceOff", "PushPowerButton")
|
||||
case "soft":
|
||||
if !on {
|
||||
return "Chassis Power is off already", nil
|
||||
}
|
||||
return reset("GracefulShutdown", "PushPowerButton")
|
||||
case "cycle":
|
||||
if !on {
|
||||
return reset("On", "PushPowerButton")
|
||||
}
|
||||
return reset("PowerCycle", "ForceRestart", "GracefulRestart")
|
||||
case "reset":
|
||||
if !on {
|
||||
return reset("On", "PushPowerButton")
|
||||
}
|
||||
return reset("ForceRestart", "PowerCycle", "GracefulRestart")
|
||||
case "bootdev":
|
||||
boot := obj{"BootSourceOverrideTarget": redfishBootTargets[arg], "BootSourceOverrideEnabled": "Once"}
|
||||
if arg == "none" || arg == "safe" {
|
||||
boot = obj{"BootSourceOverrideEnabled": "Disabled"}
|
||||
}
|
||||
if _, err := r.do(ctx, "PATCH", path, obj{"Boot": boot}, nil); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "Set Boot Device to " + arg + " (redfish " + str(boot["BootSourceOverrideTarget"]) + ", once)", nil
|
||||
}
|
||||
return "", fmt.Errorf("redfish: %s is not supported", op)
|
||||
}
|
||||
|
||||
// redfishProbe asks whether the BMC speaks Redfish with these credentials and
|
||||
// suggests the driver for its vendor. The answer is JSON.
|
||||
func redfishProbe(ctx context.Context, c *bmcCreds) (string, error) {
|
||||
r := newRF(c)
|
||||
var root obj
|
||||
if _, err := r.do(ctx, "GET", "/redfish/v1/", nil, &root); err != nil {
|
||||
return "", fmt.Errorf("no Redfish service: %w", err)
|
||||
}
|
||||
res := obj{"redfish": str(root["RedfishVersion"]), "vendor": str(root["Vendor"])}
|
||||
if res["vendor"] == "" {
|
||||
for k := range asObj(root["Oem"]) {
|
||||
res["vendor"] = k
|
||||
}
|
||||
}
|
||||
path, sys, err := r.system(ctx, &bmcCreds{Driver: "redfish", Extra: c.Extra, User: c.User, Password: c.Password, Address: c.Address, Port: c.Port})
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("Redfish %s answers, but not to these credentials: %w", res["redfish"], err)
|
||||
}
|
||||
res["system"], res["manufacturer"], res["model"], res["power"] = path, str(sys["Manufacturer"]), str(sys["Model"]), str(sys["PowerState"])
|
||||
v := strings.ToLower(str(res["vendor"]) + " " + str(res["manufacturer"]))
|
||||
switch {
|
||||
case strings.Contains(v, "dell"):
|
||||
res["driver"] = "idrac"
|
||||
case strings.Contains(v, "hpe"), strings.Contains(v, "hewlett"), strings.Contains(v, "hp "):
|
||||
res["driver"] = "ilo"
|
||||
case strings.Contains(v, "supermicro"):
|
||||
res["driver"] = "supermicro"
|
||||
case strings.Contains(v, "lenovo"):
|
||||
res["driver"] = "xcc"
|
||||
case strings.Contains(v, "openbmc"):
|
||||
res["driver"] = "openbmc"
|
||||
default:
|
||||
res["driver"] = "redfish"
|
||||
}
|
||||
b, _ := json.Marshal(res)
|
||||
return string(b), nil
|
||||
}
|
||||
|
||||
func asObj(v any) obj {
|
||||
o, _ := v.(obj)
|
||||
return o
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- Proxmox VE
|
||||
|
||||
func proxmoxOp(ctx context.Context, c *bmcCreds, op, arg string) (string, error) {
|
||||
node, vmid := c.Extra["node"], c.Extra["vmid"]
|
||||
if node == "" || vmid == "" {
|
||||
return "", errors.New("proxmox driver needs extra node and vmid")
|
||||
}
|
||||
base := bmcBase(c, "https", 8006) + "/api2/json/nodes/" + url.PathEscape(node) + "/qemu/" + url.PathEscape(vmid)
|
||||
hc := bmcHTTPClient(c)
|
||||
call := func(method, path string, form url.Values) (obj, error) {
|
||||
var rd io.Reader
|
||||
if form != nil {
|
||||
rd = strings.NewReader(form.Encode())
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, base+path, rd)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Authorization", "PVEAPIToken="+c.User+"="+c.Password)
|
||||
if form != nil {
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
}
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
data, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode >= 300 {
|
||||
return nil, fmt.Errorf("proxmox %s %s: %s %s", method, path, resp.Status, oneLine(string(data)))
|
||||
}
|
||||
var out obj
|
||||
_ = json.Unmarshal(data, &out)
|
||||
return out, nil
|
||||
}
|
||||
cur, err := call("GET", "/status/current", nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
on := str(dig(cur, "data", "status")) == "running"
|
||||
act := func(what string) (string, error) {
|
||||
if _, err := call("POST", "/status/"+what, url.Values{}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "Chassis Power Control: " + what + " (proxmox)", nil
|
||||
}
|
||||
switch op {
|
||||
case "status", "probe":
|
||||
if on {
|
||||
return "Chassis Power is on", nil
|
||||
}
|
||||
return "Chassis Power is off", nil
|
||||
case "on":
|
||||
if on {
|
||||
return "Chassis Power is on already", nil
|
||||
}
|
||||
return act("start")
|
||||
case "off":
|
||||
if !on {
|
||||
return "Chassis Power is off already", nil
|
||||
}
|
||||
return act("stop")
|
||||
case "soft":
|
||||
return act("shutdown")
|
||||
case "cycle", "reset":
|
||||
if !on {
|
||||
return act("start")
|
||||
}
|
||||
return act("reset")
|
||||
case "bootdev":
|
||||
cfg, err := call("GET", "/config", nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
d, _ := cfg["data"].(obj)
|
||||
var disks, nets, cds []string
|
||||
keys := make([]string, 0, len(d))
|
||||
for k := range d {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
v := str(d[k])
|
||||
switch {
|
||||
case strings.HasPrefix(k, "net") && len(k) > 3 && k[3] >= '0' && k[3] <= '9':
|
||||
nets = append(nets, k)
|
||||
case strings.Contains(v, "media=cdrom"):
|
||||
cds = append(cds, k)
|
||||
case hasTag([]string{"scsi0", "virtio0", "sata0", "ide0", "nvme0"}, k):
|
||||
disks = append(disks, k)
|
||||
}
|
||||
}
|
||||
var order []string
|
||||
switch arg {
|
||||
case "pxe":
|
||||
order = append(append(append(order, nets...), disks...), cds...)
|
||||
case "cdrom":
|
||||
order = append(append(append(order, cds...), disks...), nets...)
|
||||
default:
|
||||
order = append(append(append(order, disks...), cds...), nets...)
|
||||
}
|
||||
if _, err := call("PUT", "/config", url.Values{"boot": {"order=" + strings.Join(order, ";")}}); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "Set Boot Device to " + arg + " (proxmox boot order " + strings.Join(order, ";") + ")", nil
|
||||
}
|
||||
return "", fmt.Errorf("proxmox: %s is not supported", op)
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeRedfish is a tiny Redfish service in the shape of one vendor.
|
||||
type fakeRedfish struct {
|
||||
mu sync.Mutex
|
||||
vendor string
|
||||
system string
|
||||
allowed []string
|
||||
power string
|
||||
resets []string
|
||||
boot map[string]any
|
||||
ifMatch []string
|
||||
}
|
||||
|
||||
func (f *fakeRedfish) handler(t *testing.T) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
u, p, ok := r.BasicAuth()
|
||||
if r.URL.Path != "/redfish/v1/" && (!ok || u != "admin" || p != "secret") {
|
||||
w.WriteHeader(401)
|
||||
_, _ = w.Write([]byte(`{"error":{"message":"bad credentials"}}`))
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case r.URL.Path == "/redfish/v1/":
|
||||
_ = json.NewEncoder(w).Encode(obj{"RedfishVersion": "1.11.0", "Vendor": f.vendor, "Systems": obj{"@odata.id": "/redfish/v1/Systems"}})
|
||||
case r.URL.Path == "/redfish/v1/Systems":
|
||||
_ = json.NewEncoder(w).Encode(obj{"Members": []obj{{"@odata.id": f.system}}})
|
||||
case r.URL.Path == f.system && r.Method == "GET":
|
||||
w.Header().Set("ETag", `W/"42"`)
|
||||
_ = json.NewEncoder(w).Encode(obj{"PowerState": f.power, "Manufacturer": f.vendor, "Model": "Test 1",
|
||||
"Actions": obj{"#ComputerSystem.Reset": obj{"target": f.system + "/Actions/ComputerSystem.Reset", "ResetType@Redfish.AllowableValues": f.allowed}}})
|
||||
case r.URL.Path == f.system && r.Method == "PATCH":
|
||||
f.ifMatch = append(f.ifMatch, r.Header.Get("If-Match"))
|
||||
var body obj
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
f.boot, _ = body["Boot"].(map[string]any)
|
||||
w.WriteHeader(204)
|
||||
case r.URL.Path == f.system+"/Actions/ComputerSystem.Reset":
|
||||
var body struct{ ResetType string }
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
if !hasTag(f.allowed, body.ResetType) {
|
||||
w.WriteHeader(400)
|
||||
_, _ = w.Write([]byte(`{"error":{"@Message.ExtendedInfo":[{"Message":"ResetType not supported"}]}}`))
|
||||
return
|
||||
}
|
||||
f.resets = append(f.resets, body.ResetType)
|
||||
switch body.ResetType {
|
||||
case "On":
|
||||
f.power = "On"
|
||||
case "ForceOff", "GracefulShutdown":
|
||||
f.power = "Off"
|
||||
}
|
||||
w.WriteHeader(204)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func creds(addr, driver string) *bmcCreds {
|
||||
u, _ := url.Parse(addr)
|
||||
return &bmcCreds{Driver: driver, Address: "https://" + u.Host, User: "admin", Password: "secret", Extra: map[string]string{}}
|
||||
}
|
||||
|
||||
func TestRedfishIDRAC(t *testing.T) {
|
||||
f := &fakeRedfish{vendor: "Dell", system: "/redfish/v1/Systems/System.Embedded.1", power: "Off",
|
||||
allowed: []string{"On", "ForceOff", "ForceRestart", "GracefulRestart", "GracefulShutdown", "PushPowerButton", "Nmi", "PowerCycle"}}
|
||||
srv := httptest.NewTLSServer(f.handler(t))
|
||||
defer srv.Close()
|
||||
c := creds(srv.URL, "idrac")
|
||||
ctx := context.Background()
|
||||
if out, _ := powerRun(ctx, "", c, "status", ""); out != "Chassis Power is off" {
|
||||
t.Errorf("status %q", out)
|
||||
}
|
||||
if _, err := powerRun(ctx, "", c, "cycle", ""); err != nil { // off: cycle means on
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := powerRun(ctx, "", c, "cycle", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := powerRun(ctx, "", c, "soft", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := strings.Join(f.resets, ","); got != "On,PowerCycle,GracefulShutdown" {
|
||||
t.Errorf("resets %s", got)
|
||||
}
|
||||
if _, err := powerRun(ctx, "", c, "bootdev", "pxe"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if f.boot["BootSourceOverrideTarget"] != "Pxe" || f.boot["BootSourceOverrideEnabled"] != "Once" || f.ifMatch[0] != `W/"42"` {
|
||||
t.Errorf("boot override %v if-match %v", f.boot, f.ifMatch)
|
||||
}
|
||||
out, err := powerRun(ctx, "", creds(srv.URL, "redfish"), "probe", "")
|
||||
if err != nil || !strings.Contains(out, `"driver":"idrac"`) || !strings.Contains(out, "System.Embedded.1") {
|
||||
t.Errorf("probe %s %v", out, err)
|
||||
}
|
||||
bad := creds(srv.URL, "redfish")
|
||||
bad.Password = "nope"
|
||||
if _, err := powerRun(ctx, "", bad, "probe", ""); err == nil || !strings.Contains(err.Error(), "credentials") {
|
||||
t.Errorf("probe with a wrong password must fail clearly: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// iLO has no PowerCycle and, on older firmware, no GracefulShutdown.
|
||||
func TestRedfishILOFallbacks(t *testing.T) {
|
||||
f := &fakeRedfish{vendor: "HPE", system: "/redfish/v1/Systems/1", power: "On", allowed: []string{"On", "ForceOff", "ForceRestart", "Nmi", "PushPowerButton"}}
|
||||
srv := httptest.NewTLSServer(f.handler(t))
|
||||
defer srv.Close()
|
||||
c := creds(srv.URL, "ilo")
|
||||
ctx := context.Background()
|
||||
for _, op := range []string{"cycle", "soft"} {
|
||||
if _, err := powerRun(ctx, "", c, op, ""); err != nil {
|
||||
t.Fatalf("%s: %v", op, err)
|
||||
}
|
||||
}
|
||||
if got := strings.Join(f.resets, ","); got != "ForceRestart,PushPowerButton" {
|
||||
t.Errorf("resets %s", got)
|
||||
}
|
||||
out, _ := powerRun(ctx, "", creds(srv.URL, "redfish"), "probe", "")
|
||||
if !strings.Contains(out, `"driver":"ilo"`) {
|
||||
t.Errorf("probe %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxmoxDriver(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
status, posts, boot := "stopped", []string{}, ""
|
||||
srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if r.Header.Get("Authorization") != "PVEAPIToken=root@pam!rc=s3cret" {
|
||||
w.WriteHeader(401)
|
||||
return
|
||||
}
|
||||
base := "/api2/json/nodes/pve1/qemu/211"
|
||||
switch {
|
||||
case r.URL.Path == base+"/status/current":
|
||||
_ = json.NewEncoder(w).Encode(obj{"data": obj{"status": status}})
|
||||
return
|
||||
case r.URL.Path == base+"/config" && r.Method == "GET":
|
||||
_ = json.NewEncoder(w).Encode(obj{"data": obj{"scsi0": "zfspool:vm-211-disk-0,size=30G", "net0": "virtio=BC:24:11:8E:1E:6C", "ide2": "local:iso/x.iso,media=cdrom"}})
|
||||
return
|
||||
case r.URL.Path == base+"/config" && r.Method == "PUT":
|
||||
_ = r.ParseForm()
|
||||
boot = r.Form.Get("boot")
|
||||
case strings.HasPrefix(r.URL.Path, base+"/status/"):
|
||||
posts = append(posts, strings.TrimPrefix(r.URL.Path, base+"/status/"))
|
||||
status = "running"
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"data":null}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
u, _ := url.Parse(srv.URL)
|
||||
c := &bmcCreds{Driver: "proxmox", Address: "https://" + u.Host, User: "root@pam!rc", Password: "s3cret", Extra: map[string]string{"node": "pve1", "vmid": "211"}}
|
||||
ctx := context.Background()
|
||||
if out, _ := powerRun(ctx, "", c, "status", ""); out != "Chassis Power is off" {
|
||||
t.Errorf("status %q", out)
|
||||
}
|
||||
_, _ = powerRun(ctx, "", c, "cycle", "")
|
||||
_, _ = powerRun(ctx, "", c, "cycle", "")
|
||||
if strings.Join(posts, ",") != "start,reset" {
|
||||
t.Errorf("posts %v", posts)
|
||||
}
|
||||
if _, err := powerRun(ctx, "", c, "bootdev", "pxe"); err != nil || boot != "order=net0;scsi0;ide2" {
|
||||
t.Errorf("boot order %q %v", boot, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManualDriver(t *testing.T) {
|
||||
c := &bmcCreds{Driver: "manual"}
|
||||
if out, err := powerRun(context.Background(), "", c, "status", ""); err != nil || !strings.Contains(out, "unknown") {
|
||||
t.Errorf("status %q %v", out, err)
|
||||
}
|
||||
if _, err := powerRun(context.Background(), "", c, "on", ""); err != errManualPower {
|
||||
t.Errorf("on: %v", err)
|
||||
}
|
||||
}
|
||||
+65
-57
@@ -13,7 +13,6 @@ package main
|
||||
// what a single-server installation uses.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/subtle"
|
||||
"encoding/json"
|
||||
@@ -23,7 +22,6 @@ import (
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
@@ -64,13 +62,16 @@ type rackState struct {
|
||||
}
|
||||
|
||||
type rackJob struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"` // ipmi
|
||||
Address string `json:"address"`
|
||||
Port int `json:"port"`
|
||||
User string `json:"user"`
|
||||
Password string `json:"password"`
|
||||
Args []string `json:"args"`
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"` // power
|
||||
Driver string `json:"driver"`
|
||||
Address string `json:"address"`
|
||||
Port int `json:"port"`
|
||||
User string `json:"user"`
|
||||
Password string `json:"password"`
|
||||
Extra map[string]string `json:"extra"`
|
||||
Op string `json:"op"`
|
||||
Arg string `json:"arg"`
|
||||
done chan rackResult
|
||||
}
|
||||
|
||||
@@ -405,40 +406,6 @@ func (a *App) rackAddresses() []string {
|
||||
|
||||
// ---------------------------------------------------------------- BMC through the machine's rack
|
||||
|
||||
var errNoBMC = errors.New("no BMC credentials stored")
|
||||
|
||||
func ipmiExec(ctx context.Context, tool string, c *bmcCreds, args ...string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
defer cancel()
|
||||
if tool == "" {
|
||||
tool = "ipmitool"
|
||||
}
|
||||
full := append([]string{"-I", "lanplus", "-H", c.Address, "-p", strconv.Itoa(c.Port), "-U", c.User, "-E"}, args...)
|
||||
cmd := exec.CommandContext(ctx, tool, full...)
|
||||
cmd.Env = []string{"IPMI_PASSWORD=" + c.Password, "PATH=/usr/sbin:/usr/bin:/sbin:/bin"}
|
||||
var out bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &out, &out
|
||||
err := cmd.Run()
|
||||
return strings.TrimSpace(out.String()), err
|
||||
}
|
||||
|
||||
// bmc runs an ipmitool command for a machine, on the rack controller that can reach its BMC.
|
||||
func (a *App) bmc(ctx context.Context, m *Machine, args ...string) (string, error) {
|
||||
c, err := a.credsFor(m)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%w: %v", errNoBMC, err)
|
||||
}
|
||||
rack := a.rackOf(m)
|
||||
if rack == a.cfg.RackName {
|
||||
return ipmiExec(ctx, a.cfg.IPMITool, c, args...)
|
||||
}
|
||||
r, ok := a.racksSnapshot()[rack]
|
||||
if !ok || !rackOnline(r, a.cfg.RackName) {
|
||||
return "", fmt.Errorf("rack controller %s is offline", rack)
|
||||
}
|
||||
return a.hub.dispatch(ctx, rack, &rackJob{Kind: "ipmi", Address: c.Address, Port: c.Port, User: c.User, Password: c.Password, Args: args})
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- desired state
|
||||
|
||||
const defaultPXETmpl = `# generated by rollcall for rack controller %s
|
||||
@@ -503,12 +470,21 @@ func (a *App) rackState(ctx context.Context, name string) (*rackState, error) {
|
||||
return st, nil
|
||||
}
|
||||
|
||||
// hostsFiles: static DHCP leases for the machines this rack serves, and names for every machine.
|
||||
// hostsFiles: static DHCP leases for the machines this rack serves, and names
|
||||
// for every machine: the addresses on the interface carrying the management
|
||||
// address get the plain name (A and AAAA), others name-<interface>, IB name-ib.
|
||||
func (a *App) hostsFiles(ctx context.Context, rack string, subnets []Subnet) (string, string, error) {
|
||||
ms, err := a.store.List(ctx)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
all, _ := a.store.IPs(ctx, "", "")
|
||||
byMachine := map[string][]IPAddr{}
|
||||
for _, x := range all {
|
||||
if x.MachineID != "" {
|
||||
byMachine[x.MachineID] = append(byMachine[x.MachineID], x)
|
||||
}
|
||||
}
|
||||
var dhcp, hosts strings.Builder
|
||||
dhcp.WriteString("# generated by rollcall\n")
|
||||
hosts.WriteString("# generated by rollcall\n")
|
||||
@@ -518,24 +494,49 @@ func (a *App) hostsFiles(ctx context.Context, rack string, subnets []Subnet) (st
|
||||
}
|
||||
return str(a.cfg.ClusterVars["domain"])
|
||||
}
|
||||
line := func(ip, name string) {
|
||||
if d := domainOf(ip); d != "" {
|
||||
fmt.Fprintf(&hosts, "%s %s.%s %s\n", ip, name, d, name)
|
||||
} else {
|
||||
fmt.Fprintf(&hosts, "%s %s\n", ip, name)
|
||||
}
|
||||
}
|
||||
sort.Slice(ms, func(i, j int) bool { return deref(ms[i].NodeName) < deref(ms[j].NodeName) })
|
||||
for _, m := range ms {
|
||||
if m.NodeName == nil || len(m.MACs) == 0 {
|
||||
continue
|
||||
}
|
||||
name := *m.NodeName
|
||||
if ip := deref(m.MgmtIP); ip != "" {
|
||||
if a.rackOf(m) == rack {
|
||||
fmt.Fprintf(&dhcp, "%s,%s,%s\n", m.MACs[0], ip, name)
|
||||
}
|
||||
if d := domainOf(ip); d != "" {
|
||||
fmt.Fprintf(&hosts, "%s %s.%s %s\n", ip, name, d, name)
|
||||
} else {
|
||||
fmt.Fprintf(&hosts, "%s %s\n", ip, name)
|
||||
name, mgmt, ibip := *m.NodeName, deref(m.MgmtIP), deref(m.IBIP)
|
||||
mine := a.rackOf(m) == rack
|
||||
mgmtIface := ""
|
||||
for _, x := range byMachine[m.ID] {
|
||||
if x.IP == mgmt {
|
||||
mgmtIface = x.Iface
|
||||
}
|
||||
}
|
||||
if ip := deref(m.IBIP); ip != "" {
|
||||
fmt.Fprintf(&hosts, "%s %s-ib\n", ip, name)
|
||||
if mgmt != "" {
|
||||
line(mgmt, name)
|
||||
if mine && familyOf(mgmt+"/32") == 4 {
|
||||
fmt.Fprintf(&dhcp, "%s,%s,%s\n", m.MACs[0], mgmt, name)
|
||||
}
|
||||
}
|
||||
for _, x := range byMachine[m.ID] {
|
||||
if x.IP == mgmt || x.IP == ibip {
|
||||
continue
|
||||
}
|
||||
v6 := strings.Contains(x.IP, ":")
|
||||
switch {
|
||||
case x.Iface == mgmtIface && mgmtIface != "":
|
||||
line(x.IP, name)
|
||||
if v6 && mine {
|
||||
fmt.Fprintf(&dhcp, "%s,[%s],%s\n", m.MACs[0], x.IP, name)
|
||||
}
|
||||
case x.Iface != "":
|
||||
line(x.IP, name+"-"+strings.NewReplacer(".", "-", ":", "-").Replace(x.Iface))
|
||||
}
|
||||
}
|
||||
if ibip != "" {
|
||||
fmt.Fprintf(&hosts, "%s %s-ib\n", ibip, name)
|
||||
}
|
||||
}
|
||||
return dhcp.String(), hosts.String(), nil
|
||||
@@ -641,7 +642,7 @@ func localInterfaces() []RackIface {
|
||||
ri := RackIface{Name: i.Name, MAC: i.HardwareAddr.String()}
|
||||
addrs, _ := i.Addrs()
|
||||
for _, ad := range addrs {
|
||||
if ipn, ok := ad.(*net.IPNet); ok && ipn.IP.To4() != nil {
|
||||
if ipn, ok := ad.(*net.IPNet); ok && !ipn.IP.IsLinkLocalUnicast() && !ipn.IP.IsLoopback() {
|
||||
ri.Addrs = append(ri.Addrs, ipn.String())
|
||||
}
|
||||
}
|
||||
@@ -741,9 +742,13 @@ func (a *App) exportDnsmasq(ctx context.Context) error {
|
||||
|
||||
func (a *App) rackAuth(next func(w http.ResponseWriter, r *http.Request, rack string)) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if name := a.peerRack(r); name != "" {
|
||||
next(w, r, name)
|
||||
return
|
||||
}
|
||||
sec, name := r.Header.Get("X-Rollcall-Rack-Secret"), r.Header.Get("X-Rollcall-Rack")
|
||||
if a.cfg.RackSecret == "" || subtle.ConstantTimeCompare([]byte(sec), []byte(a.cfg.RackSecret)) != 1 || name == "" {
|
||||
fail(w, 401, errors.New("rack controller secret required"))
|
||||
fail(w, 401, errors.New("rack controller certificate (or legacy secret) required"))
|
||||
return
|
||||
}
|
||||
next(w, r, name)
|
||||
@@ -752,6 +757,9 @@ func (a *App) rackAuth(next func(w http.ResponseWriter, r *http.Request, rack st
|
||||
|
||||
// fromRack names the rack controller a node request came through (it proxies them).
|
||||
func (a *App) fromRack(r *http.Request) string {
|
||||
if name := a.peerRack(r); name != "" {
|
||||
return name
|
||||
}
|
||||
sec, name := r.Header.Get("X-Rollcall-Rack-Secret"), r.Header.Get("X-Rollcall-Rack")
|
||||
if a.cfg.RackSecret != "" && name != "" && subtle.ConstantTimeCompare([]byte(sec), []byte(a.cfg.RackSecret)) == 1 {
|
||||
return name
|
||||
|
||||
+243
-20
@@ -10,15 +10,24 @@ package main
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/user"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
@@ -28,16 +37,19 @@ import (
|
||||
)
|
||||
|
||||
type RackConfig struct {
|
||||
Name string `json:"name"` // default: hostname
|
||||
Region string `json:"region"` // the region's URL, e.g. http://10.0.0.234
|
||||
Secret string `json:"secret"` // the region's rack_secret
|
||||
URL string `json:"url"` // how machines reach this rack, e.g. http://10.213.78.1
|
||||
Listen string `json:"listen"` // default ":80"
|
||||
TFTPRoot string `json:"tftp_root"` // default /srv/tftp
|
||||
DnsmasqDir string `json:"dnsmasq_dir"` // default /var/lib/rollcall/dnsmasq
|
||||
CacheDir string `json:"cache_dir"` // default /var/cache/rollcall
|
||||
IPMITool string `json:"ipmitool"` // default ipmitool
|
||||
DHCPInterfaces []string `json:"dhcp_interfaces"` // default: the interfaces with an address in a served subnet
|
||||
Name string `json:"name,omitempty"` // default: hostname
|
||||
Region string `json:"region,omitempty"` // the region's URL, e.g. http://10.0.0.234
|
||||
Secret string `json:"secret,omitempty"` // the region's rack_secret
|
||||
URL string `json:"url,omitempty"` // how machines reach this rack, e.g. http://10.213.78.1
|
||||
Listen string `json:"listen,omitempty"` // default ":80"
|
||||
TFTPRoot string `json:"tftp_root,omitempty"` // default /srv/tftp
|
||||
DnsmasqDir string `json:"dnsmasq_dir,omitempty"` // default /var/lib/rollcall/dnsmasq
|
||||
CacheDir string `json:"cache_dir,omitempty"` // default /var/cache/rollcall
|
||||
IPMITool string `json:"ipmitool,omitempty"` // default ipmitool
|
||||
DHCPInterfaces []string `json:"dhcp_interfaces,omitempty"` // default: the interfaces with an address in a served subnet
|
||||
CAFile string `json:"ca_file,omitempty"` // set by rollcall rack join: the region's CA,
|
||||
CertFile string `json:"cert_file,omitempty"` // this rack controller's certificate
|
||||
KeyFile string `json:"key_file,omitempty"` // and key; with them the secret is not needed
|
||||
}
|
||||
|
||||
type rackd struct {
|
||||
@@ -45,7 +57,11 @@ type rackd struct {
|
||||
region *url.URL
|
||||
hc *http.Client // long polls
|
||||
fast *http.Client
|
||||
big *http.Client // image downloads, no timeout
|
||||
gen int64
|
||||
tlsc *tls.Config
|
||||
certMu sync.Mutex
|
||||
cert *tls.Certificate
|
||||
|
||||
headMu sync.Mutex
|
||||
heads map[string]headInfo
|
||||
@@ -86,14 +102,206 @@ func loadRackConfig(p string) (RackConfig, error) {
|
||||
c.IPMITool = "ipmitool"
|
||||
}
|
||||
c.Region, c.URL = strings.TrimRight(c.Region, "/"), strings.TrimRight(c.URL, "/")
|
||||
for _, k := range []struct{ v, n string }{{c.Region, "region"}, {c.Secret, "secret"}, {c.URL, "url"}} {
|
||||
for _, k := range []struct{ v, n string }{{c.Region, "region"}, {c.URL, "url"}} {
|
||||
if k.v == "" {
|
||||
return c, fmt.Errorf("%s: %s is required", p, k.n)
|
||||
}
|
||||
}
|
||||
if c.Secret == "" && c.CertFile == "" {
|
||||
return c, fmt.Errorf("%s: not enrolled: run rollcall rack join CODE (or set the legacy secret)", p)
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// clients builds the HTTP clients; with a certificate they speak mutual TLS
|
||||
// and pick up a renewed certificate without being rebuilt.
|
||||
func (r *rackd) clients() error {
|
||||
r.tlsc = &tls.Config{MinVersion: tls.VersionTLS12}
|
||||
if r.cfg.CertFile != "" {
|
||||
cab, err := os.ReadFile(r.cfg.CAFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
pool := x509.NewCertPool()
|
||||
if !pool.AppendCertsFromPEM(cab) {
|
||||
return fmt.Errorf("%s: no certificate", r.cfg.CAFile)
|
||||
}
|
||||
pair, err := tls.LoadX509KeyPair(r.cfg.CertFile, r.cfg.KeyFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
pair.Leaf, _ = x509.ParseCertificate(pair.Certificate[0])
|
||||
r.cert = &pair
|
||||
r.tlsc.RootCAs = pool
|
||||
r.tlsc.GetClientCertificate = func(*tls.CertificateRequestInfo) (*tls.Certificate, error) {
|
||||
r.certMu.Lock()
|
||||
defer r.certMu.Unlock()
|
||||
return r.cert, nil
|
||||
}
|
||||
}
|
||||
tr := func() *http.Transport {
|
||||
return &http.Transport{TLSClientConfig: r.tlsc, Proxy: http.ProxyFromEnvironment, MaxIdleConnsPerHost: 8,
|
||||
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext}
|
||||
}
|
||||
r.hc = &http.Client{Timeout: 45 * time.Second, Transport: tr()}
|
||||
r.fast = &http.Client{Timeout: 30 * time.Second, Transport: tr()}
|
||||
r.big = &http.Client{Transport: tr()}
|
||||
return nil
|
||||
}
|
||||
|
||||
// renewLoop asks for a new certificate a month before the current one ends.
|
||||
func (r *rackd) renewLoop() {
|
||||
for {
|
||||
r.certMu.Lock()
|
||||
leaf := r.cert.Leaf
|
||||
r.certMu.Unlock()
|
||||
if leaf != nil && time.Until(leaf.NotAfter) < renewBefore {
|
||||
if err := r.renew(); err != nil {
|
||||
log.Printf("certificate renewal: %v", err)
|
||||
}
|
||||
}
|
||||
time.Sleep(12 * time.Hour)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *rackd) renew() error {
|
||||
key, csr, err := newCSR("rack:" + r.cfg.Name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var resp struct {
|
||||
Cert string `json:"cert"`
|
||||
}
|
||||
if _, err := r.call(r.fast, "POST", "/api/v1/rack/renew", enrollRequest{CSR: csr}, &resp); err != nil {
|
||||
return err
|
||||
}
|
||||
pair, err := tls.X509KeyPair([]byte(resp.Cert), key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
pair.Leaf, _ = x509.ParseCertificate(pair.Certificate[0])
|
||||
if err := os.WriteFile(r.cfg.KeyFile, key, 0o640); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(r.cfg.CertFile, []byte(resp.Cert), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
r.certMu.Lock()
|
||||
r.cert = &pair
|
||||
r.certMu.Unlock()
|
||||
log.Printf("renewed the rack certificate, valid until %s", pair.Leaf.NotAfter.Format("2006-01-02"))
|
||||
return nil
|
||||
}
|
||||
|
||||
func newCSR(cn string) (keyPEM []byte, csrPEM string, err error) {
|
||||
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
der, err := x509.CreateCertificateRequest(rand.Reader, &x509.CertificateRequest{Subject: pkix.Name{CommonName: cn}}, key)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
kder, _ := x509.MarshalECPrivateKey(key)
|
||||
return pemEncode("EC PRIVATE KEY", kder), string(pemEncode("CERTIFICATE REQUEST", der)), nil
|
||||
}
|
||||
|
||||
// rackJoin enrolls this host as a rack controller with a code from
|
||||
// `rollcall controller join-code` and writes rack.json and the certificate.
|
||||
func rackJoin(args []string) error {
|
||||
if len(args) < 1 || strings.HasPrefix(args[0], "-") {
|
||||
return errors.New("usage: rollcall rack join CODE -url http://ADDR [-name N] [-listen ADDR:80] [-config /etc/rollcall/rack.json]")
|
||||
}
|
||||
code, err := parseJoinCode(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fs := newFlags("rack join")
|
||||
host, _ := os.Hostname()
|
||||
name := fs.String("name", host, "rack controller name")
|
||||
u := fs.String("url", "", "how machines reach this rack controller, e.g. http://10.213.78.1 (required)")
|
||||
listen := fs.String("listen", "", "listen address (default: the url's host, port 80)")
|
||||
cfgPath := fs.String("config", "/etc/rollcall/rack.json", "configuration to write")
|
||||
dir := fs.String("dir", "/etc/rollcall/rack", "where the certificate goes")
|
||||
_ = fs.Parse(args[1:])
|
||||
if *u == "" {
|
||||
return errors.New("-url is required: the address machines on this segment reach this host at")
|
||||
}
|
||||
cfg := RackConfig{}
|
||||
if b, err := os.ReadFile(*cfgPath); err == nil {
|
||||
_ = json.Unmarshal(b, &cfg)
|
||||
}
|
||||
cfg.Name, cfg.URL, cfg.Region, cfg.Secret = *name, strings.TrimRight(*u, "/"), "https://"+code.Addr, ""
|
||||
if *listen != "" {
|
||||
cfg.Listen = *listen
|
||||
} else if cfg.Listen == "" {
|
||||
if pu, err := url.Parse(cfg.URL); err == nil {
|
||||
cfg.Listen = net.JoinHostPort(pu.Hostname(), "80")
|
||||
}
|
||||
}
|
||||
cfg.CAFile, cfg.CertFile, cfg.KeyFile = filepath.Join(*dir, "ca.pem"), filepath.Join(*dir, "rack.pem"), filepath.Join(*dir, "rack-key.pem")
|
||||
if cfg.TFTPRoot == "" {
|
||||
cfg.TFTPRoot = "/srv/tftp"
|
||||
}
|
||||
key, csr, err := newCSR("rack:" + cfg.Name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req := enrollRequest{Secret: base64.RawURLEncoding.EncodeToString(code.Secret[:]), Name: cfg.Name, CSR: csr,
|
||||
Rack: RackController{URL: cfg.URL, Version: version, Interfaces: localInterfaces(), DHCPInterfaces: cfg.DHCPInterfaces, TFTPRoot: cfg.TFTPRoot}}
|
||||
body, _ := json.Marshal(req)
|
||||
hc := &http.Client{Timeout: 30 * time.Second, Transport: &http.Transport{TLSClientConfig: pinnedTLS(code.Fingerprint)}}
|
||||
resp, err := hc.Post(cfg.Region+"/api/v1/rack/enroll", "application/json", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot reach the region at %s: %w", cfg.Region, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
data, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != 200 {
|
||||
var e apiError
|
||||
_ = json.Unmarshal(data, &e)
|
||||
return fmt.Errorf("the region refused: %s %s", resp.Status, e.Error)
|
||||
}
|
||||
var out struct {
|
||||
Name, Cert, CA string
|
||||
}
|
||||
if err := json.Unmarshal(data, &out); err != nil {
|
||||
return err
|
||||
}
|
||||
cfg.Name = out.Name
|
||||
if err := os.MkdirAll(*dir, 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, f := range []struct {
|
||||
p string
|
||||
b []byte
|
||||
mode os.FileMode
|
||||
}{{cfg.CAFile, []byte(out.CA), 0o644}, {cfg.CertFile, []byte(out.Cert), 0o644}, {cfg.KeyFile, key, 0o640}} {
|
||||
if err := os.WriteFile(f.p, f.b, f.mode); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
cb, _ := json.MarshalIndent(cfg, "", " ")
|
||||
if err := os.MkdirAll(filepath.Dir(*cfgPath), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(*cfgPath, append(cb, '\n'), 0o640); err != nil {
|
||||
return err
|
||||
}
|
||||
// the service runs as user rollcall: give its group the key and the configuration
|
||||
if g, err := user.LookupGroup("rollcall"); err == nil {
|
||||
if gid, err := strconv.Atoi(g.Gid); err == nil {
|
||||
for _, p := range []string{*dir, cfg.CAFile, cfg.CertFile, cfg.KeyFile, *cfgPath} {
|
||||
_ = os.Chown(p, 0, gid)
|
||||
}
|
||||
}
|
||||
}
|
||||
ca, _ := parseCertPEM([]byte(out.CA))
|
||||
fmt.Printf("joined the region at %s as rack controller %s\n CA %s\n wrote %s and %s\nnext: sh install-rack.sh (first time) or systemctl restart rollcall-rack\n",
|
||||
cfg.Region, cfg.Name, fpString(certFingerprint(ca)), *cfgPath, *dir)
|
||||
return nil
|
||||
}
|
||||
|
||||
func runRack(args []string) error {
|
||||
fs := newFlags("rack")
|
||||
cfgPath := fs.String("config", "/etc/rollcall/rack.json", "rack controller configuration")
|
||||
@@ -106,7 +314,15 @@ func runRack(args []string) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("region: %w", err)
|
||||
}
|
||||
r := &rackd{cfg: cfg, region: ru, hc: &http.Client{Timeout: 45 * time.Second}, fast: &http.Client{Timeout: 30 * time.Second}, heads: map[string]headInfo{}}
|
||||
r := &rackd{cfg: cfg, region: ru, heads: map[string]headInfo{}}
|
||||
if err := r.clients(); err != nil {
|
||||
return fmt.Errorf("tls: %w", err)
|
||||
}
|
||||
if r.cert != nil {
|
||||
go r.renewLoop()
|
||||
} else {
|
||||
log.Printf("warning: using the legacy shared secret; enroll with rollcall rack join for mutual TLS")
|
||||
}
|
||||
for _, d := range []string{filepath.Join(cfg.TFTPRoot, "pxelinux.cfg"), cfg.CacheDir, filepath.Join(cfg.DnsmasqDir, "conf.d")} {
|
||||
if err := os.MkdirAll(d, 0o755); err != nil {
|
||||
return err
|
||||
@@ -122,9 +338,13 @@ func runRack(args []string) error {
|
||||
Rewrite: func(pr *httputil.ProxyRequest) {
|
||||
pr.SetURL(ru)
|
||||
pr.SetXForwarded()
|
||||
pr.Out.Header.Del("X-Rollcall-Rack-Secret")
|
||||
pr.Out.Header.Set("X-Rollcall-Rack", cfg.Name)
|
||||
pr.Out.Header.Set("X-Rollcall-Rack-Secret", cfg.Secret)
|
||||
if cfg.Secret != "" {
|
||||
pr.Out.Header.Set("X-Rollcall-Rack-Secret", cfg.Secret)
|
||||
}
|
||||
},
|
||||
Transport: r.big.Transport,
|
||||
FlushInterval: 100 * time.Millisecond,
|
||||
}
|
||||
for _, p := range []string{"/api/v1/commission", "/api/v1/bmc-hint", "/api/v1/jobs/self", "/api/v1/jobs/self/progress",
|
||||
@@ -150,7 +370,9 @@ func (r *rackd) req(ctx context.Context, method, p string, body any) (*http.Requ
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("X-Rollcall-Rack", r.cfg.Name)
|
||||
req.Header.Set("X-Rollcall-Rack-Secret", r.cfg.Secret)
|
||||
if r.cfg.Secret != "" {
|
||||
req.Header.Set("X-Rollcall-Rack-Secret", r.cfg.Secret)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
return req, nil
|
||||
}
|
||||
@@ -243,19 +465,20 @@ func (r *rackd) workLoop() {
|
||||
go func(j rackJob) {
|
||||
res := rackResult{ID: j.ID}
|
||||
switch j.Kind {
|
||||
case "ipmi":
|
||||
out, err := ipmiExec(context.Background(), r.cfg.IPMITool, &bmcCreds{Address: j.Address, Port: j.Port, User: j.User, Password: j.Password}, j.Args...)
|
||||
case "power":
|
||||
out, err := powerRun(context.Background(), r.cfg.IPMITool, &bmcCreds{Driver: j.Driver, Address: j.Address, Port: j.Port,
|
||||
User: j.User, Password: j.Password, Extra: j.Extra}, j.Op, j.Arg)
|
||||
res.Output = out
|
||||
if err != nil {
|
||||
res.Error = err.Error()
|
||||
if out != "" {
|
||||
res.Error = out
|
||||
res.Error = out + " (" + err.Error() + ")"
|
||||
}
|
||||
}
|
||||
default:
|
||||
res.Error = "unknown job kind " + j.Kind
|
||||
res.Error = "unknown job kind " + j.Kind + " (update this rack controller)"
|
||||
}
|
||||
log.Printf("job %s %s %s: %s", j.ID, j.Kind, strings.Join(j.Args, " "), oneLine(res.Output+" "+res.Error))
|
||||
log.Printf("job %s %s %s %s %s: %s", j.ID, j.Kind, j.Driver, j.Op, j.Arg, oneLine(res.Output+" "+res.Error))
|
||||
if _, err := r.call(r.fast, "POST", "/api/v1/rack/result", res, nil); err != nil {
|
||||
log.Printf("result %s: %v", j.ID, err)
|
||||
}
|
||||
@@ -318,7 +541,7 @@ func (r *rackd) fill(rel string, w http.ResponseWriter) error {
|
||||
if !ok {
|
||||
return errors.New("bad path")
|
||||
}
|
||||
resp, err := http.Get(r.cfg.Region + rel) // no timeout: images are large
|
||||
resp, err := r.big.Get(r.cfg.Region + rel) // no timeout: images are large
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
+84
-6
@@ -3,6 +3,8 @@ package main
|
||||
import (
|
||||
"context"
|
||||
"crypto/subtle"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"embed"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
@@ -10,7 +12,9 @@ import (
|
||||
"io"
|
||||
"io/fs"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -25,6 +29,7 @@ type App struct {
|
||||
store *Store
|
||||
sealer *Sealer
|
||||
hub *rackHub
|
||||
ca *CA
|
||||
|
||||
netMu sync.RWMutex
|
||||
racks map[string]RackController
|
||||
@@ -59,8 +64,29 @@ func (a *App) isAdmin(r *http.Request) bool {
|
||||
return strings.HasPrefix(h, "Bearer ") && tokenEqual(strings.TrimPrefix(h, "Bearer "), a.cfg.AdminToken)
|
||||
}
|
||||
|
||||
// adminOverHTTP applies admin_http to requests that did not come over TLS.
|
||||
func (a *App) adminOverHTTP(r *http.Request) error {
|
||||
if r.TLS != nil || a.cfg.AdminHTTP == "allow" {
|
||||
return nil
|
||||
}
|
||||
host, _, _ := net.SplitHostPort(r.RemoteAddr)
|
||||
if ip := net.ParseIP(host); a.cfg.AdminHTTP == "loopback" && ip != nil && ip.IsLoopback() {
|
||||
return nil
|
||||
}
|
||||
_, port, _ := net.SplitHostPort(a.cfg.TLSListen)
|
||||
h := r.Host
|
||||
if hh, _, err := net.SplitHostPort(h); err == nil {
|
||||
h = hh
|
||||
}
|
||||
return fmt.Errorf("the admin API is HTTPS-only here: https://%s (trust the CA from /ca.pem)", net.JoinHostPort(strings.Trim(h, "[]"), port))
|
||||
}
|
||||
|
||||
func (a *App) adminAuth(next http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if err := a.adminOverHTTP(r); err != nil {
|
||||
fail(w, 403, err)
|
||||
return
|
||||
}
|
||||
if !a.isAdmin(r) {
|
||||
w.Header().Set("WWW-Authenticate", `Bearer realm="rollcall"`)
|
||||
fail(w, http.StatusUnauthorized, errors.New("admin token required"))
|
||||
@@ -111,6 +137,9 @@ func (a *App) routes() http.Handler {
|
||||
mux.HandleFunc("GET /api/v1/machines/{id}/power", a.adminAuth(a.handlePower))
|
||||
mux.HandleFunc("POST /api/v1/machines/{id}/power", a.adminAuth(a.handlePower))
|
||||
mux.HandleFunc("POST /api/v1/machines/{id}/bootdev", a.adminAuth(a.handleBootdev))
|
||||
mux.HandleFunc("GET /api/v1/machines/{id}/bmc", a.adminAuth(a.handleBMC))
|
||||
mux.HandleFunc("POST /api/v1/machines/{id}/bmc", a.adminAuth(a.handleBMC))
|
||||
mux.HandleFunc("GET /api/v1/power-drivers", a.adminAuth(func(w http.ResponseWriter, r *http.Request) { writeJSON(w, 200, powerDrivers) }))
|
||||
mux.HandleFunc("GET /api/v1/whoami", a.adminAuth(func(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, 200, map[string]any{"ok": true})
|
||||
}))
|
||||
@@ -173,6 +202,10 @@ func (a *App) routes() http.Handler {
|
||||
mux.HandleFunc("GET /api/v1/controllers", a.adminAuth(a.handleControllers))
|
||||
mux.HandleFunc("DELETE /api/v1/controllers/{name}", a.adminAuth(a.handleControllerDelete))
|
||||
mux.HandleFunc("GET /api/v1/controllers/{name}/dhcp", a.adminAuth(a.handleRackDHCP))
|
||||
mux.HandleFunc("GET /ca.pem", a.handleCAPEM)
|
||||
mux.HandleFunc("POST /api/v1/controllers/join", a.adminAuth(a.handleJoinCode))
|
||||
mux.HandleFunc("POST /api/v1/rack/enroll", a.handleRackEnroll) // join-code secret inside, TLS only
|
||||
mux.HandleFunc("POST /api/v1/rack/renew", a.rackAuth(a.handleRackRenew))
|
||||
mux.HandleFunc("POST /api/v1/rack/register", a.rackAuth(a.handleRackRegister))
|
||||
mux.HandleFunc("GET /api/v1/rack/state", a.rackAuth(a.handleRackState))
|
||||
mux.HandleFunc("GET /api/v1/rack/work", a.rackAuth(a.handleRackWork))
|
||||
@@ -327,7 +360,11 @@ func summarize(m *Machine) Summary {
|
||||
if m.BMC != nil {
|
||||
var b obj
|
||||
if json.Unmarshal(m.BMC, &b) == nil {
|
||||
s.BMC = &obj{"source": b["source"], "address": b["address"], "user": b["user"],
|
||||
drv := str(b["driver"])
|
||||
if drv == "" {
|
||||
drv = "ipmi"
|
||||
}
|
||||
s.BMC = &obj{"source": b["source"], "address": b["address"], "user": b["user"], "driver": drv,
|
||||
"configured": b["configured"], "verified_oob": b["verified_oob"]}
|
||||
}
|
||||
}
|
||||
@@ -429,6 +466,21 @@ func (a *App) handleCommission(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
m.Inventory, _ = json.Marshal(inv)
|
||||
|
||||
// a BMC an operator configured (driver_set) survives commissioning; what the agent
|
||||
// found is kept next to it for reference
|
||||
if old, err := a.store.Get(r.Context(), m.ID); err == nil && old.BMC != nil {
|
||||
var ob obj
|
||||
if json.Unmarshal(old.BMC, &ob) == nil && ob["driver_set"] == true {
|
||||
var found obj
|
||||
_ = json.Unmarshal(m.BMC, &found)
|
||||
delete(found, "password")
|
||||
ob["commissioning"] = found
|
||||
m.BMC, _ = json.Marshal(ob)
|
||||
secret = old.bmcSecret
|
||||
detail += " (operator BMC settings kept)"
|
||||
}
|
||||
}
|
||||
|
||||
isNew, err := a.store.UpsertCommissioned(r.Context(), m, secret)
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
@@ -436,6 +488,9 @@ func (a *App) handleCommission(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
_ = a.store.AddEvent(r.Context(), m.ID, "commissioned", detail)
|
||||
_ = a.store.SetRackController(r.Context(), m.ID, a.fromRack(r))
|
||||
if full, err := a.store.Get(r.Context(), m.ID); err == nil {
|
||||
go a.probeBMC(full)
|
||||
}
|
||||
if full, err := a.store.Get(r.Context(), m.ID); err == nil {
|
||||
m = full // carries profile / node token when this is a re-commission of a stateless node
|
||||
}
|
||||
@@ -558,12 +613,11 @@ func (a *App) handlePower(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
action = req.Action
|
||||
}
|
||||
args, ok := powerActions[action]
|
||||
if !ok {
|
||||
if !powerOps[action] || action == "bootdev" || action == "probe" {
|
||||
fail(w, 400, fmt.Errorf("unknown power action %q", action))
|
||||
return
|
||||
}
|
||||
out, err := a.bmc(r.Context(), m, args...)
|
||||
out, err := a.power(r.Context(), m, action, "")
|
||||
if errors.Is(err, errNoBMC) {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
@@ -587,7 +641,7 @@ func (a *App) handleBootdev(w http.ResponseWriter, r *http.Request) {
|
||||
fail(w, 400, fmt.Errorf("device must be one of pxe, disk, cdrom, bios, none, safe"))
|
||||
return
|
||||
}
|
||||
out, err := a.bmc(r.Context(), m, "chassis", "bootdev", req.Device)
|
||||
out, err := a.power(r.Context(), m, "bootdev", req.Device)
|
||||
if errors.Is(err, errNoBMC) {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
@@ -612,13 +666,37 @@ func serve(cfgPath string) error {
|
||||
return fmt.Errorf("database: %w", err)
|
||||
}
|
||||
app := &App{cfg: cfg, store: store, sealer: sealer, hub: newRackHub()}
|
||||
if app.ca, err = loadOrCreateCA(cfg.CADir, "rollcall CA"); err != nil {
|
||||
return fmt.Errorf("certificate authority: %w", err)
|
||||
}
|
||||
ctx2, cancel2 := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
err = app.startRacks(ctx2)
|
||||
cancel2()
|
||||
if err != nil {
|
||||
return fmt.Errorf("rack controllers: %w", err)
|
||||
}
|
||||
srv := &http.Server{Addr: cfg.Listen, Handler: app.routes(), ReadHeaderTimeout: 10 * time.Second}
|
||||
handler := app.routes()
|
||||
if cfg.TLSListen != "" {
|
||||
names := append([]string{}, cfg.TLSNames...)
|
||||
if u, err := url.Parse(cfg.ServerURL); err == nil && u.Hostname() != "" {
|
||||
names = append(names, u.Hostname())
|
||||
}
|
||||
pair, err := regionCert(app.ca, cfg.CADir, names)
|
||||
if err != nil {
|
||||
return fmt.Errorf("region certificate: %w", err)
|
||||
}
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(app.ca.Cert)
|
||||
tsrv := &http.Server{Addr: cfg.TLSListen, Handler: handler, ReadHeaderTimeout: 10 * time.Second,
|
||||
TLSConfig: &tls.Config{MinVersion: tls.VersionTLS12, Certificates: []tls.Certificate{pair}, ClientCAs: pool, ClientAuth: tls.VerifyClientCertIfGiven}}
|
||||
go func() {
|
||||
log.Printf("rollcall HTTPS on %s (CA %s)", cfg.TLSListen, app.ca.Fingerprint())
|
||||
if err := tsrv.ListenAndServeTLS("", ""); err != nil {
|
||||
log.Printf("HTTPS listener: %v", err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
srv := &http.Server{Addr: cfg.Listen, Handler: handler, ReadHeaderTimeout: 10 * time.Second}
|
||||
log.Printf("rollcall listening on %s (image=%s tftp=%s agent_token=%v)", cfg.Listen, cfg.ImageDir, cfg.TFTPRoot, cfg.AgentToken != "")
|
||||
return srv.ListenAndServe()
|
||||
}
|
||||
@@ -297,6 +297,7 @@ type overlayCtx struct {
|
||||
Nodes []NodeCtx // all machines with a profile, sorted by name
|
||||
Profile Profile
|
||||
Overlays []string // profile overlays plus those of the node's tags
|
||||
CA string // the installation's CA certificate (PEM)
|
||||
Vars map[string]any // cluster_vars overlaid with profile vars
|
||||
Server string
|
||||
Now string
|
||||
@@ -466,7 +467,11 @@ func (a *App) buildCtx(ctx context.Context, m *Machine) (*overlayCtx, error) {
|
||||
}
|
||||
}
|
||||
}
|
||||
return &overlayCtx{Node: me, Nodes: nodes, Profile: *p, Overlays: overlays, Vars: vars, Server: a.urlFor(m),
|
||||
caPEM := ""
|
||||
if a.ca != nil {
|
||||
caPEM = string(a.ca.PEM)
|
||||
}
|
||||
return &overlayCtx{Node: me, Nodes: nodes, Profile: *p, Overlays: overlays, CA: caPEM, Vars: vars, Server: a.urlFor(m),
|
||||
Now: time.Now().Format(time.RFC3339)}, nil
|
||||
}
|
||||
|
||||
@@ -786,17 +791,17 @@ func (a *App) handleAssign(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// pxeBoot sets bootdev pxe and power-cycles (or powers on) through the BMC.
|
||||
func (a *App) pxeBoot(ctx context.Context, m *Machine) (string, bool) {
|
||||
o1, e1 := a.bmc(ctx, m, "chassis", "bootdev", "pxe")
|
||||
o1, e1 := a.power(ctx, m, "bootdev", "pxe")
|
||||
if errors.Is(e1, errNoBMC) {
|
||||
return e1.Error(), false
|
||||
}
|
||||
st, _ := a.bmc(ctx, m, "chassis", "power", "status")
|
||||
st, _ := a.power(ctx, m, "status", "")
|
||||
var o2 string
|
||||
var e2 error
|
||||
if strings.Contains(st, "off") {
|
||||
o2, e2 = a.bmc(ctx, m, "chassis", "power", "on")
|
||||
o2, e2 = a.power(ctx, m, "on", "")
|
||||
} else {
|
||||
o2, e2 = a.bmc(ctx, m, "chassis", "power", "cycle")
|
||||
o2, e2 = a.power(ctx, m, "cycle", "")
|
||||
}
|
||||
_ = a.store.AddEvent(ctx, m.ID, "pxe boot", strings.TrimSpace(o1+"\n"+o2))
|
||||
return strings.TrimSpace(o1 + "\n" + o2), e1 == nil && e2 == nil
|
||||
|
||||
@@ -163,6 +163,15 @@ CREATE TABLE IF NOT EXISTS rack_controllers (
|
||||
last_seen timestamptz,
|
||||
registered timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
ALTER TABLE rack_controllers ADD COLUMN IF NOT EXISTS cert_serial text;
|
||||
ALTER TABLE rack_controllers ADD COLUMN IF NOT EXISTS cert_expires timestamptz;
|
||||
CREATE TABLE IF NOT EXISTS join_tokens (
|
||||
hash text PRIMARY KEY,
|
||||
name text,
|
||||
expires timestamptz NOT NULL,
|
||||
used timestamptz,
|
||||
created timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS pxe_files (
|
||||
mac text PRIMARY KEY,
|
||||
machine_id text,
|
||||
|
||||
+131
-8
@@ -8,9 +8,12 @@ package main
|
||||
// terminal, a bare `rollcall` prints its help as before.
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
@@ -34,6 +37,7 @@ func interactive() bool {
|
||||
type clientConf struct {
|
||||
URL string `json:"url"`
|
||||
Token string `json:"token"`
|
||||
CA string `json:"ca,omitempty"` // PEM of the region's CA, trusted when it was first seen
|
||||
}
|
||||
|
||||
func clientConfPath() string {
|
||||
@@ -115,8 +119,11 @@ type (
|
||||
tuiTickMsg struct{ gen int }
|
||||
tuiResultMsg struct{ title, text string }
|
||||
tuiConnectMsg struct {
|
||||
c *client
|
||||
err error
|
||||
c *client
|
||||
err error
|
||||
caPEM string // the server's CA when it is one this computer does not know
|
||||
caFP string
|
||||
caBase string
|
||||
}
|
||||
)
|
||||
|
||||
@@ -311,6 +318,8 @@ type tuiModel struct {
|
||||
cTok textField
|
||||
cFocus int
|
||||
cErr string
|
||||
cCA string // a CA seen on first connect, trusted on the next Enter
|
||||
cCAURL string
|
||||
}
|
||||
|
||||
func newTUIModel(c *client) tuiModel {
|
||||
@@ -419,11 +428,17 @@ func (m tuiModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
|
||||
return mm, cmd
|
||||
case tuiConnectMsg:
|
||||
m.busy = ""
|
||||
if msg.caPEM != "" {
|
||||
m.cCA, m.cCAURL = msg.caPEM, msg.caBase
|
||||
m.cErr = "This server's certificate comes from an authority this computer does not know:\n\n " + msg.caFP +
|
||||
"\n\nCompare it with `rollcall ca show` on the server. Enter trusts it for this server; Esc quits."
|
||||
return m, nil
|
||||
}
|
||||
if msg.err != nil {
|
||||
m.cErr = msg.err.Error()
|
||||
return m, nil
|
||||
}
|
||||
if err := saveClientConf(clientConf{URL: msg.c.base, Token: msg.c.token}); err != nil {
|
||||
if err := saveClientConf(clientConf{URL: msg.c.base, Token: msg.c.token, CA: msg.c.ca}); err != nil {
|
||||
m.cErr = "connected, but could not save " + clientConfPath() + ": " + err.Error()
|
||||
}
|
||||
m.c, m.connect = msg.c, false
|
||||
@@ -889,7 +904,14 @@ func (m tuiModel) actionMenu(ids []string) *tuiDialog {
|
||||
{label: "Power cycle", run: power("cycle", true)},
|
||||
{label: "Reset", run: power("reset", true)},
|
||||
{label: "Power status", run: power("status", false)},
|
||||
{label: "Boot device: PXE", sep: true, run: boot("pxe")},
|
||||
{label: "BMC settings…", sep: true, run: func(m tuiModel) (tuiModel, tea.Cmd) { return m.bmcDialog(ids) }},
|
||||
{label: "Detect Redfish", run: func(m tuiModel) (tuiModel, tea.Cmd) {
|
||||
return m.run("detect Redfish", ids, apiPost("/bmc", obj{"probe": true}, func(d obj) string {
|
||||
b, _ := d["bmc"].(obj)
|
||||
return "driver " + orDash(str(b["driver"]))
|
||||
}))
|
||||
}},
|
||||
{label: "Boot device: PXE", run: boot("pxe")},
|
||||
{label: "Boot device: disk", run: boot("disk")},
|
||||
{label: "Deploy OS…", sep: true, run: func(m tuiModel) (tuiModel, tea.Cmd) { return m.deployDialog(ids) }},
|
||||
{label: "Assign stateless profile…", run: func(m tuiModel) (tuiModel, tea.Cmd) { return m.assignDialog(ids) }},
|
||||
@@ -1066,6 +1088,66 @@ func (m tuiModel) locationDialog(ids []string) (tuiModel, tea.Cmd) {
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (m tuiModel) bmcDialog(ids []string) (tuiModel, tea.Cmd) {
|
||||
var cur obj
|
||||
if len(ids) == 1 && m.detail != nil && m.detail.Summary.ID == ids[0] {
|
||||
cur = m.detail.BMC
|
||||
}
|
||||
var opts, vals []string
|
||||
idx := 0
|
||||
for i, d := range powerDrivers {
|
||||
opts = append(opts, d.Name+" "+sDim.Render(trunc(d.Description, 44)))
|
||||
vals = append(vals, d.Name)
|
||||
if d.Name == str(cur["driver"]) {
|
||||
idx = i
|
||||
}
|
||||
}
|
||||
var extra []string
|
||||
if ex, ok := cur["extra"].(obj); ok {
|
||||
for k, v := range ex {
|
||||
extra = append(extra, k+"="+str(v))
|
||||
}
|
||||
}
|
||||
pw := textInput("Password", "", "empty keeps the stored one")
|
||||
pw.text.Mask = true
|
||||
fields := []dlgField{{label: "Driver", kind: fkChoice, opts: opts, vals: vals, idx: idx}}
|
||||
if len(ids) == 1 {
|
||||
fields = append(fields, textInput("Address", str(cur["address"]), "host, IP or https://host:port"))
|
||||
}
|
||||
fields = append(fields, textInput("Port", pick(num(cur["port"]) > 0, str(cur["port"]), ""), "driver default"),
|
||||
textInput("User", str(cur["user"]), ""), pw, textInput("Settings", strings.Join(extra, " "), "KEY=VALUE … (system, verify_tls, cipher, node, vmid)"),
|
||||
dlgField{label: "Verify first", kind: fkToggle, on: true})
|
||||
m.dlg = &tuiDialog{title: "BMC · " + m.who(ids), note: "The password is sealed on the region and travels to the rack controller over mutual TLS.", fields: fields, okLabel: "save",
|
||||
onOK: func(m tuiModel, f []dlgField) (tuiModel, tea.Cmd) {
|
||||
req := obj{"driver": field(f, "Driver"), "verify": field(f, "Verify first") != ""}
|
||||
for _, k := range []string{"Address", "User"} {
|
||||
if v := field(f, k); v != "" {
|
||||
req[strings.ToLower(k)] = v
|
||||
}
|
||||
}
|
||||
if v := field(f, "Password"); v != "" {
|
||||
req["password"] = v
|
||||
}
|
||||
if p, err := strconv.Atoi(field(f, "Port")); err == nil {
|
||||
req["port"] = p
|
||||
}
|
||||
if s := field(f, "Settings"); s != "" {
|
||||
ex := map[string]string{}
|
||||
for _, kv := range strings.Fields(s) {
|
||||
k, v, _ := strings.Cut(kv, "=")
|
||||
ex[k] = v
|
||||
}
|
||||
req["extra"] = ex
|
||||
}
|
||||
return m.run("BMC settings", ids, apiPost("/bmc", req, func(d obj) string {
|
||||
b, _ := d["bmc"].(obj)
|
||||
return "driver " + str(b["driver"]) + " " + oneLine(str(d["output"]))
|
||||
}))
|
||||
}}
|
||||
m.dlg.setFocus(0)
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (m tuiModel) layoutDialog(ids []string) (tuiModel, tea.Cmd) {
|
||||
opts, vals := []string{"(from tags, else the boot NIC on its subnet)"}, []string{""}
|
||||
for _, l := range m.d.layouts {
|
||||
@@ -1116,10 +1198,21 @@ func (m tuiModel) connectKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
m.cErr = "a server URL and the admin token are needed"
|
||||
return m, nil
|
||||
}
|
||||
c := &client{base: u, token: strings.TrimSpace(m.cTok.Value()), http: &http.Client{Timeout: 90 * time.Second}}
|
||||
ca := ""
|
||||
if m.cCA != "" && m.cCAURL == u {
|
||||
ca = m.cCA // the operator compared the fingerprint and pressed Enter again
|
||||
}
|
||||
c := &client{base: u, token: strings.TrimSpace(m.cTok.Value()), http: newHTTPClient(ca), ca: ca}
|
||||
m.busy, m.cErr = "connecting", ""
|
||||
return m, tea.Batch(m.spin.Tick, func() tea.Msg {
|
||||
return tuiConnectMsg{c: c, err: c.do("GET", "/api/v1/whoami", nil, nil)}
|
||||
err := c.do("GET", "/api/v1/whoami", nil, nil)
|
||||
var ua x509.UnknownAuthorityError
|
||||
if err != nil && ca == "" && strings.HasPrefix(u, "https://") && (errors.As(err, &ua) || strings.Contains(err.Error(), "unknown authority")) {
|
||||
if pemB, fp, e := fetchCA(u); e == nil {
|
||||
return tuiConnectMsg{c: c, err: err, caPEM: pemB, caFP: fp, caBase: u}
|
||||
}
|
||||
}
|
||||
return tuiConnectMsg{c: c, err: err}
|
||||
})
|
||||
default:
|
||||
if m.cFocus == 0 {
|
||||
@@ -1692,6 +1785,9 @@ func (m tuiModel) machinesPanel(pw int) (string, string, string, int) {
|
||||
dot = sOK.Render("●")
|
||||
}
|
||||
bmc = dot + " " + str(b["address"])
|
||||
if d := str(b["driver"]); d != "" && d != "ipmi" {
|
||||
bmc += sDim.Render(" " + d)
|
||||
}
|
||||
}
|
||||
rows = append(rows, []string{mark, sumName(s), stateCell(s), locShort(s.Location), roleOf(s), truncateWidth(strings.Join(s.Tags, ","), 22),
|
||||
truncateWidth(strings.TrimSpace(deref(s.Manufacturer)+" "+deref(s.Product)), 22), hwSummary(s), bmc, agoP(s.LastSeen)})
|
||||
@@ -1764,7 +1860,11 @@ func (m tuiModel) detailPanel(pw int) (string, string, string, int) {
|
||||
if bmc["configured"] == true {
|
||||
dot = sOK.Render("●")
|
||||
}
|
||||
kv("BMC", fmt.Sprintf("%s %s %s · %s", dot, str(bmc["address"]), str(bmc["user"]), pick(d.HasBMCSecret, "credentials stored", sBad.Render("no credentials"))))
|
||||
drv := str(bmc["driver"])
|
||||
if drv == "" {
|
||||
drv = "ipmi"
|
||||
}
|
||||
kv("BMC", fmt.Sprintf("%s %s %s %s · %s", dot, drv, str(bmc["address"]), str(bmc["user"]), pick(d.HasBMCSecret, "credentials stored", sBad.Render("no credentials"))))
|
||||
} else {
|
||||
kv("BMC", sBad.Render("none"))
|
||||
}
|
||||
@@ -2142,8 +2242,12 @@ func (m tuiModel) networkBody(pw int) (string, int) {
|
||||
if s.Size > 0 {
|
||||
pct = float64(s.Used) * 100 / float64(s.Size)
|
||||
}
|
||||
size := strconv.Itoa(s.Size)
|
||||
if s.Size == 0 {
|
||||
size = "∞"
|
||||
}
|
||||
rows = append(rows, []string{s.CIDR, s.Purpose, fmt.Sprintf("%s/%d", orDash(s.Fabric), s.VID), dhcp, orDash(s.Gateway),
|
||||
fmt.Sprintf("%s %d/%d", bar(pct, 8), s.Used, s.Size), strings.Join(rs, " ")})
|
||||
fmt.Sprintf("%s %d/%s", bar(pct, 8), s.Used, size), strings.Join(rs, " ")})
|
||||
}
|
||||
b.WriteString(table([]string{"Subnet", "Purpose", "Fabric/VLAN", "DHCP", "Gateway", "Used", "Ranges"}, rows, m.row, pw-2, m.inPanel))
|
||||
b.WriteString("\n " + sTitle.Render("Interface layouts") + "\n")
|
||||
@@ -2154,3 +2258,22 @@ func (m tuiModel) networkBody(pw int) (string, int) {
|
||||
b.WriteString(table([]string{"Layout", "Machines", "Interfaces", "Comment"}, rows, -1, pw-2, false))
|
||||
return b.String(), head
|
||||
}
|
||||
|
||||
// fetchCA reads the region's CA without trusting it yet, for the operator to compare.
|
||||
func fetchCA(base string) (string, string, error) {
|
||||
hc := &http.Client{Timeout: 15 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}} //nolint:gosec // shown to the operator, not trusted
|
||||
resp, err := hc.Get(base + "/ca.pem")
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
c, err := parseCertPEM(b)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
return string(b), fpString(certFingerprint(c)), nil
|
||||
}
|
||||
+41
-5
@@ -25,7 +25,15 @@ async function api(method, path, body) {
|
||||
if (!r.ok) throw new Error(data.error || r.statusText);
|
||||
return data;
|
||||
}
|
||||
function askToken() { const d = $('#login'); if (!d.open) d.showModal(); }
|
||||
function askToken() {
|
||||
const d = $('#login');
|
||||
if (location.protocol === 'http:') {
|
||||
$('#http-warn').hidden = false;
|
||||
$('#https-link').href = 'https://' + location.hostname + location.pathname + location.hash;
|
||||
fetch('/ca.pem').then(r => { const fp = r.headers.get('X-Rollcall-CA-Fingerprint'); if (fp) $('#ca-fp').textContent = fp; }).catch(() => {});
|
||||
}
|
||||
if (!d.open) d.showModal();
|
||||
}
|
||||
$('#login form').addEventListener('submit', () => { token = $('#token').value.trim(); try { localStorage.setItem('rollcall.token', token); } catch (e) { /* ignore */ } $('#token').value = ''; refresh(true); });
|
||||
$('#logout').onclick = () => { token = ''; try { localStorage.removeItem('rollcall.token'); } catch (e) { /* ignore */ } askToken(); };
|
||||
|
||||
@@ -205,7 +213,7 @@ function pageMachines() {
|
||||
<td>${tagPills(m)}</td>
|
||||
<td>${esc(((m.manufacturer || '') + ' ' + (m.product || '')).trim().slice(0, 26))}<br><span class="muted mono">${esc((m.serial || '').slice(0, 20))}</span></td>
|
||||
<td>${hwSummary(m)}</td>
|
||||
<td><span class="dot ${m.bmc ? (m.bmc.configured ? 'ok' : 'bad') : ''}"></span><span class="mono">${esc(m.bmc && m.bmc.address || '—')}</span></td>
|
||||
<td><span class="dot ${m.bmc ? (m.bmc.configured ? 'ok' : 'bad') : ''}"></span><span class="mono">${esc(m.bmc && m.bmc.address || '—')}</span>${m.bmc && m.bmc.driver && m.bmc.driver !== 'ipmi' ? `<br><span class="muted">${esc(m.bmc.driver)}</span>` : ''}</td>
|
||||
<td class="muted" title="${esc(when(m.last_seen))}">${ago(m.last_seen)}</td></tr>`).join('')}</tbody></table></div>
|
||||
<div class="pager">${pages > 1 ? `<button class="btn sm" id="prev"${S.page === 0 ? ' disabled' : ''}>‹</button> page ${S.page + 1} / ${pages} <button class="btn sm" id="next"${S.page >= pages - 1 ? ' disabled' : ''}>›</button>` : ''}
|
||||
<select id="per">${[25, 50, 100, 250].map(n => `<option${n === S.per ? ' selected' : ''}>${n}</option>`).join('')}</select> per page</div>`;
|
||||
@@ -413,7 +421,10 @@ function pageMachine(d, tab) {
|
||||
${card('LLDP neighbours (switch ports)', table(['If', 'Switch', 'Descr', 'Port', 'Port descr', 'VLAN'], lldpRows), true)}
|
||||
</div>`;
|
||||
} else if (tab === 'bmc') {
|
||||
body = `<div class="grid">${card('BMC', kv([['Source', esc(b.source)], ['Address', `<span class="mono">${esc(b.address)}${b.port && b.port !== 623 ? ':' + esc(b.port) : ''}</span>`], ['MAC', `<span class="mono">${esc(b.mac)}</span>`], ['User', esc(b.user)], ['Password', d.has_bmc_secret ? '<span class="ok">stored encrypted on the server</span>' : '<span class="bad">none</span>'], ['Configured', b.configured ? '<span class="ok">yes</span>' : '<span class="bad">no</span>'], ['Verified out-of-band', b.verified_oob === true ? '<span class="ok">yes</span>' : b.verified_oob === false ? '<span class="bad">no</span>' : ''], ['Channel', esc(b.channel)], ['LAN', b.lan ? `<span class="mono">${esc(JSON.stringify(b.lan))}</span>` : ''], ['Firmware', esc((b.mc_info || {})['Firmware Revision'])], ['Manufacturer', esc((b.mc_info || {})['Manufacturer Name'])], ['Errors', (b.errors || []).length ? `<span class="bad">${esc(b.errors.join('; '))}</span>` : '']]))}
|
||||
body = `<div class="actions" style="margin-bottom:12px"><button class="btn primary" id="bmc-edit">Edit BMC…</button><button class="btn" id="bmc-probe" title="ask the BMC whether it speaks Redfish with the stored account">Detect Redfish</button><span class="muted">Commands run on rack controller ${esc(s.rack_controller || '')}</span></div>
|
||||
<div class="grid">${card('BMC', kv([['Driver', `<b>${esc(b.driver || 'ipmi')}</b>${b.driver_set ? ' <span class="muted">(set by operator)</span>' : ''}${b.redfish ? ` · Redfish ${esc(b.redfish.redfish)} ${esc(b.redfish.manufacturer || '')} ${esc(b.redfish.model || '')}` : ''}`],
|
||||
['Settings', b.extra && Object.keys(b.extra).length ? `<span class="mono">${esc(Object.entries(b.extra).map(([k, v]) => k + '=' + v).join(' '))}</span>` : ''],
|
||||
['Source', esc(b.source)], ['Address', `<span class="mono">${esc(b.address)}${b.port && b.port !== 623 ? ':' + esc(b.port) : ''}</span>`], ['MAC', `<span class="mono">${esc(b.mac)}</span>`], ['User', esc(b.user)], ['Password', d.has_bmc_secret ? '<span class="ok">stored encrypted on the server</span>' : '<span class="bad">none</span>'], ['Configured', b.configured ? '<span class="ok">yes</span>' : '<span class="bad">no</span>'], ['Verified out-of-band', b.verified_oob === true ? '<span class="ok">yes</span>' : b.verified_oob === false ? '<span class="bad">no</span>' : ''], ['Channel', esc(b.channel)], ['LAN', b.lan ? `<span class="mono">${esc(JSON.stringify(b.lan))}</span>` : ''], ['Firmware', esc((b.mc_info || {})['Firmware Revision'])], ['Manufacturer', esc((b.mc_info || {})['Manufacturer Name'])], ['Errors', (b.errors || []).length ? `<span class="bad">${esc(b.errors.join('; '))}</span>` : '']]))}
|
||||
${(b.users_before || []).length ? card('User table before commissioning', table(['ID', 'Name', 'Priv', 'IPMI'], b.users_before.map(u => [u.id, esc(u.name), esc(u.priv), u.ipmi_msg ? 'yes' : 'no']))) : ''}
|
||||
${b.fru ? card('FRU', `<pre class="json">${esc(JSON.stringify(b.fru, null, 1))}</pre>`) : ''}</div>`;
|
||||
} else if (tab === 'jobs') body = jobsTable(d.jobs || [], false);
|
||||
@@ -445,6 +456,8 @@ function wireMachine(d) {
|
||||
$$('#dmore [data-item]').forEach(b => b.onclick = () => act.more(ids, b.dataset.item));
|
||||
const t = $('#d-tag'); if (t) t.onclick = () => act.tag(ids);
|
||||
const lo = $('#d-loc'); if (lo) lo.onclick = () => act.location(ids);
|
||||
const be = $('#bmc-edit'); if (be) be.onclick = () => editBMC(id, d.bmc || {});
|
||||
const bp = $('#bmc-probe'); if (bp) bp.onclick = async () => { try { const r = await api('POST', `/api/v1/machines/${id}/bmc`, { probe: true }); toast('driver: ' + ((r.bmc || {}).driver || 'ipmi')); await loadDetail(id); } catch (e) { toast(e.message, true); } };
|
||||
$$('[data-untag]').forEach(a => a.onclick = ev => { ev.preventDefault(); each(ids, 'untag', async () => (await api('POST', `/api/v1/machines/${id}/tags`, { remove: [a.dataset.untag] })).effective.join(' ')); });
|
||||
}
|
||||
|
||||
@@ -605,16 +618,17 @@ function pageNetwork() {
|
||||
`<span class="muted" title="${esc(when(c.last_seen))}">${ago(c.last_seen)}</span>`, `<span class="mono muted">${esc((c.interfaces || []).map(i => i.name + ' ' + (i.addrs || []).join(',')).join(' · '))}</span>`, `<button class="btn sm" data-dhcp="${esc(c.name)}">dnsmasq</button>${c.kind === 'remote' ? ` <button class="btn sm danger" data-rmctl="${esc(c.name)}">remove</button>` : ''}`]));
|
||||
const sub = table(['Subnet', 'Purpose', 'Fabric / VLAN', 'DHCP', 'Gateway', 'DNS', 'Usage', 'Ranges', ''], S.subnets.map(s => {
|
||||
const pct = s.size ? Math.round(s.used * 100 / s.size) : 0;
|
||||
const sizeTxt = s.size ? s.size : '∞';
|
||||
const ranges = (s.ranges || []).map(r => `<span class="pill tag" title="${esc(r.comment || '')}">${esc(r.kind)} ${esc(r.start)}–${esc(r.end.split('.').pop())} <a href="#" data-rmrange="${r.id}" title="remove">×</a></span>`).join(' ');
|
||||
return [`<b class="mono">${esc(s.cidr)}</b>${s.name ? `<br><span class="muted">${esc(s.name)}</span>` : ''}`, esc(s.purpose), `${esc(s.fabric || '—')} / ${s.vid || 'untagged'}${s.mtu && s.mtu !== 1500 ? ` · mtu ${s.mtu}` : ''}`,
|
||||
s.dhcp ? `<span class="ok">on</span> · ${esc(s.primary_rack)}${s.secondary_rack ? ' + ' + esc(s.secondary_rack) : ''}` : '<span class="muted">off</span>', `<span class="mono">${esc(s.gateway || '—')}</span>`,
|
||||
`<span class="mono">${esc((s.dns || []).join(', '))}</span>${s.domain ? `<br><span class="muted">${esc(s.domain)}</span>` : ''}`, `<div class="bar" title="${s.used}/${s.size}"><span style="width:${pct}%"></span></div><span class="muted">${s.used} / ${s.size}</span>`,
|
||||
`<span class="mono">${esc((s.dns || []).join(', '))}</span>${s.domain ? `<br><span class="muted">${esc(s.domain)}</span>` : ''}`, `<div class="bar" title="${s.used}/${s.size}"><span style="width:${pct}%"></span></div><span class="muted">${s.used} / ${sizeTxt}</span>${s.family === 6 ? ' <span class="pill tag">IPv6</span>' : ''}`,
|
||||
ranges + ` <button class="btn sm" data-addrange="${esc(s.cidr)}">+ range</button>`, `<button class="btn sm" data-ips="${esc(s.cidr)}">addresses</button> <button class="btn sm" data-editsubnet="${esc(s.cidr)}">edit</button> <button class="btn sm danger" data-rmsubnet="${esc(s.cidr)}">delete</button>`];
|
||||
}));
|
||||
const lay = table(['Layout', 'Machines', 'Interfaces', 'Comment', ''], S.layouts.map(l => [`<b>${esc(l.name)}</b>`, l.machines || 0, `<span class="mono">${esc(layoutSummary(l))}</span>`, esc(l.comment || ''), `<button class="btn sm" data-editlayout="${esc(l.name)}">edit</button> <button class="btn sm danger" data-rmlayout="${esc(l.name)}">delete</button>`]));
|
||||
return `<div class="page-head"><h1>Network</h1><span class="sub">fabrics, VLANs and subnets (MAAS-style); rack controllers serve DHCP/PXE on their VLANs</span>
|
||||
<div class="right"><button class="btn primary" id="new-subnet">New subnet…</button><button class="btn" id="new-layout">New layout…</button></div></div>
|
||||
<div class="card wide" style="margin-bottom:12px"><h3>Rack controllers</h3><div class="tw">${ctl}</div><p class="muted" style="margin:8px 0 0">Add one next to a segment: <code>deploy/install-rack.sh</code> with <code>/etc/rollcall/rack.json</code> (region URL, rack_secret, its own URL). It registers itself.</p></div>
|
||||
<div class="card wide" style="margin-bottom:12px"><h3>Rack controllers</h3><div class="tw">${ctl}</div><p class="muted" style="margin:8px 0 0">Add one next to a segment: <code>rollcall controller join-code -name NAME -addr REGION:443</code> here, then on that host <code>./rollcall rack join CODE -url http://ITS-ADDRESS</code> and <code>sh install-rack.sh</code>. It talks to the region over mutual TLS.</p></div>
|
||||
<div class="card wide" style="margin-bottom:12px"><h3>Subnets</h3><div class="tw">${sub}</div></div>
|
||||
<div class="card wide"><h3>Interface layouts</h3>${lay}<p class="muted" style="margin:8px 0 0">A layout applies to a machine directly (More → Network layout…) or through a tag (Tags → layout). Without one, the boot NIC gets an address on its subnet.</p></div>`;
|
||||
}
|
||||
@@ -683,3 +697,25 @@ function wireNetwork() {
|
||||
$$('[data-editlayout]').forEach(b => b.onclick = () => editLayout(S.layouts.find(l => l.name === b.dataset.editlayout)));
|
||||
$$('[data-rmlayout]').forEach(b => b.onclick = () => { if (confirm(`Delete layout ${b.dataset.rmlayout}?`)) del('/api/v1/layouts/' + b.dataset.rmlayout, 'deleted ' + b.dataset.rmlayout); });
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- BMC driver settings
|
||||
const powerDriverNames = [['ipmi', 'IPMI 2.0 (ipmitool)'], ['redfish', 'Redfish, any vendor'], ['idrac', 'Dell iDRAC 8/9'], ['ilo', 'HPE iLO 4/5/6'], ['supermicro', 'Supermicro (Redfish)'], ['xcc', 'Lenovo XClarity'], ['openbmc', 'OpenBMC'], ['proxmox', 'Proxmox VE VM'], ['manual', 'manual (no power control)']];
|
||||
async function editBMC(id, b) {
|
||||
const v = await dialog('BMC settings', [
|
||||
{ name: 'driver', label: 'Driver', type: 'select', value: b.driver || 'ipmi', options: powerDriverNames },
|
||||
{ name: 'address', label: 'Address', value: b.address || '', placeholder: '10.0.0.5, bmc.example or https://host:8443' },
|
||||
{ name: 'port', label: 'Port', value: b.port ? String(b.port) : '', placeholder: "driver's default" },
|
||||
{ name: 'user', label: 'User', value: b.user || '', help: 'proxmox: token id, e.g. root@pam!rollcall' },
|
||||
{ name: 'password', label: 'Password', placeholder: 'leave empty to keep the stored one', help: 'proxmox: the token secret' },
|
||||
{ name: 'extra', label: 'Settings', type: 'textarea', rows: 3, value: Object.entries(b.extra || {}).map(([k, x]) => `${k}=${x}`).join('\n'), placeholder: 'system=/redfish/v1/Systems/1\nverify_tls=true\ncipher=17\nnode=pve1\nvmid=211', help: 'one KEY=VALUE per line; a line KEY= removes it' },
|
||||
{ name: 'verify', label: 'Check with a status call before saving', type: 'checkbox', value: true },
|
||||
], 'The password is sealed on the region and only travels to the rack controller over mutual TLS.', 'Save');
|
||||
if (!v) return;
|
||||
const extra = {};
|
||||
for (const l of v.extra.split('\n').map(x => x.trim()).filter(Boolean)) { const i = l.indexOf('='); if (i > 0) extra[l.slice(0, i).trim()] = l.slice(i + 1).trim(); }
|
||||
for (const k of Object.keys(b.extra || {})) if (!(k in extra)) extra[k] = '';
|
||||
const body = { driver: v.driver, address: v.address.trim(), user: v.user.trim(), extra, verify: v.verify };
|
||||
if (v.port) body.port = Number(v.port);
|
||||
if (v.password) body.password = v.password;
|
||||
try { const r = await api('POST', `/api/v1/machines/${id}/bmc`, body); toast(`BMC saved${r.output ? ': ' + r.output : ''}`); await loadDetail(id); refresh(); } catch (e) { toast(e.message, true); }
|
||||
}
|
||||
@@ -31,6 +31,7 @@
|
||||
<form method="dialog">
|
||||
<h2>Admin token</h2>
|
||||
<p class="muted">The <code>admin_token</code> from the server's <code>config.json</code>. Stored only in this browser.</p>
|
||||
<p id="http-warn" class="warn" hidden>This page came over plain HTTP: the token crosses the network unencrypted. Use <a id="https-link" href="#">HTTPS</a> after trusting the installation's <a href="/ca.pem">CA certificate</a> <span id="ca-fp" class="mono muted"></span>.</p>
|
||||
<input id="token" type="password" autocomplete="off" placeholder="token" required>
|
||||
<menu><button class="btn primary" value="ok">Sign in</button></menu>
|
||||
</form>
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"listen": "10.213.77.1:80",
|
||||
"listen": ":80",
|
||||
"image_dir": "/srv/rollcall/image",
|
||||
"tftp_root": "/srv/tftp",
|
||||
"database_url": "postgres://rollcall:<PGPASS>@127.0.0.1:5432/rollcall?sslmode=disable",
|
||||
@@ -13,7 +13,9 @@
|
||||
"overlays_dir": "/etc/rollcall/overlays",
|
||||
"dnsmasq_dir": "/var/lib/rollcall/dnsmasq",
|
||||
"rack_name": "pxe-server",
|
||||
"rack_secret": "<RACK-SECRET: rollcall keygen; only needed for remote rack controllers>",
|
||||
"tls_listen": ":443",
|
||||
"tls_names": ["rollcall.example.com"],
|
||||
"admin_http": "loopback",
|
||||
"os_dir": "/srv/rollcall/os",
|
||||
"os_sources": "/etc/rollcall/os-sources.json",
|
||||
"node_pools": {"mgmt": "10.213.77.50-10.213.77.99"},
|
||||
|
||||
+13
-7
@@ -1,25 +1,31 @@
|
||||
#!/bin/sh
|
||||
# Install a rollcall rack controller on this host (Debian/Ubuntu). Run as root
|
||||
# from this directory with the rollcall binary next to it and
|
||||
# /etc/rollcall/rack.json written (see rack.example.json). The region needs a
|
||||
# rack_secret in its config.json; the same value goes into rack.json.
|
||||
# from this directory with the rollcall binary next to it, after enrolling:
|
||||
#
|
||||
# region$ rollcall controller join-code -name hall-b -addr REGION:443
|
||||
# rack$ ./rollcall rack join <code> -url http://ADDR-ON-THE-SEGMENT
|
||||
#
|
||||
# which writes /etc/rollcall/rack.json and the rack's certificate; from then on
|
||||
# rack and region talk over mutual TLS (rack.example.json shows the fields).
|
||||
#
|
||||
# The rack controller serves DHCP/TFTP (dnsmasq) on the VLANs the region
|
||||
# assigns to it, keeps boot and OS images in a local cache, runs BMC commands
|
||||
# and passes the node API through to the region.
|
||||
set -eu
|
||||
cd "$(dirname "$0")"
|
||||
[ -r /etc/rollcall/rack.json ] || { echo "write /etc/rollcall/rack.json first (see rack.example.json)" >&2; exit 1; }
|
||||
[ -r /etc/rollcall/rack.json ] || { echo "enroll first: ./rollcall rack join CODE -url http://ADDR (code from: rollcall controller join-code on the region)" >&2; exit 1; }
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get install -y -q dnsmasq ipmitool pxelinux syslinux-common syslinux-efi jq >/dev/null
|
||||
|
||||
getent passwd rollcall >/dev/null || useradd --system --no-create-home --shell /usr/sbin/nologin --user-group rollcall
|
||||
[ -x ./rollcall ] && install -m 0755 rollcall /usr/local/bin/rollcall
|
||||
chgrp rollcall /etc/rollcall/rack.json && chmod 0640 /etc/rollcall/rack.json
|
||||
# certificate from rollcall rack join: readable by the service's group, the key by nobody else
|
||||
if [ -d /etc/rollcall/rack ]; then chgrp -R rollcall /etc/rollcall/rack && chmod 0750 /etc/rollcall/rack && chmod 0640 /etc/rollcall/rack/*-key.pem; fi
|
||||
|
||||
TFTP=$(jq -r '.tftp_root // "/srv/tftp"' /etc/rollcall/rack.json)
|
||||
CACHE=$(jq -r '.cache_dir // "/var/cache/rollcall"' /etc/rollcall/rack.json)
|
||||
DNS=$(jq -r '.dnsmasq_dir // "/var/lib/rollcall/dnsmasq"' /etc/rollcall/rack.json)
|
||||
TFTP=$(jq -r '(.tftp_root // "") | if . == "" then "/srv/tftp" else . end' /etc/rollcall/rack.json)
|
||||
CACHE=$(jq -r '(.cache_dir // "") | if . == "" then "/var/cache/rollcall" else . end' /etc/rollcall/rack.json)
|
||||
DNS=$(jq -r '(.dnsmasq_dir // "") | if . == "" then "/var/lib/rollcall/dnsmasq" else . end' /etc/rollcall/rack.json)
|
||||
|
||||
# image cache: kernels/initrds are prefetched (TFTP cannot fetch on demand), the rest fills on first use
|
||||
install -d -o rollcall -g rollcall "$CACHE" "$CACHE/rollcall" "$CACHE/images" "$CACHE/os"
|
||||
|
||||
@@ -1,12 +1,15 @@
|
||||
{
|
||||
"_comment": "written by: rollcall rack join <code from: rollcall controller join-code> -url http://10.213.78.1",
|
||||
"name": "hall-b",
|
||||
"region": "http://10.0.0.234",
|
||||
"secret": "<rack_secret from the region's config.json>",
|
||||
"region": "https://10.0.0.234:443",
|
||||
"url": "http://10.213.78.1",
|
||||
"listen": "10.213.78.1:80",
|
||||
"tftp_root": "/srv/tftp",
|
||||
"dnsmasq_dir": "/var/lib/rollcall/dnsmasq",
|
||||
"cache_dir": "/var/cache/rollcall",
|
||||
"ipmitool": "ipmitool",
|
||||
"dhcp_interfaces": []
|
||||
"dhcp_interfaces": [],
|
||||
"ca_file": "/etc/rollcall/rack/ca.pem",
|
||||
"cert_file": "/etc/rollcall/rack/rack.pem",
|
||||
"key_file": "/etc/rollcall/rack/rack-key.pem"
|
||||
}
|
||||
@@ -5,8 +5,9 @@ set -eu
|
||||
cd "$(dirname "$0")"
|
||||
getent passwd pve-vbmc >/dev/null || useradd --system --no-create-home --shell /usr/sbin/nologin --user-group pve-vbmc
|
||||
install -m 0755 pve_vbmc.py /usr/local/bin/pve_vbmc.py
|
||||
install -m 0755 pve_redfish.py /usr/local/bin/pve_redfish.py
|
||||
install -m 0755 pve-vbmc-addrs.sh /usr/local/bin/pve-vbmc-addrs.sh
|
||||
install -m 0644 pve-vbmc@.service pve-vbmc-addrs.service /etc/systemd/system/
|
||||
install -m 0644 pve-vbmc@.service pve-redfish@.service pve-vbmc-addrs.service /etc/systemd/system/
|
||||
chgrp pve-vbmc /etc/pve-vbmc/config.json && chmod 0640 /etc/pve-vbmc/config.json
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now pve-vbmc-addrs.service
|
||||
@@ -16,4 +17,9 @@ systemctl enable --now pve-vbmc-addrs.service
|
||||
for v in $(python3 -c 'import json;print(" ".join(str(b["vmid"]) for b in json.load(open("/etc/pve-vbmc/config.json"))["bmcs"]))'); do
|
||||
systemctl enable --now "pve-vbmc@$v.service"
|
||||
done
|
||||
systemctl --no-pager --type=service | grep pve-vbmc
|
||||
# BMC entries with "redfish": "idrac" | "ilo" | "generic" also get a Redfish service on port 443
|
||||
for v in $(python3 -c 'import json;print(" ".join(str(b["vmid"]) for b in json.load(open("/etc/pve-vbmc/config.json"))["bmcs"] if b.get("redfish")))'); do
|
||||
systemctl enable --now "pve-redfish@$v.service"
|
||||
systemctl restart "pve-redfish@$v.service"
|
||||
done
|
||||
systemctl --no-pager --type=service | grep -E "pve-vbmc|pve-redfish"
|
||||
@@ -0,0 +1,21 @@
|
||||
[Unit]
|
||||
Description=pve-redfish Redfish service for Proxmox VM %i (lab BMC driver tests)
|
||||
After=network-online.target pve-vbmc-addrs.service
|
||||
Wants=network-online.target
|
||||
Requires=pve-vbmc-addrs.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/local/bin/pve_redfish.py --config /etc/pve-vbmc/config.json --vmid %i
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
AmbientCapabilities=CAP_NET_BIND_SERVICE
|
||||
User=pve-vbmc
|
||||
Group=pve-vbmc
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
PrivateTmp=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,239 @@
|
||||
#!/usr/bin/env python3
|
||||
"""pve-redfish: a small Redfish service for one Proxmox VM, for testing BMC drivers.
|
||||
|
||||
It answers on the same address as the VM's pve-vbmc (port 443 instead of UDP
|
||||
623) and shapes itself like a vendor so rollcall's Redfish probe and the
|
||||
idrac / ilo / generic drivers see what real hardware shows:
|
||||
|
||||
idrac Vendor "Dell", system /redfish/v1/Systems/System.Embedded.1, PowerCycle
|
||||
ilo Vendor "HPE", system /redfish/v1/Systems/1, no PowerCycle, PushPowerButton
|
||||
generic no vendor, system /redfish/v1/Systems/1, AllowableValues via @Redfish.ActionInfo
|
||||
|
||||
Config: the pve-vbmc config.json; a bmcs entry may carry "redfish": "idrac"|"ilo"|"generic".
|
||||
Basic auth with the same user/password as the IPMI side. Self-signed TLS.
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import ipaddress
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
import requests
|
||||
import urllib3
|
||||
|
||||
log = logging.getLogger("pve-redfish")
|
||||
|
||||
PROFILES = {
|
||||
"idrac": {"vendor": "Dell", "manufacturer": "Dell Inc.", "model": "PowerEdge R760 (pve)", "id": "System.Embedded.1",
|
||||
"resets": ["On", "ForceOff", "ForceRestart", "GracefulRestart", "GracefulShutdown", "PushPowerButton", "Nmi", "PowerCycle"],
|
||||
"actioninfo": False, "etag": True},
|
||||
"ilo": {"vendor": "HPE", "manufacturer": "HPE", "model": "ProLiant DL380 Gen11 (pve)", "id": "1",
|
||||
"resets": ["On", "ForceOff", "ForceRestart", "Nmi", "PushPowerButton"], "actioninfo": False, "etag": False},
|
||||
"generic": {"vendor": "", "manufacturer": "QEMU", "model": "Standard PC (pve)", "id": "1",
|
||||
"resets": ["On", "ForceOff", "GracefulShutdown", "ForceRestart"], "actioninfo": True, "etag": False},
|
||||
}
|
||||
|
||||
|
||||
class Pve:
|
||||
def __init__(self, cfg, vmid):
|
||||
self.api, self.node, self.vmid = cfg["api"].rstrip("/"), cfg["node"], int(vmid)
|
||||
self.http = requests.Session()
|
||||
self.http.headers["Authorization"] = "PVEAPIToken=" + cfg["token"]
|
||||
self.http.verify = cfg.get("verify_tls", False)
|
||||
if not self.http.verify:
|
||||
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
|
||||
|
||||
def _url(self, path):
|
||||
return f"{self.api}/api2/json/nodes/{self.node}/qemu/{self.vmid}/{path}"
|
||||
|
||||
def get(self, path):
|
||||
r = self.http.get(self._url(path), timeout=10)
|
||||
r.raise_for_status()
|
||||
return r.json()["data"]
|
||||
|
||||
def post(self, path, **data):
|
||||
r = self.http.post(self._url(path), data=data, timeout=30)
|
||||
r.raise_for_status()
|
||||
return r.json()["data"]
|
||||
|
||||
def put(self, path, **data):
|
||||
r = self.http.put(self._url(path), data=data, timeout=30)
|
||||
r.raise_for_status()
|
||||
|
||||
def running(self):
|
||||
return self.get("status/current")["status"] == "running"
|
||||
|
||||
def boot(self, target):
|
||||
cfg = self.get("config")
|
||||
disks = [k for k in ("scsi0", "virtio0", "sata0", "ide0") if k in cfg]
|
||||
cds = sorted(k for k, v in cfg.items() if "media=cdrom" in str(v))
|
||||
nets = sorted(k for k in cfg if k.startswith("net") and k[3:].isdigit())
|
||||
order = {"Pxe": nets + disks + cds, "Cd": cds + disks + nets}.get(target, disks + cds + nets)
|
||||
self.put("config", boot="order=" + ";".join(order))
|
||||
return order
|
||||
|
||||
|
||||
def make_handler(pve, prof, user, password):
|
||||
sysid = prof["id"]
|
||||
syspath = f"/redfish/v1/Systems/{sysid}"
|
||||
state = {"override": "None", "enabled": "Disabled", "etag": 1}
|
||||
want = "Basic " + base64.b64encode(f"{user}:{password}".encode()).decode()
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
server_version = "pve-redfish"
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
log.info("%s %s", self.address_string(), fmt % args)
|
||||
|
||||
def send(self, code, body=None, headers=None):
|
||||
data = json.dumps(body).encode() if body is not None else b""
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("OData-Version", "4.0")
|
||||
for k, v in (headers or {}).items():
|
||||
self.send_header(k, v)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def error(self, code, msg):
|
||||
self.send(code, {"error": {"code": "Base.1.0.GeneralError", "message": msg,
|
||||
"@Message.ExtendedInfo": [{"Message": msg}]}})
|
||||
|
||||
def authed(self):
|
||||
if self.path.rstrip("/") == "/redfish/v1" or self.path.rstrip("/") == "/redfish":
|
||||
return True
|
||||
if self.headers.get("Authorization") != want:
|
||||
self.error(401, "authentication required")
|
||||
return False
|
||||
return True
|
||||
|
||||
def body(self):
|
||||
n = int(self.headers.get("Content-Length") or 0)
|
||||
return json.loads(self.rfile.read(n) or b"{}")
|
||||
|
||||
def system(self):
|
||||
reset = {"target": syspath + "/Actions/ComputerSystem.Reset"}
|
||||
if prof["actioninfo"]:
|
||||
reset["@Redfish.ActionInfo"] = syspath + "/ResetActionInfo"
|
||||
else:
|
||||
reset["ResetType@Redfish.AllowableValues"] = prof["resets"]
|
||||
return {"@odata.id": syspath, "Id": sysid, "Name": "System", "Manufacturer": prof["manufacturer"], "Model": prof["model"],
|
||||
"PowerState": "On" if pve.running() else "Off",
|
||||
"Boot": {"BootSourceOverrideTarget": state["override"], "BootSourceOverrideEnabled": state["enabled"],
|
||||
"BootSourceOverrideTarget@Redfish.AllowableValues": ["None", "Pxe", "Hdd", "Cd", "BiosSetup"]},
|
||||
"Actions": {"#ComputerSystem.Reset": reset}}
|
||||
|
||||
def do_GET(self):
|
||||
if not self.authed():
|
||||
return
|
||||
p = self.path.split("?")[0].rstrip("/")
|
||||
if p in ("/redfish/v1", "/redfish"):
|
||||
root = {"@odata.id": "/redfish/v1/", "RedfishVersion": "1.15.0", "Systems": {"@odata.id": "/redfish/v1/Systems"}}
|
||||
if prof["vendor"]:
|
||||
root["Vendor"] = prof["vendor"]
|
||||
root["Oem"] = {prof["vendor"]: {}}
|
||||
return self.send(200, root)
|
||||
if p == "/redfish/v1/Systems":
|
||||
return self.send(200, {"Members": [{"@odata.id": syspath}], "Members@odata.count": 1})
|
||||
if p == syspath:
|
||||
hdr = {"ETag": f'W/"{state["etag"]}"'} if prof["etag"] else None
|
||||
return self.send(200, self.system(), hdr)
|
||||
if p == syspath + "/ResetActionInfo":
|
||||
return self.send(200, {"Parameters": [{"Name": "ResetType", "Required": True, "AllowableValues": prof["resets"]}]})
|
||||
self.error(404, "not found")
|
||||
|
||||
def do_PATCH(self):
|
||||
if not self.authed():
|
||||
return
|
||||
if self.path.rstrip("/") != syspath:
|
||||
return self.error(404, "not found")
|
||||
if prof["etag"] and self.headers.get("If-Match") not in (f'W/"{state["etag"]}"', "*"):
|
||||
return self.error(412, "If-Match does not match the current ETag")
|
||||
boot = self.body().get("Boot", {})
|
||||
if "BootSourceOverrideEnabled" in boot:
|
||||
state["enabled"] = boot["BootSourceOverrideEnabled"]
|
||||
if "BootSourceOverrideTarget" in boot:
|
||||
state["override"] = boot["BootSourceOverrideTarget"]
|
||||
if state["enabled"] != "Disabled":
|
||||
order = pve.boot(state["override"])
|
||||
log.info("boot override %s -> order %s", state["override"], order)
|
||||
state["etag"] += 1
|
||||
self.send(204)
|
||||
|
||||
def do_POST(self):
|
||||
if not self.authed():
|
||||
return
|
||||
if self.path.rstrip("/") != syspath + "/Actions/ComputerSystem.Reset":
|
||||
return self.error(404, "not found")
|
||||
t = self.body().get("ResetType", "")
|
||||
if t not in prof["resets"]:
|
||||
return self.error(400, f"ResetType {t} is not supported")
|
||||
on = pve.running()
|
||||
log.info("reset %s (was %s)", t, "on" if on else "off")
|
||||
if t == "On" or (t == "PushPowerButton" and not on):
|
||||
if not on:
|
||||
pve.post("status/start")
|
||||
elif t == "ForceOff":
|
||||
if on:
|
||||
pve.post("status/stop")
|
||||
elif t in ("GracefulShutdown",) or (t == "PushPowerButton" and on):
|
||||
pve.post("status/shutdown", forceStop=1, timeout=60)
|
||||
elif t in ("ForceRestart", "GracefulRestart", "PowerCycle"):
|
||||
pve.post("status/reset" if on else "status/start")
|
||||
self.send(204)
|
||||
|
||||
return H
|
||||
|
||||
|
||||
def self_signed(addr):
|
||||
d = tempfile.mkdtemp()
|
||||
crt, key = os.path.join(d, "c.pem"), os.path.join(d, "k.pem")
|
||||
subprocess.run(["openssl", "req", "-x509", "-newkey", "ec", "-pkeyopt", "ec_paramgen_curve:prime256v1", "-nodes", "-days", "3650",
|
||||
"-subj", "/CN=pve-redfish", "-addext", f"subjectAltName=IP:{addr}", "-keyout", key, "-out", crt],
|
||||
check=True, capture_output=True)
|
||||
return crt, key
|
||||
|
||||
|
||||
class Server6(ThreadingHTTPServer):
|
||||
address_family = socket.AF_INET6
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--config", default="/etc/pve-vbmc/config.json")
|
||||
ap.add_argument("--vmid", type=int, required=True)
|
||||
args = ap.parse_args()
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
|
||||
cfg = json.load(open(args.config))
|
||||
entry = next((b for b in cfg["bmcs"] if int(b["vmid"]) == args.vmid), None)
|
||||
if entry is None:
|
||||
log.error("vmid %s not in config", args.vmid)
|
||||
sys.exit(2)
|
||||
prof = PROFILES[entry.get("redfish", "generic")]
|
||||
addr = entry["address"]
|
||||
user = entry.get("user", cfg.get("user", "admin"))
|
||||
password = entry.get("password", cfg.get("password", "password"))
|
||||
handler = make_handler(Pve(cfg, args.vmid), prof, user, password)
|
||||
ip = ipaddress.ip_address(addr)
|
||||
cls = Server6 if ip.version == 6 else ThreadingHTTPServer
|
||||
srv = cls((addr, int(entry.get("redfish_port", 443))), handler)
|
||||
crt, key = self_signed(addr)
|
||||
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
ctx.load_cert_chain(crt, key)
|
||||
srv.socket = ctx.wrap_socket(srv.socket, server_side=True)
|
||||
log.info("Redfish (%s profile) for VM %s on https://%s:%s", entry.get("redfish", "generic"), args.vmid, addr, entry.get("redfish_port", 443))
|
||||
srv.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user