Encrypt backups to backup_recipients with age

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 10:38:47 +02:00
1 parent 3b05ec0538
commit 12efb7d085
10 files changed
+204 -12

No files matched your search

+5
View File
@@ -159,6 +159,11 @@ APIs change between minors without a deprecation period.
where every user on the node could read it: it reaches the engine in its
environment, and a short launcher hands it to SGLang inside the process.
Both take effect as engines restart under an upgraded agent.
- **Backups can be encrypted.** `backup_recipients` in server.yaml (age
recipients or SSH public keys) has `llmgw server backup` encrypt each
archive to them with age, as `.tar.gz.age`: an archive holds the CA's key
and every secret in the store, and is meant to be copied off the machine.
Open one with `age -d -i <key>`.
### Everything that can be changed on a running cluster, from the console
+16
View File
@@ -1187,6 +1187,20 @@ The control plane publishes when the newest archive was written as
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
it is more than 26 hours old — including when there has never been one.
**Encrypt them.** Name who may open them in `/etc/llmgw/server.yaml`, and every
archive is encrypted to those keys with [age](https://age-encryption.org) and
named `.tar.gz.age`:
```yaml
backup_recipients:
- age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p # age-keygen's public key
- ssh-ed25519 AAAAC3Nza... ops@example # or an SSH public key
```
Keep the matching private keys **off** this machine — an archive whose key
sits beside it is not protected by it. Encrypted, a copy is safe wherever it
lands.
**Copy the archives off the machine.** They sit on the disk they back up and do
not survive losing it. Anything that copies files will do — rsync from the
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
@@ -1206,6 +1220,8 @@ them:
```
sudo systemctl stop llmgw-server
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
# an encrypted one: open it with the private key, from wherever it is kept
age -d -i key.txt llmgw-backup-<host>-<time>.tar.gz.age | sudo tar -xz -C /tmp/restore
cat /tmp/restore/MANIFEST # version and time it was taken
# /etc/llmgw: configuration, secrets, certificates and the CA
+12
View File
@@ -185,6 +185,7 @@ rather than repeated.
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `backup_recipients` | `[]string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
@@ -1492,6 +1493,17 @@ BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw-server.
### `backup_recipients`
`[]string`
BackupRecipients encrypt every archive `llmgw server backup` writes, to
whoever holds one of their keys: age recipients (age1…) or SSH public
keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
the certificate authority's key and every secret in the store, and is
copied off the machine; encrypted, the copy is safe wherever it lands.
Empty writes archives unencrypted, readable by root only.
### `releases`
`string`
+4 -1
View File
@@ -4,10 +4,12 @@ go 1.26.6
require (
connectrpc.com/connect v1.21.0
filippo.io/age v1.3.2
github.com/NVIDIA/go-nvml v0.13.4-0
github.com/cespare/xxhash/v2 v2.3.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/charmbracelet/x/ansi v0.11.6
github.com/jackc/pgx/v5 v5.11.0
github.com/knadh/koanf/parsers/yaml v1.1.1
github.com/knadh/koanf/providers/env/v2 v2.0.1
@@ -32,11 +34,12 @@ require (
)
require (
filippo.io/edwards25519 v1.2.0 // indirect
filippo.io/hpke v0.4.0 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/charmbracelet/colorprofile v0.4.1 // indirect
github.com/charmbracelet/x/ansi v0.11.6 // indirect
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
github.com/charmbracelet/x/term v0.2.2 // indirect
github.com/clipperhouse/displaywidth v0.9.0 // indirect
+8
View File
@@ -1,5 +1,13 @@
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d h1:Blprhc2SbChNZtWcU+BLTM4YdoqYAS9V7cJgOwJKyAs=
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
filippo.io/age v1.3.2 h1:r6RSZLFSMm6rzKepZ7ZAYkKCu14f3/Me8c7uKYh7C8c=
filippo.io/age v1.3.2/go.mod h1:TH/Yr2sSRhCKbaH4XPxpUV0Us8Gv6txYUpiZQWz8Evk=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
+37 -9
View File
@@ -15,6 +15,7 @@ import (
"strings"
"time"
agecrypt "filippo.io/age"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
@@ -44,9 +45,11 @@ func newBackupCmd() *cobra.Command {
"people, quotas, usage, the models added at runtime, the secret engine\n" +
"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
"store's address and the certificate authority's key.\n\n" +
"The archive holds that key, so it is written readable by root only. It\n" +
"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
"not survive the loss of this machine.\n\n" +
"The archive holds that key, so it is written readable by root only, and\n" +
"encrypted to backup_recipients in server.yaml when they are set (age\n" +
"recipients or SSH public keys; open it with `age -d -i <key>`). It is on\n" +
"the same disk as what it backs up: copy it elsewhere, or it does not\n" +
"survive the loss of this machine.\n\n" +
"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
"BackupTooOld fires when it is more than a day old.",
@@ -55,7 +58,11 @@ func newBackupCmd() *cobra.Command {
if err != nil {
return err
}
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
to, err := config.ParseBackupRecipients(cfg.BackupRecipients)
if err != nil {
return err
}
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir, to)
if err != nil {
return err
}
@@ -87,7 +94,9 @@ const backupPrefix = "llmgw-backup-"
// writeBackup writes one archive and returns its path. Written beside its
// final name and renamed, so a backup interrupted halfway is never the newest
// one the control plane reports.
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
//
// With recipients, the archive is encrypted to them, and named .tar.gz.age.
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string, recipients []agecrypt.Recipient) (string, error) {
// Listable by anyone, so the control plane — not root — can see when the
// newest archive was written. The archives themselves are root's alone.
if err := os.MkdirAll(dir, 0o755); err != nil {
@@ -95,6 +104,9 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
}
host, _ := os.Hostname()
name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
if len(recipients) > 0 {
name += ".age"
}
final := filepath.Join(dir, name)
part := final + ".part"
@@ -108,7 +120,13 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
_ = os.Remove(part)
}
}()
gz := gzip.NewWriter(f)
var sealed io.WriteCloser = nopCloser{f}
if len(recipients) > 0 {
if sealed, err = agecrypt.Encrypt(f, recipients...); err != nil {
return "", err
}
}
gz := gzip.NewWriter(sealed)
tw := tar.NewWriter(gz)
manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
@@ -124,7 +142,7 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
return "", fmt.Errorf("%s: %w", etcDir, err)
}
if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
if err := errors.Join(tw.Close(), gz.Close(), sealed.Close(), f.Sync(), f.Close()); err != nil {
return "", err
}
if err := os.Rename(part, final); err != nil {
@@ -134,6 +152,16 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
return final, nil
}
type nopCloser struct{ io.Writer }
func (nopCloser) Close() error { return nil }
// isBackup reports whether a file name is an archive this command wrote.
func isBackup(name string) bool {
return strings.HasPrefix(name, backupPrefix) &&
(strings.HasSuffix(name, ".tar.gz") || strings.HasSuffix(name, ".tar.gz.age"))
}
// addStore dumps the store into the archive.
func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
addr, err := store.ResolveAddress(addr)
@@ -280,7 +308,7 @@ func pruneBackups(dir string, keep int) (int, error) {
}
var names []string
for _, e := range entries {
if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
if n := e.Name(); isBackup(n) {
names = append(names, n)
}
}
@@ -306,7 +334,7 @@ func NewestBackup(dir string) time.Time {
}
var newest time.Time
for _, e := range entries {
if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
if n := e.Name(); !isBackup(n) {
continue
}
if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
+78 -2
View File
@@ -3,6 +3,7 @@ package cli
import (
"archive/tar"
"compress/gzip"
"crypto/ed25519"
"io"
"os"
"path/filepath"
@@ -10,9 +11,84 @@ import (
"testing"
"time"
agecrypt "filippo.io/age"
"golang.org/x/crypto/ssh"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func TestABackupIsEncryptedToItsRecipients(t *testing.T) {
tmp := t.TempDir()
etc := filepath.Join(tmp, "etc")
_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("THE-CA-KEY"), 0o600)
id, err := agecrypt.GenerateX25519Identity()
if err != nil {
t.Fatal(err)
}
to, err := config.ParseBackupRecipients([]string{id.Recipient().String()})
if err != nil {
t.Fatal(err)
}
out := filepath.Join(tmp, "backups")
path, err := writeBackup(t.Context(), "", etc, out, to)
if err != nil {
t.Fatal(err)
}
if !strings.HasSuffix(path, ".tar.gz.age") {
t.Errorf("named %s", filepath.Base(path))
}
raw, _ := os.ReadFile(path)
if strings.Contains(string(raw), "THE-CA-KEY") {
t.Fatal("the CA key is in the archive in the clear")
}
f, _ := os.Open(path)
defer f.Close()
plain, err := agecrypt.Decrypt(f, id)
if err != nil {
t.Fatalf("the recipient cannot open it: %v", err)
}
gz, err := gzip.NewReader(plain)
if err != nil {
t.Fatal(err)
}
tr := tar.NewReader(gz)
found := false
for {
h, err := tr.Next()
if err != nil {
break
}
found = found || h.Name == "etc/llmgw/pki/ca-key.pem"
}
if !found {
t.Error("the opened archive lacks the CA key")
}
// Seen and pruned like any other.
if NewestBackup(out).IsZero() {
t.Error("the control plane would not see an encrypted archive")
}
}
func TestBackupRecipientsMustBeKeys(t *testing.T) {
if _, err := config.ParseBackupRecipients([]string{"not a key"}); err == nil {
t.Error("nonsense was taken as a recipient")
}
pub, _, _ := ed25519.GenerateKey(nil)
sp, err := ssh.NewPublicKey(pub)
if err != nil {
t.Fatal(err)
}
if _, err := config.ParseBackupRecipients([]string{
strings.TrimSpace(string(ssh.MarshalAuthorizedKey(sp))) + " ops@example",
}); err != nil {
t.Errorf("an SSH key was refused: %v", err)
}
}
func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
tmp := t.TempDir()
dbPath := filepath.Join(tmp, "store.db")
@@ -32,7 +108,7 @@ func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
out := filepath.Join(tmp, "backups")
path, err := writeBackup(t.Context(), dbPath, etc, out)
path, err := writeBackup(t.Context(), dbPath, etc, out, nil)
if err != nil {
t.Fatal(err)
}
@@ -129,7 +205,7 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
}
_ = db.Close()
out := filepath.Join(tmp, "backups")
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out, nil); err != nil {
t.Fatal(err)
}
entries, _ := os.ReadDir(out)
+32
View File
@@ -0,0 +1,32 @@
package config
import (
"fmt"
"strings"
"filippo.io/age"
"filippo.io/age/agessh"
)
// ParseBackupRecipients reads backup_recipients: age recipients, or SSH
// public keys.
func ParseBackupRecipients(list []string) ([]age.Recipient, error) {
var out []age.Recipient
for i, s := range list {
s = strings.TrimSpace(s)
if strings.HasPrefix(s, "ssh-") {
r, err := agessh.ParseRecipient(s)
if err != nil {
return nil, fmt.Errorf("backup_recipients[%d]: %w", i, err)
}
out = append(out, r)
continue
}
rs, err := age.ParseRecipients(strings.NewReader(s))
if err != nil || len(rs) != 1 {
return nil, fmt.Errorf("backup_recipients[%d] is not an age recipient (age1…) or an SSH public key", i)
}
out = append(out, rs[0])
}
return out, nil
}
+11
View File
@@ -121,6 +121,14 @@ type Server struct {
// Empty means /var/backups/llmgw-server.
BackupDir string `koanf:"backup_dir"`
// BackupRecipients encrypt every archive `llmgw server backup` writes, to
// whoever holds one of their keys: age recipients (age1…) or SSH public
// keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
// the certificate authority's key and every secret in the store, and is
// copied off the machine; encrypted, the copy is safe wherever it lands.
// Empty writes archives unencrypted, readable by root only.
BackupRecipients []string `koanf:"backup_recipients"`
// Releases is where `llmgw ctl upgrade` fetches release binaries from,
// for this machine and — through their agents — every node. Empty means
// the project's own releases.
@@ -619,6 +627,9 @@ func (s Server) Validate() error {
if s.Listen == "" {
return fmt.Errorf("listen is empty")
}
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
return err
}
if err := s.AgentTLS.Validate(); err != nil {
return err
}
+1
View File
@@ -225,6 +225,7 @@ var Reference = []Block{
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},