server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d29f694aaf
commit
10466416df
11 files changed
+889
-25
No files matched your search
+7
-1
@@ -465,7 +465,13 @@ portal is not done — the page shows; the console and ctl change.
|
||||
thinking blocks); prompt caching; the account page's "Connect your
|
||||
editor" does not offer Claude Code yet.
|
||||
**Anthropic `/v1/messages` (M).** Translate like `responses.go`.
|
||||
12. **`server restore` (S).** Restore is manual shell today
|
||||
12. **Done (branch `features-c2`):** `server restore` (dry run, version
|
||||
check, decryption, modes and owners, SQLite and PostgreSQL),
|
||||
`backup_copy_to` (directory or ssh), the drill in docs/install.md.
|
||||
Left: restoring a replicated store's members (restore into the primary;
|
||||
Patroni replicates it); the console's Back up now does not offer
|
||||
restore; an ssh copy holds the archive in memory while it sends it.
|
||||
**`server restore` (S).** Restore is manual shell today
|
||||
(`docs/install.md:1187`); add the command with version check, dry run,
|
||||
off-host copy target; a documented DR drill.
|
||||
13. **SLO-based admission and token-level metrics (S–M).** TPOT and
|
||||
|
||||
@@ -71,6 +71,20 @@ APIs change between minors without a deprecation period.
|
||||
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
|
||||
and **E** on the console's Usage view.
|
||||
|
||||
### Restoring a backup, and copies off the machine
|
||||
|
||||
- **`llmgw server restore ARCHIVE`** puts a backup back: checks it first
|
||||
(whole, opens with `--identity`, not newer than this llmgw, where the store
|
||||
goes), says so with `--dry-run`, then moves `/etc/llmgw` aside, restores it
|
||||
with modes and owners, restores the store (SQLite, or PostgreSQL with
|
||||
`pg_restore --clean`), and starts the server. A drill is in
|
||||
docs/install.md.
|
||||
- **`backup_copy_to`** copies each archive to a directory or over ssh as it
|
||||
is written, keeping as many there as here; refused without
|
||||
`backup_recipients`.
|
||||
- Archives now record each file's owner by name, so a restore on another
|
||||
machine gives the llmgw user its files back.
|
||||
|
||||
### Anthropic's Messages API
|
||||
|
||||
- **`/v1/messages`** (and `/v1/messages/count_tokens`) translated to chat
|
||||
|
||||
+47
-19
@@ -1529,35 +1529,63 @@ anywhere else.** Restoring an old store restores which keys exist and who owns
|
||||
them, not their secrets. A key created after the backup is gone and has to be
|
||||
reissued.
|
||||
|
||||
### Copying archives off the machine
|
||||
|
||||
```yaml
|
||||
backup_recipients: [age1…] # required: an archive leaves only encrypted
|
||||
backup_copy_to:
|
||||
- /mnt/backups/llmgw # a mounted share or another disk
|
||||
- ssh://backup@10.0.0.20/srv/llmgw-backups
|
||||
backup_copy_key: /etc/llmgw/secrets/backup-ssh-key
|
||||
```
|
||||
|
||||
Each archive is copied there as soon as it is written, and the same number is
|
||||
kept there as here (`--keep`, 14). An ssh host's key is learnt the first time
|
||||
and checked every time after, in `/etc/llmgw/backup_known_hosts`. A copy that
|
||||
fails makes the backup fail, so the timer's unit shows it.
|
||||
|
||||
### Restoring
|
||||
|
||||
On a machine with llmgw installed (the same version or newer) and, for
|
||||
PostgreSQL, an empty database and role as `server init --postgres` would create
|
||||
them:
|
||||
On a machine with llmgw installed — the same version as the archive's, or
|
||||
newer — and, for PostgreSQL, the database and role the store's address names:
|
||||
|
||||
```
|
||||
sudo systemctl stop llmgw-server
|
||||
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
|
||||
# an encrypted one: open it with the private key, from wherever it is kept
|
||||
age -d -i key.txt llmgw-backup-<host>-<time>.tar.gz.age | sudo tar -xz -C /tmp/restore
|
||||
cat /tmp/restore/MANIFEST # version and time it was taken
|
||||
|
||||
# /etc/llmgw: configuration, secrets, certificates and the CA
|
||||
sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
|
||||
|
||||
# the store — PostgreSQL
|
||||
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
|
||||
# the store — SQLite
|
||||
sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
|
||||
|
||||
sudo systemctl start llmgw-server
|
||||
sudo rm -rf /tmp/restore
|
||||
sudo llmgw server restore /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz --dry-run
|
||||
sudo llmgw server restore llmgw-backup-<host>-<time>.tar.gz.age --identity key.txt
|
||||
```
|
||||
|
||||
It checks before it touches anything — that the archive is whole and opens,
|
||||
that this llmgw is not older than the one that wrote it, where the store goes
|
||||
and that `pg_restore` is there for a PostgreSQL dump — and `--dry-run` stops
|
||||
there. Then it stops `llmgw-server`, moves `/etc/llmgw` aside to
|
||||
`/etc/llmgw.before-restore-<time>`, puts the archive's in its place with each
|
||||
file's mode and owner, restores the store (a SQLite file replaced, the old one
|
||||
kept beside it; PostgreSQL by `pg_restore --clean` in one transaction), starts
|
||||
the server and waits for it to answer. `--store` restores the store somewhere
|
||||
else than the archive's `server.yaml` says.
|
||||
|
||||
Running engines keep serving throughout: agents reconnect to the restored
|
||||
control plane and their replicas are adopted, and because the engine-key secret
|
||||
came back with the store, the router can still reach them.
|
||||
|
||||
### A disaster recovery drill
|
||||
|
||||
Worth doing once before it is needed, on a spare machine, and again after a
|
||||
major upgrade:
|
||||
|
||||
1. Install the same llmgw version on the spare machine (step 1).
|
||||
2. Copy the newest archive there from wherever `backup_copy_to` put it, and
|
||||
the private key that opens it.
|
||||
3. `sudo llmgw server restore <archive> --identity <key> --dry-run`, and read
|
||||
what it says. For PostgreSQL, create the database and role first, or point
|
||||
`--store` at a scratch database.
|
||||
4. Without `--dry-run`. Then `llmgw ctl status`, `llmgw keys list --all`, and
|
||||
a request with a key that existed before.
|
||||
5. Note how long it took: that is the number to plan an outage around.
|
||||
|
||||
The drill machine now holds the cluster's CA key and secrets: take it down
|
||||
afterwards, or keep it as the standby it has become.
|
||||
|
||||
**The mirror's directory** is the one thing you can rebuild, by downloading it
|
||||
all again. A pinned version may no longer be upstream to download: that is
|
||||
exactly what pinning it protected you from, and a copy of the directory is what
|
||||
|
||||
@@ -245,6 +245,8 @@ rather than repeated.
|
||||
| `notify` | `notify.Config` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `backup_copy_to` | `[]string` | [Server](#server) |
|
||||
| `backup_copy_key` | `string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
| `control` | `Control` | [Server](#server) |
|
||||
@@ -2008,6 +2010,23 @@ the certificate authority's key and every secret in the store, and is
|
||||
copied off the machine; encrypted, the copy is safe wherever it lands.
|
||||
Empty writes archives unencrypted, readable by root only.
|
||||
|
||||
### `backup_copy_to`
|
||||
|
||||
`[]string`
|
||||
|
||||
BackupCopyTo is where each archive is copied once written, so it
|
||||
survives this machine: a directory (a mounted share, another disk) or
|
||||
ssh://user@host[:port]/path, reached with BackupCopyKey and checked
|
||||
against /etc/llmgw/backup_known_hosts, which learns a host the first
|
||||
time. The same number of archives is kept there as here. Refused
|
||||
without backup_recipients: an archive leaves the machine encrypted.
|
||||
|
||||
### `backup_copy_key`
|
||||
|
||||
`string`
|
||||
|
||||
BackupCopyKey is the SSH private key for ssh:// copies.
|
||||
|
||||
### `releases`
|
||||
|
||||
`string`
|
||||
|
||||
+27
-4
@@ -74,6 +74,16 @@ func newBackupCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(cfg.BackupCopyTo) > 0 {
|
||||
if err := copyBackup(cmd.Context(), path, cfg.BackupCopyTo, cfg.BackupCopyKey,
|
||||
filepath.Join(etcDir, "backup_known_hosts"), keep,
|
||||
func(s string) { fmt.Fprintln(cmd.OutOrStdout(), s) }); err != nil {
|
||||
// The archive is here; that it did not travel is said, and
|
||||
// the exit says it failed, so the timer's unit shows it.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "wrote %s\n", path)
|
||||
return fmt.Errorf("the archive was not copied off this machine: %w", err)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "wrote %s", path)
|
||||
if removed > 0 {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "; removed %d older", removed)
|
||||
@@ -263,8 +273,9 @@ func addTree(tw *tar.Writer, root, as string) error {
|
||||
}
|
||||
switch {
|
||||
case d.IsDir():
|
||||
return tw.WriteHeader(&tar.Header{Name: name + "/", Mode: int64(info.Mode().Perm()),
|
||||
Typeflag: tar.TypeDir, ModTime: info.ModTime()})
|
||||
h := ownedHeader(info)
|
||||
h.Name, h.Mode, h.Typeflag, h.ModTime = name+"/", int64(info.Mode().Perm()), tar.TypeDir, info.ModTime()
|
||||
return tw.WriteHeader(h)
|
||||
case info.Mode().IsRegular():
|
||||
return addFile(tw, p, name)
|
||||
}
|
||||
@@ -282,14 +293,26 @@ func addFile(tw *tar.Writer, path, name string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: int64(info.Mode().Perm()),
|
||||
Size: info.Size(), ModTime: info.ModTime(), Typeflag: tar.TypeReg}); err != nil {
|
||||
h := ownedHeader(info)
|
||||
h.Name, h.Mode, h.Size, h.ModTime, h.Typeflag = name, int64(info.Mode().Perm()), info.Size(), info.ModTime(), tar.TypeReg
|
||||
if err := tw.WriteHeader(h); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(tw, f)
|
||||
return err
|
||||
}
|
||||
|
||||
// ownedHeader is a header with the file's owner and group by name, which is
|
||||
// what a restore on another machine can give back: the llmgw user there has
|
||||
// another number.
|
||||
func ownedHeader(info fs.FileInfo) *tar.Header {
|
||||
h, err := tar.FileInfoHeader(info, "")
|
||||
if err != nil {
|
||||
return &tar.Header{}
|
||||
}
|
||||
return &tar.Header{Uid: h.Uid, Gid: h.Gid, Uname: h.Uname, Gname: h.Gname}
|
||||
}
|
||||
|
||||
func addBytes(tw *tar.Writer, name string, b []byte) error {
|
||||
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(b)),
|
||||
ModTime: time.Now(), Typeflag: tar.TypeReg}); err != nil {
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
)
|
||||
|
||||
// copyBackup puts an archive where server.yaml's backup_copy_to says, and
|
||||
// keeps as many there as here. It says each copy, and fails on the first
|
||||
// that cannot be made.
|
||||
func copyBackup(ctx context.Context, archive string, targets []string, key, knownHosts string, keep int, say func(string)) error {
|
||||
for _, t := range targets {
|
||||
if strings.HasPrefix(t, "ssh://") {
|
||||
if err := copyOverSSH(ctx, archive, t, key, knownHosts, keep); err != nil {
|
||||
return fmt.Errorf("%s: %w", t, err)
|
||||
}
|
||||
} else {
|
||||
if err := copyToDir(archive, t, keep); err != nil {
|
||||
return fmt.Errorf("%s: %w", t, err)
|
||||
}
|
||||
}
|
||||
say("copied to " + t)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func copyToDir(archive, dir string, keep int) error {
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := os.ReadFile(archive) //nolint:gosec // the archive just written
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
final := filepath.Join(dir, filepath.Base(archive))
|
||||
if err := os.WriteFile(final+".part", b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(final+".part", final); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = pruneBackups(dir, keep)
|
||||
return err
|
||||
}
|
||||
|
||||
func copyOverSSH(ctx context.Context, archive, target, key, knownHosts string, keep int) error {
|
||||
u, err := url.Parse(target)
|
||||
if err != nil || u.User == nil || u.Host == "" || u.Path == "" {
|
||||
return fmt.Errorf("not ssh://user@host/path")
|
||||
}
|
||||
port := 22
|
||||
if p := u.Port(); p != "" {
|
||||
port, _ = strconv.Atoi(p)
|
||||
}
|
||||
b, err := os.ReadFile(archive) //nolint:gosec // the archive just written
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c := &onboard.GoSSH{Addr: u.Hostname(), Port: port, User: u.User.Username(), KeyFile: key, KnownHosts: knownHosts}
|
||||
defer c.Close()
|
||||
dir, name := shellQuote(u.Path), shellQuote(filepath.Base(archive))
|
||||
// Written beside its name and renamed, then the oldest beyond keep
|
||||
// removed: only archives this command wrote, by their names.
|
||||
script := fmt.Sprintf(`set -e
|
||||
umask 077
|
||||
mkdir -p %[1]s
|
||||
cat > %[1]s/%[2]s.part
|
||||
mv %[1]s/%[2]s.part %[1]s/%[2]s
|
||||
ls -1t %[1]s | grep -E '^llmgw-backup-.*\.tar\.gz(\.age)?$' | tail -n +%[3]d | while read -r f; do rm -f %[1]s/"$f"; done
|
||||
`, dir, name, keep+1)
|
||||
if out, err := c.Run(ctx, script, b); err != nil {
|
||||
return fmt.Errorf("%w: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func shellQuote(s string) string { return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" }
|
||||
@@ -0,0 +1,546 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
agecrypt "filippo.io/age"
|
||||
"filippo.io/age/agessh"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
|
||||
// restore is `llmgw server restore`: an archive `server backup` wrote, put
|
||||
// back on this machine.
|
||||
type restore struct {
|
||||
archive string
|
||||
identities []string
|
||||
etcDir string
|
||||
storeAddr string // overrides the restored server.yaml's
|
||||
dryRun bool
|
||||
force bool
|
||||
// service is the unit stopped before and started after; "" touches no
|
||||
// service, for tests and machines without systemd.
|
||||
service string
|
||||
out io.Writer
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
func newRestoreCmd() *cobra.Command {
|
||||
rs := restore{etcDir: "/etc/llmgw", service: "llmgw-server", now: time.Now}
|
||||
c := &cobra.Command{
|
||||
Use: "restore ARCHIVE",
|
||||
Short: "Put a backup back: /etc/llmgw and the store, on this machine",
|
||||
Long: "Reads an archive `llmgw server backup` wrote — opening an encrypted one\n" +
|
||||
"with --identity, an age identity file or an SSH private key — and checks\n" +
|
||||
"it before anything is touched: that it is whole, that this llmgw is not\n" +
|
||||
"older than the one that wrote it, and where the store goes. --dry-run\n" +
|
||||
"stops there and says what it would do.\n\n" +
|
||||
"Then: the server is stopped, /etc/llmgw is moved aside to\n" +
|
||||
"/etc/llmgw.before-restore-<time> and the archive's put in its place, the\n" +
|
||||
"store is restored — a SQLite file replaced, the old one kept beside it;\n" +
|
||||
"PostgreSQL by pg_restore --clean into the database the address names,\n" +
|
||||
"which must exist — and the server is started and asked whether it\n" +
|
||||
"answers.\n\n" +
|
||||
"Running engines keep serving throughout: their agents reconnect, and the\n" +
|
||||
"engine-key secret came back with the store. For a whole drill, see\n" +
|
||||
"\"Restoring\" in docs/install.md.",
|
||||
Example: " sudo llmgw server restore /var/backups/llmgw-server/llmgw-backup-ctl-20261004T020000Z.tar.gz --dry-run\n" +
|
||||
" sudo llmgw server restore backup.tar.gz.age --identity ~/llmgw-backup-key.txt",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
rs.archive, rs.out = args[0], cmd.OutOrStdout()
|
||||
return rs.run(cmd.Context())
|
||||
},
|
||||
}
|
||||
f := c.Flags()
|
||||
f.StringArrayVar(&rs.identities, "identity", nil, "a key that opens an encrypted archive: an age identity file or an SSH private key; repeatable")
|
||||
f.StringVar(&rs.etcDir, "etc", rs.etcDir, "the configuration directory to restore into")
|
||||
f.StringVar(&rs.storeAddr, "store", "", "restore the store here rather than where the archive's server.yaml says: a SQLite path or a postgres:// address")
|
||||
f.BoolVar(&rs.dryRun, "dry-run", false, "check the archive and say what would be done; change nothing")
|
||||
f.BoolVar(&rs.force, "force", false, "restore an archive written by a newer llmgw")
|
||||
f.StringVar(&rs.service, "service", rs.service, "the unit stopped before and started after; empty touches none")
|
||||
return c
|
||||
}
|
||||
|
||||
func (rs *restore) say(format string, a ...any) { fmt.Fprintf(rs.out, format+"\n", a...) }
|
||||
|
||||
func (rs *restore) run(ctx context.Context) error {
|
||||
tmp, err := os.MkdirTemp("", "llmgw-restore-")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(tmp)
|
||||
|
||||
// 1. The archive, opened and unpacked where only this user can read it.
|
||||
man, err := rs.unpack(tmp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rs.say("archive written by llmgw %s on %s at %s", man.version, man.host, man.time)
|
||||
if newer(man.version, version.Version) && !rs.force {
|
||||
return fmt.Errorf("the archive was written by llmgw %s, newer than this one (%s): its store may have "+
|
||||
"changes this one does not know. Install %s or newer first, or pass --force", man.version, version.Version, man.version)
|
||||
}
|
||||
restoredEtc := filepath.Join(tmp, "etc", "llmgw")
|
||||
if _, err := os.Stat(filepath.Join(restoredEtc, "server.yaml")); err != nil {
|
||||
return errors.New("the archive has no etc/llmgw/server.yaml: not a control plane's backup")
|
||||
}
|
||||
n := 0
|
||||
_ = filepath.WalkDir(restoredEtc, func(_ string, d os.DirEntry, err error) error {
|
||||
if err == nil && !d.IsDir() {
|
||||
n++
|
||||
}
|
||||
return nil
|
||||
})
|
||||
rs.say("config %d files for %s", n, rs.etcDir)
|
||||
|
||||
// 2. Where the store goes, and what it is.
|
||||
dump, kind := filepath.Join(tmp, "store.sqlite"), "sqlite"
|
||||
if _, err := os.Stat(dump); err != nil {
|
||||
dump, kind = filepath.Join(tmp, "store.pgdump"), "postgres"
|
||||
if _, err := os.Stat(dump); err != nil {
|
||||
dump, kind = "", ""
|
||||
}
|
||||
}
|
||||
target, err := rs.storeTarget(restoredEtc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case kind == "":
|
||||
rs.say("store none in the archive; nothing to restore")
|
||||
case kind == "postgres" && !store.IsPostgres(target):
|
||||
return fmt.Errorf("the archive holds a PostgreSQL dump and the store goes to %s: "+
|
||||
"give --store a postgres:// address", target)
|
||||
case kind == "sqlite" && store.IsPostgres(target):
|
||||
return fmt.Errorf("the archive holds a SQLite store and the store goes to PostgreSQL: give --store a file, " +
|
||||
"then `llmgw keys move-store` copies it into PostgreSQL")
|
||||
case kind == "postgres":
|
||||
if _, err := exec.LookPath("pg_restore"); err != nil {
|
||||
return errors.New("the store is a PostgreSQL dump and pg_restore is not installed here (apt-get install postgresql-client)")
|
||||
}
|
||||
rs.say("store PostgreSQL dump into %s, with pg_restore --clean", redactDSN(target))
|
||||
default:
|
||||
rs.say("store SQLite file to %s%s", target, aside(target, rs.now()))
|
||||
}
|
||||
if rs.service != "" {
|
||||
if exec.Command("systemctl", "is-active", "--quiet", rs.service).Run() == nil { //nolint:gosec // our unit
|
||||
rs.say("service %s is running: stopped first, started after", rs.service)
|
||||
} else {
|
||||
rs.say("service %s is not running: started after", rs.service)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(rs.etcDir); err == nil {
|
||||
rs.say("current %s moved aside to %s", rs.etcDir, rs.etcDir+".before-restore-"+stamp(rs.now()))
|
||||
}
|
||||
if rs.dryRun {
|
||||
rs.say("\ndry run: nothing was changed")
|
||||
return nil
|
||||
}
|
||||
|
||||
// 3. Done, in an order that can be undone by hand at every step.
|
||||
if rs.service != "" {
|
||||
_ = exec.CommandContext(ctx, "systemctl", "stop", rs.service).Run() //nolint:gosec // our unit
|
||||
}
|
||||
if _, err := os.Stat(rs.etcDir); err == nil {
|
||||
if err := os.Rename(rs.etcDir, rs.etcDir+".before-restore-"+stamp(rs.now())); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := copyTree(restoredEtc, rs.etcDir); err != nil {
|
||||
return fmt.Errorf("putting back %s: %w", rs.etcDir, err)
|
||||
}
|
||||
rs.say("ok %s restored", rs.etcDir)
|
||||
switch kind {
|
||||
case "sqlite":
|
||||
if err := restoreSQLite(dump, target, rs.now()); err != nil {
|
||||
return fmt.Errorf("the store: %w", err)
|
||||
}
|
||||
rs.say("ok store restored to %s", target)
|
||||
case "postgres":
|
||||
if err := pgRestore(ctx, target, dump); err != nil {
|
||||
return fmt.Errorf("the store: %w", err)
|
||||
}
|
||||
rs.say("ok store restored into %s", redactDSN(target))
|
||||
}
|
||||
if rs.service != "" {
|
||||
if err := exec.CommandContext(ctx, "systemctl", "start", rs.service).Run(); err != nil { //nolint:gosec // our unit
|
||||
return fmt.Errorf("starting %s: %w; `journalctl -u %s` says why", rs.service, err, rs.service)
|
||||
}
|
||||
if err := rs.waitHealthy(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
rs.say("ok %s answers", rs.service)
|
||||
}
|
||||
rs.say("\nRestored. `llmgw ctl status` shows the cluster as it comes back; agents reconnect by themselves.")
|
||||
return nil
|
||||
}
|
||||
|
||||
type manifest struct{ version, host, time string }
|
||||
|
||||
// unpack opens the archive — decrypting it when it is sealed — and writes
|
||||
// its files under dir, keeping their modes and, by name, their owners.
|
||||
func (rs *restore) unpack(dir string) (manifest, error) {
|
||||
f, err := os.Open(rs.archive)
|
||||
if err != nil {
|
||||
return manifest{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
var r io.Reader = bufio.NewReader(f)
|
||||
if strings.HasSuffix(rs.archive, ".age") {
|
||||
if len(rs.identities) == 0 {
|
||||
return manifest{}, errors.New("the archive is encrypted: give the key that opens it with --identity")
|
||||
}
|
||||
ids, err := readIdentities(rs.identities)
|
||||
if err != nil {
|
||||
return manifest{}, err
|
||||
}
|
||||
if r, err = agecrypt.Decrypt(r, ids...); err != nil {
|
||||
return manifest{}, fmt.Errorf("opening the archive: %w", err)
|
||||
}
|
||||
}
|
||||
gz, err := gzip.NewReader(r)
|
||||
if err != nil {
|
||||
return manifest{}, fmt.Errorf("not an archive server backup wrote: %w", err)
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
var man manifest
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return manifest{}, fmt.Errorf("the archive is damaged: %w", err)
|
||||
}
|
||||
name := filepath.Clean(h.Name)
|
||||
if filepath.IsAbs(name) || strings.HasPrefix(name, "..") {
|
||||
return manifest{}, fmt.Errorf("the archive names %q, outside itself", h.Name)
|
||||
}
|
||||
p := filepath.Join(dir, name)
|
||||
switch h.Typeflag {
|
||||
case tar.TypeDir:
|
||||
if err := os.MkdirAll(p, 0o700); err != nil {
|
||||
return manifest{}, err
|
||||
}
|
||||
case tar.TypeReg:
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
|
||||
return manifest{}, err
|
||||
}
|
||||
out, err := os.OpenFile(p, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
|
||||
if err != nil {
|
||||
return manifest{}, err
|
||||
}
|
||||
_, err = io.Copy(out, tr)
|
||||
if cerr := out.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
return manifest{}, err
|
||||
}
|
||||
if name == "MANIFEST" {
|
||||
b, _ := os.ReadFile(p)
|
||||
man = parseManifest(b)
|
||||
}
|
||||
default:
|
||||
continue
|
||||
}
|
||||
// The mode and owner, applied when the tree is copied into place.
|
||||
if err := writeMeta(p, h); err != nil {
|
||||
return manifest{}, err
|
||||
}
|
||||
}
|
||||
if man.version == "" {
|
||||
return manifest{}, errors.New("the archive has no MANIFEST: not one server backup wrote, or cut short")
|
||||
}
|
||||
return man, nil
|
||||
}
|
||||
|
||||
func parseManifest(b []byte) manifest {
|
||||
var m manifest
|
||||
for _, l := range strings.Split(string(b), "\n") {
|
||||
k, v, _ := strings.Cut(l, " ")
|
||||
switch k {
|
||||
case "llmgw":
|
||||
m.version = v
|
||||
case "host":
|
||||
m.host = v
|
||||
case "time":
|
||||
m.time = v
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// meta is a restored file's mode and owner, kept beside it until it is
|
||||
// copied into place.
|
||||
const metaSuffix = ".llmgw-restore-meta"
|
||||
|
||||
func writeMeta(p string, h *tar.Header) error {
|
||||
return os.WriteFile(p+metaSuffix, fmt.Appendf(nil, "%o %s %s %d %d", h.Mode&0o7777, h.Uname, h.Gname, h.Uid, h.Gid), 0o600)
|
||||
}
|
||||
|
||||
// copyTree puts the unpacked tree in place with each file's mode and owner.
|
||||
func copyTree(from, to string) error {
|
||||
return filepath.WalkDir(from, func(p string, d os.DirEntry, err error) error {
|
||||
if err != nil || strings.HasSuffix(p, metaSuffix) {
|
||||
return err
|
||||
}
|
||||
rel, _ := filepath.Rel(from, p)
|
||||
dst := filepath.Join(to, rel)
|
||||
mode, uid, gid := readMeta(p, d.IsDir())
|
||||
if d.IsDir() {
|
||||
if err := os.MkdirAll(dst, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
b, err := os.ReadFile(p) //nolint:gosec // our own temporary copy
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(dst, b, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := os.Chmod(dst, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() == 0 && uid >= 0 {
|
||||
return os.Lchown(dst, uid, gid)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// readMeta is a file's mode, and its owner resolved by name on this machine,
|
||||
// falling back to the number; -1 leaves it as it is.
|
||||
func readMeta(p string, dir bool) (os.FileMode, int, int) {
|
||||
mode := os.FileMode(0o640)
|
||||
if dir {
|
||||
mode = 0o750
|
||||
}
|
||||
b, err := os.ReadFile(p + metaSuffix)
|
||||
if err != nil {
|
||||
return mode, -1, -1
|
||||
}
|
||||
var m uint32
|
||||
var un, gn string
|
||||
var uid, gid int
|
||||
if _, err := fmt.Sscanf(string(b), "%o %s %s %d %d", &m, &un, &gn, &uid, &gid); err != nil {
|
||||
// Names may be empty: read what is there.
|
||||
f := strings.Fields(string(b))
|
||||
if len(f) == 0 {
|
||||
return mode, -1, -1
|
||||
}
|
||||
v, _ := strconv.ParseUint(f[0], 8, 32)
|
||||
return os.FileMode(v), -1, -1
|
||||
}
|
||||
if u, err := user.Lookup(un); err == nil {
|
||||
uid, _ = strconv.Atoi(u.Uid)
|
||||
}
|
||||
if g, err := user.LookupGroup(gn); err == nil {
|
||||
gid, _ = strconv.Atoi(g.Gid)
|
||||
}
|
||||
return os.FileMode(m), uid, gid
|
||||
}
|
||||
|
||||
func readIdentities(paths []string) ([]agecrypt.Identity, error) {
|
||||
var out []agecrypt.Identity
|
||||
for _, p := range paths {
|
||||
b, err := os.ReadFile(p) //nolint:gosec // the operator's key file
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if bytes.Contains(b, []byte("PRIVATE KEY")) {
|
||||
id, err := agessh.ParseIdentity(b)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", p, err)
|
||||
}
|
||||
out = append(out, id)
|
||||
continue
|
||||
}
|
||||
ids, err := agecrypt.ParseIdentities(bytes.NewReader(b))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s is neither an age identity file nor an SSH private key: %w", p, err)
|
||||
}
|
||||
out = append(out, ids...)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// storeTarget is where the store is restored: --store, or the restored
|
||||
// server.yaml's auth.store, read from where it was unpacked.
|
||||
func (rs *restore) storeTarget(restoredEtc string) (string, error) {
|
||||
if rs.storeAddr != "" {
|
||||
return store.ResolveAddress(rs.storeAddr)
|
||||
}
|
||||
cfg, err := config.Load(filepath.Join(restoredEtc, "server.yaml"))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the archive's server.yaml: %w", err)
|
||||
}
|
||||
addr := cfg.Auth.Store
|
||||
if addr == "" {
|
||||
return "/var/lib/llmgw/llmgw.db", nil
|
||||
}
|
||||
// "@/etc/llmgw/secrets/…" names a file the archive brought back.
|
||||
if f, ok := strings.CutPrefix(addr, "@"); ok && strings.HasPrefix(f, rs.etcDir+"/") {
|
||||
addr = "@" + filepath.Join(restoredEtc, strings.TrimPrefix(f, rs.etcDir+"/"))
|
||||
}
|
||||
return store.ResolveAddress(addr)
|
||||
}
|
||||
|
||||
func restoreSQLite(dump, target string, now time.Time) error {
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := os.Stat(target); err == nil {
|
||||
if err := os.Rename(target, target+".before-restore-"+stamp(now)); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, side := range []string{"-wal", "-shm"} {
|
||||
_ = os.Remove(target + side)
|
||||
}
|
||||
}
|
||||
b, err := os.ReadFile(dump) //nolint:gosec // our own temporary copy
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(target, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if os.Geteuid() == 0 {
|
||||
if u, err := user.Lookup("llmgw"); err == nil {
|
||||
uid, _ := strconv.Atoi(u.Uid)
|
||||
gid, _ := strconv.Atoi(u.Gid)
|
||||
return os.Chown(target, uid, gid)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// pgRestore loads a dump into the database the address names, as its
|
||||
// owner, on the primary.
|
||||
func pgRestore(ctx context.Context, dsn, dump string) error {
|
||||
env, err := pgEnv(dsn)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i, e := range env {
|
||||
if strings.HasPrefix(e, "PGTARGETSESSIONATTRS=") {
|
||||
env[i] = "PGTARGETSESSIONATTRS=read-write"
|
||||
}
|
||||
}
|
||||
env = append(env, "PGTARGETSESSIONATTRS=read-write")
|
||||
db := ""
|
||||
for _, e := range env {
|
||||
if v, ok := strings.CutPrefix(e, "PGDATABASE="); ok {
|
||||
db = v
|
||||
}
|
||||
}
|
||||
if db == "" {
|
||||
return errors.New("the store's address names no database")
|
||||
}
|
||||
// Without --dbname pg_restore writes SQL out rather than restoring; the
|
||||
// rest of the connection comes from the environment, the password with it.
|
||||
cmd := exec.CommandContext(ctx, "pg_restore", "--clean", "--if-exists", "--no-owner", "--single-transaction", //nolint:gosec // fixed binary, our own file
|
||||
"--exit-on-error", "--dbname="+db, dump)
|
||||
cmd.Env = append(os.Environ(), env...)
|
||||
if b, err := cmd.CombinedOutput(); err != nil {
|
||||
return fmt.Errorf("pg_restore: %v: %s", err, strings.TrimSpace(string(b)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// waitHealthy waits for the restored server to answer, for a minute.
|
||||
func (rs *restore) waitHealthy(ctx context.Context) error {
|
||||
cfg, err := config.Load(filepath.Join(rs.etcDir, "server.yaml"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c := newCtlClient(cfg.AdminListen, adminTokenPath(filepath.Join(rs.etcDir, "server.yaml"), cfg))
|
||||
deadline := time.Now().Add(time.Minute)
|
||||
for {
|
||||
var v map[string]any
|
||||
if err := c.get(ctx, "/api/v1/summary", &v); err == nil {
|
||||
return nil
|
||||
} else if time.Now().After(deadline) {
|
||||
return fmt.Errorf("%s started but does not answer: %w", rs.service, err)
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
func stamp(t time.Time) string { return t.UTC().Format("20060102T150405Z") }
|
||||
|
||||
func aside(target string, now time.Time) string {
|
||||
if _, err := os.Stat(target); err == nil {
|
||||
return "; the current one kept as " + filepath.Base(target) + ".before-restore-" + stamp(now)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// redactDSN is an address without its password, for saying where.
|
||||
func redactDSN(dsn string) string {
|
||||
if i := strings.Index(dsn, "://"); i >= 0 {
|
||||
if j := strings.Index(dsn[i+3:], "@"); j >= 0 {
|
||||
cred := dsn[i+3 : i+3+j]
|
||||
if k := strings.IndexByte(cred, ':'); k >= 0 {
|
||||
return dsn[:i+3] + cred[:k] + ":…" + dsn[i+3+j:]
|
||||
}
|
||||
}
|
||||
}
|
||||
return dsn
|
||||
}
|
||||
|
||||
// newer says whether version a is newer than b, by their leading
|
||||
// major.minor.patch; anything unreadable is not newer.
|
||||
func newer(a, b string) bool {
|
||||
pa, oka := semverOf(a)
|
||||
pb, okb := semverOf(b)
|
||||
if !oka || !okb {
|
||||
return false
|
||||
}
|
||||
for i := range pa {
|
||||
if pa[i] != pb[i] {
|
||||
return pa[i] > pb[i]
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func semverOf(v string) ([3]int, bool) {
|
||||
v = strings.TrimPrefix(v, "v")
|
||||
if i := strings.IndexAny(v, "-+"); i >= 0 {
|
||||
v = v[:i]
|
||||
}
|
||||
parts := strings.Split(v, ".")
|
||||
var out [3]int
|
||||
if len(parts) != 3 {
|
||||
return out, false
|
||||
}
|
||||
for i, p := range parts {
|
||||
n, err := strconv.Atoi(p)
|
||||
if err != nil {
|
||||
return out, false
|
||||
}
|
||||
out[i] = n
|
||||
}
|
||||
return out, true
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
agecrypt "filippo.io/age"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestABackupIsRestoredWhereItSays(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
etc := filepath.Join(tmp, "etc")
|
||||
_ = os.MkdirAll(filepath.Join(etc, "secrets"), 0o750)
|
||||
dbPath := filepath.Join(tmp, "llmgw.db")
|
||||
_ = os.WriteFile(filepath.Join(etc, "server.yaml"), []byte("auth:\n enabled: true\n store: "+dbPath+"\n"), 0o640)
|
||||
_ = os.WriteFile(filepath.Join(etc, "secrets", "admin-token"), []byte(strings.Repeat("t", 40)), 0o600)
|
||||
|
||||
db, err := store.Open(t.Context(), dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := db.CreateOrg(t.Context(), "acme"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = db.Close()
|
||||
|
||||
id, _ := agecrypt.GenerateX25519Identity()
|
||||
to, _ := config.ParseBackupRecipients([]string{id.Recipient().String()})
|
||||
archive, err := writeBackup(t.Context(), dbPath, etc, filepath.Join(tmp, "backups"), to)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key := filepath.Join(tmp, "key.txt")
|
||||
_ = os.WriteFile(key, []byte(id.String()+"\n"), 0o600)
|
||||
|
||||
// Onto another machine, as it were: an empty place for both.
|
||||
newEtc, newDB := filepath.Join(tmp, "new-etc"), filepath.Join(tmp, "new", "llmgw.db")
|
||||
var out bytes.Buffer
|
||||
rs := restore{archive: archive, identities: []string{key}, etcDir: newEtc, storeAddr: newDB,
|
||||
dryRun: true, out: &out, now: time.Now}
|
||||
if err := rs.run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(newEtc); err == nil || !strings.Contains(out.String(), "dry run") {
|
||||
t.Fatalf("a dry run changed something:\n%s", out.String())
|
||||
}
|
||||
out.Reset()
|
||||
rs.dryRun = false
|
||||
if err := rs.run(t.Context()); err != nil {
|
||||
t.Fatalf("%v\n%s", err, out.String())
|
||||
}
|
||||
if b, _ := os.ReadFile(filepath.Join(newEtc, "secrets", "admin-token")); len(b) != 40 {
|
||||
t.Error("the configuration did not come back")
|
||||
}
|
||||
if fi, err := os.Stat(filepath.Join(newEtc, "secrets", "admin-token")); err != nil || fi.Mode().Perm() != 0o600 {
|
||||
t.Errorf("modes: %v %v", fi.Mode(), err)
|
||||
}
|
||||
db, err = store.Open(t.Context(), newDB)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if orgs, _ := db.AllOrgs(t.Context()); len(orgs) != 1 || orgs[0].Name != "acme" {
|
||||
t.Errorf("the store did not come back: %+v", orgs)
|
||||
}
|
||||
|
||||
// Again over what is there: the current one is kept aside, not lost.
|
||||
out.Reset()
|
||||
if err := rs.run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
matches, _ := filepath.Glob(newEtc + ".before-restore-*")
|
||||
if len(matches) != 1 {
|
||||
t.Errorf("the configuration replaced was not kept: %v", matches)
|
||||
}
|
||||
|
||||
// Without the key it is said, not guessed.
|
||||
rs.identities = nil
|
||||
if err := rs.run(t.Context()); err == nil || !strings.Contains(err.Error(), "--identity") {
|
||||
t.Errorf("%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewerArchiveIsRefused(t *testing.T) {
|
||||
if !newer("v0.12.0", "v0.11.1-106-gabc") || newer("v0.11.1", "v0.11.1-5-g1") || newer("dev", "v1.0.0") {
|
||||
t.Error("version comparison")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnArchiveIsCopiedAndTheOldestPruned(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
dest := filepath.Join(tmp, "share")
|
||||
for i := range 4 {
|
||||
p := filepath.Join(tmp, "llmgw-backup-h-2026100"+string(rune('1'+i))+"T000000Z.tar.gz.age")
|
||||
_ = os.WriteFile(p, []byte("x"), 0o600)
|
||||
_ = os.Chtimes(p, time.Now().Add(time.Duration(i)*time.Hour), time.Now().Add(time.Duration(i)*time.Hour))
|
||||
if err := copyToDir(p, dest, 2); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Chtimes(filepath.Join(dest, filepath.Base(p)), time.Now().Add(time.Duration(i)*time.Hour), time.Now().Add(time.Duration(i)*time.Hour))
|
||||
}
|
||||
left, _ := filepath.Glob(filepath.Join(dest, "llmgw-backup-*"))
|
||||
if len(left) != 2 {
|
||||
t.Errorf("kept %v", left)
|
||||
}
|
||||
c := config.Default()
|
||||
c.BackupCopyTo = []string{"/mnt/x"}
|
||||
if err := c.Validate(); err == nil || !strings.Contains(err.Error(), "backup_recipients") {
|
||||
t.Errorf("an unencrypted copy off the machine was allowed: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -576,7 +576,7 @@ func newServerCmd() *cobra.Command {
|
||||
f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
|
||||
|
||||
cmd.AddCommand(newServerInitCmd(), newServerAddCmd())
|
||||
cmd.AddCommand(newBackupCmd())
|
||||
cmd.AddCommand(newBackupCmd(), newRestoreCmd())
|
||||
return cmd
|
||||
}
|
||||
|
||||
|
||||
@@ -148,6 +148,16 @@ type Server struct {
|
||||
// Empty writes archives unencrypted, readable by root only.
|
||||
BackupRecipients []string `koanf:"backup_recipients"`
|
||||
|
||||
// BackupCopyTo is where each archive is copied once written, so it
|
||||
// survives this machine: a directory (a mounted share, another disk) or
|
||||
// ssh://user@host[:port]/path, reached with BackupCopyKey and checked
|
||||
// against /etc/llmgw/backup_known_hosts, which learns a host the first
|
||||
// time. The same number of archives is kept there as here. Refused
|
||||
// without backup_recipients: an archive leaves the machine encrypted.
|
||||
BackupCopyTo []string `koanf:"backup_copy_to"`
|
||||
// BackupCopyKey is the SSH private key for ssh:// copies.
|
||||
BackupCopyKey string `koanf:"backup_copy_key"`
|
||||
|
||||
// Releases is where `llmgw ctl upgrade` fetches release binaries from,
|
||||
// for this machine and — through their agents — every node. Empty means
|
||||
// the project's own releases.
|
||||
@@ -686,6 +696,20 @@ func (s Server) Validate() error {
|
||||
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
|
||||
return err
|
||||
}
|
||||
for i, t := range s.BackupCopyTo {
|
||||
switch {
|
||||
case strings.HasPrefix(t, "ssh://"):
|
||||
if s.BackupCopyKey == "" {
|
||||
return fmt.Errorf("backup_copy_to[%d] is over ssh: give backup_copy_key", i)
|
||||
}
|
||||
case !strings.HasPrefix(t, "/"):
|
||||
return fmt.Errorf("backup_copy_to[%d] %q: an absolute directory or ssh://user@host/path", i, t)
|
||||
}
|
||||
}
|
||||
if len(s.BackupCopyTo) > 0 && len(s.BackupRecipients) == 0 {
|
||||
return fmt.Errorf("backup_copy_to needs backup_recipients: an archive holds the certificate authority's key " +
|
||||
"and every secret in the store, and leaves this machine only encrypted")
|
||||
}
|
||||
if _, err := s.FirewallAllow(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -303,6 +303,8 @@ var Reference = []Block{
|
||||
{Key: "notify", Type: "notify.Config", Doc: "Notify sends what needs a person — a model parked, a node lost, a\nbudget spent, new weights published — to webhooks, signed, without a\nPrometheus and Alertmanager in between. Empty sends nothing.", Nested: "notify.Config"},
|
||||
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
|
||||
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
|
||||
{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},
|
||||
{Key: "backup_copy_key", Type: "string", Doc: "BackupCopyKey is the SSH private key for ssh:// copies.", Nested: ""},
|
||||
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
|
||||
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
|
||||
{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
|
||||
|
||||
Reference in new issue
Block a user