Encrypt backups to backup_recipients with age
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
3b05ec0538
commit
12efb7d085
10 files changed
+204
-12
No files matched your search
@@ -159,6 +159,11 @@ APIs change between minors without a deprecation period.
|
||||
where every user on the node could read it: it reaches the engine in its
|
||||
environment, and a short launcher hands it to SGLang inside the process.
|
||||
Both take effect as engines restart under an upgraded agent.
|
||||
- **Backups can be encrypted.** `backup_recipients` in server.yaml (age
|
||||
recipients or SSH public keys) has `llmgw server backup` encrypt each
|
||||
archive to them with age, as `.tar.gz.age`: an archive holds the CA's key
|
||||
and every secret in the store, and is meant to be copied off the machine.
|
||||
Open one with `age -d -i <key>`.
|
||||
|
||||
### Everything that can be changed on a running cluster, from the console
|
||||
|
||||
|
||||
@@ -1187,6 +1187,20 @@ The control plane publishes when the newest archive was written as
|
||||
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
|
||||
it is more than 26 hours old — including when there has never been one.
|
||||
|
||||
**Encrypt them.** Name who may open them in `/etc/llmgw/server.yaml`, and every
|
||||
archive is encrypted to those keys with [age](https://age-encryption.org) and
|
||||
named `.tar.gz.age`:
|
||||
|
||||
```yaml
|
||||
backup_recipients:
|
||||
- age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p # age-keygen's public key
|
||||
- ssh-ed25519 AAAAC3Nza... ops@example # or an SSH public key
|
||||
```
|
||||
|
||||
Keep the matching private keys **off** this machine — an archive whose key
|
||||
sits beside it is not protected by it. Encrypted, a copy is safe wherever it
|
||||
lands.
|
||||
|
||||
**Copy the archives off the machine.** They sit on the disk they back up and do
|
||||
not survive losing it. Anything that copies files will do — rsync from the
|
||||
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
|
||||
@@ -1206,6 +1220,8 @@ them:
|
||||
```
|
||||
sudo systemctl stop llmgw-server
|
||||
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
|
||||
# an encrypted one: open it with the private key, from wherever it is kept
|
||||
age -d -i key.txt llmgw-backup-<host>-<time>.tar.gz.age | sudo tar -xz -C /tmp/restore
|
||||
cat /tmp/restore/MANIFEST # version and time it was taken
|
||||
|
||||
# /etc/llmgw: configuration, secrets, certificates and the CA
|
||||
|
||||
@@ -185,6 +185,7 @@ rather than repeated.
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
| `control` | `Control` | [Server](#server) |
|
||||
@@ -1492,6 +1493,17 @@ BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
Empty means /var/backups/llmgw-server.
|
||||
|
||||
### `backup_recipients`
|
||||
|
||||
`[]string`
|
||||
|
||||
BackupRecipients encrypt every archive `llmgw server backup` writes, to
|
||||
whoever holds one of their keys: age recipients (age1…) or SSH public
|
||||
keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
|
||||
the certificate authority's key and every secret in the store, and is
|
||||
copied off the machine; encrypted, the copy is safe wherever it lands.
|
||||
Empty writes archives unencrypted, readable by root only.
|
||||
|
||||
### `releases`
|
||||
|
||||
`string`
|
||||
|
||||
@@ -4,10 +4,12 @@ go 1.26.6
|
||||
|
||||
require (
|
||||
connectrpc.com/connect v1.21.0
|
||||
filippo.io/age v1.3.2
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0
|
||||
github.com/cespare/xxhash/v2 v2.3.0
|
||||
github.com/charmbracelet/bubbletea v1.3.10
|
||||
github.com/charmbracelet/lipgloss v1.1.0
|
||||
github.com/charmbracelet/x/ansi v0.11.6
|
||||
github.com/jackc/pgx/v5 v5.11.0
|
||||
github.com/knadh/koanf/parsers/yaml v1.1.1
|
||||
github.com/knadh/koanf/providers/env/v2 v2.0.1
|
||||
@@ -32,11 +34,12 @@ require (
|
||||
)
|
||||
|
||||
require (
|
||||
filippo.io/edwards25519 v1.2.0 // indirect
|
||||
filippo.io/hpke v0.4.0 // indirect
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||
github.com/charmbracelet/colorprofile v0.4.1 // indirect
|
||||
github.com/charmbracelet/x/ansi v0.11.6 // indirect
|
||||
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
|
||||
github.com/charmbracelet/x/term v0.2.2 // indirect
|
||||
github.com/clipperhouse/displaywidth v0.9.0 // indirect
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d h1:Blprhc2SbChNZtWcU+BLTM4YdoqYAS9V7cJgOwJKyAs=
|
||||
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
|
||||
connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
|
||||
connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
|
||||
filippo.io/age v1.3.2 h1:r6RSZLFSMm6rzKepZ7ZAYkKCu14f3/Me8c7uKYh7C8c=
|
||||
filippo.io/age v1.3.2/go.mod h1:TH/Yr2sSRhCKbaH4XPxpUV0Us8Gv6txYUpiZQWz8Evk=
|
||||
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
||||
|
||||
+37
-9
@@ -15,6 +15,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
agecrypt "filippo.io/age"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
@@ -44,9 +45,11 @@ func newBackupCmd() *cobra.Command {
|
||||
"people, quotas, usage, the models added at runtime, the secret engine\n" +
|
||||
"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
|
||||
"store's address and the certificate authority's key.\n\n" +
|
||||
"The archive holds that key, so it is written readable by root only. It\n" +
|
||||
"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
|
||||
"not survive the loss of this machine.\n\n" +
|
||||
"The archive holds that key, so it is written readable by root only, and\n" +
|
||||
"encrypted to backup_recipients in server.yaml when they are set (age\n" +
|
||||
"recipients or SSH public keys; open it with `age -d -i <key>`). It is on\n" +
|
||||
"the same disk as what it backs up: copy it elsewhere, or it does not\n" +
|
||||
"survive the loss of this machine.\n\n" +
|
||||
"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
|
||||
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
|
||||
"BackupTooOld fires when it is more than a day old.",
|
||||
@@ -55,7 +58,11 @@ func newBackupCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
|
||||
to, err := config.ParseBackupRecipients(cfg.BackupRecipients)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir, to)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -87,7 +94,9 @@ const backupPrefix = "llmgw-backup-"
|
||||
// writeBackup writes one archive and returns its path. Written beside its
|
||||
// final name and renamed, so a backup interrupted halfway is never the newest
|
||||
// one the control plane reports.
|
||||
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
|
||||
//
|
||||
// With recipients, the archive is encrypted to them, and named .tar.gz.age.
|
||||
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string, recipients []agecrypt.Recipient) (string, error) {
|
||||
// Listable by anyone, so the control plane — not root — can see when the
|
||||
// newest archive was written. The archives themselves are root's alone.
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
@@ -95,6 +104,9 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
|
||||
if len(recipients) > 0 {
|
||||
name += ".age"
|
||||
}
|
||||
final := filepath.Join(dir, name)
|
||||
part := final + ".part"
|
||||
|
||||
@@ -108,7 +120,13 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
_ = os.Remove(part)
|
||||
}
|
||||
}()
|
||||
gz := gzip.NewWriter(f)
|
||||
var sealed io.WriteCloser = nopCloser{f}
|
||||
if len(recipients) > 0 {
|
||||
if sealed, err = agecrypt.Encrypt(f, recipients...); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
gz := gzip.NewWriter(sealed)
|
||||
tw := tar.NewWriter(gz)
|
||||
|
||||
manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
|
||||
@@ -124,7 +142,7 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
|
||||
return "", fmt.Errorf("%s: %w", etcDir, err)
|
||||
}
|
||||
if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
|
||||
if err := errors.Join(tw.Close(), gz.Close(), sealed.Close(), f.Sync(), f.Close()); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.Rename(part, final); err != nil {
|
||||
@@ -134,6 +152,16 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
return final, nil
|
||||
}
|
||||
|
||||
type nopCloser struct{ io.Writer }
|
||||
|
||||
func (nopCloser) Close() error { return nil }
|
||||
|
||||
// isBackup reports whether a file name is an archive this command wrote.
|
||||
func isBackup(name string) bool {
|
||||
return strings.HasPrefix(name, backupPrefix) &&
|
||||
(strings.HasSuffix(name, ".tar.gz") || strings.HasSuffix(name, ".tar.gz.age"))
|
||||
}
|
||||
|
||||
// addStore dumps the store into the archive.
|
||||
func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
|
||||
addr, err := store.ResolveAddress(addr)
|
||||
@@ -280,7 +308,7 @@ func pruneBackups(dir string, keep int) (int, error) {
|
||||
}
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
|
||||
if n := e.Name(); isBackup(n) {
|
||||
names = append(names, n)
|
||||
}
|
||||
}
|
||||
@@ -306,7 +334,7 @@ func NewestBackup(dir string) time.Time {
|
||||
}
|
||||
var newest time.Time
|
||||
for _, e := range entries {
|
||||
if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
|
||||
if n := e.Name(); !isBackup(n) {
|
||||
continue
|
||||
}
|
||||
if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
|
||||
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"crypto/ed25519"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -10,9 +11,84 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
agecrypt "filippo.io/age"
|
||||
"golang.org/x/crypto/ssh"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestABackupIsEncryptedToItsRecipients(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
etc := filepath.Join(tmp, "etc")
|
||||
_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
|
||||
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("THE-CA-KEY"), 0o600)
|
||||
|
||||
id, err := agecrypt.GenerateX25519Identity()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
to, err := config.ParseBackupRecipients([]string{id.Recipient().String()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out := filepath.Join(tmp, "backups")
|
||||
path, err := writeBackup(t.Context(), "", etc, out, to)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasSuffix(path, ".tar.gz.age") {
|
||||
t.Errorf("named %s", filepath.Base(path))
|
||||
}
|
||||
raw, _ := os.ReadFile(path)
|
||||
if strings.Contains(string(raw), "THE-CA-KEY") {
|
||||
t.Fatal("the CA key is in the archive in the clear")
|
||||
}
|
||||
|
||||
f, _ := os.Open(path)
|
||||
defer f.Close()
|
||||
plain, err := agecrypt.Decrypt(f, id)
|
||||
if err != nil {
|
||||
t.Fatalf("the recipient cannot open it: %v", err)
|
||||
}
|
||||
gz, err := gzip.NewReader(plain)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
found := false
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
found = found || h.Name == "etc/llmgw/pki/ca-key.pem"
|
||||
}
|
||||
if !found {
|
||||
t.Error("the opened archive lacks the CA key")
|
||||
}
|
||||
// Seen and pruned like any other.
|
||||
if NewestBackup(out).IsZero() {
|
||||
t.Error("the control plane would not see an encrypted archive")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRecipientsMustBeKeys(t *testing.T) {
|
||||
if _, err := config.ParseBackupRecipients([]string{"not a key"}); err == nil {
|
||||
t.Error("nonsense was taken as a recipient")
|
||||
}
|
||||
pub, _, _ := ed25519.GenerateKey(nil)
|
||||
sp, err := ssh.NewPublicKey(pub)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := config.ParseBackupRecipients([]string{
|
||||
strings.TrimSpace(string(ssh.MarshalAuthorizedKey(sp))) + " ops@example",
|
||||
}); err != nil {
|
||||
t.Errorf("an SSH key was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
dbPath := filepath.Join(tmp, "store.db")
|
||||
@@ -32,7 +108,7 @@ func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
|
||||
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
|
||||
|
||||
out := filepath.Join(tmp, "backups")
|
||||
path, err := writeBackup(t.Context(), dbPath, etc, out)
|
||||
path, err := writeBackup(t.Context(), dbPath, etc, out, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -129,7 +205,7 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
|
||||
}
|
||||
_ = db.Close()
|
||||
out := filepath.Join(tmp, "backups")
|
||||
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
|
||||
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := os.ReadDir(out)
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"filippo.io/age"
|
||||
"filippo.io/age/agessh"
|
||||
)
|
||||
|
||||
// ParseBackupRecipients reads backup_recipients: age recipients, or SSH
|
||||
// public keys.
|
||||
func ParseBackupRecipients(list []string) ([]age.Recipient, error) {
|
||||
var out []age.Recipient
|
||||
for i, s := range list {
|
||||
s = strings.TrimSpace(s)
|
||||
if strings.HasPrefix(s, "ssh-") {
|
||||
r, err := agessh.ParseRecipient(s)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("backup_recipients[%d]: %w", i, err)
|
||||
}
|
||||
out = append(out, r)
|
||||
continue
|
||||
}
|
||||
rs, err := age.ParseRecipients(strings.NewReader(s))
|
||||
if err != nil || len(rs) != 1 {
|
||||
return nil, fmt.Errorf("backup_recipients[%d] is not an age recipient (age1…) or an SSH public key", i)
|
||||
}
|
||||
out = append(out, rs[0])
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -121,6 +121,14 @@ type Server struct {
|
||||
// Empty means /var/backups/llmgw-server.
|
||||
BackupDir string `koanf:"backup_dir"`
|
||||
|
||||
// BackupRecipients encrypt every archive `llmgw server backup` writes, to
|
||||
// whoever holds one of their keys: age recipients (age1…) or SSH public
|
||||
// keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
|
||||
// the certificate authority's key and every secret in the store, and is
|
||||
// copied off the machine; encrypted, the copy is safe wherever it lands.
|
||||
// Empty writes archives unencrypted, readable by root only.
|
||||
BackupRecipients []string `koanf:"backup_recipients"`
|
||||
|
||||
// Releases is where `llmgw ctl upgrade` fetches release binaries from,
|
||||
// for this machine and — through their agents — every node. Empty means
|
||||
// the project's own releases.
|
||||
@@ -619,6 +627,9 @@ func (s Server) Validate() error {
|
||||
if s.Listen == "" {
|
||||
return fmt.Errorf("listen is empty")
|
||||
}
|
||||
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.AgentTLS.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -225,6 +225,7 @@ var Reference = []Block{
|
||||
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
|
||||
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
|
||||
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
|
||||
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
|
||||
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
|
||||
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
|
||||
{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
|
||||
|
||||
Reference in new issue
Block a user