diff --git a/BACKLOG.md b/BACKLOG.md index 6868f70..0c75b41 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -490,7 +490,14 @@ portal is not done — the page shows; the console and ctl change. **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max, pre-warm, queue-depth scaling in `catalog.Scaling`, `control/reconciler/demand.go`, `headroom.go`. -15. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the +15. **Done (branch `features-c2`):** `env:` and `vault:` references + wherever a secret file is named (store address, api_key_file, OIDC, + SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key` + sealing cluster_secrets (Vault transit or a local key). Left: the + certificate authority's key is still a file in /etc/llmgw (signing in + Vault's PKI engine instead is its own change); rotating the master key; + `backup_copy_key` and `admin_token_file` are files only. + **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`); envelope-encrypt them in backups (with A8 backups). 16. **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage diff --git a/CHANGELOG.md b/CHANGELOG.md index a978ad3..d814001 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -71,6 +71,16 @@ APIs change between minors without a deprecation period. - **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`) and **E** on the console's Usage view. +### Secrets in Vault + +- **Secrets from Vault or the environment.** Every setting naming a secret's + file also takes `env:VARIABLE` or `vault:mount/path#field` (HashiCorp + Vault or OpenBao, KV v1 or v2; a token, AppRole, or VAULT_TOKEN). +- **The store's own secrets sealed** under `secrets.master_key`: Vault's + transit engine (`vault-transit:KEY`), or a 32-byte key read from a + reference. A store, or a backup of it, then does not hold what engine keys + are derived from. Values from before are sealed as they are next read. + ### Scaling by the clock - **`scaling.schedule`**: windows of the week with their own `min` and diff --git a/docs/data.md b/docs/data.md index 1b6d427..1605c2f 100644 --- a/docs/data.md +++ b/docs/data.md @@ -44,6 +44,10 @@ never stored — so a leaked database hands over no working credential, and neither llmgw nor an administrator can recover a key somebody lost. A prefix is kept so a key can be identified in a listing without revealing it. +**The store's own secrets**, like the one engine keys are derived from, plain +unless `secrets.master_key` seals them — with Vault's transit engine, so that +neither the store nor its backups hold them. + **People and tenancy**: accounts, the organisation/team/project tree, and the limits attached to it. Passwords are hashed. diff --git a/docs/install.md b/docs/install.md index ef4da74..c5923ce 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1394,6 +1394,43 @@ and point it at `https:///scim/v2` with that token as the bearer. join and leave them. Membership in a group is removed when someone leaves it. +## Secrets in Vault + +Every setting that names a file holding a secret — the store's `@address`, +`api_key_file`, `auth.oidc.client_secret_file`, `auth.scim.token_file`, a +webhook's `secret_file` — also takes `env:VARIABLE` or a field of a secret in +HashiCorp Vault or OpenBao: + +```yaml +auth: + store: "@vault:secret/llmgw/store#dsn" +secrets: + vault: + addr: https://vault.internal:8200 + ca_file: /etc/llmgw/vault-ca.pem + role_id_file: /etc/llmgw/secrets/vault-role-id # AppRole; or token_file, + secret_id_file: /etc/llmgw/secrets/vault-secret-id # or VAULT_TOKEN + master_key: vault-transit:llmgw +``` + +These are read when the gateway starts (an external model's key, each time +the catalog changes): a secret rotated in Vault is taken up at the next +restart. + +**The store's own secrets** — what every engine's key is derived from — are +kept in the store, and so in its backups. `secrets.master_key` seals them: + +- `vault-transit:KEY` seals with Vault's transit engine. The key never leaves + Vault, so neither the store nor any backup of it opens without Vault. +- Anything else is a reference to 32 random bytes (`openssl rand -hex 32`), + used here: better than nothing, but a key in `/etc/llmgw` is in every backup + beside what it seals. Keep it in Vault or the environment. + +Values kept plain from before are sealed the first time they are read after +the key is set. The key cannot be changed once set without losing them: a new +one cannot open what the old one sealed (the engines are then restarted with +new keys: `llmgw ctl restart --all`). + ## The firewall `server init`, `node add`, `server add` and `mirror init` each keep that diff --git a/docs/reference.md b/docs/reference.md index 4610f68..67325c4 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -230,6 +230,8 @@ rather than repeated. | `target_inflight` | `float64` | [Scoring](#scoring) | | `kv_ceiling` | `float64` | [Scoring](#scoring) | | `session_load_factor` | `float64` | [Scoring](#scoring) | +| `vault` | `secrets.Vault` | [Secrets](#secrets) | +| `master_key` | `string` | [Secrets](#secrets) | | `listen` | `string` | [Server](#server) | | `admin_listen` | `string` | [Server](#server) | | `metrics_listen` | `string` | [Server](#server) | @@ -250,6 +252,7 @@ rather than repeated. | `engine_overrides` | `[]EngineOverride` | [Server](#server) | | `backup_dir` | `string` | [Server](#server) | | `guard` | `guard.Config` | [Server](#server) | +| `secrets` | `Secrets` | [Server](#server) | | `notify` | `notify.Config` | [Server](#server) | | `firewall_networks` | `[]string` | [Server](#server) | | `backup_recipients` | `[]string` | [Server](#server) | @@ -279,6 +282,12 @@ rather than repeated. | `bin` | `string` | [VLLMConfig](#vllmconfig) | | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) | | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) | +| `addr` | `string` | [Vault](#vault) | +| `token_file` | `string` | [Vault](#vault) | +| `role_id_file` | `string` | [Vault](#vault) | +| `secret_id_file` | `string` | [Vault](#vault) | +| `ca_file` | `string` | [Vault](#vault) | +| `namespace` | `string` | [Vault](#vault) | | `url` | `string` | [Webhook](#webhook) | | `format` | `string` | [Webhook](#webhook) | | `events` | `[]string` | [Webhook](#webhook) | @@ -1859,6 +1868,27 @@ arrive at nine are the reason for the window. `float64` +## Secrets + +### `vault` + +`secrets.Vault` + +Vault is how vault: references and vault-transit: are reached. + +Takes a [Vault](#vault) block. + +### `master_key` + +`string` + +MasterKey seals the secrets the store keeps for itself — what every +engine's key is derived from — so a copy of the store, or of a backup, +does not hold them: vault-transit:KEY seals with Vault's transit engine, +which never lets the key out; otherwise a reference to 32 random bytes +(openssl rand -hex 32), which then must be kept as safely as the store. +Values from before it was set are sealed the first time they are read. + ## Server ### `listen` @@ -2048,6 +2078,15 @@ guard.default, and to parts of the tenancy tree by attaching them Takes a [guard.Config](#guardconfig) block. +### `secrets` + +`Secrets` + +Secrets is where secrets kept outside files are read from, and the key +the store's own secrets are sealed under. + +Takes a [Secrets](#secrets) block. + ### `notify` `notify.Config` @@ -2263,6 +2302,42 @@ node and has not been checked against a real cluster. RayBin is used when LaunchMode is "ray". +## Vault + +### `addr` + +`string` + +Addr is Vault's address, like https://vault.internal:8200; empty +takes VAULT_ADDR. + +### `token_file` + +`string` + +TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with +AppRole when RoleIDFile and SecretIDFile are set. + +### `role_id_file` + +`string` + +### `secret_id_file` + +`string` + +### `ca_file` + +`string` + +CAFile verifies Vault's certificate; empty uses the system's. + +### `namespace` + +`string` + +Namespace is Vault Enterprise's or OpenBao's namespace. + ## Webhook ### `url` diff --git a/internal/cli/server.go b/internal/cli/server.go index e6ef2f5..d155582 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -48,6 +48,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/obs" "gitea.mynodes.xyz/ilker/llmgw/internal/promq" "gitea.mynodes.xyz/ilker/llmgw/internal/router" + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store" "gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf" "gitea.mynodes.xyz/ilker/llmgw/internal/version" @@ -120,6 +121,15 @@ func newServerCmd() *cobra.Command { if err != nil { return err } + // The store's own secrets sealed under a key kept outside it, + // before anything reads them. + if st := rt.Store(); st != nil && cfg.Secrets.MasterKey != "" { + sealer, err := secrets.NewSealer(cmd.Context(), cfg.Secrets.MasterKey) + if err != nil { + return err + } + st.SetSealer(sealer) + } // The longest any key may live, where every way of making one // reads it: the store. if st := rt.Store(); st != nil { diff --git a/internal/config/config.go b/internal/config/config.go index 8b2612e..23c0f55 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -20,6 +20,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/guard" "gitea.mynodes.xyz/ilker/llmgw/internal/notify" + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" "gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf" ) @@ -129,6 +130,10 @@ type Server struct { // (llmgw ctl guard attach). Empty checks nothing. Guard guard.Config `koanf:"guard"` + // Secrets is where secrets kept outside files are read from, and the key + // the store's own secrets are sealed under. + Secrets Secrets `koanf:"secrets"` + // Notify sends what needs a person — a model parked, a node lost, a // budget spent, new weights published — to webhooks, signed, without a // Prometheus and Alertmanager in between. Empty sends nothing. @@ -571,6 +576,24 @@ type Endpoint struct { Priority int `koanf:"priority"` } +// Secrets is where secrets are read from besides files, and how the +// store's own are sealed. +// +// Wherever a setting names a file holding a secret — the store's @address, +// api_key_file, client_secret_file, token_file, secret_file — it may name +// env:VARIABLE or vault:mount/path#field instead. +type Secrets struct { + // Vault is how vault: references and vault-transit: are reached. + Vault secrets.Vault `koanf:"vault"` + // MasterKey seals the secrets the store keeps for itself — what every + // engine's key is derived from — so a copy of the store, or of a backup, + // does not hold them: vault-transit:KEY seals with Vault's transit engine, + // which never lets the key out; otherwise a reference to 32 random bytes + // (openssl rand -hex 32), which then must be kept as safely as the store. + // Values from before it was set are sealed the first time they are read. + MasterKey string `koanf:"master_key"` +} + // Batch is how the Files and Batch APIs are bounded. type Batch struct { // MaxFileBytes bounds one uploaded file; 100 MB unless set. @@ -679,6 +702,10 @@ func Load(path string) (Server, error) { if err := k.UnmarshalWithConf("", &cfg, koanf.UnmarshalConf{Tag: "koanf"}); err != nil { return cfg, fmt.Errorf("decoding configuration: %w", err) } + // So every secret reference in it, from here on, can be read. + if err := secrets.Configure(cfg.Secrets.Vault); err != nil { + return cfg, err + } return cfg, cfg.Validate() } diff --git a/internal/control/control.go b/internal/control/control.go index 72d663b..b75bfa2 100644 --- a/internal/control/control.go +++ b/internal/control/control.go @@ -9,7 +9,6 @@ import ( "encoding/hex" "fmt" "log/slog" - "os" "strings" "sync" "time" @@ -19,6 +18,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory" "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" "gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint" + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" ) // EndpointSink receives the endpoint list. The router implements it. @@ -356,7 +356,7 @@ func (c *Controller) externalEndpoints() []endpoint.Info { } key := "" if f := m.External.APIKeyFile; f != "" { - b, err := os.ReadFile(f) //nolint:gosec // the operator's own path + b, err := secrets.Read(context.Background(), f) if err != nil { c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err) continue diff --git a/internal/docgen/reference_gen.go b/internal/docgen/reference_gen.go index eb4ad02..4503a72 100644 --- a/internal/docgen/reference_gen.go +++ b/internal/docgen/reference_gen.go @@ -289,6 +289,10 @@ var Reference = []Block{ {Key: "kv_ceiling", Type: "float64", Doc: "", Nested: ""}, {Key: "session_load_factor", Type: "float64", Doc: "", Nested: ""}, }}, + {Package: "config", Name: "Secrets", Title: "Secrets", Doc: "", Fields: []Field{ + {Key: "vault", Type: "secrets.Vault", Doc: "Vault is how vault: references and vault-transit: are reached.", Nested: "secrets.Vault"}, + {Key: "master_key", Type: "string", Doc: "MasterKey seals the secrets the store keeps for itself — what every\nengine's key is derived from — so a copy of the store, or of a backup,\ndoes not hold them: vault-transit:KEY seals with Vault's transit engine,\nwhich never lets the key out; otherwise a reference to 32 random bytes\n(openssl rand -hex 32), which then must be kept as safely as the store.\nValues from before it was set are sealed the first time they are read.", Nested: ""}, + }}, {Package: "config", Name: "Server", Title: "Server", Doc: "", Fields: []Field{ {Key: "listen", Type: "string", Doc: "Listen is the client-facing OpenAI API address.", Nested: ""}, {Key: "admin_listen", Type: "string", Doc: "AdminListen serves the dashboard, the management API, health and pprof.\n\nLoopback by default, and it should stay there: pprof can dump the process\nheap, and API keys are in that heap in plaintext. Every comment beside\nthis setting has always said not to expose it, while the default bound\nevery interface — a default that contradicts its own documentation is a\nfootgun, so the default now matches the advice.", Nested: ""}, @@ -310,6 +314,7 @@ var Reference = []Block{ {Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""}, {Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""}, {Key: "guard", Type: "guard.Config", Doc: "Guard is the library of guardrail policies: what requests and answers\nare checked for, and what is done. Applied to every request through\nguard.default, and to parts of the tenancy tree by attaching them\n(llmgw ctl guard attach). Empty checks nothing.", Nested: "guard.Config"}, + {Key: "secrets", Type: "Secrets", Doc: "Secrets is where secrets kept outside files are read from, and the key\nthe store's own secrets are sealed under.", Nested: "config.Secrets"}, {Key: "notify", Type: "notify.Config", Doc: "Notify sends what needs a person — a model parked, a node lost, a\nbudget spent, new weights published — to webhooks, signed, without a\nPrometheus and Alertmanager in between. Empty sends nothing.", Nested: "notify.Config"}, {Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""}, {Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""}, @@ -352,6 +357,14 @@ var Reference = []Block{ {Key: "launch_mode", Type: "string", Doc: "LaunchMode selects the multi-node mechanism: \"mp\", the default, or\n\"ray\".\n\nmp is what vLLM documents today — every node runs serve with its rank,\nthe workers headless — and needs nothing but vLLM. ray is the older way,\nkept for a vLLM too old for mp's multi-node flags; it needs Ray on every\nnode and has not been checked against a real cluster.", Nested: ""}, {Key: "ray_bin", Type: "string", Doc: "RayBin is used when LaunchMode is \"ray\".", Nested: ""}, }}, + {Package: "secrets", Name: "Vault", Title: "Vault", Doc: "", Fields: []Field{ + {Key: "addr", Type: "string", Doc: "Addr is Vault's address, like https://vault.internal:8200; empty\ntakes VAULT_ADDR.", Nested: ""}, + {Key: "token_file", Type: "string", Doc: "TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with\nAppRole when RoleIDFile and SecretIDFile are set.", Nested: ""}, + {Key: "role_id_file", Type: "string", Doc: "", Nested: ""}, + {Key: "secret_id_file", Type: "string", Doc: "", Nested: ""}, + {Key: "ca_file", Type: "string", Doc: "CAFile verifies Vault's certificate; empty uses the system's.", Nested: ""}, + {Key: "namespace", Type: "string", Doc: "Namespace is Vault Enterprise's or OpenBao's namespace.", Nested: ""}, + }}, {Package: "notify", Name: "Webhook", Title: "Webhook", Doc: "", Fields: []Field{ {Key: "url", Type: "string", Doc: "", Nested: ""}, {Key: "format", Type: "string", Doc: "Format is json (the event, with X-Llmgw-Signature when SecretFile is\nset) or slack ({\"text\": …}, which Slack, Mattermost and Rocket.Chat\nincoming webhooks take).", Nested: ""}, diff --git a/internal/docgen/render.go b/internal/docgen/render.go index 2f01358..89abf06 100644 --- a/internal/docgen/render.go +++ b/internal/docgen/render.go @@ -30,6 +30,7 @@ var Sources = []string{ "internal/guard/judge.go", "internal/config/oidc.go", "internal/notify/notify.go", + "internal/secrets/secrets.go", } // notReference are files with koanf tags that an operator never writes. diff --git a/internal/notify/notify.go b/internal/notify/notify.go index 0eca9b5..72c148b 100644 --- a/internal/notify/notify.go +++ b/internal/notify/notify.go @@ -27,6 +27,8 @@ import ( "strings" "sync" "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" ) // Event kinds. @@ -138,7 +140,7 @@ func New(ctx context.Context, c Config, log *slog.Logger) (*Notifier, error) { for _, w := range c.Webhooks { h := hook{Webhook: w} if w.SecretFile != "" { - b, err := os.ReadFile(w.SecretFile) + b, err := secrets.Read(ctx, w.SecretFile) if err != nil { return nil, fmt.Errorf("notify: %w", err) } diff --git a/internal/router/handlers.go b/internal/router/handlers.go index b313eaa..cb597d0 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -10,7 +10,6 @@ import ( "fmt" "io" "net/http" - "os" "slices" "strconv" "strings" @@ -25,6 +24,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/router/admission" "gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy" "gitea.mynodes.xyz/ilker/llmgw/internal/scim" + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" "gitea.mynodes.xyz/ilker/llmgw/internal/store" "gitea.mynodes.xyz/ilker/llmgw/internal/version" "gitea.mynodes.xyz/ilker/llmgw/internal/web" @@ -211,7 +211,7 @@ func (r *Router) Handler() http.Handler { } } if r.scim.TokenFile != "" { - if b, err := os.ReadFile(r.scim.TokenFile); err != nil || strings.TrimSpace(string(b)) == "" { + if b, err := secrets.Read(context.Background(), r.scim.TokenFile); err != nil || strings.TrimSpace(string(b)) == "" { r.log.Error("provisioning (SCIM) is off: its token cannot be read", "file", r.scim.TokenFile, "err", err) } else { scim.New(r.store, strings.TrimSpace(string(b)), diff --git a/internal/secrets/seal.go b/internal/secrets/seal.go new file mode 100644 index 0000000..0520bb8 --- /dev/null +++ b/internal/secrets/seal.go @@ -0,0 +1,136 @@ +package secrets + +import ( + "context" + "crypto/aes" + "crypto/cipher" + "crypto/rand" + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "errors" + "fmt" + "io" + "net/http" + "strings" + + "golang.org/x/crypto/hkdf" +) + +// Sealer encrypts the store's own secrets — what engine keys are derived +// from — under a key the store does not hold, so a copy of the store or of +// its backup does not hold them either. +type Sealer interface { + Seal(ctx context.Context, plain []byte) (string, error) + Open(ctx context.Context, sealed string) ([]byte, error) +} + +// Sealed values say how they were sealed, so a plain value from before is +// told apart, and read. +const ( + localPrefix = "llmgw:local:v1:" + transitPrefix = "llmgw:transit:v1:" +) + +// IsSealed says whether a stored value is sealed. +func IsSealed(v string) bool { + return strings.HasPrefix(v, localPrefix) || strings.HasPrefix(v, transitPrefix) +} + +// NewSealer is the sealer secrets.master_key names: vault-transit:KEY seals +// with Vault's transit engine, which never lets the key out; anything else +// is a reference to 32 bytes (hex, base64 or raw) used here. +func NewSealer(ctx context.Context, ref string) (Sealer, error) { + if key, ok := strings.CutPrefix(ref, "vault-transit:"); ok { + mount, name := "transit", key + if m, n, ok := strings.Cut(key, "/"); ok { + mount, name = m, n + } + return transit{mount: mount, key: name}, nil + } + raw, err := Read(ctx, ref) + if err != nil { + return nil, fmt.Errorf("secrets.master_key: %w", err) + } + key := decodeKey(raw) + if len(key) < 32 { + return nil, errors.New("secrets.master_key is shorter than 32 bytes: make one with `openssl rand -hex 32`") + } + k := make([]byte, 32) + if _, err := io.ReadFull(hkdf.New(sha256.New, key, nil, []byte("llmgw cluster secrets v1")), k); err != nil { + return nil, err + } + block, err := aes.NewCipher(k) + if err != nil { + return nil, err + } + gcm, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + return local{gcm: gcm}, nil +} + +func decodeKey(b []byte) []byte { + s := strings.TrimSpace(string(b)) + if k, err := hex.DecodeString(s); err == nil { + return k + } + if k, err := base64.StdEncoding.DecodeString(s); err == nil { + return k + } + return []byte(s) +} + +type local struct{ gcm cipher.AEAD } + +func (l local) Seal(_ context.Context, plain []byte) (string, error) { + nonce := make([]byte, l.gcm.NonceSize()) + if _, err := rand.Read(nonce); err != nil { + return "", err + } + return localPrefix + base64.StdEncoding.EncodeToString(l.gcm.Seal(nonce, nonce, plain, nil)), nil +} + +func (l local) Open(_ context.Context, sealed string) ([]byte, error) { + b, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(sealed, localPrefix)) + if err != nil || !strings.HasPrefix(sealed, localPrefix) || len(b) < l.gcm.NonceSize() { + return nil, errors.New("not a value this key sealed") + } + out, err := l.gcm.Open(nil, b[:l.gcm.NonceSize()], b[l.gcm.NonceSize():], nil) + if err != nil { + return nil, errors.New("the master key does not open this value: it was sealed under another") + } + return out, nil +} + +type transit struct{ mount, key string } + +func (t transit) Seal(ctx context.Context, plain []byte) (string, error) { + var out struct { + Data struct { + Ciphertext string `json:"ciphertext"` + } `json:"data"` + } + if err := call(ctx, http.MethodPost, "/v1/"+t.mount+"/encrypt/"+t.key, + map[string]string{"plaintext": base64.StdEncoding.EncodeToString(plain)}, &out); err != nil { + return "", fmt.Errorf("sealing with Vault's transit key %s: %w", t.key, err) + } + return transitPrefix + out.Data.Ciphertext, nil +} + +func (t transit) Open(ctx context.Context, sealed string) ([]byte, error) { + if !strings.HasPrefix(sealed, transitPrefix) { + return nil, errors.New("not a value Vault's transit engine sealed") + } + var out struct { + Data struct { + Plaintext string `json:"plaintext"` + } `json:"data"` + } + if err := call(ctx, http.MethodPost, "/v1/"+t.mount+"/decrypt/"+t.key, + map[string]string{"ciphertext": strings.TrimPrefix(sealed, transitPrefix)}, &out); err != nil { + return nil, fmt.Errorf("opening with Vault's transit key %s: %w", t.key, err) + } + return base64.StdEncoding.DecodeString(out.Data.Plaintext) +} diff --git a/internal/secrets/secrets.go b/internal/secrets/secrets.go new file mode 100644 index 0000000..e62b427 --- /dev/null +++ b/internal/secrets/secrets.go @@ -0,0 +1,278 @@ +// Package secrets reads the secrets llmgw is configured with from where +// they are kept: a file, as always, an environment variable, or HashiCorp +// Vault (or OpenBao) — and seals the store's own secrets under a key that +// stays outside it. +// +// A reference is one of: +// +// /etc/llmgw/secrets/store-dsn a file, trimmed +// env:LLMGW_STORE_DSN an environment variable +// vault:secret/llmgw/store#dsn a field of a KV secret in Vault +// +// A Vault path's first segment is its KV v2 mount; secret/data/… is taken +// as written. KV v1 mounts work too. +package secrets + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "os" + "strings" + "sync" + "time" +) + +// Vault is how Vault is reached. +type Vault struct { + // Addr is Vault's address, like https://vault.internal:8200; empty + // takes VAULT_ADDR. + Addr string `koanf:"addr"` + // TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with + // AppRole when RoleIDFile and SecretIDFile are set. + TokenFile string `koanf:"token_file"` + RoleIDFile string `koanf:"role_id_file"` + SecretIDFile string `koanf:"secret_id_file"` + // CAFile verifies Vault's certificate; empty uses the system's. + CAFile string `koanf:"ca_file"` + // Namespace is Vault Enterprise's or OpenBao's namespace. + Namespace string `koanf:"namespace"` +} + +var ( + mu sync.Mutex + vault Vault + client = &http.Client{Timeout: 10 * time.Second} + token string + expiry time.Time + cache = map[string]cached{} +) + +type cached struct { + b []byte + at time.Time +} + +// cacheFor is how long a value read from Vault is used before it is read +// again: a secret rotated there reaches llmgw within it. +const cacheFor = time.Minute + +// Configure says how to reach Vault. Called once configuration is read. +func Configure(v Vault) error { + mu.Lock() + defer mu.Unlock() + vault, token, expiry, cache = v, "", time.Time{}, map[string]cached{} + if v.CAFile != "" { + pem, err := os.ReadFile(v.CAFile) + if err != nil { + return fmt.Errorf("secrets.vault.ca_file: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(pem) { + return fmt.Errorf("secrets.vault.ca_file %s holds no certificate", v.CAFile) + } + client = &http.Client{Timeout: 10 * time.Second, + Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}} + } + return nil +} + +// IsReference says whether s names a secret somewhere other than a file. +func IsReference(s string) bool { + return strings.HasPrefix(s, "env:") || strings.HasPrefix(s, "vault:") +} + +// Read reads a secret by reference, trimmed of surrounding space. +func Read(ctx context.Context, ref string) ([]byte, error) { + switch { + case strings.HasPrefix(ref, "env:"): + name := strings.TrimPrefix(ref, "env:") + v, ok := os.LookupEnv(name) + if !ok { + return nil, fmt.Errorf("the environment has no %s", name) + } + return []byte(strings.TrimSpace(v)), nil + case strings.HasPrefix(ref, "vault:"): + return readVault(ctx, strings.TrimPrefix(ref, "vault:")) + } + b, err := os.ReadFile(ref) //nolint:gosec // a path the operator configured + if err != nil { + return nil, err + } + return bytes.TrimSpace(b), nil +} + +// ReadString is Read as a string. +func ReadString(ctx context.Context, ref string) (string, error) { + b, err := Read(ctx, ref) + return string(b), err +} + +func readVault(ctx context.Context, ref string) ([]byte, error) { + path, field, ok := strings.Cut(ref, "#") + if !ok || path == "" || field == "" { + return nil, fmt.Errorf("vault:%s: write it as vault:mount/path#field", ref) + } + mu.Lock() + if c, ok := cache[ref]; ok && time.Since(c.at) < cacheFor { + mu.Unlock() + return c.b, nil + } + mu.Unlock() + + api := path + if mount, rest, ok := strings.Cut(path, "/"); ok && !strings.HasPrefix(rest, "data/") { + api = mount + "/data/" + rest + } + var out struct { + Data map[string]any `json:"data"` + } + if err := call(ctx, http.MethodGet, "/v1/"+api, nil, &out); err != nil { + // A KV v1 mount has no /data/ in its path. + if api != path && errors.Is(err, errNotFound) { + err = call(ctx, http.MethodGet, "/v1/"+path, nil, &out) + } + if err != nil { + return nil, fmt.Errorf("vault:%s: %w", ref, err) + } + } + data := out.Data + if inner, ok := data["data"].(map[string]any); ok { + data = inner // KV v2 + } + v, ok := data[field] + if !ok { + return nil, fmt.Errorf("vault:%s: the secret has no field %q", ref, field) + } + s, ok := v.(string) + if !ok { + return nil, fmt.Errorf("vault:%s: field %q is not a string", ref, field) + } + b := []byte(strings.TrimSpace(s)) + mu.Lock() + cache[ref] = cached{b: b, at: time.Now()} + mu.Unlock() + return b, nil +} + +var errNotFound = errors.New("not found") + +// call makes one request of Vault's API with the token, logging in first +// when there is none. +func call(ctx context.Context, method, path string, body, out any) error { + tok, addr, err := currentToken(ctx) + if err != nil { + return err + } + return do(ctx, addr, tok, method, path, body, out) +} + +func do(ctx context.Context, addr, tok, method, path string, body, out any) error { + var rd io.Reader + if body != nil { + b, _ := json.Marshal(body) + rd = bytes.NewReader(b) + } + req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(addr, "/")+path, rd) + if err != nil { + return err + } + if tok != "" { + req.Header.Set("X-Vault-Token", tok) + } + mu.Lock() + ns, c := vault.Namespace, client + mu.Unlock() + if ns != "" { + req.Header.Set("X-Vault-Namespace", ns) + } + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + resp, err := c.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + switch { + case resp.StatusCode == http.StatusNotFound: + return errNotFound + case resp.StatusCode >= 300: + var e struct { + Errors []string `json:"errors"` + } + _ = json.Unmarshal(b, &e) + return fmt.Errorf("vault answered %s: %s", resp.Status, strings.Join(e.Errors, "; ")) + } + if out != nil { + return json.Unmarshal(b, out) + } + return nil +} + +// currentToken is the token to send, logging in with AppRole when that is +// how it is configured and the last login has expired. +func currentToken(ctx context.Context) (string, string, error) { + mu.Lock() + v := vault + tok, exp := token, expiry + mu.Unlock() + addr := v.Addr + if addr == "" { + addr = os.Getenv("VAULT_ADDR") + } + if addr == "" { + return "", "", errors.New("no Vault to ask: set secrets.vault.addr or VAULT_ADDR") + } + if tok != "" && (exp.IsZero() || time.Now().Before(exp)) { + return tok, addr, nil + } + switch { + case v.TokenFile != "": + b, err := os.ReadFile(v.TokenFile) + if err != nil { + return "", "", fmt.Errorf("secrets.vault.token_file: %w", err) + } + tok, exp = strings.TrimSpace(string(b)), time.Time{} + case v.RoleIDFile != "" && v.SecretIDFile != "": + role, err := os.ReadFile(v.RoleIDFile) + if err != nil { + return "", "", err + } + sec, err := os.ReadFile(v.SecretIDFile) + if err != nil { + return "", "", err + } + var out struct { + Auth struct { + ClientToken string `json:"client_token"` + LeaseDuration int `json:"lease_duration"` + } `json:"auth"` + } + if err := do(ctx, addr, "", http.MethodPost, "/v1/auth/approle/login", map[string]string{ + "role_id": strings.TrimSpace(string(role)), "secret_id": strings.TrimSpace(string(sec))}, &out); err != nil { + return "", "", fmt.Errorf("logging in to Vault with AppRole: %w", err) + } + tok = out.Auth.ClientToken + if d := out.Auth.LeaseDuration; d > 0 { + // Renewed by logging in again, a little before it runs out. + exp = time.Now().Add(time.Duration(d) * time.Second * 9 / 10) + } + default: + tok, exp = os.Getenv("VAULT_TOKEN"), time.Time{} + } + if tok == "" { + return "", "", errors.New("no Vault token: set secrets.vault.token_file, AppRole's files, or VAULT_TOKEN") + } + mu.Lock() + token, expiry = tok, exp + mu.Unlock() + return tok, addr, nil +} diff --git a/internal/secrets/secrets_test.go b/internal/secrets/secrets_test.go new file mode 100644 index 0000000..c011c4d --- /dev/null +++ b/internal/secrets/secrets_test.go @@ -0,0 +1,142 @@ +package secrets + +import ( + "encoding/base64" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +// fakeVault answers KV v2 reads, AppRole logins and transit, and counts +// what it was asked. +func fakeVault(t *testing.T) (*httptest.Server, *int) { + t.Helper() + reads := 0 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/auth/approle/login" { + var in map[string]string + _ = json.NewDecoder(r.Body).Decode(&in) + if in["role_id"] != "role" || in["secret_id"] != "sid" { + http.Error(w, `{"errors":["denied"]}`, 400) + return + } + _, _ = w.Write([]byte(`{"auth":{"client_token":"approle-token","lease_duration":3600}}`)) + return + } + if tok := r.Header.Get("X-Vault-Token"); tok != "root" && tok != "approle-token" { + http.Error(w, `{"errors":["permission denied"]}`, 403) + return + } + switch { + case r.URL.Path == "/v1/secret/data/llmgw/store": + reads++ + _, _ = w.Write([]byte(`{"data":{"data":{"dsn":"postgres://u:p@h/db"}}}`)) + case strings.HasPrefix(r.URL.Path, "/v1/transit/encrypt/llmgw"): + var in map[string]string + _ = json.NewDecoder(r.Body).Decode(&in) + _, _ = w.Write([]byte(`{"data":{"ciphertext":"vault:v1:` + in["plaintext"] + `"}}`)) + case strings.HasPrefix(r.URL.Path, "/v1/transit/decrypt/llmgw"): + var in map[string]string + _ = json.NewDecoder(r.Body).Decode(&in) + _, _ = w.Write([]byte(`{"data":{"plaintext":"` + strings.TrimPrefix(in["ciphertext"], "vault:v1:") + `"}}`)) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + return srv, &reads +} + +func TestReferencesAreRead(t *testing.T) { + ctx := t.Context() + f := filepath.Join(t.TempDir(), "s") + _ = os.WriteFile(f, []byte(" from-file\n"), 0o600) + if v, err := ReadString(ctx, f); err != nil || v != "from-file" { + t.Errorf("file: %q %v", v, err) + } + t.Setenv("LLMGW_TEST_SECRET", "from-env") + if v, err := ReadString(ctx, "env:LLMGW_TEST_SECRET"); err != nil || v != "from-env" { + t.Errorf("env: %q %v", v, err) + } + if _, err := Read(ctx, "env:LLMGW_NOT_SET_ANYWHERE"); err == nil { + t.Error("an unset variable read as empty") + } + + srv, reads := fakeVault(t) + t.Setenv("VAULT_TOKEN", "root") + if err := Configure(Vault{Addr: srv.URL}); err != nil { + t.Fatal(err) + } + for range 3 { + if v, err := ReadString(ctx, "vault:secret/llmgw/store#dsn"); err != nil || v != "postgres://u:p@h/db" { + t.Fatalf("vault: %q %v", v, err) + } + } + if *reads != 1 { + t.Errorf("read from Vault %d times; cached for a minute", *reads) + } + if _, err := Read(ctx, "vault:secret/llmgw/store#nope"); err == nil { + t.Error("a missing field read") + } + if _, err := Read(ctx, "vault:secret/llmgw/store"); err == nil { + t.Error("a reference without a field read") + } + + // AppRole, with no token given. + t.Setenv("VAULT_TOKEN", "") + dir := t.TempDir() + _ = os.WriteFile(filepath.Join(dir, "role"), []byte("role\n"), 0o600) + _ = os.WriteFile(filepath.Join(dir, "sid"), []byte("sid\n"), 0o600) + _ = Configure(Vault{Addr: srv.URL, RoleIDFile: filepath.Join(dir, "role"), SecretIDFile: filepath.Join(dir, "sid")}) + if v, err := ReadString(ctx, "vault:secret/llmgw/store#dsn"); err != nil || v == "" { + t.Errorf("approle: %q %v", v, err) + } +} + +func TestSealersSealAndOpen(t *testing.T) { + ctx := t.Context() + keyFile := filepath.Join(t.TempDir(), "master") + _ = os.WriteFile(keyFile, []byte(strings.Repeat("ab", 32)), 0o600) + l, err := NewSealer(ctx, keyFile) + if err != nil { + t.Fatal(err) + } + sealed, err := l.Seal(ctx, []byte("the engine key")) + if err != nil || !IsSealed(sealed) || strings.Contains(sealed, "engine key") { + t.Fatalf("%q %v", sealed, err) + } + if got, err := l.Open(ctx, sealed); err != nil || string(got) != "the engine key" { + t.Errorf("open: %q %v", got, err) + } + other := filepath.Join(t.TempDir(), "other") + _ = os.WriteFile(other, []byte(strings.Repeat("cd", 32)), 0o600) + l2, _ := NewSealer(ctx, other) + if _, err := l2.Open(ctx, sealed); err == nil { + t.Error("another key opened it") + } + short := filepath.Join(t.TempDir(), "short") + _ = os.WriteFile(short, []byte("abc"), 0o600) + if _, err := NewSealer(ctx, short); err == nil { + t.Error("a short key was taken") + } + + srv, _ := fakeVault(t) + t.Setenv("VAULT_TOKEN", "root") + _ = Configure(Vault{Addr: srv.URL}) + tr, err := NewSealer(ctx, "vault-transit:llmgw") + if err != nil { + t.Fatal(err) + } + sealed, err = tr.Seal(ctx, []byte("secret")) + if err != nil || !strings.HasPrefix(sealed, transitPrefix) || + !strings.Contains(sealed, base64.StdEncoding.EncodeToString([]byte("secret"))) { + t.Fatalf("%q %v", sealed, err) + } + if got, err := tr.Open(ctx, sealed); err != nil || string(got) != "secret" { + t.Errorf("transit open: %q %v", got, err) + } +} diff --git a/internal/store/batches_test.go b/internal/store/batches_test.go index 45dd72d..e9692fc 100644 --- a/internal/store/batches_test.go +++ b/internal/store/batches_test.go @@ -2,8 +2,13 @@ package store import ( "errors" + "os" + "path/filepath" + "strings" "testing" "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" ) func TestFilesAndBatches(t *testing.T) { @@ -78,3 +83,40 @@ func checkFilesAndBatches(t *testing.T, db *DB) { t.Errorf("missing: %v", err) } } + +func TestClusterSecretsAreSealedUnderAMasterKey(t *testing.T) { + db := capacityStore(t) + ctx := t.Context() + plain, err := db.ClusterSecret(ctx, "old") + if err != nil { + t.Fatal(err) + } + key := filepath.Join(t.TempDir(), "master") + _ = os.WriteFile(key, []byte(strings.Repeat("ab", 32)), 0o600) + sealer, err := secrets.NewSealer(ctx, key) + if err != nil { + t.Fatal(err) + } + db.SetSealer(sealer) + if again, err := db.ClusterSecret(ctx, "old"); err != nil || again != plain { + t.Fatalf("a secret from before changed: %q %v", again, err) + } + fresh, err := db.ClusterSecret(ctx, "new") + if err != nil { + t.Fatal(err) + } + for _, name := range []string{"old", "new"} { + var v string + _ = db.db.QueryRowContext(ctx, `SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v) + if !secrets.IsSealed(v) { + t.Errorf("%s is kept plain: %q", name, v) + } + } + if again, _ := db.ClusterSecret(ctx, "new"); again != fresh { + t.Error("a sealed secret read back differently") + } + db.SetSealer(nil) + if _, err := db.ClusterSecret(ctx, "new"); err == nil { + t.Error("a sealed secret was read without its key") + } +} diff --git a/internal/store/open.go b/internal/store/open.go index 606019f..d267b8f 100644 --- a/internal/store/open.go +++ b/internal/store/open.go @@ -3,8 +3,9 @@ package store import ( "context" "fmt" - "os" "strings" + + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" ) // Open picks a backend from what the address looks like. @@ -49,7 +50,8 @@ func resolveAddress(address string) (string, error) { return address, nil } path := strings.TrimPrefix(address, "@") - b, err := os.ReadFile(path) + // A file, or env: or vault: (the secrets package says how). + b, err := secrets.Read(context.Background(), path) if err != nil { return "", fmt.Errorf("reading the store address from %s: %w", path, err) } diff --git a/internal/store/secrets.go b/internal/store/secrets.go index 1dc0bbb..f371d64 100644 --- a/internal/store/secrets.go +++ b/internal/store/secrets.go @@ -4,6 +4,8 @@ import ( "context" "crypto/rand" "encoding/hex" + "fmt" + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" "time" ) @@ -18,14 +20,44 @@ func (s *DB) ClusterSecret(ctx context.Context, name string) (string, error) { if _, err := rand.Read(b); err != nil { return "", err } + fresh := hex.EncodeToString(b) + stored := fresh + if s.sealer != nil { + var err error + if stored, err = s.sealer.Seal(ctx, []byte(fresh)); err != nil { + return "", err + } + } if _, err := s.db.ExecContext(ctx, `INSERT INTO cluster_secrets (name, value, created_at) VALUES (?, ?, ?) ON CONFLICT (name) DO NOTHING`, - name, hex.EncodeToString(b), time.Now().UnixMilli()); err != nil { + name, stored, time.Now().UnixMilli()); err != nil { return "", err } var v string - err := s.db.QueryRowContext(ctx, - `SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v) - return v, err + if err := s.db.QueryRowContext(ctx, + `SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v); err != nil { + return "", err + } + if secrets.IsSealed(v) { + if s.sealer == nil { + return "", fmt.Errorf("the store's secret %q is sealed, and secrets.master_key is not set to open it", name) + } + plain, err := s.sealer.Open(ctx, v) + if err != nil { + return "", fmt.Errorf("the store's secret %q: %w", name, err) + } + return string(plain), nil + } + // Kept plain from before a master key: sealed now, the value unchanged. + if s.sealer != nil { + if sealed, err := s.sealer.Seal(ctx, []byte(v)); err == nil { + _, _ = s.db.ExecContext(ctx, `UPDATE cluster_secrets SET value = ? WHERE name = ? AND value = ?`, sealed, name, v) + } + } + return v, nil } + +// SetSealer seals the store's own secrets with s from now on; values kept +// plain are sealed as they are next read. +func (s *DB) SetSealer(x secrets.Sealer) { s.sealer = x } diff --git a/internal/store/sqlite.go b/internal/store/sqlite.go index 85b2ed5..babd8ae 100644 --- a/internal/store/sqlite.go +++ b/internal/store/sqlite.go @@ -13,6 +13,8 @@ import ( "time" _ "modernc.org/sqlite" // the cgo-free driver, so the binary stays static + + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" ) // DB is the store, on either dialect. @@ -35,6 +37,9 @@ type DB struct { // the path nobody exercised before deploying. db bound + // sealer seals the store's own secrets; nil keeps them plain. + sealer secrets.Sealer + // mu guards the default limits, which are set once at wiring but read on // every authentication. mu sync.RWMutex diff --git a/internal/web/oidc.go b/internal/web/oidc.go index 0e87d1f..6085142 100644 --- a/internal/web/oidc.go +++ b/internal/web/oidc.go @@ -10,7 +10,6 @@ import ( "errors" "fmt" "net/http" - "os" "slices" "strings" "sync" @@ -21,6 +20,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/audit" "gitea.mynodes.xyz/ilker/llmgw/internal/config" + "gitea.mynodes.xyz/ilker/llmgw/internal/secrets" "gitea.mynodes.xyz/ilker/llmgw/internal/store" ) @@ -53,7 +53,7 @@ const oidcCookie = "llmgw_oidc" func (p *AccountPages) RegisterOIDC(mux *http.ServeMux, cfg config.OIDC, st OIDCStore) error { secret := "" if cfg.ClientSecretFile != "" { - b, err := os.ReadFile(cfg.ClientSecretFile) + b, err := secrets.Read(context.Background(), cfg.ClientSecretFile) if err != nil { return fmt.Errorf("auth.oidc.client_secret_file: %w", err) }