Secrets in Vault: env: and vault: references wherever a secret file is named, and the store's own secrets sealed under a master key — Vault transit or a local key

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:21:44 +02:00
1 parent bce54f0941
commit 35657c1417
20 files changed
+837 -14

No files matched your search

+8 -1
View File
@@ -490,7 +490,14 @@ portal is not done — the page shows; the console and ctl change.
**Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
pre-warm, queue-depth scaling in `catalog.Scaling`,
`control/reconciler/demand.go`, `headroom.go`.
15. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
15. **Done (branch `features-c2`):** `env:` and `vault:` references
wherever a secret file is named (store address, api_key_file, OIDC,
SCIM, webhooks), Vault KV with token or AppRole, `secrets.master_key`
sealing cluster_secrets (Vault transit or a local key). Left: the
certificate authority's key is still a file in /etc/llmgw (signing in
Vault's PKI engine instead is its own change); rotating the master key;
`backup_copy_key` and `admin_token_file` are files only.
**Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
CA key, `cluster_secrets` (`store/secrets.go`, `store/seal.go`);
envelope-encrypt them in backups (with A8 backups).
16. **GDPR and residency (S).** "Forget user" anonymises `user_id` in usage
+10
View File
@@ -71,6 +71,16 @@ APIs change between minors without a deprecation period.
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
and **E** on the console's Usage view.
### Secrets in Vault
- **Secrets from Vault or the environment.** Every setting naming a secret's
file also takes `env:VARIABLE` or `vault:mount/path#field` (HashiCorp
Vault or OpenBao, KV v1 or v2; a token, AppRole, or VAULT_TOKEN).
- **The store's own secrets sealed** under `secrets.master_key`: Vault's
transit engine (`vault-transit:KEY`), or a 32-byte key read from a
reference. A store, or a backup of it, then does not hold what engine keys
are derived from. Values from before are sealed as they are next read.
### Scaling by the clock
- **`scaling.schedule`**: windows of the week with their own `min` and
+4
View File
@@ -44,6 +44,10 @@ never stored — so a leaked database hands over no working credential, and
neither llmgw nor an administrator can recover a key somebody lost. A prefix is
kept so a key can be identified in a listing without revealing it.
**The store's own secrets**, like the one engine keys are derived from, plain
unless `secrets.master_key` seals them — with Vault's transit engine, so that
neither the store nor its backups hold them.
**People and tenancy**: accounts, the organisation/team/project tree, and the
limits attached to it. Passwords are hashed.
+37
View File
@@ -1394,6 +1394,43 @@ and point it at `https://<gateway>/scim/v2` with that token as the bearer.
join and leave them. Membership in a group is removed when someone leaves
it.
## Secrets in Vault
Every setting that names a file holding a secret — the store's `@address`,
`api_key_file`, `auth.oidc.client_secret_file`, `auth.scim.token_file`, a
webhook's `secret_file` — also takes `env:VARIABLE` or a field of a secret in
HashiCorp Vault or OpenBao:
```yaml
auth:
store: "@vault:secret/llmgw/store#dsn"
secrets:
vault:
addr: https://vault.internal:8200
ca_file: /etc/llmgw/vault-ca.pem
role_id_file: /etc/llmgw/secrets/vault-role-id # AppRole; or token_file,
secret_id_file: /etc/llmgw/secrets/vault-secret-id # or VAULT_TOKEN
master_key: vault-transit:llmgw
```
These are read when the gateway starts (an external model's key, each time
the catalog changes): a secret rotated in Vault is taken up at the next
restart.
**The store's own secrets** — what every engine's key is derived from — are
kept in the store, and so in its backups. `secrets.master_key` seals them:
- `vault-transit:KEY` seals with Vault's transit engine. The key never leaves
Vault, so neither the store nor any backup of it opens without Vault.
- Anything else is a reference to 32 random bytes (`openssl rand -hex 32`),
used here: better than nothing, but a key in `/etc/llmgw` is in every backup
beside what it seals. Keep it in Vault or the environment.
Values kept plain from before are sealed the first time they are read after
the key is set. The key cannot be changed once set without losing them: a new
one cannot open what the old one sealed (the engines are then restarted with
new keys: `llmgw ctl restart --all`).
## The firewall
`server init`, `node add`, `server add` and `mirror init` each keep that
+75
View File
@@ -230,6 +230,8 @@ rather than repeated.
| `target_inflight` | `float64` | [Scoring](#scoring) |
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
| `session_load_factor` | `float64` | [Scoring](#scoring) |
| `vault` | `secrets.Vault` | [Secrets](#secrets) |
| `master_key` | `string` | [Secrets](#secrets) |
| `listen` | `string` | [Server](#server) |
| `admin_listen` | `string` | [Server](#server) |
| `metrics_listen` | `string` | [Server](#server) |
@@ -250,6 +252,7 @@ rather than repeated.
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `guard` | `guard.Config` | [Server](#server) |
| `secrets` | `Secrets` | [Server](#server) |
| `notify` | `notify.Config` | [Server](#server) |
| `firewall_networks` | `[]string` | [Server](#server) |
| `backup_recipients` | `[]string` | [Server](#server) |
@@ -279,6 +282,12 @@ rather than repeated.
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
| `addr` | `string` | [Vault](#vault) |
| `token_file` | `string` | [Vault](#vault) |
| `role_id_file` | `string` | [Vault](#vault) |
| `secret_id_file` | `string` | [Vault](#vault) |
| `ca_file` | `string` | [Vault](#vault) |
| `namespace` | `string` | [Vault](#vault) |
| `url` | `string` | [Webhook](#webhook) |
| `format` | `string` | [Webhook](#webhook) |
| `events` | `[]string` | [Webhook](#webhook) |
@@ -1859,6 +1868,27 @@ arrive at nine are the reason for the window.
`float64`
## Secrets
### `vault`
`secrets.Vault`
Vault is how vault: references and vault-transit: are reached.
Takes a [Vault](#vault) block.
### `master_key`
`string`
MasterKey seals the secrets the store keeps for itself — what every
engine's key is derived from — so a copy of the store, or of a backup,
does not hold them: vault-transit:KEY seals with Vault's transit engine,
which never lets the key out; otherwise a reference to 32 random bytes
(openssl rand -hex 32), which then must be kept as safely as the store.
Values from before it was set are sealed the first time they are read.
## Server
### `listen`
@@ -2048,6 +2078,15 @@ guard.default, and to parts of the tenancy tree by attaching them
Takes a [guard.Config](#guardconfig) block.
### `secrets`
`Secrets`
Secrets is where secrets kept outside files are read from, and the key
the store's own secrets are sealed under.
Takes a [Secrets](#secrets) block.
### `notify`
`notify.Config`
@@ -2263,6 +2302,42 @@ node and has not been checked against a real cluster.
RayBin is used when LaunchMode is "ray".
## Vault
### `addr`
`string`
Addr is Vault's address, like https://vault.internal:8200; empty
takes VAULT_ADDR.
### `token_file`
`string`
TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with
AppRole when RoleIDFile and SecretIDFile are set.
### `role_id_file`
`string`
### `secret_id_file`
`string`
### `ca_file`
`string`
CAFile verifies Vault's certificate; empty uses the system's.
### `namespace`
`string`
Namespace is Vault Enterprise's or OpenBao's namespace.
## Webhook
### `url`
+10
View File
@@ -48,6 +48,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -120,6 +121,15 @@ func newServerCmd() *cobra.Command {
if err != nil {
return err
}
// The store's own secrets sealed under a key kept outside it,
// before anything reads them.
if st := rt.Store(); st != nil && cfg.Secrets.MasterKey != "" {
sealer, err := secrets.NewSealer(cmd.Context(), cfg.Secrets.MasterKey)
if err != nil {
return err
}
st.SetSealer(sealer)
}
// The longest any key may live, where every way of making one
// reads it: the store.
if st := rt.Store(); st != nil {
+27
View File
@@ -20,6 +20,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
@@ -129,6 +130,10 @@ type Server struct {
// (llmgw ctl guard attach). Empty checks nothing.
Guard guard.Config `koanf:"guard"`
// Secrets is where secrets kept outside files are read from, and the key
// the store's own secrets are sealed under.
Secrets Secrets `koanf:"secrets"`
// Notify sends what needs a person — a model parked, a node lost, a
// budget spent, new weights published — to webhooks, signed, without a
// Prometheus and Alertmanager in between. Empty sends nothing.
@@ -571,6 +576,24 @@ type Endpoint struct {
Priority int `koanf:"priority"`
}
// Secrets is where secrets are read from besides files, and how the
// store's own are sealed.
//
// Wherever a setting names a file holding a secret — the store's @address,
// api_key_file, client_secret_file, token_file, secret_file — it may name
// env:VARIABLE or vault:mount/path#field instead.
type Secrets struct {
// Vault is how vault: references and vault-transit: are reached.
Vault secrets.Vault `koanf:"vault"`
// MasterKey seals the secrets the store keeps for itself — what every
// engine's key is derived from — so a copy of the store, or of a backup,
// does not hold them: vault-transit:KEY seals with Vault's transit engine,
// which never lets the key out; otherwise a reference to 32 random bytes
// (openssl rand -hex 32), which then must be kept as safely as the store.
// Values from before it was set are sealed the first time they are read.
MasterKey string `koanf:"master_key"`
}
// Batch is how the Files and Batch APIs are bounded.
type Batch struct {
// MaxFileBytes bounds one uploaded file; 100 MB unless set.
@@ -679,6 +702,10 @@ func Load(path string) (Server, error) {
if err := k.UnmarshalWithConf("", &cfg, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
return cfg, fmt.Errorf("decoding configuration: %w", err)
}
// So every secret reference in it, from here on, can be read.
if err := secrets.Configure(cfg.Secrets.Vault); err != nil {
return cfg, err
}
return cfg, cfg.Validate()
}
+2 -2
View File
@@ -9,7 +9,6 @@ import (
"encoding/hex"
"fmt"
"log/slog"
"os"
"strings"
"sync"
"time"
@@ -19,6 +18,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
)
// EndpointSink receives the endpoint list. The router implements it.
@@ -356,7 +356,7 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
}
key := ""
if f := m.External.APIKeyFile; f != "" {
b, err := os.ReadFile(f) //nolint:gosec // the operator's own path
b, err := secrets.Read(context.Background(), f)
if err != nil {
c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
continue
+13
View File
@@ -289,6 +289,10 @@ var Reference = []Block{
{Key: "kv_ceiling", Type: "float64", Doc: "", Nested: ""},
{Key: "session_load_factor", Type: "float64", Doc: "", Nested: ""},
}},
{Package: "config", Name: "Secrets", Title: "Secrets", Doc: "", Fields: []Field{
{Key: "vault", Type: "secrets.Vault", Doc: "Vault is how vault: references and vault-transit: are reached.", Nested: "secrets.Vault"},
{Key: "master_key", Type: "string", Doc: "MasterKey seals the secrets the store keeps for itself — what every\nengine's key is derived from — so a copy of the store, or of a backup,\ndoes not hold them: vault-transit:KEY seals with Vault's transit engine,\nwhich never lets the key out; otherwise a reference to 32 random bytes\n(openssl rand -hex 32), which then must be kept as safely as the store.\nValues from before it was set are sealed the first time they are read.", Nested: ""},
}},
{Package: "config", Name: "Server", Title: "Server", Doc: "", Fields: []Field{
{Key: "listen", Type: "string", Doc: "Listen is the client-facing OpenAI API address.", Nested: ""},
{Key: "admin_listen", Type: "string", Doc: "AdminListen serves the dashboard, the management API, health and pprof.\n\nLoopback by default, and it should stay there: pprof can dump the process\nheap, and API keys are in that heap in plaintext. Every comment beside\nthis setting has always said not to expose it, while the default bound\nevery interface — a default that contradicts its own documentation is a\nfootgun, so the default now matches the advice.", Nested: ""},
@@ -310,6 +314,7 @@ var Reference = []Block{
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
{Key: "guard", Type: "guard.Config", Doc: "Guard is the library of guardrail policies: what requests and answers\nare checked for, and what is done. Applied to every request through\nguard.default, and to parts of the tenancy tree by attaching them\n(llmgw ctl guard attach). Empty checks nothing.", Nested: "guard.Config"},
{Key: "secrets", Type: "Secrets", Doc: "Secrets is where secrets kept outside files are read from, and the key\nthe store's own secrets are sealed under.", Nested: "config.Secrets"},
{Key: "notify", Type: "notify.Config", Doc: "Notify sends what needs a person — a model parked, a node lost, a\nbudget spent, new weights published — to webhooks, signed, without a\nPrometheus and Alertmanager in between. Empty sends nothing.", Nested: "notify.Config"},
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
@@ -352,6 +357,14 @@ var Reference = []Block{
{Key: "launch_mode", Type: "string", Doc: "LaunchMode selects the multi-node mechanism: \"mp\", the default, or\n\"ray\".\n\nmp is what vLLM documents today — every node runs serve with its rank,\nthe workers headless — and needs nothing but vLLM. ray is the older way,\nkept for a vLLM too old for mp's multi-node flags; it needs Ray on every\nnode and has not been checked against a real cluster.", Nested: ""},
{Key: "ray_bin", Type: "string", Doc: "RayBin is used when LaunchMode is \"ray\".", Nested: ""},
}},
{Package: "secrets", Name: "Vault", Title: "Vault", Doc: "", Fields: []Field{
{Key: "addr", Type: "string", Doc: "Addr is Vault's address, like https://vault.internal:8200; empty\ntakes VAULT_ADDR.", Nested: ""},
{Key: "token_file", Type: "string", Doc: "TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with\nAppRole when RoleIDFile and SecretIDFile are set.", Nested: ""},
{Key: "role_id_file", Type: "string", Doc: "", Nested: ""},
{Key: "secret_id_file", Type: "string", Doc: "", Nested: ""},
{Key: "ca_file", Type: "string", Doc: "CAFile verifies Vault's certificate; empty uses the system's.", Nested: ""},
{Key: "namespace", Type: "string", Doc: "Namespace is Vault Enterprise's or OpenBao's namespace.", Nested: ""},
}},
{Package: "notify", Name: "Webhook", Title: "Webhook", Doc: "", Fields: []Field{
{Key: "url", Type: "string", Doc: "", Nested: ""},
{Key: "format", Type: "string", Doc: "Format is json (the event, with X-Llmgw-Signature when SecretFile is\nset) or slack ({\"text\": …}, which Slack, Mattermost and Rocket.Chat\nincoming webhooks take).", Nested: ""},
+1
View File
@@ -30,6 +30,7 @@ var Sources = []string{
"internal/guard/judge.go",
"internal/config/oidc.go",
"internal/notify/notify.go",
"internal/secrets/secrets.go",
}
// notReference are files with koanf tags that an operator never writes.
+3 -1
View File
@@ -27,6 +27,8 @@ import (
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
)
// Event kinds.
@@ -138,7 +140,7 @@ func New(ctx context.Context, c Config, log *slog.Logger) (*Notifier, error) {
for _, w := range c.Webhooks {
h := hook{Webhook: w}
if w.SecretFile != "" {
b, err := os.ReadFile(w.SecretFile)
b, err := secrets.Read(ctx, w.SecretFile)
if err != nil {
return nil, fmt.Errorf("notify: %w", err)
}
+2 -2
View File
@@ -10,7 +10,6 @@ import (
"fmt"
"io"
"net/http"
"os"
"slices"
"strconv"
"strings"
@@ -25,6 +24,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
"gitea.mynodes.xyz/ilker/llmgw/internal/scim"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
@@ -211,7 +211,7 @@ func (r *Router) Handler() http.Handler {
}
}
if r.scim.TokenFile != "" {
if b, err := os.ReadFile(r.scim.TokenFile); err != nil || strings.TrimSpace(string(b)) == "" {
if b, err := secrets.Read(context.Background(), r.scim.TokenFile); err != nil || strings.TrimSpace(string(b)) == "" {
r.log.Error("provisioning (SCIM) is off: its token cannot be read", "file", r.scim.TokenFile, "err", err)
} else {
scim.New(r.store, strings.TrimSpace(string(b)),
+136
View File
@@ -0,0 +1,136 @@
package secrets
import (
"context"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"strings"
"golang.org/x/crypto/hkdf"
)
// Sealer encrypts the store's own secrets — what engine keys are derived
// from — under a key the store does not hold, so a copy of the store or of
// its backup does not hold them either.
type Sealer interface {
Seal(ctx context.Context, plain []byte) (string, error)
Open(ctx context.Context, sealed string) ([]byte, error)
}
// Sealed values say how they were sealed, so a plain value from before is
// told apart, and read.
const (
localPrefix = "llmgw:local:v1:"
transitPrefix = "llmgw:transit:v1:"
)
// IsSealed says whether a stored value is sealed.
func IsSealed(v string) bool {
return strings.HasPrefix(v, localPrefix) || strings.HasPrefix(v, transitPrefix)
}
// NewSealer is the sealer secrets.master_key names: vault-transit:KEY seals
// with Vault's transit engine, which never lets the key out; anything else
// is a reference to 32 bytes (hex, base64 or raw) used here.
func NewSealer(ctx context.Context, ref string) (Sealer, error) {
if key, ok := strings.CutPrefix(ref, "vault-transit:"); ok {
mount, name := "transit", key
if m, n, ok := strings.Cut(key, "/"); ok {
mount, name = m, n
}
return transit{mount: mount, key: name}, nil
}
raw, err := Read(ctx, ref)
if err != nil {
return nil, fmt.Errorf("secrets.master_key: %w", err)
}
key := decodeKey(raw)
if len(key) < 32 {
return nil, errors.New("secrets.master_key is shorter than 32 bytes: make one with `openssl rand -hex 32`")
}
k := make([]byte, 32)
if _, err := io.ReadFull(hkdf.New(sha256.New, key, nil, []byte("llmgw cluster secrets v1")), k); err != nil {
return nil, err
}
block, err := aes.NewCipher(k)
if err != nil {
return nil, err
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return nil, err
}
return local{gcm: gcm}, nil
}
func decodeKey(b []byte) []byte {
s := strings.TrimSpace(string(b))
if k, err := hex.DecodeString(s); err == nil {
return k
}
if k, err := base64.StdEncoding.DecodeString(s); err == nil {
return k
}
return []byte(s)
}
type local struct{ gcm cipher.AEAD }
func (l local) Seal(_ context.Context, plain []byte) (string, error) {
nonce := make([]byte, l.gcm.NonceSize())
if _, err := rand.Read(nonce); err != nil {
return "", err
}
return localPrefix + base64.StdEncoding.EncodeToString(l.gcm.Seal(nonce, nonce, plain, nil)), nil
}
func (l local) Open(_ context.Context, sealed string) ([]byte, error) {
b, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(sealed, localPrefix))
if err != nil || !strings.HasPrefix(sealed, localPrefix) || len(b) < l.gcm.NonceSize() {
return nil, errors.New("not a value this key sealed")
}
out, err := l.gcm.Open(nil, b[:l.gcm.NonceSize()], b[l.gcm.NonceSize():], nil)
if err != nil {
return nil, errors.New("the master key does not open this value: it was sealed under another")
}
return out, nil
}
type transit struct{ mount, key string }
func (t transit) Seal(ctx context.Context, plain []byte) (string, error) {
var out struct {
Data struct {
Ciphertext string `json:"ciphertext"`
} `json:"data"`
}
if err := call(ctx, http.MethodPost, "/v1/"+t.mount+"/encrypt/"+t.key,
map[string]string{"plaintext": base64.StdEncoding.EncodeToString(plain)}, &out); err != nil {
return "", fmt.Errorf("sealing with Vault's transit key %s: %w", t.key, err)
}
return transitPrefix + out.Data.Ciphertext, nil
}
func (t transit) Open(ctx context.Context, sealed string) ([]byte, error) {
if !strings.HasPrefix(sealed, transitPrefix) {
return nil, errors.New("not a value Vault's transit engine sealed")
}
var out struct {
Data struct {
Plaintext string `json:"plaintext"`
} `json:"data"`
}
if err := call(ctx, http.MethodPost, "/v1/"+t.mount+"/decrypt/"+t.key,
map[string]string{"ciphertext": strings.TrimPrefix(sealed, transitPrefix)}, &out); err != nil {
return nil, fmt.Errorf("opening with Vault's transit key %s: %w", t.key, err)
}
return base64.StdEncoding.DecodeString(out.Data.Plaintext)
}
+278
View File
@@ -0,0 +1,278 @@
// Package secrets reads the secrets llmgw is configured with from where
// they are kept: a file, as always, an environment variable, or HashiCorp
// Vault (or OpenBao) — and seals the store's own secrets under a key that
// stays outside it.
//
// A reference is one of:
//
// /etc/llmgw/secrets/store-dsn a file, trimmed
// env:LLMGW_STORE_DSN an environment variable
// vault:secret/llmgw/store#dsn a field of a KV secret in Vault
//
// A Vault path's first segment is its KV v2 mount; secret/data/… is taken
// as written. KV v1 mounts work too.
package secrets
import (
"bytes"
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strings"
"sync"
"time"
)
// Vault is how Vault is reached.
type Vault struct {
// Addr is Vault's address, like https://vault.internal:8200; empty
// takes VAULT_ADDR.
Addr string `koanf:"addr"`
// TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with
// AppRole when RoleIDFile and SecretIDFile are set.
TokenFile string `koanf:"token_file"`
RoleIDFile string `koanf:"role_id_file"`
SecretIDFile string `koanf:"secret_id_file"`
// CAFile verifies Vault's certificate; empty uses the system's.
CAFile string `koanf:"ca_file"`
// Namespace is Vault Enterprise's or OpenBao's namespace.
Namespace string `koanf:"namespace"`
}
var (
mu sync.Mutex
vault Vault
client = &http.Client{Timeout: 10 * time.Second}
token string
expiry time.Time
cache = map[string]cached{}
)
type cached struct {
b []byte
at time.Time
}
// cacheFor is how long a value read from Vault is used before it is read
// again: a secret rotated there reaches llmgw within it.
const cacheFor = time.Minute
// Configure says how to reach Vault. Called once configuration is read.
func Configure(v Vault) error {
mu.Lock()
defer mu.Unlock()
vault, token, expiry, cache = v, "", time.Time{}, map[string]cached{}
if v.CAFile != "" {
pem, err := os.ReadFile(v.CAFile)
if err != nil {
return fmt.Errorf("secrets.vault.ca_file: %w", err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return fmt.Errorf("secrets.vault.ca_file %s holds no certificate", v.CAFile)
}
client = &http.Client{Timeout: 10 * time.Second,
Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}
}
return nil
}
// IsReference says whether s names a secret somewhere other than a file.
func IsReference(s string) bool {
return strings.HasPrefix(s, "env:") || strings.HasPrefix(s, "vault:")
}
// Read reads a secret by reference, trimmed of surrounding space.
func Read(ctx context.Context, ref string) ([]byte, error) {
switch {
case strings.HasPrefix(ref, "env:"):
name := strings.TrimPrefix(ref, "env:")
v, ok := os.LookupEnv(name)
if !ok {
return nil, fmt.Errorf("the environment has no %s", name)
}
return []byte(strings.TrimSpace(v)), nil
case strings.HasPrefix(ref, "vault:"):
return readVault(ctx, strings.TrimPrefix(ref, "vault:"))
}
b, err := os.ReadFile(ref) //nolint:gosec // a path the operator configured
if err != nil {
return nil, err
}
return bytes.TrimSpace(b), nil
}
// ReadString is Read as a string.
func ReadString(ctx context.Context, ref string) (string, error) {
b, err := Read(ctx, ref)
return string(b), err
}
func readVault(ctx context.Context, ref string) ([]byte, error) {
path, field, ok := strings.Cut(ref, "#")
if !ok || path == "" || field == "" {
return nil, fmt.Errorf("vault:%s: write it as vault:mount/path#field", ref)
}
mu.Lock()
if c, ok := cache[ref]; ok && time.Since(c.at) < cacheFor {
mu.Unlock()
return c.b, nil
}
mu.Unlock()
api := path
if mount, rest, ok := strings.Cut(path, "/"); ok && !strings.HasPrefix(rest, "data/") {
api = mount + "/data/" + rest
}
var out struct {
Data map[string]any `json:"data"`
}
if err := call(ctx, http.MethodGet, "/v1/"+api, nil, &out); err != nil {
// A KV v1 mount has no /data/ in its path.
if api != path && errors.Is(err, errNotFound) {
err = call(ctx, http.MethodGet, "/v1/"+path, nil, &out)
}
if err != nil {
return nil, fmt.Errorf("vault:%s: %w", ref, err)
}
}
data := out.Data
if inner, ok := data["data"].(map[string]any); ok {
data = inner // KV v2
}
v, ok := data[field]
if !ok {
return nil, fmt.Errorf("vault:%s: the secret has no field %q", ref, field)
}
s, ok := v.(string)
if !ok {
return nil, fmt.Errorf("vault:%s: field %q is not a string", ref, field)
}
b := []byte(strings.TrimSpace(s))
mu.Lock()
cache[ref] = cached{b: b, at: time.Now()}
mu.Unlock()
return b, nil
}
var errNotFound = errors.New("not found")
// call makes one request of Vault's API with the token, logging in first
// when there is none.
func call(ctx context.Context, method, path string, body, out any) error {
tok, addr, err := currentToken(ctx)
if err != nil {
return err
}
return do(ctx, addr, tok, method, path, body, out)
}
func do(ctx context.Context, addr, tok, method, path string, body, out any) error {
var rd io.Reader
if body != nil {
b, _ := json.Marshal(body)
rd = bytes.NewReader(b)
}
req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(addr, "/")+path, rd)
if err != nil {
return err
}
if tok != "" {
req.Header.Set("X-Vault-Token", tok)
}
mu.Lock()
ns, c := vault.Namespace, client
mu.Unlock()
if ns != "" {
req.Header.Set("X-Vault-Namespace", ns)
}
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := c.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
switch {
case resp.StatusCode == http.StatusNotFound:
return errNotFound
case resp.StatusCode >= 300:
var e struct {
Errors []string `json:"errors"`
}
_ = json.Unmarshal(b, &e)
return fmt.Errorf("vault answered %s: %s", resp.Status, strings.Join(e.Errors, "; "))
}
if out != nil {
return json.Unmarshal(b, out)
}
return nil
}
// currentToken is the token to send, logging in with AppRole when that is
// how it is configured and the last login has expired.
func currentToken(ctx context.Context) (string, string, error) {
mu.Lock()
v := vault
tok, exp := token, expiry
mu.Unlock()
addr := v.Addr
if addr == "" {
addr = os.Getenv("VAULT_ADDR")
}
if addr == "" {
return "", "", errors.New("no Vault to ask: set secrets.vault.addr or VAULT_ADDR")
}
if tok != "" && (exp.IsZero() || time.Now().Before(exp)) {
return tok, addr, nil
}
switch {
case v.TokenFile != "":
b, err := os.ReadFile(v.TokenFile)
if err != nil {
return "", "", fmt.Errorf("secrets.vault.token_file: %w", err)
}
tok, exp = strings.TrimSpace(string(b)), time.Time{}
case v.RoleIDFile != "" && v.SecretIDFile != "":
role, err := os.ReadFile(v.RoleIDFile)
if err != nil {
return "", "", err
}
sec, err := os.ReadFile(v.SecretIDFile)
if err != nil {
return "", "", err
}
var out struct {
Auth struct {
ClientToken string `json:"client_token"`
LeaseDuration int `json:"lease_duration"`
} `json:"auth"`
}
if err := do(ctx, addr, "", http.MethodPost, "/v1/auth/approle/login", map[string]string{
"role_id": strings.TrimSpace(string(role)), "secret_id": strings.TrimSpace(string(sec))}, &out); err != nil {
return "", "", fmt.Errorf("logging in to Vault with AppRole: %w", err)
}
tok = out.Auth.ClientToken
if d := out.Auth.LeaseDuration; d > 0 {
// Renewed by logging in again, a little before it runs out.
exp = time.Now().Add(time.Duration(d) * time.Second * 9 / 10)
}
default:
tok, exp = os.Getenv("VAULT_TOKEN"), time.Time{}
}
if tok == "" {
return "", "", errors.New("no Vault token: set secrets.vault.token_file, AppRole's files, or VAULT_TOKEN")
}
mu.Lock()
token, expiry = tok, exp
mu.Unlock()
return tok, addr, nil
}
+142
View File
@@ -0,0 +1,142 @@
package secrets
import (
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
// fakeVault answers KV v2 reads, AppRole logins and transit, and counts
// what it was asked.
func fakeVault(t *testing.T) (*httptest.Server, *int) {
t.Helper()
reads := 0
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1/auth/approle/login" {
var in map[string]string
_ = json.NewDecoder(r.Body).Decode(&in)
if in["role_id"] != "role" || in["secret_id"] != "sid" {
http.Error(w, `{"errors":["denied"]}`, 400)
return
}
_, _ = w.Write([]byte(`{"auth":{"client_token":"approle-token","lease_duration":3600}}`))
return
}
if tok := r.Header.Get("X-Vault-Token"); tok != "root" && tok != "approle-token" {
http.Error(w, `{"errors":["permission denied"]}`, 403)
return
}
switch {
case r.URL.Path == "/v1/secret/data/llmgw/store":
reads++
_, _ = w.Write([]byte(`{"data":{"data":{"dsn":"postgres://u:p@h/db"}}}`))
case strings.HasPrefix(r.URL.Path, "/v1/transit/encrypt/llmgw"):
var in map[string]string
_ = json.NewDecoder(r.Body).Decode(&in)
_, _ = w.Write([]byte(`{"data":{"ciphertext":"vault:v1:` + in["plaintext"] + `"}}`))
case strings.HasPrefix(r.URL.Path, "/v1/transit/decrypt/llmgw"):
var in map[string]string
_ = json.NewDecoder(r.Body).Decode(&in)
_, _ = w.Write([]byte(`{"data":{"plaintext":"` + strings.TrimPrefix(in["ciphertext"], "vault:v1:") + `"}}`))
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
return srv, &reads
}
func TestReferencesAreRead(t *testing.T) {
ctx := t.Context()
f := filepath.Join(t.TempDir(), "s")
_ = os.WriteFile(f, []byte(" from-file\n"), 0o600)
if v, err := ReadString(ctx, f); err != nil || v != "from-file" {
t.Errorf("file: %q %v", v, err)
}
t.Setenv("LLMGW_TEST_SECRET", "from-env")
if v, err := ReadString(ctx, "env:LLMGW_TEST_SECRET"); err != nil || v != "from-env" {
t.Errorf("env: %q %v", v, err)
}
if _, err := Read(ctx, "env:LLMGW_NOT_SET_ANYWHERE"); err == nil {
t.Error("an unset variable read as empty")
}
srv, reads := fakeVault(t)
t.Setenv("VAULT_TOKEN", "root")
if err := Configure(Vault{Addr: srv.URL}); err != nil {
t.Fatal(err)
}
for range 3 {
if v, err := ReadString(ctx, "vault:secret/llmgw/store#dsn"); err != nil || v != "postgres://u:p@h/db" {
t.Fatalf("vault: %q %v", v, err)
}
}
if *reads != 1 {
t.Errorf("read from Vault %d times; cached for a minute", *reads)
}
if _, err := Read(ctx, "vault:secret/llmgw/store#nope"); err == nil {
t.Error("a missing field read")
}
if _, err := Read(ctx, "vault:secret/llmgw/store"); err == nil {
t.Error("a reference without a field read")
}
// AppRole, with no token given.
t.Setenv("VAULT_TOKEN", "")
dir := t.TempDir()
_ = os.WriteFile(filepath.Join(dir, "role"), []byte("role\n"), 0o600)
_ = os.WriteFile(filepath.Join(dir, "sid"), []byte("sid\n"), 0o600)
_ = Configure(Vault{Addr: srv.URL, RoleIDFile: filepath.Join(dir, "role"), SecretIDFile: filepath.Join(dir, "sid")})
if v, err := ReadString(ctx, "vault:secret/llmgw/store#dsn"); err != nil || v == "" {
t.Errorf("approle: %q %v", v, err)
}
}
func TestSealersSealAndOpen(t *testing.T) {
ctx := t.Context()
keyFile := filepath.Join(t.TempDir(), "master")
_ = os.WriteFile(keyFile, []byte(strings.Repeat("ab", 32)), 0o600)
l, err := NewSealer(ctx, keyFile)
if err != nil {
t.Fatal(err)
}
sealed, err := l.Seal(ctx, []byte("the engine key"))
if err != nil || !IsSealed(sealed) || strings.Contains(sealed, "engine key") {
t.Fatalf("%q %v", sealed, err)
}
if got, err := l.Open(ctx, sealed); err != nil || string(got) != "the engine key" {
t.Errorf("open: %q %v", got, err)
}
other := filepath.Join(t.TempDir(), "other")
_ = os.WriteFile(other, []byte(strings.Repeat("cd", 32)), 0o600)
l2, _ := NewSealer(ctx, other)
if _, err := l2.Open(ctx, sealed); err == nil {
t.Error("another key opened it")
}
short := filepath.Join(t.TempDir(), "short")
_ = os.WriteFile(short, []byte("abc"), 0o600)
if _, err := NewSealer(ctx, short); err == nil {
t.Error("a short key was taken")
}
srv, _ := fakeVault(t)
t.Setenv("VAULT_TOKEN", "root")
_ = Configure(Vault{Addr: srv.URL})
tr, err := NewSealer(ctx, "vault-transit:llmgw")
if err != nil {
t.Fatal(err)
}
sealed, err = tr.Seal(ctx, []byte("secret"))
if err != nil || !strings.HasPrefix(sealed, transitPrefix) ||
!strings.Contains(sealed, base64.StdEncoding.EncodeToString([]byte("secret"))) {
t.Fatalf("%q %v", sealed, err)
}
if got, err := tr.Open(ctx, sealed); err != nil || string(got) != "secret" {
t.Errorf("transit open: %q %v", got, err)
}
}
+42
View File
@@ -2,8 +2,13 @@ package store
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
)
func TestFilesAndBatches(t *testing.T) {
@@ -78,3 +83,40 @@ func checkFilesAndBatches(t *testing.T, db *DB) {
t.Errorf("missing: %v", err)
}
}
func TestClusterSecretsAreSealedUnderAMasterKey(t *testing.T) {
db := capacityStore(t)
ctx := t.Context()
plain, err := db.ClusterSecret(ctx, "old")
if err != nil {
t.Fatal(err)
}
key := filepath.Join(t.TempDir(), "master")
_ = os.WriteFile(key, []byte(strings.Repeat("ab", 32)), 0o600)
sealer, err := secrets.NewSealer(ctx, key)
if err != nil {
t.Fatal(err)
}
db.SetSealer(sealer)
if again, err := db.ClusterSecret(ctx, "old"); err != nil || again != plain {
t.Fatalf("a secret from before changed: %q %v", again, err)
}
fresh, err := db.ClusterSecret(ctx, "new")
if err != nil {
t.Fatal(err)
}
for _, name := range []string{"old", "new"} {
var v string
_ = db.db.QueryRowContext(ctx, `SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v)
if !secrets.IsSealed(v) {
t.Errorf("%s is kept plain: %q", name, v)
}
}
if again, _ := db.ClusterSecret(ctx, "new"); again != fresh {
t.Error("a sealed secret read back differently")
}
db.SetSealer(nil)
if _, err := db.ClusterSecret(ctx, "new"); err == nil {
t.Error("a sealed secret was read without its key")
}
}
+4 -2
View File
@@ -3,8 +3,9 @@ package store
import (
"context"
"fmt"
"os"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
)
// Open picks a backend from what the address looks like.
@@ -49,7 +50,8 @@ func resolveAddress(address string) (string, error) {
return address, nil
}
path := strings.TrimPrefix(address, "@")
b, err := os.ReadFile(path)
// A file, or env: or vault: (the secrets package says how).
b, err := secrets.Read(context.Background(), path)
if err != nil {
return "", fmt.Errorf("reading the store address from %s: %w", path, err)
}
+36 -4
View File
@@ -4,6 +4,8 @@ import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
"time"
)
@@ -18,14 +20,44 @@ func (s *DB) ClusterSecret(ctx context.Context, name string) (string, error) {
if _, err := rand.Read(b); err != nil {
return "", err
}
fresh := hex.EncodeToString(b)
stored := fresh
if s.sealer != nil {
var err error
if stored, err = s.sealer.Seal(ctx, []byte(fresh)); err != nil {
return "", err
}
}
if _, err := s.db.ExecContext(ctx,
`INSERT INTO cluster_secrets (name, value, created_at) VALUES (?, ?, ?)
ON CONFLICT (name) DO NOTHING`,
name, hex.EncodeToString(b), time.Now().UnixMilli()); err != nil {
name, stored, time.Now().UnixMilli()); err != nil {
return "", err
}
var v string
err := s.db.QueryRowContext(ctx,
`SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v)
return v, err
if err := s.db.QueryRowContext(ctx,
`SELECT value FROM cluster_secrets WHERE name = ?`, name).Scan(&v); err != nil {
return "", err
}
if secrets.IsSealed(v) {
if s.sealer == nil {
return "", fmt.Errorf("the store's secret %q is sealed, and secrets.master_key is not set to open it", name)
}
plain, err := s.sealer.Open(ctx, v)
if err != nil {
return "", fmt.Errorf("the store's secret %q: %w", name, err)
}
return string(plain), nil
}
// Kept plain from before a master key: sealed now, the value unchanged.
if s.sealer != nil {
if sealed, err := s.sealer.Seal(ctx, []byte(v)); err == nil {
_, _ = s.db.ExecContext(ctx, `UPDATE cluster_secrets SET value = ? WHERE name = ? AND value = ?`, sealed, name, v)
}
}
return v, nil
}
// SetSealer seals the store's own secrets with s from now on; values kept
// plain are sealed as they are next read.
func (s *DB) SetSealer(x secrets.Sealer) { s.sealer = x }
+5
View File
@@ -13,6 +13,8 @@ import (
"time"
_ "modernc.org/sqlite" // the cgo-free driver, so the binary stays static
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
)
// DB is the store, on either dialect.
@@ -35,6 +37,9 @@ type DB struct {
// the path nobody exercised before deploying.
db bound
// sealer seals the store's own secrets; nil keeps them plain.
sealer secrets.Sealer
// mu guards the default limits, which are set once at wiring but read on
// every authentication.
mu sync.RWMutex
+2 -2
View File
@@ -10,7 +10,6 @@ import (
"errors"
"fmt"
"net/http"
"os"
"slices"
"strings"
"sync"
@@ -21,6 +20,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -53,7 +53,7 @@ const oidcCookie = "llmgw_oidc"
func (p *AccountPages) RegisterOIDC(mux *http.ServeMux, cfg config.OIDC, st OIDCStore) error {
secret := ""
if cfg.ClientSecretFile != "" {
b, err := os.ReadFile(cfg.ClientSecretFile)
b, err := secrets.Read(context.Background(), cfg.ClientSecretFile)
if err != nil {
return fmt.Errorf("auth.oidc.client_secret_file: %w", err)
}