Ilker ManapandClaude Opus 5.5
37a80a9b44
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not know — was started 155 times in eight hours, failing the same way each time, and phi-4, which fitted on no node, logged "no placement" 250 times in two days. Neither showed up anywhere an operator looked: the backoff never gave up, no metric said a model was failing or unplaceable, and the one alert for a model with nothing serving could not fire, because its series was published only while a replica was ready. After five failures in a row a model is parked: not placed again until its definition changes — a new source or profile — or `llmgw ctl retry` is run. Status, the model list and the admin API show it with its last error. The placement warning is logged when its reason changes or every ten minutes. A request for a parked or unplaceable model is answered at once with why, rather than "loading; retry shortly", which a client retries forever. Each request still clears the previous placement result and triggers a fresh attempt, so a model that fits once room is made is not refused on a stale answer; the router waits for that attempt, not for its full activation window. Metrics now cover model health (desired, ready at 0 for what should run, consecutive start failures, parked, unplaceable), engine installs per node, leadership, and certificate expiry read from disk at each scrape, with alerts for each. `ctl status` counts a model against what it should have now, so an on-demand model that was asked for and fitted nowhere no longer reads as satisfied. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
59 lines
1.6 KiB
Go
59 lines
1.6 KiB
Go
package tlsconf
|
|
|
|
import (
|
|
"crypto/x509"
|
|
"encoding/pem"
|
|
"os"
|
|
|
|
"github.com/prometheus/client_golang/prometheus"
|
|
)
|
|
|
|
// ExpiryCollector publishes when each certificate in a set of files expires.
|
|
//
|
|
// Certificates here last a year and nothing renews them; the first sign of one
|
|
// running out was going to be every agent refusing to connect on the same day.
|
|
// Read at every scrape rather than once, so a certificate replaced on disk is
|
|
// seen without a restart. A file that cannot be read publishes nothing, which
|
|
// is its own thing to notice.
|
|
func ExpiryCollector(files map[string]string) prometheus.Collector {
|
|
return &expiry{files: files, desc: prometheus.NewDesc("llmgw_certificate_expiry_timestamp_seconds",
|
|
"When a certificate this control plane uses expires, as a Unix time.",
|
|
[]string{"cert", "subject"}, nil)}
|
|
}
|
|
|
|
type expiry struct {
|
|
files map[string]string
|
|
desc *prometheus.Desc
|
|
}
|
|
|
|
func (e *expiry) Describe(ch chan<- *prometheus.Desc) { ch <- e.desc }
|
|
|
|
func (e *expiry) Collect(ch chan<- prometheus.Metric) {
|
|
for name, path := range e.files {
|
|
if path == "" {
|
|
continue
|
|
}
|
|
b, err := os.ReadFile(path) //nolint:gosec // configured certificate paths
|
|
if err != nil {
|
|
continue
|
|
}
|
|
// Every certificate in the file: a CA bundle may hold several.
|
|
for {
|
|
var blk *pem.Block
|
|
blk, b = pem.Decode(b)
|
|
if blk == nil {
|
|
break
|
|
}
|
|
if blk.Type != "CERTIFICATE" {
|
|
continue
|
|
}
|
|
c, err := x509.ParseCertificate(blk.Bytes)
|
|
if err != nil {
|
|
continue
|
|
}
|
|
ch <- prometheus.MustNewConstMetric(e.desc, prometheus.GaugeValue,
|
|
float64(c.NotAfter.Unix()), name, c.Subject.CommonName)
|
|
}
|
|
}
|
|
}
|