Model mirror

{{.Addr}}{{if .Unreachable}} did not answer{{end}}

{{/* Links rather than a script's panels: each tab has an address, the back button works, and only the one shown asks anything of the registry. */}} {{if .Error}}

{{end}} {{if eq .Tab "held"}} {{if .Unreachable}} {{/* Said outright rather than shown as an empty table. A mirror holding nothing and a mirror that is not there look the same in a list of zero rows, and they call for opposite responses. */}}

Nothing was read from the mirror, so this page is not a list of what it holds — it is not currently reachable at {{.Addr}}. Check that llmgw-mirror is running there and that its management port is reachable from this machine.

{{if .Rows}} {{range .Groups}}{{$grouped := gt .Models 1}} {{if $grouped}} {{end}} {{range .Rows}}{{$row0 := .}} {{if .Err}}{{end}} {{with .Detail}}{{end}} {{with .Serve}} {{end}} {{end}} {{end}}

Model	Files	Size	Runs
{{.Family}} {{.Models}} models · {{bytes .Bytes}} {{if .Fetching}}downloading{{end}} {{if .Failed}}failed{{end}} {{if .InUse}}in use{{end}}
{{if eq (printf "%s" .State) "present"}}{{.Model}} {{else}}{{.Model}}{{end}} {{.Version}}	{{if gt .Files 1}}{{.Files}} files{{else}}`{{.File}}`{{end}}	{{if eq (printf "%s" .State) "fetching"}} {{if .HasPercent}}{{.Percent}}% of {{bytes .BytesTotal}} {{else}}{{/* No total came from upstream, so no percentage is invented — a bar that guesses its own end and then stops moving is worse than a number that only goes up. */}} {{bytes .BytesDone}} so far{{end}} {{else if eq (printf "%s" .State) "failed"}}— {{else}}{{bytes .BytesDone}}{{end}}	{{if eq (printf "%s" .State) "fetching"}}downloading {{else if eq (printf "%s" .State) "failed"}}failed {{else if .UsedBy}}{{$row := .}}{{range .UsedBy}} {{/* Named only when the catalog calls it something other than the mirror does; otherwise it is the model column again. /}} {{if ne .Name $row.Model}}as {{.Name}}{{end}} {{if .Declared}} {{- if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}} · yaml {{else}} {{/ Submitted on change: the choice is the action, and a second click to confirm it would only be a way to forget to. */}} {{end}} {{if .Fleet}} {{- if .Parked}}parked {{- else if and (eq .Wanted 0) (eq .Ready 0)}}none running {{- else}}{{.Ready}}/{{.Wanted}}{{if .Starting}} +{{.Starting}}{{end}}{{end}} {{- if .Unplaceable}} · no room{{end}} {{- with .Nodes}} · {{range $i, $n := .}}{{if $i}}, {{end}}{{$n.Short}}{{end}}{{end}}{{end}} {{end}} {{else}}not served{{end}}	{{if eq (printf "%s" .State) "failed"}} {{/* Retry, not remove. A fetch that failed left nothing on disk, so the remove would come back "the mirror holds nothing for this" — a button whose only outcome is a confusing error. Starting the same download again is what somebody looking at a failed row wants, and the mirror allows it: only a download still running is refused. */}} {{else if eq (printf "%s" .State) "present"}} {{if and $.HasCatalog (not .UsedBy)}}{{if .Serve}}close {{else}}serve{{end}}{{end}} {{end}}
{{.Err}}
{{template "mirror_held_detail" dict "D" . "Row" $row0 "CSRF" $.CSRF}}
{{if .Refusal}} {{/* Offered and then failing at placement is worse than not offered at all. /}} Cannot be served as it stands: {{.Refusal}}. {{else}} {{if .Warning}} {{.Warning}}. {{end}} Serving `{{.Source}}` with {{.Engine}}. {{if .Files}}The whole version, {{.Files}} files: the node fetches them into one directory and the engine is given the directory.{{end}} Name Aliases comma separated Replicas Runs {{/ Reaches clients through /v1/models, which for most of them is the only view of what this gateway serves. Nine names and nothing to choose between them is a choice made by guesswork. */}} What it is for shown to clients in /v1/models Context a request may use Output limit optional Engine context pool divided across slots Profile name Needs RAM (GB) for the context pool beside it CPU cores reserved; the engine's threads Only nodes labelled key / value, optional The context pool is what the engine is given; llama.cpp divides it across its parallel slots, so the number one request may use is the smaller one above. Memory: {{.MemoryBasis}}. Change the context pool and this changes with it, by the KV cache's share — an estimate, and editable. It is filled in rather than left blank because a CPU profile has no card to be short of: with no figure, every ready node is a candidate and the first one picked is whichever the scheduler reached, not whichever can hold the model. A node label does the same job by name if you would rather pin it. {{end}}

“copy address” gives what a catalog entry's source should say — the file, or for a version of several files the directory, with its trailing slash. It is rendered rather than assembled by hand because the mirror and the catalog have to spell a file the same way, which is also how this page knows which models use it.

The mirror is reachable and holds nothing yet. {{if .HasRegistry}}Explore Hugging Face for a model, or {{end}}fetch one by URL.

{{end}} {{end}} {{if .RegistryErr}} {{/* Its own line, not the page's error: the registry is somebody else's server and it being down says nothing about the mirror, which is what this page is for and has to keep working when Hugging Face does not. */}}

The registry: {{.RegistryErr}}

Repository	Size	Architecture	Context	Downloads	Likes	Updated
`{{.ID}}` {{if .New}}new{{end}}{{if .Gated.Is}}gated{{end}} {{if or .BaseModel .License}}{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}{{end}}	{{.ParamsLabel}}	{{if .Architecture}}{{.Architecture}}{{if .Verdict.Checked}}{{if .Verdict.Servable}} ✓{{else}} ✗{{end}}{{end}}{{else}}—{{end}}	{{.ContextLabel}}	{{compact .Downloads}}	{{compact .Likes}}	{{.Age}}

{{.Hidden}} more {{if eq .Hidden 1}}is{{else}}are{{end}} hidden: their architecture is one the nodes' llama.cpp cannot load. Untick “only what these nodes can run” to see them.

A gated repository needs its terms accepted with a Hugging Face account, and this browses anonymously — the files will list, and fetching one will fail at the mirror. Download it by hand and put it in the mirror directory, or give the mirror a token.

{{if $.Query}}Nothing matched “{{$.Query}}”{{else}}Nothing to list{{end}}{{if .Hidden}} that these nodes can run — {{.Hidden}} {{if eq .Hidden 1}}result was{{else}}results were{{end}} hidden for an architecture llama.cpp cannot load{{end}}.

By hand, for anything the search does not reach.

The mirror answers as soon as the download is under way, not when it finishes — a model is tens of gigabytes and nothing here holds a connection open for that. The Held tab shows it arriving. A version is never overwritten: fetching into one that exists is refused, so a new build is a new version and the old weights stay where they are.

{{.Repo.ID}} at {{.Repo.ShortRevision}}

{{if .Inline}}at {{.Repo.ShortRevision}}{{end}} {{with .Repo.BaseModel}}from {{range $i, $b := .}}{{if $i}}, {{end}}{{$b}}{{end}}{{end}} {{with .Repo.License}}license {{.}}{{end}} {{with .Repo.Languages}}{{join .}}{{end}} on Hugging Face ↗

Only the weights are listed; a repository is mostly a README and a tokenizer. Shards of one file are one row and one button — a fifteen-shard model is one decision, not fifteen. Two separate files are two rows because they are usually alternatives: one quantisation or the other, not both.

What	Size	Published checksum
`{{.Name}}` {{if .Shards}}{{.Shards}} shards, fetched together{{end}}	{{bytes .Bytes}}	{{with .SHA256}}`{{slice . 0 12}}`{{else}}—{{end}}	{{if not .Verdict.Servable}} cannot be served {{.Verdict.Reason}} {{else}} {{if .Verdict.Architecture}}{{.Verdict.Architecture}}{{if .Verdict.Checked}} ✓{{end}}{{end}} {{range .Sources}} {{end}} {{end}}

The source URLs are pinned to {{.Repo.ShortRevision}} rather than to a branch, so what arrives is what is listed here even if the branch moves in between — and a single file's checksum above is what the registry publishes, to compare with what the mirror computes once it has it.

{{if gt $row.Files 1}}{{$row.Files}} files{{else}}{{$row.File}}{{end}} {{bytes $row.BytesDone}} {{with $row.SHA256}}sha256 {{slice . 0 12}}{{end}} {{$row.URL}}

Not in the catalog: nothing runs this version. {{if $row.Serve}}{{else}}Use serve on its row to put it into service.{{end}}

{{.Name}} {{with .View}}{{.Ready}} of {{.Desired}} ready{{if ne .Total .Ready}} · {{.Total}} in all{{end}} {{if .Parked}}parked{{else if .Unplaceable}}does not fit{{end}}{{end}}

Replica	Node	State	Age	CPU	Memory
{{if eq $i 0}}`{{$rep.ID}}`{{end}}	{{$s.Node}}{{if $s.MemoryGB}} {{$s.MemoryGB}} GB{{end}}	{{if eq $i 0}}{{$rep.State}} {{with $rep.Restart}}restart {{.}}{{end}}{{end}}	{{if eq $i 0}}{{age $rep.Age}}{{end}}	{{if $s.CPU}}{{printf "%.1f" $s.CPU}} cores{{else}}—{{end}}	{{if $s.MemBytes}}{{bytes $s.MemBytes}}{{else}}—{{end}}

No replica runs now{{if eq .Mode "ondemand"}}: it starts when somebody asks for it{{end}}.

{{if .Limited}}Room for at least {{.More}} more replicas. {{else if .More}}Room for {{.More}} more now{{if .Nodes}}, on {{range $i, $n := .Nodes}}{{if $i}}, {{end}}{{join $n}}{{end}}{{end}}. {{else}}No free room for another replica now.{{end}} {{if .ByPreempting}}
{{if .More}}And {{end}}{{.ByPreempting}} {{if .More}}more {{end}}by stopping replicas its class outranks: {{join .Preempts}}.{{end}}

Why not {{if .More}}more{{else}}another{{end}}

{{.}}

Set in server.yaml, which is read again on every start: change its replicas there.

Always running Up to, under load Class {{if $scopes}}{{$owner := .Owner}}Owner {{end}}

Memory GB per replica Cores per replica, its threads

{{if .MostNow}}At most {{.MostNow}} can always run now{{if .Headroom}}{{if .Headroom.ByPreempting}}, counting what its class may stop to make room{{end}}{{end}}. {{end}}0 always running with a higher limit loads it when asked for; 0 and 0 stops it. A replica is added under load when requests wait, and taken away after ten quiet minutes. A higher class may stop replicas of lower ones to make room; memory and cores apply to replicas started after the change.