Commit Graph
353 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 220110ed0d v0.16.2: changelog and README
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 08:32:34 +01:00
Ilker ManapandClaude Opus 5.5 a88624747e A set wanted by several replicas on one node is fetched once
Single files were shared between the shards and replicas asking for them
at the same time; sets were not, and four replicas on one node's four cards
wrote the same .part files over one another until all but one failed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 08:29:25 +01:00
Ilker ManapandClaude Opus 5.5 540b518fde Rate limits counted across the control planes with router.active_active
Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 17:30:08 +01:00
Ilker ManapandClaude Opus 5.5 8b7eefef16 Standbys that answer the client API themselves, behind router.active_active
A standby follows the leader's Ready replicas over the agent port, every
two seconds, and reports back the replicas its router used and the
on-demand models it was asked for. No engine keys travel: every control
plane derives them from the cluster secret. Budgets are raised to the
store's every 30 s. Unheard from the leader for a minute, a standby
passes /v1 on as before. Off by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 13:47:47 +01:00
Ilker ManapandClaude Opus 5.5 31f440ab6e A served model's longest answer changed in place, and Strata's context
max_output on POST /api/v1/models/{name}/context, --max-output on ctl model
context, and a field in the console's Context length… form. Alone it changes
only the catalog's limit, which the router holds max_tokens to, and starts no
replica again; with a context it is set with it, never past it.

Strata's context is now changed like the others', through its setup's
--context, which writes the engine's start script at each start. Neither
needs a model removed and added again any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 00:15:38 +01:00
Ilker ManapandClaude Opus 5.5 f4b34dbf11 FreeToken, an engine for models larger than the card
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.

ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.

Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.

FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 22:10:05 +01:00
Ilker ManapandClaude Opus 5.5 c866b3a8ee How much of each GPU works, beside whether anything does
The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.

Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:11:26 +01:00
Ilker ManapandClaude Opus 5.5 7bfe70c8df BACKLOG: the router profiled, the write per token removed, what is left
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 15:50:15 +01:00
Ilker ManapandClaude Opus 5.5 9ff49cad8e Requests missing the cache for the same key at once share one read and one allowance
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 09:30:18 +01:00
Ilker ManapandClaude Opus 5.5 524f029603 External providers left out of admission's saturation; the scale-out section in the backlog
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 19:13:15 +01:00
Ilker ManapandClaude Opus 5.5 be54f1617b A mirror download cut off carries on from where it got to, and is tried again
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 17:47:32 +01:00
Ilker ManapandClaude Opus 5.5 76746715ec A set's files checked four at once, the check said as it goes and not taken for a stuck download
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 14:58:44 +01:00
Ilker ManapandClaude Opus 5.5 d5dcf78836 A set's download reported as the whole set's, not the part being fetched
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 14:48:45 +01:00
Ilker ManapandClaude Opus 5.5 a5fefbdaaa v0.16.1: changelog and README
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:14:25 +01:00
Ilker ManapandClaude Opus 5.5 f41c96d30f Strata's DeepSeek not offered on cards its engine was not built for, and refused there by the agent
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:00:21 +01:00
Ilker ManapandClaude Opus 5.5 104b60fbf8 A repository's files read from every directory, not only its top
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:16:11 +01:00
Ilker ManapandClaude Opus 5.5 845de8eef4 The mirror page's Strata tab; Strata's ready-made packs served with Strata fetching them, strata-ple named
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:06:46 +01:00
Ilker ManapandClaude Opus 5.5 4ee9ba54ef v0.16.0: changelog and README
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:21:32 +01:00
Ilker ManapandClaude Opus 5.5 12464ce0f6 Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:14:02 +01:00
Ilker Manap d6f8f433f4 CHANGELOG: one list 2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 b8ed368e31 Weights from Hugging Face pinned to a commit and checked against the sha256 it published
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 07:33:15 +01:00
Ilker ManapandClaude Opus 5.5 29d7f54cfd A path escaped as it came reaches the leader through a standby; F5 in the backlog
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 22:54:27 +01:00
Ilker ManapandClaude Opus 5.5 cacaa35c3f The gateway embeds the documents llmgw-kb sends as text
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 21:56:05 +01:00
Ilker ManapandClaude Opus 5.5 01ad8c9207 v0.15.3: changelog and README
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 20:54:53 +01:00
Ilker ManapandClaude Opus 5.5 59187cdae5 A model can be stopped and kept, or removed with its files from the mirror and the nodes
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 20:45:48 +01:00
Ilker ManapandClaude Opus 5.5 435ffe1171 A file removed outside the running mirror is no longer reported, and its removal reaches the peers
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 19:25:00 +01:00
Ilker ManapandClaude Opus 5.5 785e06ecbf A mirror with peers records the checksums of files that had none, so they are copied too
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 18:36:59 +01:00
Ilker ManapandClaude Opus 5.5 0033a530c7 mirror add-peer on a mirror machine of its own says what to run on a control plane
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 18:30:30 +01:00
Ilker ManapandClaude Opus 5.5 8cd376615b v0.15.2: changelog and README
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:46:15 +01:00
Ilker ManapandClaude Opus 5.5 e1cff0d4e3 A served model can be moved to more cards, or fewer
llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:40:49 +01:00
Ilker ManapandClaude Opus 5.5 51aeadae3e v0.15.1: changelog and README
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 15:59:12 +01:00
Ilker ManapandClaude Opus 5.5 986981ac81 A served model's context can be changed, engine and memory with it
ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context
and the console's Context length…: the catalog, the engine's arguments and
the KV memory change together, refused past the trained context or what the
cards hold, then the replicas restart one by one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 15:50:04 +01:00
Ilker ManapandClaude Opus 5.5 1c16f2337d A second mirror: llmgw mirror add-peer, and on a store machine at setup
mirror add-peer, run on the mirror's machine, sets up a mirror on
another over ssh, kept in step with this one: llmgw there if it has
none, mirror init --peer, this mirror told of it and allowed to be read
by it, and mirror.peers in the cluster's settings. The setup screens
offer one on either database machine, as server init --second-mirror.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 13:43:27 +01:00
Ilker ManapandClaude Opus 5.5 c09d5ad4ee Outside services read answers redacted and whole; a card's own lanes
From the second review.
- An output service reads the answer with the values restore put back
  taken out again (guard.Subject.Restored), as an input service reads the
  request redacted.
- With n above 1, the service reads the first choice once it is whole,
  not at whichever choice finished first.
- Restored values go into an answer asked for as JSON (response_format)
  escaped for it, as into tool-call arguments.
- The embedding failure is logged once a minute through an outage, not
  once.
- A card's own lane count and fastest generation come from the kernel
  (max_link_width, max_link_speed): the driver's width.max is what card
  and slot allow together, so an x16 card in an x8 slot read x8 of x8 and
  was never said.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 13:22:39 +01:00
Ilker ManapandClaude Opus 5.5 ecf5545a7a VS Code's sign-in keeps its place; recordings checked for secrets
From review.
- /profile asked before signing in keeps its query for ten minutes, and
  the sign-in (password or OIDC) returns to it: VS Code's sign-in state
  came back lost, and the confirmation came every time. Only /profile is
  ever returned to.
- The extension keeps an https gateway at https when a link names it by
  plain http (a proxy ending TLS unannounced), so the key is never sent in
  the clear. The install command says --force, since dev builds share a
  version and VS Code kept the old one.
- Every frame of every install recording is checked for a key, a
  password or a private key, not only the page.
- The admin API document again matches the API: f94d902's
  MCPServerRequest.clear was not in it.
- CHANGELOG for this round's fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 11:40:19 +01:00
Ilker ManapandClaude Opus 5.5 721705b29d A card in the wrong PCIe slot is said so
At the user's asking: we know what a card's link could be, so a card
linked slower is reported. Each agent reads, apart from the GPU query so
that a driver not knowing a field costs nothing else, the fastest PCIe
generation card and slot allow together, the card's own, and the lanes in
use and the card's most (nvidia-smi; NVML in the nvml build). The current
generation is not compared: it drops while a card idles. A card on fewer
lanes, or a slot of an older generation, is a node warning with the GB/s
it carries against what it could — on the dashboard, the console and in
llmgw doctor, which points at lspci's LnkCap and LnkSta. MIG instances are
said of their card. GPU proto fields 23-26.

The PoC's Quadro RTX 6000 reads Gen3 of Gen3, x16 of x16, idle at Gen1:
nothing to say.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:49:16 +01:00
Ilker ManapandClaude Opus 5.5 b986ea180d Fixes from review: setup's memory check, MCP secrets, VS Code, bandwidth
- server init run again on a machine already set up is not refused its
  memory (the docs fix a certificate with --host so), and judges count
  only when one will be installed (judgeComes: not without a mirror, nor
  from a bundle without it).
- A server's sealed secrets that it no longer reads are removed when it is
  put or deleted; replacing a credential replaces every header it was
  sent, rather than adding one beside the old.
- llmgw.gatewayUrl is a machine setting and restricted in untrusted
  workspaces: a cloned repository's settings cannot send a sign-in to
  another gateway.
- The bandwidth buffer is at least eight times the CPU's last cache and at
  most a quarter of free memory, else nothing is measured until there is
  room; a registration carries the figure measured so far.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:23:00 +01:00
Ilker ManapandClaude Opus 5.5 54c6b66cb9 Fixes from review: the similar cache, restore, outside services, latency
- The similar-question cache embeds a question as its project would: a
  project kept in the cluster has external fallbacks refused, and an
  embedding model outside the cluster is not asked at all. The vectors are
  compared outside the cache's lock. The question's bound is in characters.
- The restorer puts values back in streamed tool calls' arguments (escaped
  for the JSON they are), completions' text and reasoning, each held across
  events on its own, and releases a held tail with the finish event. A
  whole answer is walked as JSON, so a value with a quote no longer breaks
  tool-call arguments.
- An outside guardrail service reads the user's turn as redacted, and a
  streamed answer once rather than at the finish chunk and again at the
  end. guardrails.md says it can stop a stream only under buffer_all.
- A replica's smoothed time to first token is forgotten after five minutes
  without an answer: one found slow and then given nothing was never tried
  again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:15:43 +01:00
Ilker ManapandClaude Opus 5.5 fd94265e08 Pickles refused however they come; a string temperature passes again
From review. A mirror set already on the node is looked at again, not
trusted for having been fetched once; Pickled cuts a query or a fragment
from any source, hf: too; .pt2 is a pickle. The flag check reads a flag as
the engine does: underscores as dashes and any beginning of a refused flag
as that flag, since argparse takes --trust-remote for --trust-remote-code;
SGLang refuses --load-format, which would undo the agent's safetensors.

TripoSR's preset says allow_pickle: without it the preset could not be
added and the presets pass failed on it each time. A shared ComfyUI
environment's hidden entry takes its first member's allow_pickle, as it
takes its source.

The GenAI trace attributes made temperature and top_p typed fields of the
request's parse, so "0.7" as a string was a 400; they are read loosely,
as they are read only for the trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:19:06 +01:00
Ilker ManapandClaude Opus 5.5 44451bacd9 Granting or taking back a server's tools leaves the rest as it was
A tool is seen where every level above that names tools names it (store
Effective.AllowsTool). The console and /admin/mcp wrote a level's list as
given, so a team that saw its organization's hr.* through having no list
was given jira.* and then saw neither; and revoking a level's last server
left [] in the store, closed to every tool, shown nowhere.

store.GrantTools and RevokeTools now do both screens' work: a level's first
list starts from the nearest list above it; taking a server away leaves the
others, and a level left naming nothing goes back to no list, or stays
closed when a level above names tools, so it opens to nothing it was kept
from. GrantTools says which levels above still keep the granted tools out
of reach, and both screens say so. The page shows closed levels and member
scopes by name, and offers no revoke for a * grant it would not change. The
console calls the store instead of keys limits with a list built from what
it had loaded, which overwrote changes made since. Found in review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:11:07 +01:00
Ilker ManapandClaude Opus 5.5 7a04248148 A reused answer never carries another caller's values
With restore: true and the answer cache on, alice's "write to
alice@example.com" and bob's "write to bob@example.com" redact to the
same request, and bob was given the answer kept for alice, her address put
back in it. A request with restore values is now never looked up in the
cache nor kept in it.

The similar-question cache compared the last message's text and blanked
its parts: two pictures asked about in the same words were one question.
A last message with any part that is not text is now reused only when the
request is the same.

Found in review; both tests fail without the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:03:32 +01:00
Ilker ManapandClaude Opus 5.5 03d3bcfd2d The company's MCP servers on a web page
/admin/mcp, for administrators only, since a server's credential reaches
whatever it guards: each server's state, who has been granted its tools,
its rate, the headers it is sent by name with the sealed ones marked, and
its address; adding one with a bearer token or another header kept in the
store; testing it with its credential and listing its tools; granting them
to a team, project, org, person or key, written into that level's limits
beside its other grants and its rate and budget; revoking; changing;
replacing the credential; turning off and on; removing; and how a person
points an agent at the gateway's /mcp with a key of their own. English and
Turkish. The secret's field is named so the audit line redacts it.

The admin API's server merge and test are package functions now,
adminapi.MCPServerFrom and adminapi.TestMCPServer, shared by both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:26:51 +01:00
Ilker ManapandClaude Opus 5.5 98c45ed28b The company's MCP servers in the console
A screen beside Knowledge: each server's state, who has been granted its
tools (read from the store's limits), its rate, the names of the headers
it is sent but never their values, and its address. A adds one, with a
bearer token or another header sent once and kept sealed. On a row: test
it and list its tools; grant them to a team, project, org, person or key,
merged with what else that level is granted, since keys limits --tools
writes the list whole; revoke; change; replace the credential; turn off or
on; remove. The screen also says how an agent is pointed at /mcp with a
person's own key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 13:23:25 +01:00
Ilker ManapandClaude Opus 5.5 f2c6f12cfb How fast a model would answer on the cards, as this cluster would lay it out
For each layout modelplan.Options offers here — one card, tensor parallel
across a node's cards, llama.cpp's layer split, a pipeline across nodes on
a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes
a token reads over each card's bandwidth, by its model name (a table, SXM
and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on
NVLink, 25 us and up on PCIe), turns between cards for a layer split, and
a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a
repository's files show the fastest, with the layout; mixed classes are
judged class by class, FP8 on 8.9 cards only.

Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB,
62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB,
144 measured, which says a model of experts reads 1.6 times its active
share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics
are not measured here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 00:19:36 +01:00
Ilker ManapandClaude Opus 5.5 84444e71c8 How fast a model would answer on a CPU, before it is fetched
Each agent measures its node's memory read rate when it starts: every CPU
sums a share of a buffer past any cache, a tenth of free memory at most,
the best of four passes, in a fraction of a second; reported in Host
(mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it
uses once, so one reply's speed is about that rate over those bytes:
modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token,
calibrated on the PoC against llama.cpp's predicted_per_second (node01
24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s,
qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active
part (ActiveShare).

The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that
holds the model; a repository's page gives it for each GGUF by its size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 00:09:47 +01:00
Ilker ManapandClaude Opus 5.5 2a45068461 The explorer lists what this cluster's hardware holds
Judged by the nodes as they are now, through modelplan.Options, the
planner that serving a model uses: GGUF at Q4_K_M from the parameter count,
safetensors at the size the registry's dtype breakdown gives (packed AWQ,
GPTQ and MXFP4 at four bits), FP8 only where a card has compute capability
8.9, MLX safetensors never. A picture model is judged against one card. A
cluster with cards asks the registry for safetensors too, a hundred at a
time; one without asks for GGUF only. The page says what it judged by: the
largest card, its capability, the most RAM on one node.

On a repository's page a file set no node holds is marked with what it
needs, against what the best node offers, and has no Fetch button.

Tried against the PoC's seven nodes and Hugging Face's trending hundred:
63 shown, the rest too big for one 24 GB Turing card or FP8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 18:30:16 +01:00
Ilker ManapandClaude Opus 5.5 e84631ce3a The explorer lists only what a node loads
A repository with neither GGUF nor safetensors (by the registry's summary
or its tags) is hidden under "runs here", which is ticked by default, and
counted with the architectures llama.cpp cannot load; unticked it is
listed. An open repository's pickle files are not offered, only counted,
and a repository of pickles only says to look for another version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 13:00:28 +01:00
Ilker ManapandClaude Opus 5.5 4466d2d15a Weights that run code are refused
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.

BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 23:17:22 +01:00
Ilker ManapandClaude Opus 5.5 6752938e6d v0.15.0: changelog and README
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:59:44 +01:00
Ilker ManapandClaude Opus 5.5 3a3db8cd20 Answers to similar questions, not only the same one
llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.

The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:47:11 +01:00