Ilker ManapandClaude Opus 5.5 2a1e0e6c4c LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters ...

A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-05 01:14:38 +02:00

..

admin-api.md

The admin API described in OpenAPI 3.1: generated from a route table by docgen, served at /api/v1/openapi.json

2026-10-04 23:47:59 +02:00

admin-api.openapi.json

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

choosing-models.md

Manage people and the tenancy tree, from the web and the CLI

2026-09-28 16:26:39 +02:00

clients.md

A docs portal at /docs and a client developer's guide

2026-10-04 23:46:42 +02:00

console.md

Merge C21 quality suites and gating

2026-10-05 00:08:33 +02:00

data.md

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

development.md

Docs: one way in — quick-install starts at the setup screens, quickstart becomes development, install.md no longer says engines are yours to install

2026-10-04 17:32:54 +02:00

fabrics.md

Measure two nodes' fabric rail by rail: llmgw ctl fabric-test

2026-09-29 21:29:23 +02:00

facility-aware-placement.md

Mark facility steps 1-5 deferred; node sensors collected

2026-09-30 20:25:50 +02:00

guardrails.md

A hardened install by default: baseline guardrails with Llama Guard 3 1B judging on a CPU, keys at most 90 days

2026-10-04 17:45:31 +02:00

install.md

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

node-onboarding.md

Add a node with one command: llmgw node add, with a terminal screen

2026-09-29 19:33:47 +02:00

onboarding.md

Merge C24 docs portal and client guide

2026-10-04 23:47:35 +02:00

quick-install.md

The baseline fails closed with two judges, and a judge not answering is a notification

2026-10-04 19:19:49 +02:00

reference.md

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00