Trained adapters published, private to their owners like the knowledge bases they come from
An adapter now has an owner (org, team, project, user or key): only callers there see it, and to anyone else it is a model that does not exist, absent from /v1/models too. A training job's owner defaults to the one collection its pairs come from, must be able to read every collection they come from, and pairs from a knowledge base are never made public. ctl train publish keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and sets the owned adapter after the usual architecture check. The knowledge bases' cache_salt follows the collections searched rather than the organization, so two teams of one organization no longer share cached passages. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
964b35ae65
commit
85025dfb03
29 files changed
+1049
-84
No files matched your search
+48
-3
@@ -317,9 +317,11 @@ Behaviour to know about:
|
||||
(Codex, Claude Code) work the same way, since they are answered as chat
|
||||
completions.
|
||||
- **Separate prefix caches.** On vLLM replicas, a request given passages
|
||||
carries a `cache_salt` derived from its organization. One organization's
|
||||
passages then never share cached blocks with another's, so how fast an
|
||||
answer starts says nothing about what another tenant asked.
|
||||
carries a `cache_salt` derived from the collections they came from.
|
||||
Requests that could be given the same passages still share cached blocks,
|
||||
and no one else's do: not another organization's, nor another team's of
|
||||
the same one. How fast an answer starts therefore says nothing about what
|
||||
someone else asked.
|
||||
- **No answer cache.** Answers from knowledge bases are never served from
|
||||
the answer cache, because two keys of one project can read different
|
||||
collections.
|
||||
@@ -340,6 +342,49 @@ a product code or a name.
|
||||
Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
|
||||
`llmgw_kb_retrieval_seconds`.
|
||||
|
||||
## Training an adapter on a collection
|
||||
|
||||
An adapter (LoRA) changes how a model *writes*: the organisation's words,
|
||||
format and tone. It does not reliably teach facts. On the PoC, a small
|
||||
model trained on nine policy documents answered 4 of 9 rephrased questions
|
||||
right and mixed facts up on the rest, while the same questions through the
|
||||
knowledge base got 9 of 9. Use both: the knowledge base for what answers
|
||||
say, an adapter for how they sound.
|
||||
|
||||
```sh
|
||||
# 1. Questions and answers from the collection, written by a model of the
|
||||
# cluster. Read the file through and correct it.
|
||||
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
|
||||
|
||||
# 2. Train on a node. --gpu-memory is what the job needs of a card; it
|
||||
# runs beside an engine when the engine leaves that much free.
|
||||
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
|
||||
--base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
|
||||
llmgw ctl train show JOB --follow
|
||||
|
||||
# 3. Make it the model's adapter, kept on the mirror.
|
||||
llmgw ctl train publish JOB --restart
|
||||
```
|
||||
|
||||
The node trains with its installed vLLM's torch and a small overlay (peft
|
||||
and accelerate), as the engines' user. For a model llama.cpp serves, it also
|
||||
converts the adapter to GGUF with the installed llama.cpp's own code. What
|
||||
it makes is kept by the control plane (`llmgw ctl train download`).
|
||||
|
||||
An adapter is private like its collection:
|
||||
|
||||
- **It has an owner.** An adapter trained on a collection carries some of
|
||||
what the collection says in its weights, so it belongs to a part of the
|
||||
tree.
|
||||
- By default the owner is the collection's owner.
|
||||
- `--owner` names another, which must be able to read every collection
|
||||
the pairs came from.
|
||||
- Only callers in the owner's part of the tree see the adapter. To anyone
|
||||
else it is a model that does not exist, absent from `/v1/models` too.
|
||||
- **It is public only by choice.** `--public` makes an adapter every
|
||||
caller's. It is refused for pairs from a knowledge base, so it is only for
|
||||
pairs written by hand.
|
||||
|
||||
## How it is kept apart
|
||||
|
||||
Passages live in PostgreSQL, partitioned by organization, with an index per
|
||||
|
||||
Reference in new issue
Block a user