Trained adapters published, private to their owners like the knowledge bases they come from
An adapter now has an owner (org, team, project, user or key): only callers there see it, and to anyone else it is a model that does not exist, absent from /v1/models too. A training job's owner defaults to the one collection its pairs come from, must be able to read every collection they come from, and pairs from a knowledge base are never made public. ctl train publish keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and sets the owned adapter after the usual architecture check. The knowledge bases' cache_salt follows the collections searched rather than the organization, so two teams of one organization no longer share cached passages. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
964b35ae65
commit
85025dfb03
29 files changed
+1049
-84
No files matched your search
@@ -3940,6 +3940,48 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/train/jobs/{id}/publish": {
|
||||
"post": {
|
||||
"operationId": "postTrainJobsIdPublish",
|
||||
"summary": "Make what a finished job made the model's adapter",
|
||||
"description": "Kept on the mirror — the GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves — and added to the model's adapters after the same check as POST /api/v1/models/{name}/adapters.",
|
||||
"tags": [
|
||||
"train"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The job's ID, as starting it answered.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/TrainPublishView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/updates": {
|
||||
"get": {
|
||||
"operationId": "getUpdates",
|
||||
@@ -5378,6 +5420,10 @@
|
||||
"ord": {
|
||||
"type": "integer"
|
||||
},
|
||||
"collection": {
|
||||
"type": "string",
|
||||
"description": "Collection is where the pair came from: an adapter trained on it belongs to a part of the tree that may read it."
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
}
|
||||
@@ -7089,6 +7135,9 @@
|
||||
"node": {
|
||||
"type": "string"
|
||||
},
|
||||
"owner": {
|
||||
"type": "string"
|
||||
},
|
||||
"gpu_memory_mib": {
|
||||
"type": "integer"
|
||||
},
|
||||
@@ -7143,6 +7192,28 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"TrainPublishView": {
|
||||
"type": "object",
|
||||
"description": "TrainPublishView is an adapter published: where the nodes fetch it from, and what to do for replicas to load it.",
|
||||
"properties": {
|
||||
"adapter": {
|
||||
"type": "string"
|
||||
},
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"source": {
|
||||
"type": "string"
|
||||
},
|
||||
"owner": {
|
||||
"type": "string",
|
||||
"description": "Owner is who sees it; empty is every caller of the model."
|
||||
},
|
||||
"note": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"TrainRequest": {
|
||||
"type": "object",
|
||||
"description": "TrainRequest queues a job.",
|
||||
@@ -7169,6 +7240,14 @@
|
||||
"gpu_memory_mib": {
|
||||
"type": "integer"
|
||||
},
|
||||
"owner": {
|
||||
"type": "string",
|
||||
"description": "Owner is who the adapter will belong to: org:, team:, project: (by path or ID), user: or key:. Only callers there see it. When the pairs come from one knowledge base, it defaults to that collection's owner; it must be able to read every collection they come from."
|
||||
},
|
||||
"public": {
|
||||
"type": "boolean",
|
||||
"description": "Public makes the adapter every caller's its model allows. Refused for pairs from a knowledge base."
|
||||
},
|
||||
"data": {
|
||||
"type": "string",
|
||||
"description": "Data is the pairs, NDJSON of {\"q\", \"a\"} (llmgw ctl kb dataset)."
|
||||
@@ -7334,6 +7413,10 @@
|
||||
"scale": {
|
||||
"type": "number",
|
||||
"description": "Scale is how strongly it is applied; zero takes 1."
|
||||
},
|
||||
"owner": {
|
||||
"type": "string",
|
||||
"description": "Owner is the part of the tenancy tree the adapter belongs to — org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in it see it: to anyone else it is a model that does not exist. An adapter trained on an organisation's documents carries some of what they say in its weights. Empty is every caller the model allows."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+48
-3
@@ -317,9 +317,11 @@ Behaviour to know about:
|
||||
(Codex, Claude Code) work the same way, since they are answered as chat
|
||||
completions.
|
||||
- **Separate prefix caches.** On vLLM replicas, a request given passages
|
||||
carries a `cache_salt` derived from its organization. One organization's
|
||||
passages then never share cached blocks with another's, so how fast an
|
||||
answer starts says nothing about what another tenant asked.
|
||||
carries a `cache_salt` derived from the collections they came from.
|
||||
Requests that could be given the same passages still share cached blocks,
|
||||
and no one else's do: not another organization's, nor another team's of
|
||||
the same one. How fast an answer starts therefore says nothing about what
|
||||
someone else asked.
|
||||
- **No answer cache.** Answers from knowledge bases are never served from
|
||||
the answer cache, because two keys of one project can read different
|
||||
collections.
|
||||
@@ -340,6 +342,49 @@ a product code or a name.
|
||||
Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
|
||||
`llmgw_kb_retrieval_seconds`.
|
||||
|
||||
## Training an adapter on a collection
|
||||
|
||||
An adapter (LoRA) changes how a model *writes*: the organisation's words,
|
||||
format and tone. It does not reliably teach facts. On the PoC, a small
|
||||
model trained on nine policy documents answered 4 of 9 rephrased questions
|
||||
right and mixed facts up on the rest, while the same questions through the
|
||||
knowledge base got 9 of 9. Use both: the knowledge base for what answers
|
||||
say, an adapter for how they sound.
|
||||
|
||||
```sh
|
||||
# 1. Questions and answers from the collection, written by a model of the
|
||||
# cluster. Read the file through and correct it.
|
||||
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
|
||||
|
||||
# 2. Train on a node. --gpu-memory is what the job needs of a card; it
|
||||
# runs beside an engine when the engine leaves that much free.
|
||||
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
|
||||
--base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
|
||||
llmgw ctl train show JOB --follow
|
||||
|
||||
# 3. Make it the model's adapter, kept on the mirror.
|
||||
llmgw ctl train publish JOB --restart
|
||||
```
|
||||
|
||||
The node trains with its installed vLLM's torch and a small overlay (peft
|
||||
and accelerate), as the engines' user. For a model llama.cpp serves, it also
|
||||
converts the adapter to GGUF with the installed llama.cpp's own code. What
|
||||
it makes is kept by the control plane (`llmgw ctl train download`).
|
||||
|
||||
An adapter is private like its collection:
|
||||
|
||||
- **It has an owner.** An adapter trained on a collection carries some of
|
||||
what the collection says in its weights, so it belongs to a part of the
|
||||
tree.
|
||||
- By default the owner is the collection's owner.
|
||||
- `--owner` names another, which must be able to read every collection
|
||||
the pairs came from.
|
||||
- Only callers in the owner's part of the tree see the adapter. To anyone
|
||||
else it is a model that does not exist, absent from `/v1/models` too.
|
||||
- **It is public only by choice.** `--public` makes an adapter every
|
||||
caller's. It is refused for pairs from a knowledge base, so it is only for
|
||||
pairs written by hand.
|
||||
|
||||
## How it is kept apart
|
||||
|
||||
Passages live in PostgreSQL, partitioned by organization, with an index per
|
||||
|
||||
@@ -18,6 +18,7 @@ rather than repeated.
|
||||
| `name` | `string` | [Adapter](#adapter) |
|
||||
| `source` | `string` | [Adapter](#adapter) |
|
||||
| `scale` | `float64` | [Adapter](#adapter) |
|
||||
| `owner` | `string` | [Adapter](#adapter) |
|
||||
| `enabled` | `bool` | [Admission](#admission) |
|
||||
| `bands` | `[]Band` | [Admission](#admission) |
|
||||
| `default_band` | `string` | [Admission](#admission) |
|
||||
@@ -426,6 +427,16 @@ weights are.
|
||||
|
||||
Scale is how strongly it is applied; zero takes 1.
|
||||
|
||||
### `owner`
|
||||
|
||||
`string`
|
||||
|
||||
Owner is the part of the tenancy tree the adapter belongs to —
|
||||
org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
|
||||
it see it: to anyone else it is a model that does not exist. An
|
||||
adapter trained on an organisation's documents carries some of what
|
||||
they say in its weights. Empty is every caller the model allows.
|
||||
|
||||
## Admission
|
||||
|
||||
### `enabled`
|
||||
|
||||
Reference in new issue
Block a user