Ilker ManapandClaude Opus 5.5
79b3e9f887
A job names the catalog model the adapter is for, the safetensors weights to train on, the node and the MiB of a card it needs, and question/answer pairs. The control plane sends it when the node's agent is there and nothing else trains on it; the agent trains with its installed vLLM's torch and a pinned overlay of peft and accelerate (no second environment), as the engines' user, on the card with that much free — beside an engine, if it leaves that much — converts the adapter to GGUF with the installed llama.cpp's own conversion code when the model is served by llama.cpp, and sends what it made back in pieces, kept in the store to download. Progress, loss and peak card memory are reported as it goes; a node that goes fails its job. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
43 lines
1.7 KiB
SQL
43 lines
1.7 KiB
SQL
-- Training jobs (BACKLOG F4): a LoRA adapter trained on a node, from a base
|
|
-- model's weights and question/answer pairs, and what it made. The pairs
|
|
-- and the adapter are kept here, in the store every control plane reads:
|
|
-- they are megabytes, and a job's record is worth little without them.
|
|
CREATE TABLE train_jobs (
|
|
id TEXT PRIMARY KEY,
|
|
-- name is the adapter's: what clients will ask for it by.
|
|
name TEXT NOT NULL,
|
|
-- model is the catalog model the adapter is for; base_repo and
|
|
-- base_revision the weights it is trained on.
|
|
model TEXT NOT NULL,
|
|
base_repo TEXT NOT NULL,
|
|
base_revision TEXT NOT NULL DEFAULT '',
|
|
node TEXT NOT NULL,
|
|
gpu_memory_mib INTEGER NOT NULL,
|
|
params TEXT NOT NULL DEFAULT '',
|
|
gguf INTEGER NOT NULL DEFAULT 0,
|
|
data BYTEA NOT NULL,
|
|
pairs INTEGER NOT NULL,
|
|
-- state is queued, starting, preparing, running, done, failed or
|
|
-- cancelled.
|
|
state TEXT NOT NULL,
|
|
step INTEGER NOT NULL DEFAULT 0,
|
|
steps INTEGER NOT NULL DEFAULT 0,
|
|
loss DOUBLE PRECISION NOT NULL DEFAULT 0,
|
|
message TEXT NOT NULL DEFAULT '',
|
|
peak_mib INTEGER NOT NULL DEFAULT 0,
|
|
created_at BIGINT NOT NULL,
|
|
started_at BIGINT NOT NULL DEFAULT 0,
|
|
finished_at BIGINT NOT NULL DEFAULT 0,
|
|
created_by TEXT NOT NULL DEFAULT ''
|
|
);
|
|
CREATE INDEX train_jobs_by_state ON train_jobs (state);
|
|
|
|
-- What a job made: the PEFT adapter as a gzipped tar, and for llama.cpp the
|
|
-- adapter as GGUF.
|
|
CREATE TABLE train_artifacts (
|
|
job_id TEXT NOT NULL REFERENCES train_jobs(id) ON DELETE CASCADE,
|
|
kind TEXT NOT NULL,
|
|
data BYTEA NOT NULL,
|
|
PRIMARY KEY (job_id, kind)
|
|
);
|