Docs: one way in — quick-install starts at the setup screens, quickstart becomes development, install.md no longer says engines are yours to install
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4979e08d64
commit
6abd510628
5 files changed
+103
-72
No files matched your search
@@ -0,0 +1,376 @@
|
||||
# Development: llmgw from source on one machine
|
||||
|
||||
Written for a developer who wants to see llmgw answer a request on a single
|
||||
workstation, built from source, with no systemd and no `sudo` — for working on
|
||||
llmgw itself. **To install llmgw on servers, read
|
||||
[quick-install.md](quick-install.md) instead.**
|
||||
|
||||
What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
|
||||
of a real model served by llama.cpp.
|
||||
|
||||
## Two ways to run
|
||||
|
||||
Steps 1 to 4 use the simple one, **router-only**: you start the engines yourself
|
||||
and tell the router where they are with `--endpoint`. No certificates, no node
|
||||
agent.
|
||||
|
||||
Step 5 uses the **control plane and node agent**, the same architecture as a real
|
||||
cluster on one machine: the agent starts and supervises the engine, and the
|
||||
control plane places the model.
|
||||
|
||||
## 1. Build
|
||||
|
||||
```
|
||||
git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
|
||||
make build
|
||||
make test
|
||||
./bin/llmgw --version
|
||||
```
|
||||
|
||||
`make build` writes `./bin/llmgw`. Go 1.26.6 or later is the only requirement (an older Go fetches it itself).
|
||||
|
||||
## 2. Check the path with a fake engine
|
||||
|
||||
Before involving a GPU, prove the router works. `sim-engine` is an
|
||||
OpenAI-compatible fake that streams placeholder words.
|
||||
|
||||
```
|
||||
./bin/llmgw sim-engine --listen :8001 --model llama &
|
||||
./bin/llmgw server --endpoint e1=http://127.0.0.1:8001=llama &
|
||||
|
||||
curl -N localhost:8080/v1/chat/completions \
|
||||
-H 'content-type: application/json' \
|
||||
-H 'x-session-id: demo' \
|
||||
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
|
||||
```
|
||||
|
||||
Chunks of `alpha beta gamma ...` ending in `data: [DONE]` mean the router is
|
||||
fine. Stop both before continuing (see [Stopping](#stopping)).
|
||||
|
||||
## 3. Serve a real model
|
||||
|
||||
Two processes, two ports. Clients only ever talk to the router; the router
|
||||
knows where the engine is because you tell it with `--endpoint`:
|
||||
|
||||
```
|
||||
curl :8080 -> llmgw router :8080 -> llama serve :8082 -> model
|
||||
(client) (the gateway) (the engine)
|
||||
```
|
||||
|
||||
Requests go to 8080. Port 8082 is the engine's own, and you only touch it to
|
||||
check that the engine is up.
|
||||
|
||||
llmgw does not install engines or weights. You need:
|
||||
|
||||
- a `llama-server` build from llama.cpp (CUDA build if you want the GPU), and
|
||||
- a `.gguf` file.
|
||||
|
||||
Check the card first with `nvidia-smi`. An 8 GB card holds a 1B model at bf16
|
||||
(2.4 GB) or a 9B model at `q4_k_m` (5.4 GB).
|
||||
|
||||
Start the engine on its own port. The model name given to `--alias` is the name
|
||||
clients will use:
|
||||
|
||||
```
|
||||
llama serve \
|
||||
-m /path/to/Llama-3.2-1B-Instruct-BF16.gguf \
|
||||
--host 0.0.0.0 --port 8082 -ngl 99 --alias llama
|
||||
```
|
||||
|
||||
`-ngl 99` offloads every layer to the GPU. Confirm it is up and knows the model:
|
||||
|
||||
```
|
||||
curl -s localhost:8082/v1/models
|
||||
```
|
||||
|
||||
Then point the router at it. The format is `id=URL=model`:
|
||||
|
||||
```
|
||||
./bin/llmgw server --endpoint e1=http://127.0.0.1:8082=llama
|
||||
```
|
||||
|
||||
## 4. Ask it something
|
||||
|
||||
Streaming:
|
||||
|
||||
```
|
||||
curl -N localhost:8080/v1/chat/completions \
|
||||
-H 'content-type: application/json' \
|
||||
-H 'x-session-id: demo' \
|
||||
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
|
||||
```
|
||||
|
||||
One reply at once:
|
||||
|
||||
```
|
||||
curl -s localhost:8080/v1/chat/completions \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"max_tokens":100}'
|
||||
```
|
||||
|
||||
Also useful: `curl -s localhost:8080/v1/models` and
|
||||
`curl -s localhost:8080/health`. `llmgw ctl` does not work in this mode: it
|
||||
talks to the control plane's admin API, which router-only mode does not have,
|
||||
so `ctl status` answers `404`.
|
||||
|
||||
On the machine this was written on, a 1B bf16 model on an RTX 4070 laptop GPU
|
||||
answered at roughly 85 tokens per second.
|
||||
|
||||
## Several replicas of one model
|
||||
|
||||
Give the router more than one `--endpoint` with the same model name and it
|
||||
spreads requests across them. `x-session-id` keeps a conversation on the same
|
||||
replica so its cache is reused:
|
||||
|
||||
```
|
||||
./bin/llmgw server \
|
||||
--endpoint e1=http://127.0.0.1:8082=llama \
|
||||
--endpoint e2=http://127.0.0.1:8083=llama
|
||||
```
|
||||
|
||||
## Stopping
|
||||
|
||||
```
|
||||
pkill -f "llmgw server"
|
||||
pkill -f "llmgw sim-engine"
|
||||
```
|
||||
|
||||
Stop the engine the way you started it (Ctrl-C in its terminal).
|
||||
|
||||
## What goes wrong
|
||||
|
||||
**`listening on :8080: bind: address already in use`.** An earlier router is
|
||||
still running. `ss -ltnp | grep 8080` names the process; stop it first. This is
|
||||
the usual result of moving from step 2 to step 3 without stopping the fake
|
||||
engine's router.
|
||||
|
||||
**`/v1/models` on the engine returns an empty list.** The engine was started
|
||||
without `-m`, so it is a router with nothing to load. Restart it with a model.
|
||||
|
||||
**`error: nothing to serve`.** `llmgw server` with no `--endpoint` and no model
|
||||
carrying `profiles` has nothing to do. In router-only mode, always pass at least
|
||||
one `--endpoint`.
|
||||
|
||||
**Requests fail with a model-not-found error.** The name in the request must
|
||||
match the name after the last `=` in `--endpoint`, and it must be a name the
|
||||
engine actually serves (its `--alias`).
|
||||
|
||||
**Answers are poor.** A 1B model is a plumbing test, not a chat assistant. Swap
|
||||
in a larger `.gguf`; nothing else changes.
|
||||
|
||||
## 5. Let llmgw run the engine (control plane and node agent)
|
||||
|
||||
Steps 2 and 3 start the engine by hand. Here a **node agent** starts and
|
||||
supervises llama.cpp for you, and a control plane decides where the model runs.
|
||||
Still one machine, still no `sudo`. Stop everything from steps 2 and 3 first:
|
||||
the GPU has to be free and port 8080 unused.
|
||||
|
||||
```
|
||||
curl :8080 -> llmgw server (control plane + router) <- llmgw agent (this node)
|
||||
|
|
||||
starts llama-server :20000
|
||||
```
|
||||
|
||||
The agent dials the control plane over mutual TLS, so this needs a small private
|
||||
certificate authority. Everything goes in a scratch directory, `devcluster/`,
|
||||
which is in `.gitignore`.
|
||||
|
||||
### Certificates
|
||||
|
||||
```
|
||||
mkdir -p devcluster/pki
|
||||
./bin/llmgw ctl cert init-ca --dir devcluster/pki
|
||||
./bin/llmgw ctl cert issue-server llmgw-ctl --host 127.0.0.1 --host localhost \
|
||||
--dir devcluster/pki --out devcluster
|
||||
./bin/llmgw ctl cert issue-client llmgw-node01 --dir devcluster/pki --out devcluster
|
||||
```
|
||||
|
||||
The name given to `issue-client` becomes the certificate's common name and must
|
||||
equal `node_id` in the agent config. The control plane refuses an agent whose
|
||||
claimed id differs from its certificate.
|
||||
|
||||
### The engine wrapper
|
||||
|
||||
The agent starts the engine as `<bin> --model ... --port ...`. A `llama` binary
|
||||
that wants `llama serve ...` refuses every flag, so put a wrapper in front of it:
|
||||
|
||||
```
|
||||
cat > devcluster/llama-server <<'EOF'
|
||||
#!/bin/sh
|
||||
exec /path/to/llama serve "$@"
|
||||
EOF
|
||||
chmod +x devcluster/llama-server
|
||||
|
||||
./bin/llmgw ctl engine-check llamacpp --bin devcluster/llama-server
|
||||
```
|
||||
|
||||
`engine-check` should end with `every flag llmgw builds is accepted`. If your
|
||||
build already ships a `llama-server` binary, point `bin` at that and skip the
|
||||
wrapper.
|
||||
|
||||
### Control plane config: `devcluster/server.yaml`
|
||||
|
||||
```yaml
|
||||
listen: "127.0.0.1:8080"
|
||||
admin_listen: "127.0.0.1:9090"
|
||||
agent_listen: "127.0.0.1:7443"
|
||||
|
||||
agent_tls:
|
||||
ca: devcluster/pki/ca.pem
|
||||
cert: devcluster/server.pem
|
||||
key: devcluster/server-key.pem
|
||||
|
||||
models:
|
||||
- name: llama
|
||||
source: /path/to/Llama-3.2-1B-Instruct-BF16.gguf
|
||||
routing_profile: chat
|
||||
context_length: 8192
|
||||
scaling: { min: 1, max: 1 }
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: llamacpp
|
||||
gpu_class: 4070-8g
|
||||
weight: 1
|
||||
priority: 100
|
||||
startup_timeout: 5m
|
||||
```
|
||||
|
||||
Three things in there are easy to get wrong:
|
||||
|
||||
- **The model must have at least one profile.** That is what switches the control
|
||||
plane on. With `models: []` the server starts as a plain router and exits with
|
||||
`nothing to serve`.
|
||||
- **`gpu_class` decides CPU or GPU.** Without it the replica is placed in the CPU
|
||||
pool and runs on the CPU. Get the class name from `./bin/llmgw agent inventory`
|
||||
(the `CLASS` column).
|
||||
- **There is no `auth:` block.** Leave it out for a local trial; with auth on,
|
||||
requests need an API key.
|
||||
|
||||
### Agent config: `devcluster/agent.yaml`
|
||||
|
||||
```yaml
|
||||
node_id: llmgw-node01
|
||||
server: "https://127.0.0.1:7443"
|
||||
mgmt_addr: "127.0.0.1"
|
||||
|
||||
tls:
|
||||
ca: devcluster/pki/ca.pem
|
||||
cert: devcluster/llmgw-node01.pem
|
||||
key: devcluster/llmgw-node01-key.pem
|
||||
|
||||
engines:
|
||||
llamacpp:
|
||||
bin: /absolute/path/to/devcluster/llama-server
|
||||
|
||||
supervisor:
|
||||
work_dir: devcluster/agent-work
|
||||
```
|
||||
|
||||
### Run it
|
||||
|
||||
Two terminals:
|
||||
|
||||
```
|
||||
./bin/llmgw server --config devcluster/server.yaml
|
||||
./bin/llmgw agent --config devcluster/agent.yaml
|
||||
```
|
||||
|
||||
The model is not placed at once. The control plane waits about 35 seconds for the
|
||||
fleet to settle after it starts, then places the replica, and the agent then
|
||||
loads the weights. Watch it with:
|
||||
|
||||
```
|
||||
./bin/llmgw ctl nodes
|
||||
./bin/llmgw ctl models
|
||||
./bin/llmgw ctl replicas
|
||||
./bin/llmgw ctl explain-placement llama
|
||||
```
|
||||
|
||||
`STATUS ready` in `ctl models` means it is serving. `nvidia-smi` should now show
|
||||
the model in GPU memory. The same curl from step 4 works against port 8080.
|
||||
|
||||
`ctl` works here, unlike in router-only mode, because there is a control plane
|
||||
for it to talk to.
|
||||
|
||||
Stop it with Ctrl-C in each terminal. The agent leaves the engine running when
|
||||
the agent itself stops, so if you restart the agent and the GPU stays full,
|
||||
`pgrep -af "llama serve"` will show the engine still up.
|
||||
|
||||
## 6. Memory: conversations and notes
|
||||
|
||||
Off unless you ask. llmgw otherwise stores nothing anyone typed. On a single
|
||||
machine with no API keys, one block in the server config turns on two things:
|
||||
|
||||
```yaml
|
||||
threads:
|
||||
store: devcluster/data/threads.db
|
||||
retain_hours: 24
|
||||
```
|
||||
|
||||
Restart the server after adding it. `retain_hours` is required: everything kept
|
||||
expires that long after it was last written.
|
||||
|
||||
Everything kept this way sits in one shared pool with no owner, and anyone who
|
||||
can reach the port and knows an id can read it. That is fine on one machine and
|
||||
wrong on a shared network. With auth on, threads and notes are per project
|
||||
instead and the block is ignored; see [install.md](install.md).
|
||||
|
||||
### Conversations (threads)
|
||||
|
||||
Send the same `x-llmgw-thread` id and only the new turn. llmgw puts the stored
|
||||
history in front of it:
|
||||
|
||||
```
|
||||
curl -s localhost:8080/v1/chat/completions \
|
||||
-H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
|
||||
-d '{"model":"llama","messages":[{"role":"user","content":"My name is Deniz."}]}'
|
||||
|
||||
curl -s localhost:8080/v1/chat/completions \
|
||||
-H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
|
||||
-d '{"model":"llama","messages":[{"role":"user","content":"What is my name?"}]}'
|
||||
|
||||
curl -X DELETE localhost:8080/v1/threads/demo # forget it
|
||||
```
|
||||
|
||||
Without the block, a request carrying `x-llmgw-thread` is refused with
|
||||
`this gateway keeps no conversations` rather than answered without its history.
|
||||
|
||||
### Notes an agent keeps for itself
|
||||
|
||||
Notes outlive any one conversation. Each is a key, some text and optional tags,
|
||||
kept per agent; name the agent with `x-llmgw-agent` (default `default`), so two
|
||||
agents using the same key do not read each other's notes.
|
||||
|
||||
```
|
||||
curl -X PUT localhost:8080/v1/memory/favorite-color \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"text":"The user likes dark green.","tags":["preference"]}'
|
||||
|
||||
curl 'localhost:8080/v1/memory?q=green' # search by words
|
||||
curl 'localhost:8080/v1/memory?tag=preference' # or by tag
|
||||
curl localhost:8080/v1/memory/favorite-color
|
||||
curl -X DELETE localhost:8080/v1/memory/favorite-color
|
||||
```
|
||||
|
||||
The same notes are offered to an agent as tools over MCP at
|
||||
`http://localhost:8080/mcp`: `remember`, `recall` and `forget`. Add that URL to
|
||||
your agent's MCP server list (HTTP transport; how you set request headers such
|
||||
as `x-llmgw-agent` depends on the client). llmgw does not put these tools into
|
||||
chat requests on its own, so an agent that is not given the server never sees
|
||||
memory.
|
||||
|
||||
Limits: a key is up to 200 bytes, a note's text up to 16 KiB, at most 16 tags,
|
||||
1000 notes per agent. Tags are lowercase letters, digits, `-` and `_`.
|
||||
|
||||
## Where this stops
|
||||
|
||||
Router-only mode (steps 2 to 4) has no authentication, no placement and no model
|
||||
management: you start and restart engines yourself. Step 5 adds placement, model
|
||||
management and a node agent, and can add authentication as `install.md`
|
||||
describes. For a real cluster, with several nodes, systemd and a persistent
|
||||
certificate authority, follow [install.md](install.md).
|
||||
|
||||
One caveat found while writing this page: `install.md` recommends starting with
|
||||
an empty `models: []` and adding models later with `ctl model add`. The control
|
||||
plane only turns on when the config declares a model with `profiles`, so that
|
||||
first start does not produce a control plane.
|
||||
Reference in new issue
Block a user