llmgw/docs/quickstart.md

# Quickstart: one machine, one model, ten minutes

Written for a developer who wants to see llmgw answer a request on a single
workstation, built from source, with no systemd and no `sudo`. **To install
llmgw on servers, read [quick-install.md](quick-install.md) instead.**

What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front
of a real model served by llama.cpp.

## Two ways to run

Steps 1 to 4 use the simple one, **router-only**: you start the engines yourself
and tell the router where they are with `--endpoint`. No certificates, no node
agent.

Step 5 uses the **control plane and node agent**, the same architecture as a real
cluster on one machine: the agent starts and supervises the engine, and the
control plane places the model.

## 1. Build

```
git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
make build
make test
./bin/llmgw --version
```

`make build` writes `./bin/llmgw`. Go 1.26 or later is the only requirement.

## 2. Check the path with a fake engine

Before involving a GPU, prove the router works. `sim-engine` is an
OpenAI-compatible fake that streams placeholder words.

```
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw server --endpoint e1=http://127.0.0.1:8001=llama &

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
```

Chunks of `alpha beta gamma ...` ending in `data: [DONE]` mean the router is
fine. Stop both before continuing (see [Stopping](#stopping)).

## 3. Serve a real model

Two processes, two ports. Clients only ever talk to the router; the router
knows where the engine is because you tell it with `--endpoint`:

```
curl :8080  ->  llmgw router :8080  ->  llama serve :8082  ->  model
 (client)        (the gateway)           (the engine)
```

Requests go to 8080. Port 8082 is the engine's own, and you only touch it to
check that the engine is up.

llmgw does not install engines or weights. You need:

- a `llama-server` build from llama.cpp (CUDA build if you want the GPU), and
- a `.gguf` file.

Check the card first with `nvidia-smi`. An 8 GB card holds a 1B model at bf16
(2.4 GB) or a 9B model at `q4_k_m` (5.4 GB).

Start the engine on its own port. The model name given to `--alias` is the name
clients will use:

```
llama serve \
  -m /path/to/Llama-3.2-1B-Instruct-BF16.gguf \
  --host 0.0.0.0 --port 8082 -ngl 99 --alias llama
```

`-ngl 99` offloads every layer to the GPU. Confirm it is up and knows the model:

```
curl -s localhost:8082/v1/models
```

Then point the router at it. The format is `id=URL=model`:

```
./bin/llmgw server --endpoint e1=http://127.0.0.1:8082=llama
```

## 4. Ask it something

Streaming:

```
curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
```

One reply at once:

```
curl -s localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"max_tokens":100}'
```

Also useful: `curl -s localhost:8080/v1/models` and
`curl -s localhost:8080/health`. `llmgw ctl` does not work in this mode: it
talks to the control plane's admin API, which router-only mode does not have,
so `ctl status` answers `404`.

On the machine this was written on, a 1B bf16 model on an RTX 4070 laptop GPU
answered at roughly 85 tokens per second.

## Several replicas of one model

Give the router more than one `--endpoint` with the same model name and it
spreads requests across them. `x-session-id` keeps a conversation on the same
replica so its cache is reused:

```
./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8082=llama \
  --endpoint e2=http://127.0.0.1:8083=llama
```

## Stopping

```
pkill -f "llmgw server"
pkill -f "llmgw sim-engine"
```

Stop the engine the way you started it (Ctrl-C in its terminal).

## What goes wrong

**`listening on :8080: bind: address already in use`.** An earlier router is
still running. `ss -ltnp | grep 8080` names the process; stop it first. This is
the usual result of moving from step 2 to step 3 without stopping the fake
engine's router.

**`/v1/models` on the engine returns an empty list.** The engine was started
without `-m`, so it is a router with nothing to load. Restart it with a model.

**`error: nothing to serve`.** `llmgw server` with no `--endpoint` and no model
carrying `profiles` has nothing to do. In router-only mode, always pass at least
one `--endpoint`.

**Requests fail with a model-not-found error.** The name in the request must
match the name after the last `=` in `--endpoint`, and it must be a name the
engine actually serves (its `--alias`).

**Answers are poor.** A 1B model is a plumbing test, not a chat assistant. Swap
in a larger `.gguf`; nothing else changes.

## 5. Let llmgw run the engine (control plane and node agent)

Steps 2 and 3 start the engine by hand. Here a **node agent** starts and
supervises llama.cpp for you, and a control plane decides where the model runs.
Still one machine, still no `sudo`. Stop everything from steps 2 and 3 first:
the GPU has to be free and port 8080 unused.

```
curl :8080  ->  llmgw server (control plane + router)  <-  llmgw agent (this node)
                                                              |
                                                       starts llama-server :20000
```

The agent dials the control plane over mutual TLS, so this needs a small private
certificate authority. Everything goes in a scratch directory, `devcluster/`,
which is in `.gitignore`.

### Certificates

```
mkdir -p devcluster/pki
./bin/llmgw ctl cert init-ca --dir devcluster/pki
./bin/llmgw ctl cert issue-server llmgw-ctl --host 127.0.0.1 --host localhost \
  --dir devcluster/pki --out devcluster
./bin/llmgw ctl cert issue-client llmgw-node01 --dir devcluster/pki --out devcluster
```

The name given to `issue-client` becomes the certificate's common name and must
equal `node_id` in the agent config. The control plane refuses an agent whose
claimed id differs from its certificate.

### The engine wrapper

The agent starts the engine as `<bin> --model ... --port ...`. A `llama` binary
that wants `llama serve ...` refuses every flag, so put a wrapper in front of it:

```
cat > devcluster/llama-server <<'EOF'
#!/bin/sh
exec /path/to/llama serve "$@"
EOF
chmod +x devcluster/llama-server

./bin/llmgw ctl engine-check llamacpp --bin devcluster/llama-server
```

`engine-check` should end with `every flag llmgw builds is accepted`. If your
build already ships a `llama-server` binary, point `bin` at that and skip the
wrapper.

### Control plane config: `devcluster/server.yaml`

```yaml
listen: "127.0.0.1:8080"
admin_listen: "127.0.0.1:9090"
agent_listen: "127.0.0.1:7443"

agent_tls:
  ca: devcluster/pki/ca.pem
  cert: devcluster/server.pem
  key: devcluster/server-key.pem

models:
  - name: llama
    source: /path/to/Llama-3.2-1B-Instruct-BF16.gguf
    routing_profile: chat
    context_length: 8192
    scaling: { min: 1, max: 1 }
    profiles:
      - name: gpu
        engine: llamacpp
        gpu_class: 4070-8g
        weight: 1
        priority: 100
        startup_timeout: 5m
```

Three things in there are easy to get wrong:

- **The model must have at least one profile.** That is what switches the control
  plane on. With `models: []` the server starts as a plain router and exits with
  `nothing to serve`.
- **`gpu_class` decides CPU or GPU.** Without it the replica is placed in the CPU
  pool and runs on the CPU. Get the class name from `./bin/llmgw agent inventory`
  (the `CLASS` column).
- **There is no `auth:` block.** Leave it out for a local trial; with auth on,
  requests need an API key.

### Agent config: `devcluster/agent.yaml`

```yaml
node_id: llmgw-node01
server: "https://127.0.0.1:7443"
mgmt_addr: "127.0.0.1"

tls:
  ca: devcluster/pki/ca.pem
  cert: devcluster/llmgw-node01.pem
  key: devcluster/llmgw-node01-key.pem

engines:
  llamacpp:
    bin: /absolute/path/to/devcluster/llama-server

supervisor:
  work_dir: devcluster/agent-work
```

### Run it

Two terminals:

```
./bin/llmgw server --config devcluster/server.yaml
./bin/llmgw agent  --config devcluster/agent.yaml
```

The model is not placed at once. The control plane waits about 35 seconds for the
fleet to settle after it starts, then places the replica, and the agent then
loads the weights. Watch it with:

```
./bin/llmgw ctl nodes
./bin/llmgw ctl models
./bin/llmgw ctl replicas
./bin/llmgw ctl explain-placement llama
```

`STATUS ready` in `ctl models` means it is serving. `nvidia-smi` should now show
the model in GPU memory. The same curl from step 4 works against port 8080.

`ctl` works here, unlike in router-only mode, because there is a control plane
for it to talk to.

Stop it with Ctrl-C in each terminal. The agent leaves the engine running when
the agent itself stops, so if you restart the agent and the GPU stays full,
`pgrep -af "llama serve"` will show the engine still up.

## 6. Memory: conversations and notes

Off unless you ask. llmgw otherwise stores nothing anyone typed. On a single
machine with no API keys, one block in the server config turns on two things:

```yaml
threads:
  store: devcluster/data/threads.db
  retain_hours: 24
```

Restart the server after adding it. `retain_hours` is required: everything kept
expires that long after it was last written.

Everything kept this way sits in one shared pool with no owner, and anyone who
can reach the port and knows an id can read it. That is fine on one machine and
wrong on a shared network. With auth on, threads and notes are per project
instead and the block is ignored; see [install.md](install.md).

### Conversations (threads)

Send the same `x-llmgw-thread` id and only the new turn. llmgw puts the stored
history in front of it:

```
curl -s localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"My name is Deniz."}]}'

curl -s localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"What is my name?"}]}'

curl -X DELETE localhost:8080/v1/threads/demo     # forget it
```

Without the block, a request carrying `x-llmgw-thread` is refused with
`this gateway keeps no conversations` rather than answered without its history.

### Notes an agent keeps for itself

Notes outlive any one conversation. Each is a key, some text and optional tags,
kept per agent; name the agent with `x-llmgw-agent` (default `default`), so two
agents using the same key do not read each other's notes.

```
curl -X PUT localhost:8080/v1/memory/favorite-color \
  -H 'content-type: application/json' \
  -d '{"text":"The user likes dark green.","tags":["preference"]}'

curl 'localhost:8080/v1/memory?q=green'          # search by words
curl 'localhost:8080/v1/memory?tag=preference'   # or by tag
curl localhost:8080/v1/memory/favorite-color
curl -X DELETE localhost:8080/v1/memory/favorite-color
```

The same notes are offered to an agent as tools over MCP at
`http://localhost:8080/mcp`: `remember`, `recall` and `forget`. Add that URL to
your agent's MCP server list (HTTP transport; how you set request headers such
as `x-llmgw-agent` depends on the client). llmgw does not put these tools into
chat requests on its own, so an agent that is not given the server never sees
memory.

Limits: a key is up to 200 bytes, a note's text up to 16 KiB, at most 16 tags,
1000 notes per agent. Tags are lowercase letters, digits, `-` and `_`.

## Where this stops

Router-only mode (steps 2 to 4) has no authentication, no placement and no model
management: you start and restart engines yourself. Step 5 adds placement, model
management and a node agent, and can add authentication as `install.md`
describes. For a real cluster, with several nodes, systemd and a persistent
certificate authority, follow [install.md](install.md).

One caveat found while writing this page: `install.md` recommends starting with
an empty `models: []` and adding models later with `ctl model add`. The control
plane only turns on when the config declares a model with `profiles`, so that
first start does not produce a control plane.