llmgw/docs/quickstart.md

Quickstart: one machine, one model, ten minutes

Written for someone who wants to see llmgw answer a request on a single workstation before reading anything else. No systemd, no sudo. For a real cluster, see install.md.

What you end up with: an OpenAI-compatible endpoint on localhost:8080 in front of a real model served by llama.cpp.

Two ways to run

Steps 1 to 4 use the simple one, router-only: you start the engines yourself and tell the router where they are with --endpoint. No certificates, no node agent.

Step 5 uses the control plane and node agent, the same architecture as a real cluster on one machine: the agent starts and supervises the engine, and the control plane places the model.

1. Build

git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
make build
make test
./bin/llmgw --version

make build writes ./bin/llmgw. Go 1.26 or later is the only requirement.

2. Check the path with a fake engine

Before involving a GPU, prove the router works. sim-engine is an OpenAI-compatible fake that streams placeholder words.

./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw server --endpoint e1=http://127.0.0.1:8001=llama &

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'

Chunks of alpha beta gamma ... ending in data: [DONE] mean the router is fine. Stop both before continuing (see Stopping).

3. Serve a real model

Two processes, two ports. Clients only ever talk to the router; the router knows where the engine is because you tell it with --endpoint:

curl :8080  ->  llmgw router :8080  ->  llama serve :8082  ->  model
 (client)        (the gateway)           (the engine)

Requests go to 8080. Port 8082 is the engine's own, and you only touch it to check that the engine is up.

llmgw does not install engines or weights. You need:

• a llama-server build from llama.cpp (CUDA build if you want the GPU), and
• a .gguf file.

Check the card first with nvidia-smi. An 8 GB card holds a 1B model at bf16 (2.4 GB) or a 9B model at q4_k_m (5.4 GB).

Start the engine on its own port. The model name given to --alias is the name clients will use:

llama serve \
  -m /path/to/Llama-3.2-1B-Instruct-BF16.gguf \
  --host 0.0.0.0 --port 8082 -ngl 99 --alias llama

-ngl 99 offloads every layer to the GPU. Confirm it is up and knows the model:

curl -s localhost:8082/v1/models

Then point the router at it. The format is id=URL=model:

./bin/llmgw server --endpoint e1=http://127.0.0.1:8082=llama

4. Ask it something

Streaming:

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'

One reply at once:

curl -s localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"max_tokens":100}'

Also useful: curl -s localhost:8080/v1/models and curl -s localhost:8080/health. llmgw ctl does not work in this mode: it talks to the control plane's admin API, which router-only mode does not have, so ctl status answers 404.

On the machine this was written on, a 1B bf16 model on an RTX 4070 laptop GPU answered at roughly 85 tokens per second.

Several replicas of one model

Give the router more than one --endpoint with the same model name and it spreads requests across them. x-session-id keeps a conversation on the same replica so its cache is reused:

./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8082=llama \
  --endpoint e2=http://127.0.0.1:8083=llama

Stopping

pkill -f "llmgw server"
pkill -f "llmgw sim-engine"

Stop the engine the way you started it (Ctrl-C in its terminal).

What goes wrong

listening on :8080: bind: address already in use. An earlier router is still running. ss -ltnp | grep 8080 names the process; stop it first. This is the usual result of moving from step 2 to step 3 without stopping the fake engine's router.

/v1/models on the engine returns an empty list. The engine was started without -m, so it is a router with nothing to load. Restart it with a model.

error: nothing to serve. llmgw server with no --endpoint and no model carrying profiles has nothing to do. In router-only mode, always pass at least one --endpoint.

Requests fail with a model-not-found error. The name in the request must match the name after the last = in --endpoint, and it must be a name the engine actually serves (its --alias).

Answers are poor. A 1B model is a plumbing test, not a chat assistant. Swap in a larger .gguf; nothing else changes.

5. Let llmgw run the engine (control plane and node agent)

Steps 2 and 3 start the engine by hand. Here a node agent starts and supervises llama.cpp for you, and a control plane decides where the model runs. Still one machine, still no sudo. Stop everything from steps 2 and 3 first: the GPU has to be free and port 8080 unused.

curl :8080  ->  llmgw server (control plane + router)  <-  llmgw agent (this node)
                                                              |
                                                       starts llama-server :20000

The agent dials the control plane over mutual TLS, so this needs a small private certificate authority. Everything goes in a scratch directory, devcluster/, which is in .gitignore.

Certificates

mkdir -p devcluster/pki
./bin/llmgw ctl cert init-ca --dir devcluster/pki
./bin/llmgw ctl cert issue-server llmgw-ctl --host 127.0.0.1 --host localhost \
  --dir devcluster/pki --out devcluster
./bin/llmgw ctl cert issue-client llmgw-node01 --dir devcluster/pki --out devcluster

The name given to issue-client becomes the certificate's common name and must equal node_id in the agent config. The control plane refuses an agent whose claimed id differs from its certificate.

The engine wrapper

The agent starts the engine as <bin> --model ... --port .... A llama binary that wants llama serve ... refuses every flag, so put a wrapper in front of it:

cat > devcluster/llama-server <<'EOF'
#!/bin/sh
exec /path/to/llama serve "$@"
EOF
chmod +x devcluster/llama-server

./bin/llmgw ctl engine-check llamacpp --bin devcluster/llama-server

engine-check should end with every flag llmgw builds is accepted. If your build already ships a llama-server binary, point bin at that and skip the wrapper.

Control plane config: devcluster/server.yaml

listen: "127.0.0.1:8080"
admin_listen: "127.0.0.1:9090"
agent_listen: "127.0.0.1:7443"

agent_tls:
  ca: devcluster/pki/ca.pem
  cert: devcluster/server.pem
  key: devcluster/server-key.pem

models:
  - name: llama
    source: /path/to/Llama-3.2-1B-Instruct-BF16.gguf
    routing_profile: chat
    context_length: 8192
    scaling: { min: 1, max: 1 }
    profiles:
      - name: gpu
        engine: llamacpp
        gpu_class: 4070-8g
        weight: 1
        priority: 100
        startup_timeout: 5m

Three things in there are easy to get wrong:

• The model must have at least one profile. That is what switches the control plane on. With models: [] the server starts as a plain router and exits with nothing to serve.
• gpu_class decides CPU or GPU. Without it the replica is placed in the CPU pool and runs on the CPU. Get the class name from ./bin/llmgw agent inventory (the CLASS column).
• There is no auth: block. Leave it out for a local trial; with auth on, requests need an API key.

Agent config: devcluster/agent.yaml

node_id: llmgw-node01
server: "https://127.0.0.1:7443"
mgmt_addr: "127.0.0.1"

tls:
  ca: devcluster/pki/ca.pem
  cert: devcluster/llmgw-node01.pem
  key: devcluster/llmgw-node01-key.pem

engines:
  llamacpp:
    bin: /absolute/path/to/devcluster/llama-server

supervisor:
  work_dir: devcluster/agent-work

Run it

Two terminals:

./bin/llmgw server --config devcluster/server.yaml
./bin/llmgw agent  --config devcluster/agent.yaml

The model is not placed at once. The control plane waits about 35 seconds for the fleet to settle after it starts, then places the replica, and the agent then loads the weights. Watch it with:

./bin/llmgw ctl nodes
./bin/llmgw ctl models
./bin/llmgw ctl replicas
./bin/llmgw ctl explain-placement llama

STATUS ready in ctl models means it is serving. nvidia-smi should now show the model in GPU memory. The same curl from step 4 works against port 8080.

ctl works here, unlike in router-only mode, because there is a control plane for it to talk to.

Stop it with Ctrl-C in each terminal. The agent leaves the engine running when the agent itself stops, so if you restart the agent and the GPU stays full, pgrep -af "llama serve" will show the engine still up.

6. Memory: conversations and notes

Off unless you ask. llmgw otherwise stores nothing anyone typed. On a single machine with no API keys, one block in the server config turns on two things:

threads:
  store: devcluster/data/threads.db
  retain_hours: 24

Restart the server after adding it. retain_hours is required: everything kept expires that long after it was last written.

Everything kept this way sits in one shared pool with no owner, and anyone who can reach the port and knows an id can read it. That is fine on one machine and wrong on a shared network. With auth on, threads and notes are per project instead and the block is ignored; see install.md.

Conversations (threads)

Send the same x-llmgw-thread id and only the new turn. llmgw puts the stored history in front of it:

curl -s localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"My name is Deniz."}]}'

curl -s localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' -H 'x-llmgw-thread: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"What is my name?"}]}'

curl -X DELETE localhost:8080/v1/threads/demo     # forget it

Without the block, a request carrying x-llmgw-thread is refused with this gateway keeps no conversations rather than answered without its history.

Notes an agent keeps for itself

Notes outlive any one conversation. Each is a key, some text and optional tags, kept per agent; name the agent with x-llmgw-agent (default default), so two agents using the same key do not read each other's notes.

curl -X PUT localhost:8080/v1/memory/favorite-color \
  -H 'content-type: application/json' \
  -d '{"text":"The user likes dark green.","tags":["preference"]}'

curl 'localhost:8080/v1/memory?q=green'          # search by words
curl 'localhost:8080/v1/memory?tag=preference'   # or by tag
curl localhost:8080/v1/memory/favorite-color
curl -X DELETE localhost:8080/v1/memory/favorite-color

The same notes are offered to an agent as tools over MCP at http://localhost:8080/mcp: remember, recall and forget. Add that URL to your agent's MCP server list (HTTP transport; how you set request headers such as x-llmgw-agent depends on the client). llmgw does not put these tools into chat requests on its own, so an agent that is not given the server never sees memory.

Limits: a key is up to 200 bytes, a note's text up to 16 KiB, at most 16 tags, 1000 notes per agent. Tags are lowercase letters, digits, - and _.

Where this stops

Router-only mode (steps 2 to 4) has no authentication, no placement and no model management: you start and restart engines yourself. Step 5 adds placement, model management and a node agent, and can add authentication as install.md describes. For a real cluster, with several nodes, systemd and a persistent certificate authority, follow install.md.

One caveat found while writing this page: install.md recommends starting with an empty models: [] and adding models later with ctl model add. The control plane only turns on when the config declares a model with profiles, so that first start does not produce a control plane.