llmgw/docs/clients.md

Using llmgw from code

Written for a developer whose program talks to a model through llmgw. There is no llmgw SDK, and there will not be one: the gateway speaks OpenAI's API and Anthropic's Messages API, so the official SDKs for both work as they are. Only two things change — the base URL and the key. Everything llmgw adds on top is an HTTP header, which every one of those SDKs lets you set.

The examples use https://llm.example.com for the gateway; use the address your operator gave you, or the one on the gateway's own front page. Every example reads the key from LLMGW_API_KEY. <model-name> is a model this gateway serves: GET /v1/models lists the ones your key may use, and sending the placeholder unchanged is answered with that list rather than with an error.

Getting a key

Keys are made on the gateway's Account page after signing in, or issued by an operator. A key is shown once and never again — only its hash is kept — so put it in a password manager straight away. GET /me with the key says what it may do and what it has spent.

The base URL and the key

SDK	Base URL	Key
OpenAI (any language)	https://llm.example.com/v1	Authorization: Bearer …, as the SDK sends it
Anthropic (any language)	https://llm.example.com — the SDK adds /v1/messages	x-api-key: …, as the SDK sends it

Both SDKs also read their usual environment variables, so a program that already uses them needs no change at all:

export OPENAI_BASE_URL=https://llm.example.com/v1
export OPENAI_API_KEY=$LLMGW_API_KEY

export ANTHROPIC_BASE_URL=https://llm.example.com
export ANTHROPIC_API_KEY=$LLMGW_API_KEY

The API described

Every endpoint is described in OpenAPI 3.1 at /v1/openapi.json. It needs no key, so a client generator can be pointed at it. /docs/api shows it in Swagger UI, where Authorize takes a key and Try it out sends a request to the gateway. People with a part in running the gateway also find the admin API's document there.

A first request

curl

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>",
       "messages": [{"role": "user", "content": "Say hello in Turkish."}]}'

Python

pip install openai

import os
from openai import OpenAI

client = OpenAI(base_url="https://llm.example.com/v1",
                api_key=os.environ["LLMGW_API_KEY"])

reply = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Say hello in Turkish."}],
)
print(reply.choices[0].message.content)

TypeScript

npm install openai

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm.example.com/v1",
  apiKey: process.env.LLMGW_API_KEY,
});

const reply = await client.chat.completions.create({
  model: "<model-name>",
  messages: [{ role: "user", content: "Say hello in Turkish." }],
});
console.log(reply.choices[0].message.content);

Go

go get github.com/openai/openai-go/v3

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/openai/openai-go/v3"
	"github.com/openai/openai-go/v3/option"
)

func main() {
	client := openai.NewClient(
		option.WithBaseURL("https://llm.example.com/v1/"),
		option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
	)
	reply, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
		Model:    "<model-name>",
		Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Say hello in Turkish.")},
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(reply.Choices[0].Message.Content)
}

The import path carries the SDK's major version; use the one your module already has.

Streaming

Set stream and read the chunks. llmgw may send a last chunk that carries only the token counts and no choices, so check for a choice before reading one — the OpenAI API does the same when usage is asked for.

stream = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Count to ten."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

const stream = await client.chat.completions.create({
  model: "<model-name>",
  messages: [{ role: "user", content: "Count to ten." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

stream := client.Chat.Completions.NewStreaming(ctx, openai.ChatCompletionNewParams{
	Model:    "<model-name>",
	Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Count to ten.")},
})
for stream.Next() {
	chunk := stream.Current()
	if len(chunk.Choices) > 0 {
		fmt.Print(chunk.Choices[0].Delta.Content)
	}
}
if err := stream.Err(); err != nil {
	log.Fatal(err)
}

With curl, add "stream": true to the body and -N so curl prints the events as they come.

Tools

Function calling is OpenAI's: describe the tools, run the ones the model asks for, send the results back. Whether a model calls tools well is the model's business, not the gateway's; ask your operator which of the served models are set up for it.

Editors' agents need such a model. Continue's Agent mode, Cline and Codex act only through tool calls: with a model whose calls come back as text — <tool_call>{...} or a JSON block in the answer, tool_calls empty — they chat but never read a file or run a command. Qwen2.5-Coder is one; Qwen3-Coder is not. The configuration the portal writes for Continue says tool_use only for a model whose catalog entry says tools, and gives Continue the model's context and longest answer, so it never sends a request the gateway refuses.

import json

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "The current weather in a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

messages = [{"role": "user", "content": "Is it raining in Ankara?"}]
reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)
msg = reply.choices[0].message

if msg.tool_calls:
    messages.append(msg)
    for call in msg.tool_calls:
        args = json.loads(call.function.arguments)
        result = {"city": args["city"], "raining": False}  # your code here
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": json.dumps(result)})
    reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)

print(reply.choices[0].message.content)

Embeddings

result = client.embeddings.create(
    model="<embedding-model>",
    input=["llmgw routes requests", "to GPUs it can reach"],
)
vectors = [d.embedding for d in result.data]

const result = await client.embeddings.create({
  model: "<embedding-model>",
  input: ["llmgw routes requests", "to GPUs it can reach"],
});
const vectors = result.data.map((d) => d.embedding);

An embedding model is a model of its own; a chat model does not answer /v1/embeddings.

The Responses API

/v1/responses is served too, for clients that speak nothing else (Codex among them). llmgw translates it to chat completions, so it works with every model the gateway serves; what has no equivalent there — OpenAI's hosted tools, reasoning summaries — is ignored rather than refused.

response = client.responses.create(
    model="<model-name>",
    instructions="Answer in one sentence.",
    input="Why is the sky blue?",
)
print(response.output_text)

Streamed, the events are the Responses API's own:

stream = client.responses.create(model="<model-name>", input="Count to ten.", stream=True)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

Batches

For work that need not finish now: OpenAI's Files and Batch APIs. A batch runs at the lowest priority, so it uses what interactive traffic leaves free, and each line is counted against the key that submitted it.

import json

with open("requests.jsonl", "w") as f:
    for i, text in enumerate(["first document", "second document"]):
        f.write(json.dumps({
            "custom_id": f"doc-{i}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {"model": "<model-name>",
                     "messages": [{"role": "user", "content": "Summarise: " + text}]},
        }) + "\n")

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

# Later:
batch = client.batches.retrieve(batch.id)
if batch.status == "completed":
    for line in client.files.content(batch.output_file_id).text.splitlines():
        result = json.loads(line)
        print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])

The Anthropic SDK

/v1/messages is Anthropic's Messages API, translated to chat completions and back, streamed or not, with tools and images. It serves the gateway's own models — model is a model this gateway serves, not a Claude model.

pip install anthropic

import os
import anthropic

client = anthropic.Anthropic(base_url="https://llm.example.com",
                             api_key=os.environ["LLMGW_API_KEY"])

message = client.messages.create(
    model="<model-name>",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Say hello in Turkish."}],
)
print(message.content[0].text)

with client.messages.stream(
    model="<model-name>",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Count to ten."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  baseURL: "https://llm.example.com",
  apiKey: process.env.LLMGW_API_KEY,
});

const message = await client.messages.create({
  model: "<model-name>",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Say hello in Turkish." }],
});
for (const block of message.content) {
  if (block.type === "text") console.log(block.text);
}

client := anthropic.NewClient(
	option.WithBaseURL("https://llm.example.com/"),
	option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
)
message, err := client.Messages.New(ctx, anthropic.MessageNewParams{
	Model:     "<model-name>",
	MaxTokens: 1024,
	Messages: []anthropic.MessageParam{
		anthropic.NewUserMessage(anthropic.NewTextBlock("Say hello in Turkish.")),
	},
})
if err != nil {
	log.Fatal(err)
}
for _, block := range message.Content {
	if text, ok := block.AsAny().(anthropic.TextBlock); ok {
		fmt.Println(text.Text)
	}
}

(github.com/anthropics/anthropic-sdk-go and its option package.)

With curl:

curl https://llm.example.com/v1/messages \
  -H "x-api-key: $LLMGW_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "max_tokens": 1024,
       "messages": [{"role": "user", "content": "Say hello in Turkish."}]}'

What llmgw adds

Each is a header, so a request stays exactly what the SDK would send anywhere else. Set one for every request on the client, or for one request:

	Every request	One request
Python (both SDKs)	OpenAI(..., default_headers={...})	create(..., extra_headers={...})
TypeScript (both SDKs)	new OpenAI({ ..., defaultHeaders: {...} })	create({...}, { headers: {...} })
Go (both SDKs)	openai.NewClient(..., option.WithHeader(k, v))	New(ctx, params, option.WithHeader(k, v))
curl		-H "k: v"

Stored conversations: x-llmgw-thread

Send the same thread id with each request and only the new turn; llmgw puts the stored conversation in front of it. The conversation continues on whichever model serves the next turn, even a different one. It must be turned on for your project by an operator (llmgw keys threads); a request that names a thread on a gateway that keeps none is refused rather than answered without its history.

import uuid

thread = str(uuid.uuid4())  # anyone in your project who knows it reads it
for turn in ["My name is Deniz.", "What is my name?"]:
    reply = client.chat.completions.create(
        model="<model-name>",
        messages=[{"role": "user", "content": turn}],
        extra_headers={"x-llmgw-thread": thread},
    )
    print(reply.choices[0].message.content)

A thread belongs to the project, not to the key, so use an id nobody can guess: at most 128 letters, digits and - _ . :. When the conversation no longer fits the model, the oldest turns are left out and the reply carries x-llmgw-thread-trimmed with how many. DELETE /v1/threads/{id} forgets one:

curl -X DELETE https://llm.example.com/v1/threads/$THREAD \
  -H "Authorization: Bearer $LLMGW_API_KEY"

Sessions: x-session-id

Requests with the same session id go to the same replica while it is healthy, so a conversation that resends its history finds it already in that replica's cache and answers sooner. Without the header, the body's user field is used. Nothing is stored; it is only a routing hint.

client = OpenAI(base_url="https://llm.example.com/v1", api_key=os.environ["LLMGW_API_KEY"],
                default_headers={"x-session-id": "chat-7f3a"})

Tags and metadata

Requests can be labelled so the usage they record can be told apart — by feature, by customer, by job. Send OpenAI's own metadata, string keys to string values, or the same as the x-llmgw-tags header for a client that cannot set metadata; the user field is recorded too. Which tags also become metric labels is the operator's choice (router.tag_labels).

reply = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Classify this ticket."}],
    metadata={"feature": "triage", "customer": "acme"},
    user="user-1234",
)

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "x-llmgw-tags: feature=triage,customer=acme" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "Classify this ticket."}]}'

Canaries: x-llmgw-variant

While an operator tries a model's newer weights on a share of its traffic, a session stays on one side. x-llmgw-variant: canary sends a request to the canary deliberately — to try it before it is promoted, or to compare the two.

reply = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "hello"}],
    extra_headers={"x-llmgw-variant": "canary"},
)

Memory notes: /v1/memory

Notes an agent keeps for itself across conversations: a key, some text, and optional tags, private to your project and to the agent named in x-llmgw-agent (default when it is not sent). An operator turns them on for a project (llmgw keys memory). They are plain REST, so any HTTP client will do:

H="Authorization: Bearer $LLMGW_API_KEY"

curl -X PUT https://llm.example.com/v1/memory/favourite-colour -H "$H" \
  -H "x-llmgw-agent: helper" -H "Content-Type: application/json" \
  -d '{"text": "The user likes dark green.", "tags": ["preference"]}'

curl "https://llm.example.com/v1/memory?q=green" -H "$H" -H "x-llmgw-agent: helper"
curl "https://llm.example.com/v1/memory?tag=preference" -H "$H" -H "x-llmgw-agent: helper"
curl https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"
curl -X DELETE https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"

A key is up to 200 bytes, a note up to 16 KiB, with at most 16 tags of lowercase letters, digits, - and _, and 1000 notes per agent.

MCP: /mcp

The same notes as tools an agent can be given — remember, recall and forget — from an MCP server at https://llm.example.com/mcp (the HTTP transport, answered as plain JSON). Add it to your agent's MCP servers with the key as a bearer token and, if you want notes kept apart, x-llmgw-agent. llmgw does not put these tools into requests by itself: an agent that is not given the server never sees its notes.

curl https://llm.example.com/mcp \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

Headers llmgw sends back

Header	Says
x-request-id	The request's id in the gateway's logs and usage records — quote it when asking an operator about a request.
x-llmgw-served-model	The model that answered, when it was not the one asked for: a spent budget can move a request to a cheaper model.
x-llmgw-queue-ms	How long the request waited for room before it was sent.
x-llmgw-thread-trimmed	How many stored turns were left out to fit the model.

When something is refused

Errors come back in the shape of the API that was called — OpenAI's {"error": {"message": …, "type": …}} or Anthropic's — so each SDK raises its usual exception with the gateway's message in it. The ones worth knowing:

Status	Usually means
400	Besides a malformed request: a prompt too long for the model, or a guardrail policy refused it (the code content_policy_violation).
401	No key, a mistyped one, or one that was revoked or has expired.
403	The key may not use that model.
429	A rate limit or budget of the key, its project or its team was reached. Retry-After says when to try again, and is absent when waiting will not help — a spent monthly budget.
503	No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be.
504	Every replica was busy for as long as the key's band may wait. Retry-After says when to send it again.

Priority

With keys on, the key decides the request's priority band and routing profile, set by an operator; the x-llmgw-band and x-llmgw-profile headers are ignored, so a client cannot promote itself. On a gateway without keys they are read from the request.