llmgw/docs/clients.md

# Using llmgw from code

Written for a developer whose program talks to a model through llmgw. There
is no llmgw SDK, and there will not be one: the gateway speaks OpenAI's API
and Anthropic's Messages API, so the official SDKs for both work as they are.
Only two things change — the base URL and the key. Everything llmgw adds on
top is an HTTP header, which every one of those SDKs lets you set.

The examples use `https://llm.example.com` for the gateway; use the address
your operator gave you, or the one on the gateway's own front page. Every
example reads the key from `LLMGW_API_KEY`. `<model-name>` is a model this
gateway serves: `GET /v1/models` lists the ones your key may use, and sending
the placeholder unchanged is answered with that list rather than with an
error.

## Getting a key

Keys are made on the gateway's **Account** page after signing in, or issued
by an operator. A key is shown once and never again — only its hash is kept —
so put it in a password manager straight away. `GET /me` with the key says
what it may do and what it has spent.

## The base URL and the key

| SDK | Base URL | Key |
|---|---|---|
| OpenAI (any language) | `https://llm.example.com/v1` | `Authorization: Bearer …`, as the SDK sends it |
| Anthropic (any language) | `https://llm.example.com` — the SDK adds `/v1/messages` | `x-api-key: …`, as the SDK sends it |

Both SDKs also read their usual environment variables, so a program that
already uses them needs no change at all:

```
export OPENAI_BASE_URL=https://llm.example.com/v1
export OPENAI_API_KEY=$LLMGW_API_KEY

export ANTHROPIC_BASE_URL=https://llm.example.com
export ANTHROPIC_API_KEY=$LLMGW_API_KEY
```

### The API described

Every endpoint is described in OpenAPI 3.1 at `/v1/openapi.json`. It needs
no key, so a client generator can be pointed at it. `/docs/api` shows it in
Swagger UI, where *Authorize* takes a key and *Try it out* sends a request to
the gateway. People with a part in running the gateway also find the admin
API's document there.

## A first request

### curl

```
curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>",
       "messages": [{"role": "user", "content": "Say hello in Turkish."}]}'
```

### Python

```
pip install openai
```

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://llm.example.com/v1",
                api_key=os.environ["LLMGW_API_KEY"])

reply = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Say hello in Turkish."}],
)
print(reply.choices[0].message.content)
```

### TypeScript

```
npm install openai
```

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm.example.com/v1",
  apiKey: process.env.LLMGW_API_KEY,
});

const reply = await client.chat.completions.create({
  model: "<model-name>",
  messages: [{ role: "user", content: "Say hello in Turkish." }],
});
console.log(reply.choices[0].message.content);
```

### Go

```
go get github.com/openai/openai-go/v3
```

```go
package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/openai/openai-go/v3"
	"github.com/openai/openai-go/v3/option"
)

func main() {
	client := openai.NewClient(
		option.WithBaseURL("https://llm.example.com/v1/"),
		option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
	)
	reply, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
		Model:    "<model-name>",
		Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Say hello in Turkish.")},
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(reply.Choices[0].Message.Content)
}
```

The import path carries the SDK's major version; use the one your module
already has.

## Streaming

Set `stream` and read the chunks. llmgw may send a last chunk that carries
only the token counts and no choices, so check for a choice before reading
one — the OpenAI API does the same when usage is asked for.

```python
stream = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Count to ten."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

```typescript
const stream = await client.chat.completions.create({
  model: "<model-name>",
  messages: [{ role: "user", content: "Count to ten." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```

```go
stream := client.Chat.Completions.NewStreaming(ctx, openai.ChatCompletionNewParams{
	Model:    "<model-name>",
	Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Count to ten.")},
})
for stream.Next() {
	chunk := stream.Current()
	if len(chunk.Choices) > 0 {
		fmt.Print(chunk.Choices[0].Delta.Content)
	}
}
if err := stream.Err(); err != nil {
	log.Fatal(err)
}
```

With curl, add `"stream": true` to the body and `-N` so curl prints the
events as they come.

## Tools

Function calling is OpenAI's: describe the tools, run the ones the model
asks for, send the results back. Whether a model calls tools well is the
model's business, not the gateway's; ask your operator which of the served
models are set up for it.

**Editors' agents need such a model.** Continue's Agent mode, Cline and
Codex act only through tool calls: with a model whose calls come back as
text — `<tool_call>{...}` or a JSON block in the answer, `tool_calls` empty —
they chat but never read a file or run a command. Qwen2.5-Coder is one;
Qwen3-Coder is not. The configuration the portal writes for Continue says
`tool_use` only for a model whose catalog entry says tools, and gives
Continue the model's context and longest answer, so it never sends a request
the gateway refuses.

```python
import json

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "The current weather in a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

messages = [{"role": "user", "content": "Is it raining in Ankara?"}]
reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)
msg = reply.choices[0].message

if msg.tool_calls:
    messages.append(msg)
    for call in msg.tool_calls:
        args = json.loads(call.function.arguments)
        result = {"city": args["city"], "raining": False}  # your code here
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": json.dumps(result)})
    reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)

print(reply.choices[0].message.content)
```

## Embeddings

```python
result = client.embeddings.create(
    model="<embedding-model>",
    input=["llmgw routes requests", "to GPUs it can reach"],
)
vectors = [d.embedding for d in result.data]
```

```typescript
const result = await client.embeddings.create({
  model: "<embedding-model>",
  input: ["llmgw routes requests", "to GPUs it can reach"],
});
const vectors = result.data.map((d) => d.embedding);
```

An embedding model is a model of its own; a chat model does not answer
`/v1/embeddings`.

## The Responses API

`/v1/responses` is served too, for clients that speak nothing else (Codex
among them). llmgw translates it to chat completions, so it works with every
model the gateway serves; what has no equivalent there — OpenAI's hosted
tools, reasoning summaries — is ignored rather than refused.

```python
response = client.responses.create(
    model="<model-name>",
    instructions="Answer in one sentence.",
    input="Why is the sky blue?",
)
print(response.output_text)
```

Streamed, the events are the Responses API's own:

```python
stream = client.responses.create(model="<model-name>", input="Count to ten.", stream=True)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
```

## Batches

For work that need not finish now: OpenAI's Files and Batch APIs. A batch runs
at the lowest priority, so it uses what interactive traffic leaves free, and
each line is counted against the key that submitted it.

```python
import json

with open("requests.jsonl", "w") as f:
    for i, text in enumerate(["first document", "second document"]):
        f.write(json.dumps({
            "custom_id": f"doc-{i}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {"model": "<model-name>",
                     "messages": [{"role": "user", "content": "Summarise: " + text}]},
        }) + "\n")

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

# Later:
batch = client.batches.retrieve(batch.id)
if batch.status == "completed":
    for line in client.files.content(batch.output_file_id).text.splitlines():
        result = json.loads(line)
        print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])
```

## The Anthropic SDK

`/v1/messages` is Anthropic's Messages API, translated to chat completions and
back, streamed or not, with tools and images. It serves the gateway's own
models — `model` is a model this gateway serves, not a Claude model.

```
pip install anthropic
```

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://llm.example.com",
                             api_key=os.environ["LLMGW_API_KEY"])

message = client.messages.create(
    model="<model-name>",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Say hello in Turkish."}],
)
print(message.content[0].text)

with client.messages.stream(
    model="<model-name>",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Count to ten."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
```

```typescript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  baseURL: "https://llm.example.com",
  apiKey: process.env.LLMGW_API_KEY,
});

const message = await client.messages.create({
  model: "<model-name>",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Say hello in Turkish." }],
});
for (const block of message.content) {
  if (block.type === "text") console.log(block.text);
}
```

```go
client := anthropic.NewClient(
	option.WithBaseURL("https://llm.example.com/"),
	option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
)
message, err := client.Messages.New(ctx, anthropic.MessageNewParams{
	Model:     "<model-name>",
	MaxTokens: 1024,
	Messages: []anthropic.MessageParam{
		anthropic.NewUserMessage(anthropic.NewTextBlock("Say hello in Turkish.")),
	},
})
if err != nil {
	log.Fatal(err)
}
for _, block := range message.Content {
	if text, ok := block.AsAny().(anthropic.TextBlock); ok {
		fmt.Println(text.Text)
	}
}
```

(`github.com/anthropics/anthropic-sdk-go` and its `option` package.)

With curl:

```
curl https://llm.example.com/v1/messages \
  -H "x-api-key: $LLMGW_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "max_tokens": 1024,
       "messages": [{"role": "user", "content": "Say hello in Turkish."}]}'
```

## What llmgw adds

Each is a header, so a request stays exactly what the SDK would send anywhere
else. Set one for every request on the client, or for one request:

| | Every request | One request |
|---|---|---|
| Python (both SDKs) | `OpenAI(..., default_headers={...})` | `create(..., extra_headers={...})` |
| TypeScript (both SDKs) | `new OpenAI({ ..., defaultHeaders: {...} })` | `create({...}, { headers: {...} })` |
| Go (both SDKs) | `openai.NewClient(..., option.WithHeader(k, v))` | `New(ctx, params, option.WithHeader(k, v))` |
| curl | | `-H "k: v"` |

### Stored conversations: `x-llmgw-thread`

Send the same thread id with each request and only the new turn; llmgw puts
the stored conversation in front of it. The conversation continues on
whichever model serves the next turn, even a different one. It must be
turned on for your project by an operator (`llmgw keys threads`); a request
that names a thread on a gateway that keeps none is refused rather than
answered without its history.

```python
import uuid

thread = str(uuid.uuid4())  # anyone in your project who knows it reads it
for turn in ["My name is Deniz.", "What is my name?"]:
    reply = client.chat.completions.create(
        model="<model-name>",
        messages=[{"role": "user", "content": turn}],
        extra_headers={"x-llmgw-thread": thread},
    )
    print(reply.choices[0].message.content)
```

A thread belongs to the project, not to the key, so use an id nobody can
guess: at most 128 letters, digits and `- _ . :`. When the conversation no
longer fits the model, the oldest turns are left out and the reply carries
`x-llmgw-thread-trimmed` with how many. `DELETE /v1/threads/{id}` forgets one:

```
curl -X DELETE https://llm.example.com/v1/threads/$THREAD \
  -H "Authorization: Bearer $LLMGW_API_KEY"
```

### Sessions: `x-session-id`

Requests with the same session id go to the same replica while it is
healthy, so a conversation that resends its history finds it already in that
replica's cache and answers sooner. Without the header, the body's `user`
field is used. Nothing is stored; it is only a routing hint.

```python
client = OpenAI(base_url="https://llm.example.com/v1", api_key=os.environ["LLMGW_API_KEY"],
                default_headers={"x-session-id": "chat-7f3a"})
```

### Tags and metadata

Requests can be labelled so the usage they record can be told apart — by
feature, by customer, by job. Send OpenAI's own `metadata`, string keys to
string values, or the same as the `x-llmgw-tags` header for a client that
cannot set `metadata`; the `user` field is recorded too. Which tags also
become metric labels is the operator's choice (`router.tag_labels`).

```python
reply = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "Classify this ticket."}],
    metadata={"feature": "triage", "customer": "acme"},
    user="user-1234",
)
```

```
curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "x-llmgw-tags: feature=triage,customer=acme" \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "Classify this ticket."}]}'
```

### Canaries: `x-llmgw-variant`

While an operator tries a model's newer weights on a share of its traffic, a
session stays on one side. `x-llmgw-variant: canary` sends a request to the
canary deliberately — to try it before it is promoted, or to compare the two.

```python
reply = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "hello"}],
    extra_headers={"x-llmgw-variant": "canary"},
)
```

### Memory notes: `/v1/memory`

Notes an agent keeps for itself across conversations: a key, some text, and
optional tags, private to your project and to the agent named in
`x-llmgw-agent` (`default` when it is not sent). An operator turns them on for
a project (`llmgw keys memory`). They are plain REST, so any HTTP client
will do:

```
H="Authorization: Bearer $LLMGW_API_KEY"

curl -X PUT https://llm.example.com/v1/memory/favourite-colour -H "$H" \
  -H "x-llmgw-agent: helper" -H "Content-Type: application/json" \
  -d '{"text": "The user likes dark green.", "tags": ["preference"]}'

curl "https://llm.example.com/v1/memory?q=green" -H "$H" -H "x-llmgw-agent: helper"
curl "https://llm.example.com/v1/memory?tag=preference" -H "$H" -H "x-llmgw-agent: helper"
curl https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"
curl -X DELETE https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"
```

A key is up to 200 bytes, a note up to 16 KiB, with at most 16 tags of
lowercase letters, digits, `-` and `_`, and 1000 notes per agent.

### MCP: `/mcp`

The same notes as tools an agent can be given — `remember`, `recall` and
`forget` — from an MCP server at `https://llm.example.com/mcp` (the HTTP
transport, answered as plain JSON). Add it to your agent's MCP servers with
the key as a bearer token and, if you want notes kept apart, `x-llmgw-agent`.
llmgw does not put these tools into requests by itself: an agent that is not
given the server never sees its notes.

```
curl https://llm.example.com/mcp \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
```

### Headers llmgw sends back

| Header | Says |
|---|---|
| `x-request-id` | The request's id in the gateway's logs and usage records — quote it when asking an operator about a request. |
| `x-llmgw-served-model` | The model that answered, when it was not the one asked for: a spent budget can move a request to a cheaper model. |
| `x-llmgw-queue-ms` | How long the request waited for room before it was sent. |
| `x-llmgw-thread-trimmed` | How many stored turns were left out to fit the model. |

## When something is refused

Errors come back in the shape of the API that was called — OpenAI's
`{"error": {"message": …, "type": …}}` or Anthropic's — so each SDK raises
its usual exception with the gateway's message in it. The ones worth knowing:

| Status | Usually means |
|---|---|
| 400 | Besides a malformed request: a prompt too long for the model, or a guardrail policy refused it (the code `content_policy_violation`). |
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
| 403 | The key may not use that model. |
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |

## Priority

With keys on, the key decides the request's priority band and routing
profile, set by an operator; the `x-llmgw-band` and `x-llmgw-profile` headers
are ignored, so a client cannot promote itself. On a gateway without keys
they are read from the request.