ilker/llmgw
2
0 Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 95 Wiki Activity

95 Releases 95 Tags

RSS Feed
  • v0.15.0 6752938e6d
    Compare

    v0.15.0
    CI / test (push) Successful in 9m0s
    Details
    Release / build (push) Successful in 4m23s
    Details
    Release / notes (push) Successful in 3s
    Details
    Stable

    gitea-actions released this 2026-10-25 18:17:57 +01:00 | 116 commits to main since this release

    The company's tools and editors through the gateway, and guardrails that
    can lean on others. Its own MCP servers sit behind llmgw's /mcp under the
    guardrails, a rate and a breaker. VS Code gets the gateway's models from an
    extension the gateway serves. A guardrail policy can ask an outside
    service, and can put redacted values back in the answer. Chat goes to the
    replica answering soonest. A similar question can get the answer already
    given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
    GenAI conventions, and Kong is written down from trying it. Also an install
    guide with every screen, at /install.

    Upgrading.

    • Migrations 0049 (mcp_servers), 0050 (their rate and breaker) and 0051
      (projects.cache_similar), applied when the control plane starts.
    • llmgw server init now refuses a machine that would serve models without
      the memory for them: the starter model and the judges need 12 GB on one
      machine, where 8 GB was said before. --no-model, --no-guard-model or
      --no-node make it fit. A running cluster is not checked again.
    • Traces carry the GenAI conventions' gen_ai.* attributes beside the
      llmgw.* ones, which stay; docs/install.md, "Tracing", lists them.
    • Nothing else changes until it is turned on: services and restore in a
      guardrail policy, keys cache --similar with
      router.cache.embedding_model, cached_per_1k in a price.

    Added

    • Answers to similar questions. llmgw keys cache --similar 0.95 also
      gives a chat question the answer already given to one like it in the same
      conversation, compared by router.cache.embedding_model, as
      x-llmgw-cache: similar with the similarity. Held in memory beside the
      exact answers; nothing written to the store but the setting.

    • A cached prompt costs the cached rate. cached_per_1k in a price
      charges the prompt tokens an engine or provider read from its cache (the
      answer's cached_tokens) at their own rate; without it they cost what any
      prompt token does, as before.

    • Chat goes to the replica answering soonest. The router keeps each
      replica's time to first token, smoothed over its last answers, and the
      chat profile scores by it beside the queue and the KV cache: of two
      replicas of one model, the one on the slower card or the busier host gets
      less, and a replica not yet measured is tried.

    • llmgw and Kong, written down (docs/kong.md): Kong 3.9
      in front of llmgw by a plain route or its ai-proxy, each team's llmgw
      key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
      no completion tokens of a streamed answer, so bill from llmgw's usage.

    • A guardrail that is someone else's service. A policy's services
      asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
      HTTP: a body template, keys from files, dotted paths to where the answer
      says unsafe and why, on input, output or both, block or flag, and the
      policy's fail when it does not answer.

    • Redacted values put back in the answer. restore: true on a guardrail
      policy's input: the model reads [EMAIL_1], the client reads its own
      address in the reply, streamed or not. Only the request's own values, held
      in memory for its life; the output rules read the answer as the client
      will.

    • Traces in the OpenTelemetry GenAI conventions. An inference request
      and each attempt at an engine carry gen_ai.operation.name, the requested
      and the answering model, max_tokens, temperature and top_p when set,
      the session as gen_ai.conversation.id, and the input, output and cached
      tokens, and an attempt the engine that answered as gen_ai.provider.name.
      Tools that read the conventions show llmgw's calls as model calls with no
      dashboard of their own. No prompt or answer text is ever in a span.

    • llmgw for VS Code. An extension of llmgw's own, served by every gateway
      at /setup/vscode/llmgw.vsix with that gateway's address in it, made from
      the binary itself: plain JavaScript, no dependencies, no build, and so no
      marketplace needed at a site with no internet. It offers the gateway's chat
      models to VS Code's language-model API — the chat's model picker, agent
      mode, and every extension that asks VS Code for a model — with the sizes
      and the tool and vision capabilities the catalog says, streamed, with tool
      calls and their results both ways. VS Code is a tool on the account
      page's Connect your editor: a key, the one-line install, and an Open
      in VS Code       link that hands the key to the extension, which keeps it in
      VS Code's secret storage. llmgw: Sign In opens that page with a state
      of its own; a link without it is confirmed first, the gateway named.

    • An install guide with every screen, at /install. From three bare
      Ubuntu machines to a first answer: what each machine needs and how to
      check it, the release fetched and checked, the setup screens one by one,
      the system's state, and a first request with a key from the account page.
      The screens are a real three-machine install, recorded and kept as
      terminal frames — sharp at any size and a few kilobytes — with each
      session played back on request; the administrator's password and every
      key were masked before it was kept. /about says more of what llmgw does
      (documents, guardrails, sign-in, high availability, pictures and 3D), and
      its install steps are the ones of today, leading to the guide.

    • The company's MCP tools under the guardrails, a rate and a breaker.

      • The answer: the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
        ticket is withheld as a pasted one would be, and personal data is redacted.
      • The arguments: they are read before they leave, so a secret pasted into one does not reach the company system.
      • Rate: ctl mcp add --rate N allows each caller N calls a minute to a server.
      • Breaker: five failures in a row take a server out for a minute, its tools off the list.

    • How fast the judge is, measured. llmgw ctl guard judge check (and POST /api/v1/guard/judges/check) works
      out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
      and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
      and what to do: more cores, more judges, a GPU judge, a longer wait, or fail: open. server init runs it once the
      judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.

    • The company's own MCP servers through llmgw's /mcp. llmgw ctl mcp add jira --url … --secret-file Authorization=FILE registers one; /mcp then lists its tools as jira.search beside llmgw's own. Only keys granted
      them see them: llmgw keys limits --tools 'jira.*', patterns at any level of the tenancy tree. None granted anywhere
      is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
      server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
      on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
      admin API has /api/v1/mcp/servers.

    • The cluster as a read-only MCP server. Registered as llmgw ctl mcp add llmgw --url llmgw:admin and granted
      as llmgw.*, its tools say what llmgw ctl does:

      • cluster_status, list_nodes, list_models, list_replicas, explain_placement, node_logs;
      • nothing that changes the cluster.

      It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
      The admin API serves the same at POST /api/v1/mcp. ctl mcp add --bearer-file keeps a token, sent as
      Authorization: Bearer.

    • Large models across cards and nodes. For a model of experts on more than one card, the plan now adds vLLM's
      --enable-expert-parallel. It reads the expert count from config.json under any of its names, and sizes a model
      that also reads pictures by its text_config. Several nodes are offered in any number, not only powers of two. Each
      way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
      shows in ctl model plan, in the console's Serve a model, and in a serve job's first step. The serve form checks that
      a longer context still fits. ctl model serve --env KEY=VALUE puts engine variables such as NCCL_DEBUG=INFO on
      every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.

    • Machines of the judges' own in a three-machine install. The setup screen asks for up to two judge machines after
      the store's two; server init takes --judge-node user@host for each. Each is added as a node labelled
      llmgw.role: judge, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
      control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
      of their own, 2 were. A node with an llmgw.role label takes only profiles whose node selector asks for that role,
      so no other model is placed on the judges' machine.

    Fixed

    • A machine too small for the models it would serve is said so first.
      The control plane that is also a node, serving the starter model and two
      judges, needs about 10.5 GB. On 8 GB the install finished every step and
      then had no room for the starter. Now the setup's checks fail its memory
      with what it would serve, and server init refuses before it installs
      anything, naming --no-model, --no-guard-model and --no-node.

    • Turkish pages are Turkish to the end.

      • Errors: those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
        catalog", the settings changed meanwhile, and others) are now in the page's language.
      • Percent: a percentage is written the language's way: %93 in Turkish, 93% in English.
      • Footer: the time at the foot of a page is in the language's date form.
      • Stale forms: they are refused with a sentence in that language rather than "bad request".

    • llmgw keys limits no longer clears the limits it was not given. Setting one limit at a scope wrote the whole
      row, so --tools 'jira.*' alone took away that scope's rate, budget and model list. Only what is given changes now.
      --replace writes every limit, as the console's limits form does.

    • The plan refused tensor parallelism past a model's KV heads. Qwen3-235B, with four KV heads, had no way to run
      on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
      offered, as is one that divides them.

    • A big model on a slow mirror was failed while it downloaded. The startup timeout counted the download, so
      hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
      The timeout now runs from when a node's download last moved.

    • The judge init installs is no longer replaced by the leader's own. Just after the control plane started, the
      leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
      a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
      after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.

    • A judge the leader serves by itself is sized as init's is. When a guard policy names a judge the catalog lacks,
      the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
      3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and fail: closed refused the
      request. It now gets what server init gives its judges: 2 GB, two cores each (fewer on a small node), and no
      prompt cache. Every judge served through modelserve runs with --cache-ram 0.

    Downloads