llmgw

5 Commits 9 Branches 95 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
ilkerandClaude Opus 5	28029eb43b	Switch the project language to English CI / test (push) Has been cancelled Details Comments, doc comments, test names, error and log messages, CLI help, README, Makefile, CI step names and the design document are now all in English. No behaviour changes; the full suite passes under the race detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-16 01:15:21 +02:00
ilkerandClaude Opus 5	b7417e1a69	Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping The code behind plan §7.3-7.6: - breaker: folds cooldown and circuit breaking into one component (FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff and half-open probes. - endpoint: live state registry. Inflight counting, staleness detection, drain rate derived from counters. A scrape failure does not trip the breaker — the engine may be serving fine while only its metrics endpoint is down. - scrape: parses the §7.6 metric contract. Errors when any of the mandatory three is missing; the metric name mapping is overridable, since names drift between versions (plan §14). - scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a set of scorer weights; chat, batch and long-context are the same code with different weights. Scorers: queue, kv (hard penalty above the ceiling), inflight, load, weight, session (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Decision carries the rationale — which scorer produced what, which filter dropped how many. Tuning the weights without that is guesswork. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-16 01:15:21 +02:00
ilkerandClaude Opus 5	e07bc46c57	Faz 0: iskelet ve sim-engine CI / test (push) Has been cancelled Details Plan v0.2'nin Faz 0 çıktısı. Yazılan sıraya göre: - cobra tabanlı CLI iskeleti; server/agent/ctl komutları henüz stub. - internal/sim: OpenAI uyumlu sahte inference engine. Router'ı GPU olmadan gerçekçi koşullarda test edebilmek için gerçek bir engine'in router açısından önemli davranışlarını modeller: * continuous batching ve KV blok muhasebesi (head-of-line blocking dahil) * zincirlenmiş blok hash'leri + LRU ile otomatik prefix cache * batch büyüdükçe artan token gecikmesi * istemci koptuğunda isteğin düşürülüp blokların bırakılması * vLLM metrik adlarıyla /metrics (plan §7.6 engine kontratı) - Testler: prefix hash zincirlemesi, LRU tahliyesi, iptal propagasyonu, kuyruklama, SSE formatı, usage/cached_tokens, metriklerin gerçek Prometheus parser'ıyla okunabilirliği. - Gitea Actions CI: gofmt, vet, race detector. Modül yolu gitea.mynodes.xyz/ilker/llmgw; `make rename` ile değiştirilebilir. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-16 01:00:03 +02:00

Author

SHA1

Message

Date

ilkerandClaude Opus 5

28029eb43b

Switch the project language to English

CI / test (push) Has been cancelled

Details

Comments, doc comments, test names, error and log messages, CLI help, README,
Makefile, CI step names and the design document are now all in English.
No behaviour changes; the full suite passes under the race detector.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-16 01:15:21 +02:00

ilkerandClaude Opus 5

b7417e1a69

Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping

The code behind plan §7.3-7.6:

- breaker: folds cooldown and circuit breaking into one component
  (FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff
  and half-open probes.
- endpoint: live state registry. Inflight counting, staleness detection, drain
  rate derived from counters. A scrape failure does not trip the breaker — the
  engine may be serving fine while only its metrics endpoint is down.
- scrape: parses the §7.6 metric contract. Errors when any of the mandatory
  three is missing; the metric name mapping is overridable, since names drift
  between versions (plan §14).
- scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a
  set of scorer weights; chat, batch and long-context are the same code with
  different weights. Scorers: queue, kv (hard penalty above the ceiling),
  inflight, load, weight, session (bounded-load consistent hashing). Pickers:
  max-score, p2c, softmax, round-robin.

Decision carries the rationale — which scorer produced what, which filter
dropped how many. Tuning the weights without that is guesswork.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-16 01:15:21 +02:00

ilkerandClaude Opus 5

e07bc46c57

Faz 0: iskelet ve sim-engine

CI / test (push) Has been cancelled

Details

Plan v0.2'nin Faz 0 çıktısı. Yazılan sıraya göre:

- cobra tabanlı CLI iskeleti; server/agent/ctl komutları henüz stub.
- internal/sim: OpenAI uyumlu sahte inference engine. Router'ı GPU olmadan
  gerçekçi koşullarda test edebilmek için gerçek bir engine'in router
  açısından önemli davranışlarını modeller:
    * continuous batching ve KV blok muhasebesi (head-of-line blocking dahil)
    * zincirlenmiş blok hash'leri + LRU ile otomatik prefix cache
    * batch büyüdükçe artan token gecikmesi
    * istemci koptuğunda isteğin düşürülüp blokların bırakılması
    * vLLM metrik adlarıyla /metrics (plan §7.6 engine kontratı)
- Testler: prefix hash zincirlemesi, LRU tahliyesi, iptal propagasyonu,
  kuyruklama, SSE formatı, usage/cached_tokens, metriklerin gerçek
  Prometheus parser'ıyla okunabilirliği.
- Gitea Actions CI: gofmt, vet, race detector.

Modül yolu gitea.mynodes.xyz/ilker/llmgw; `make rename` ile değiştirilebilir.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-16 01:00:03 +02:00