Jobs are served by the leader, and a moment of fitting nowhere is waited out
CI / test (push) Failing after 8m47s

A 3D job leased by the standby control plane, whose router has no endpoints
and which a job replayed in the process never leaves, was told "loading"
for seven minutes while the leader served the jobs behind it. Jobs and
batches are claimed only while leading.

Five jobs asked for while cuda01's agent restarted failed for good as
"cannot be placed": re-registering, the node had no engines for a few
seconds. A job asks again for five minutes before that is its end.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-10 23:53:33 +02:00
1 parent 9b8ec89247
commit 95c61daf1b
6 files changed
+107 -3

No files matched your search

+4 -1
View File
@@ -2086,7 +2086,10 @@ curl -X POST http://gateway:8080/v1/3d/generations/job_…/cancel -H "Authorizat
- **How it is served:** the request is kept as it was sent and replayed by
whichever gateway holds the job's lease, through the same path a
client's request takes — limits, admission in the key's band, guardrails,
accounting. It is one usage record under the job's id, tagged
accounting — by the leading control plane, since a standby has nothing
to send it to. A model said to fit nowhere is asked for again for five
minutes before that is the job's error: an agent re-registering says so
for a few seconds. It is one usage record under the job's id, tagged
`job=<id>`, charged like any request; `GET` shows the usage once it is
done. A model still loading, a rate limit or a full queue is waited out
for as long as the job has: nobody is holding a connection, which is the