Skip to content

Wake-on-chat

Issue: dashecorp/rig-agent-runtime#679 — part 4 of rig-conductor#2008 (volume-first wake), after #674/#677 (the openai-api provider resolves its endpoint from the conductor and throws providerNotReady when the pod is not running).

Problem

The GPU pod auto-stops after 15 idle minutes. A message to Rig-E then ended in chat.js's generic 502 { error: 'the agent provider failed' }, and the operator woke the pod by hand (POST /api/pods/qwen/start in-cluster — the cockpit worker does not proxy /api/pods/*). Daily use needs message → wake → "waking, ~9 min" → ask again → answer.

Decisions

# Decision
D1 Pure policy decideWakeOnNotReady({ notReady, now, state }) in src/chat/wake-policy.js (no I/O). First not-ready → wake; not-ready again within the 15-minute window → wait with whole minutes elapsed; beyond the window → wake again (the conductor joins an in-flight wake, so re-asking is safe); lookup: UNREADABLE → wait, nothing woken. Replies are plain text, no markdown, no URLs, nothing from the error but the pod name.
D2 Adapter: createConductorPodClient gains startPod(name) → POST {conductorUrl}/api/pods/{name}/start, Accept: application/json, empty JSON body, 3 s AbortSignal.timeout, no bearer (in-cluster). Returns { requested, status } — requested is true for 200/202 — and resolves { requested: false, status: null } on abort or network error. Never throws. A client-side abort still leaves the conductor waking: it runs the wake on its host lifetime token.
D3 The resolver attaches wake() to providerNotReady as a non-enumerable function (() => podClient.startPod(pod)), so JSON.stringify and log lines stay clean and agent.js's final re-throw — which copies providerNotReady by reference — carries it through. chat.js needs no new DI.
D4 chat.js catch: if error.providerNotReady is present (and retryable !== false) → run the policy with the handler's per-pod state (a Map in the handler factory closure, not module-global) → on wake, call wake() fire-and-forget (logged on settle, never awaited in the response path) → respond 200 { reply, sessionId }, the same shape as a successful turn, so the web SPA and the iOS app render it as an assistant message unchanged. The single in-flight slot is released as before.
D5 Config-class errors (EndpointConfigError, no providerNotReady) and providerNotReady with retryable: false keep the generic 502 byte-for-byte and never wake.
D6 The completion POST carries AbortSignal.timeout(llm.timeoutMs ?? 120000). After the idle auto-stop the resolver still holds the pre-stop proxy URL, so the next turn POSTs there first. If that proxy answers 404/502/503/504 or resets, shouldRefreshEndpoint re-resolves and the not-ready → wake path runs; if it holds the connection instead (RunPod's behaviour for an EXITED pod is not verified), the abort has no httpStatus and takes the same path rather than pinning the single in-flight slot. The proxy itself closes with a 524 at 100 s (RunPod docs), so the 120 s default never cuts a request the proxy would have completed.
D7 Three states, not two (#685). A gateway status (502/503/504) or a genuine transport failure that survives the one-shot re-resolve, on the conductor path, while the conductor has just confirmed the pod RUNNING with a proxy URL (D8), is providerNotReady with modelLoading: true — built by the same exported notReadyError(pod, details, podClient) factory as the asleep case, so the wake capability and the fallbackEligible: false / retryable: true flags are identical. modelLoading is set only when true, so the asleep error's shape is byte-for-byte what it was. Everything else — 400/401/429/500, a malformed body, the literal-baseUrl path — keeps today's behaviour, and there is still exactly one re-resolve, one retry, no sleep. The policy answers wait and never wakes: the pod is already up.
D8 Fresh, not merely ready (#685 review). When the conductor is unreachable the resolver serves its last-known resolution, which still says RUNNING/FOUND about a pod that may have auto-stopped minutes ago; ready === true alone is tautological (the resolver returns a ready resolution or throws). So that fallback is tagged fromCache: true and the model-loading branch requires fromCache !== true. During a conductor outage the operator gets the raw 502 they can act on, never "the GPU is up" about an EXITED pod.
D9 The loading claim has a ceiling (#685 review). The policy stamps modelLoadingSince (falling back to wakeRequestedAt) into the per-pod state on the first loading turn. Past MODEL_LOADING_CEILING_MS = 20 min — the conductor's boot grace, rig-conductor#2052 — the reply becomes "The model server has been up for N minutes and still is not answering — it may be stuck. Someone needs to look at the pod." and decision.stuck makes chat.js log it at error level. Without it, a RUNNING pod whose vLLM crashed answers "a few more minutes" on every turn forever and nothing is ever logged as a fault. Any non-loading not-ready clears the clock.
D10 "Nothing is listening" is narrow. Only a gateway status, or a failure where the connection never produced a response, counts as loading. createChatCompletion marks the fetch's own throw transportFailure: true (an abort/timeout is excluded — something held the connection for two minutes, which an empty proxy does not do) and wraps the body reads so a 200 that dies mid-body or fails to parse carries its real httpStatus. A vLLM that OOMs on long contexts is therefore a visible provider failure, not an agent that says "still loading" on some turns and answers on others — and the crashing payload is not re-sent.
D11 The userMessage is branched too. chat.js never reads it, but index.js's Discord path replies with error.userMessage (carried through agent.js's final re-throw) and never runs the wake policy. The loading variant says "The GPU is up and the model is still loading — try again shortly." instead of the asleep variant's "The model server is not running right now." fallbackEligible: false stays on both: opting out is what keeps providerNotReady reaching chat.js on a chained character (D3).

Sequence

  1. POST /api/chat → agent.process() → openai-api → resolver.current() → conductor says isRunning: false.
  2. The resolver throws AgentProviderError { providerNotReady: { pod, status, lookup, podId, wake() } }; agent.js re-throws it with the fields intact.
  3. chat.js runs decideWakeOnNotReady → wake → wake() → POST /api/pods/qwen/start (3 s cap) → log [chat] wake requested for pod qwen: requested=true status=202.
  4. Reply 200: "Waking the GPU brain — that takes about 9 minutes. Ask me again in a few minutes."
  5. The user asks again in 3 minutes → still not running → "Still waking — 3 min elapsed, usually ready by ~9. Ask again shortly." (no second start request).
  6. ~3 minutes in, the conductor reports the pod RUNNING — but vLLM is still loading, so the completion POST gets the RunPod proxy's 502 HTML, the single re-resolve returns the same correct endpoint, and the retry gets 502 again. That is modelLoading (D7): reply 200 "Still waking — 6 min elapsed; the GPU is up and the model is loading. Ask again shortly.", no second wake.
  7. ~9 minutes in, vLLM is serving: the turn answers normally. The per-pod wake state is simply left behind; the next not-ready more than 15 minutes later starts a fresh wake.

Three states, and the timeline that proved it (#685)

The pod is not binary. asleep → waking → model loading → ready are four points, and only the first was handled by #679/#682, so the window the operator is most likely to type in — the minutes right after "ask me again in a few minutes" — was the one that still showed a raw ERROR HTTP 502.

State Conductor says The proxy answers Reply
Asleep isRunning: false (EXITED) — (never reached) "Waking the GPU brain — that takes about 9 minutes…" + one POST /api/pods/qwen/start
Waking isRunning: false, pod being created — "Still waking — N min elapsed, usually ready by ~9. Ask again shortly."
Model loading isRunning: true (RUNNING), proxyUrl set 502 HTML (or a reset) — nothing on :8000 yet "The GPU is up and the model is still loading — that takes a few more minutes. Ask again shortly." (with a wake on the clock: "Still waking — N min elapsed; the GPU is up and the model is loading. Ask again shortly.")
Stuck isRunning: true, > 20 min loading still 502 / nothing "The model server has been up for N minutes and still is not answering — it may be stuck…" + an error-level [chat] … → wait — STUCK: up but not answering log line
Conductor unreachable nothing — the cached answer is stale (fromCache) anything the generic 502; the loading reply is never given about a pod nobody confirmed
Ready isRunning: true the completion the model's answer

Measured on the first live wake-on-chat, 2026-09-09 (UTC):

Time Event
12:26:25 message 1, pod EXITED → "Waking the GPU brain…", wake fired
12:26:40 the conductor created the replacement pod kcbbg9igl0xdaz
12:33:08 message 2 — conductor RUNNING, completion POST → RunPod 502 <!DOCTYPE html>
12:33:10 the one-shot re-resolve returned the same correct endpoint; the retry got 502 again → before #685 this reached chat.js with no providerNotReady and became the generic 502
12:33:56 vLLM /health answered 200 — 48 s after the failed turn

So RUNNING leads readiness by roughly six minutes, and the whole wake is ~9. Nothing polls or retries inside the turn; the user asks again, and now gets told which of the three states the brain is in.

What this does NOT do

  • No polling, streaming or retry inside the turn — the user asks again.
  • No push notification when the pod is ready.
  • No "ready in about N more minutes" — the reply says "shortly". GET /api/pods/{name}.readySince (rig-conductor#2056) is what would make that number real, and would also let a proxy 502 against an already-ready pod be classified as a crashed model rather than a loading one instead of waiting out the D9 ceiling; the cockpit status line is rig-cockpit-worker#82.
  • No fallback to the next provider while the model loads (D11). A chained character stays on openai-api for that turn, because dropping the opt-out would lose providerNotReady on agent.js's final re-throw and with it the whole three-state reply.
  • Idle grace so the woken pod stays up long enough to be used: rig-conductor#2052.
  • The conductor endpoints stay unauthenticated in-cluster; nothing new carries or logs the API key.

Tests

  • src/chat/wake-policy.test.js — table over D1: first wake, in-window wait with minute arithmetic (window edge inclusive), re-wake beyond the window, UNREADABLE, plain-text replies. D7/D9: modelLoading → wait with and without a wake on the clock, the loading clock stamped and seeded from the wake, the 20-minute ceiling (edge inclusive) flipping the reply to stuck, a non-loading not-ready clearing the clock, state never mutated, never wake even past the window.
  • src/agent/providers/endpoint-resolver.test.js — notReadyError shape for both variants (asleep byte-for-byte; modelLoading present only when true) and their two distinct userMessages (D11); a resolution served from cache during a conductor outage is tagged fromCache without mutating the cached object (D8) with wake() absent from JSON.stringify; startPod URL/method/headers/body/timeout signal, 200/202 vs 404/409/503, abort and network errors never throw; the not-ready error carries a callable wake() for the right pod that is absent from JSON.stringify.
  • src/agent.test.js — wake() survives processWithProviders' final re-throw.
  • src/agent/providers/openai-api.test.js — a hanging completion is aborted by llm.timeoutMs, re-resolves once, and throws providerNotReady with wake() (D6). D7: a 502 and a transport failure that survive the re-resolve while the pod is RUNNING → modelLoading: true; the same 502 whose re-resolve says EXITED → the asleep error, unchanged; a 500 after the re-resolve → the plain provider error; a literal-baseUrl 502 → unchanged. D8/D10: a 502 whose re-resolve could not reach the conductor → the raw 502, not modelLoading; a 200 whose body cannot be read → httpStatus: 200, no re-resolve, no re-sent payload; an aborted completion that survives the re-resolve while RUNNING → not modelLoading.
  • src/chat.test.js — first not-ready → 200 waking reply + one wake; second within the window → "still waking", no wake; after >15 min → wake again; UNREADABLE → no wake; retryable: false → 502 unchanged; plain error → 502 byte-for-byte; in-flight slot released and a rejecting wake() contained; modelLoading → 200 loading reply with no wake and the distinguishing …, model loading) → wait log line; a pod still loading 25 minutes on → the stuck reply on console.error (D9).