Wake-on-chat

Issue: dashecorp/rig-agent-runtime#679 — part 4 of rig-conductor#2008 (volume-first wake), after #674/#677 (the openai-api provider resolves its endpoint from the conductor and throws providerNotReady when the pod is not running).

Problem

The GPU pod auto-stops after 15 idle minutes. A message to Rig-E then ended in chat.js's generic 502 { error: 'the agent provider failed' }, and the operator woke the pod by hand (POST /api/pods/qwen/start in-cluster). The cockpit worker proxies GET /api/pods/{name} and, since worker#84, POST …/stop, but never start. Daily use needs message → wake → "waking, ~9 min" → ask again → answer.

Decisions

#	Decision
D1	Pure policy decideWakeOnNotReady({ notReady, now, state }) in src/chat/wake-policy.js (no I/O). First not-ready → wake; not-ready again within the 15-minute window → wait with whole minutes elapsed; beyond the window → wake again (the conductor joins an in-flight wake, so re-asking is safe); lookup: UNREADABLE → wait, nothing woken. Replies are plain text, no markdown, no URLs, nothing from the error but the pod name.
D2	Adapter: createConductorPodClient gains startPod(name) → POST {conductorUrl}/api/pods/{name}/start, Accept: application/json, empty JSON body, 3 s AbortSignal.timeout, no bearer (in-cluster). Returns { requested, status } — requested is true for 200/202, false with the status for any other HTTP answer (a definite refusal). Never throws. A client-side abort still leaves the conductor waking: it runs the wake on its host lifetime token, and its start blocks until RUNNING, so the 3 s cap aborts on every real wake. #684: a TimeoutError/AbortError therefore resolves { requested: true, status: null, timedOut: true, timeoutMs } (in flight), and only a transport failure before the request is sent resolves { requested: false, status: null, error: 'TypeError (ECONNREFUSED)' }. See Wake log lines.
D3	The resolver attaches wake() to providerNotReady as a non-enumerable function (() => podClient.startPod(pod)), so JSON.stringify and log lines stay clean and agent.js's final re-throw — which copies providerNotReady by reference — carries it through. chat.js needs no new DI.
D4	chat.js catch: if error.providerNotReady is present (and retryable !== false) → run the policy with the handler's per-pod state (a Map in the handler factory closure, not module-global) → on wake, call wake() fire-and-forget (logged on settle, never awaited in the response path) → respond 200 { reply, sessionId }, the same shape as a successful turn, so the web SPA and the iOS app render it as an assistant message unchanged. The single in-flight slot is released as before.
D5	Config-class errors (EndpointConfigError, no providerNotReady) and providerNotReady with retryable: false keep the generic 502 byte-for-byte and never wake.
D6	The completion POST carries AbortSignal.timeout(llm.timeoutMs ?? 120000). After the idle auto-stop the resolver still holds the pre-stop proxy URL, so the next turn POSTs there first. If that proxy answers 404/502/503/504 or resets, shouldRefreshEndpoint re-resolves and the not-ready → wake path runs; if it holds the connection instead (RunPod's behaviour for an EXITED pod is not verified), the abort has no httpStatus and takes the same path rather than pinning the single in-flight slot. The proxy itself closes with a 524 at 100 s (RunPod docs), so the 120 s default never cuts a request the proxy would have completed.
D7	Three states, not two (#685). A gateway status (502/503/504) or a genuine transport failure that survives the one-shot re-resolve, on the conductor path, while the conductor has just confirmed the pod RUNNING with a proxy URL (D8), is providerNotReady with modelLoading: true — built by the same exported notReadyError(pod, details, podClient) factory as the asleep case, so the wake capability and the fallbackEligible: false / retryable: true flags are identical. modelLoading is set only when true, so the asleep error's shape is byte-for-byte what it was. Everything else — 400/401/429/500, a malformed body, the literal-baseUrl path — keeps today's behaviour, and there is still exactly one re-resolve, one retry, no sleep. The policy answers wait and never wakes: the pod is already up.
D8	Fresh, not merely ready (#685 review). When the conductor is unreachable the resolver serves its last-known resolution, which still says RUNNING/FOUND about a pod that may have auto-stopped minutes ago; ready === true alone is tautological (the resolver returns a ready resolution or throws). So that fallback is tagged fromCache: true and the model-loading branch requires fromCache !== true. During a conductor outage the operator gets the raw 502 they can act on, never "the GPU is up" about an EXITED pod.
D9	The loading claim has a ceiling (#685 review). The policy stamps modelLoadingSince (falling back to wakeRequestedAt) into the per-pod state on the first loading turn. Past MODEL_LOADING_CEILING_MS = 20 min — the conductor's boot grace, rig-conductor#2052 — the reply becomes "The model server has been up for N minutes and still is not answering — it may be stuck. Someone needs to look at the pod." and decision.stuck makes chat.js log it at error level. Without it, a RUNNING pod whose vLLM crashed answers "a few more minutes" on every turn forever and nothing is ever logged as a fault. Any non-loading not-ready clears the clock.
D10	"Nothing is listening" is narrow. Only a gateway status, or a failure where the connection never produced a response, counts as loading. createChatCompletion marks the fetch's own throw transportFailure: true (an abort/timeout is excluded — something held the connection for two minutes, which an empty proxy does not do) and wraps the body reads so a 200 that dies mid-body or fails to parse carries its real httpStatus. A vLLM that OOMs on long contexts is therefore a visible provider failure, not an agent that says "still loading" on some turns and answers on others — and the crashing payload is not re-sent.
D11	The userMessage is branched too. chat.js never reads it, but index.js's Discord path replies with error.userMessage (carried through agent.js's final re-throw) and never runs the wake policy. The loading variant says "The GPU is up and the model is still loading — try again shortly." instead of the asleep variant's "The model server is not running right now." fallbackEligible: false stays on both: opting out is what keeps providerNotReady reaching chat.js on a chained character (D3).
D12	Name the wake's outcome, not the gap since the last ask (#695). chat.js watches each wake it issues: notReadyError attaches a non-enumerable readPod() (GET /api/pods/{name}) beside wake(), and the handler polls it every 60 s (unref'd timer) until readySince (rig-conductor#2056) appears, or records never-ready after WAKE_WATCH_MS = 35 min (window + loading ceiling). A start answered with a non-200/202 HTTP status is recorded as refused and not watched; the 3 s client abort (status: null) is watched, because the conductor keeps waking. Outcomes are written only while wakeRequestedAt still matches, so a superseded watch is inert; a readySince more than 60 s before the wake is an earlier boot and ignored. The policy then answers ready + pod not RUNNING → "The GPU woke and was ready at HH:MM UTC, then stopped — it shuts down after 15 idle minutes. Waking it again — about 9 minutes." (date-prefixed when not today, UTC); never-ready → "The wake requested at HH:MM UTC did not complete — the GPU never became ready. Waking it again…"; refused → "The last wake request was refused by the conductor (HTTP N). Waking it again…" — all wake, regardless of window. Unknown outcome keeps D1 and #712. No elapsed is ever measured from a clock an ask reset.

Sequence

1. POST /api/chat → agent.process() → openai-api → resolver.current() → conductor says isRunning: false.
2. The resolver throws AgentProviderError { providerNotReady: { pod, status, lookup, podId, wake() } }; agent.js re-throws it with the fields intact.
3. chat.js runs decideWakeOnNotReady → wake → wake() → POST /api/pods/qwen/start (3 s cap) → the client times out while the conductor keeps waking → log [chat] wake requested for pod qwen: in flight (client timed out after 3 s; the conductor continues) (#684).
4. Reply 200: "Waking the GPU brain — that takes about 9 minutes. Ask me again in a few minutes."
5. The user asks again in 3 minutes → still not running → "Still waking — 3 min elapsed, usually ready by ~9. Ask again shortly." (no second start request).
6. ~3 minutes in, the conductor reports the pod RUNNING — but vLLM is still loading, so the completion POST gets the RunPod proxy's 502 HTML, the single re-resolve returns the same correct endpoint, and the retry gets 502 again. That is modelLoading (D7): reply 200 "Still waking — 6 min elapsed; the GPU is up and the model is loading. Ask again shortly.", no second wake.
7. ~9 minutes in, vLLM is serving: the turn answers normally. The per-pod wake state is simply left behind; the next not-ready more than 15 minutes later starts a fresh wake.

Expired Wake Window (#712, Refs #695)

Beyond the 15-minute window, D1 still chooses wake and resets wakeRequestedAt. Its reply is now: "The GPU brain is not ready. I'll try to wake it again. Ask again in a few minutes." The old request timestamp is not evidence of continuous downtime: successful turns leave it behind, and the GPU may have stopped since. The reply expresses intent, not confirmation that Conductor accepted the request or that the GPU started. D4 still fires the wake asynchronously when available.

The exact 15-minute boundary still waits. First-wake, loading and unreadable branches are unchanged, as are session IDs and the lack of automatic prompt replay. Broader wake-outcome tracking (#695) and refusal/timeout handling (#684) remain open; this wording change does not clear wake state on successful turns.

695 adds the outcome tracking (D12): on 2026-09-10 four asks produced four wakes that each

became ready, served nobody and idled out, and the replies read as a single wake failing for ten hours. Now the reply after such a wake says when it was ready and that it stopped.

Three states, and the timeline that proved it (#685)

The pod is not binary. asleep → waking → model loading → ready are four points, and only the first was handled by #679/#682, so the window the operator is most likely to type in — the minutes right after "ask me again in a few minutes" — was the one that still showed a raw ERROR HTTP 502.

State	Conductor says	The proxy answers	Reply
Asleep	isRunning: false (EXITED)	— (never reached)	"Waking the GPU brain — that takes about 9 minutes…" + one POST /api/pods/qwen/start
Waking	isRunning: false, pod being created	—	"Still waking — N min elapsed, usually ready by ~9. Ask again shortly."
Model loading	isRunning: true (RUNNING), proxyUrl set	502 HTML (or a reset) — nothing on :8000 yet	"The GPU is up and the model is still loading — that takes a few more minutes. Ask again shortly." (with a wake on the clock: "Still waking — N min elapsed; the GPU is up and the model is loading. Ask again shortly.")
Stuck	isRunning: true, > 20 min loading	still 502 / nothing	"The model server has been up for N minutes and still is not answering — it may be stuck…" + an error-level [chat] … → wait — STUCK: up but not answering log line
Conductor unreachable	nothing — the cached answer is stale (fromCache)	anything	the generic 502; the loading reply is never given about a pod nobody confirmed
Ready	isRunning: true	the completion	the model's answer

Measured on the first live wake-on-chat, 2026-09-09 (UTC):

Time	Event
12:26:25	message 1, pod EXITED → "Waking the GPU brain…", wake fired
12:26:40	the conductor created the replacement pod kcbbg9igl0xdaz
12:33:08	message 2 — conductor RUNNING, completion POST → RunPod 502 <!DOCTYPE html>
12:33:10	the one-shot re-resolve returned the same correct endpoint; the retry got 502 again → before #685 this reached chat.js with no providerNotReady and became the generic 502
12:33:56	vLLM /health answered 200 — 48 s after the failed turn

So RUNNING leads readiness by roughly six minutes, and the whole wake is ~9. Nothing polls or retries inside the turn; the user asks again, and now gets told which of the three states the brain is in.

Wake log lines (#684)

The first live wake (2026-09-09 12:26Z) logged requested=false status=null while the conductor went on to create a replacement pod: the conductor's start blocks until RUNNING, so the 3 s client cap aborts on every real wake. describeWakeResult (src/chat/wake-policy.js) now gives each startPod() outcome its own line:

Outcome	startPod() result	Level	Line
In flight (client timed out)	{ requested: true, status: null, timedOut: true, timeoutMs: 3000 }	log	[chat] wake requested for pod qwen: in flight (client timed out after 3 s; the conductor continues)
Accepted	{ requested: true, status: 202 }	log	[chat] wake requested for pod qwen: accepted status=202
Refused	{ requested: false, status: 409 }	error	[chat] wake refused for pod qwen: status=409
Failed (never sent)	{ requested: false, status: null, error: 'TypeError (ECONNREFUSED)' }	error	[chat] wake failed for pod qwen: TypeError (ECONNREFUSED)

To check whether a wake fired, grep wake requested for pod. To alert, use wake refused / wake failed. Do not alert on a timeout. A 202-first contract on the conductor (rig-conductor#2052) would turn the in-flight line into accepted, and this side would need no change.

What this does NOT do

The legacy flow below still applies when wakeMode is omitted. Since #715, foreground Cockpit callers can opt into wakeMode: "wait"; a preflight-only pendingWake marker lets them retain the prompt until fresh readiness and send one wakeMode: "resume" continuation. Resume cannot wake again, and neither mode retries a failed completion or changes providers after failure. The HTTP contract defines the safety conditions. No pending prompt is stored in the runtime.

• No polling, streaming or retry inside the turn — the user asks again.
• No push notification when the pod is ready.
• No "ready in about N more minutes" — the reply says "shortly". GET /api/pods/{name}.readySince (rig-conductor#2056) is what would make that number real, and would also let a proxy 502 against an already-ready pod be classified as a crashed model rather than a loading one instead of waiting out the D9 ceiling; the cockpit status line is rig-cockpit-worker#82.
• No fallback to the next provider while the model loads (D11). A chained character stays on openai-api for that turn, because dropping the opt-out would lose providerNotReady on agent.js's final re-throw and with it the whole three-state reply.
• Idle grace so the woken pod stays up long enough to be used: rig-conductor#2052.
• The conductor endpoints stay unauthenticated in-cluster; nothing new carries or logs the API key. Update, rig-conductor#2369: the conductor now puts an operator-token gate on POST /api/pods/{name}/start. It is report-only until enforced. startPod sends RIG_WAKE_TOKEN as X-Rig-Operator-Token when that env var is set. It is a wake-only token: the conductor accepts it on start and nowhere else, and getPod never sends it. A 401/403 is logged as an error that names RIG_WAKE_TOKEN, instead of vanishing into requested=false, so an enforced gate without the token cannot silently break wake.

Tests

• 695 (D12): src/chat/wake-policy.test.js — ready-then-stopped (inside and beyond the window, same-day and date-prefixed time), RUNNING not called stopped, never-ready, refused, unknown outcome unchanged, outcome carried through loading/unreadable. src/chat.test.js — the watch records ready and stops, gives up at exactly 35 min with an error log, a refused start is not watched while a client abort is, a pre-wake readySince is ignored and a superseded watch is inert. endpoint-resolver.test.js — readPod() reads this pod and is hidden from JSON.

• src/chat/wake-continuation.test.js exercises real handler/provider composition with synthetic fetch: asleep to ready, exact original prompt and session, no early execution/history, one wake, stale cache, loading, invalid readiness, missing key, unreadable state, stop race and no ambiguous completion retry.

• src/chat/wake-provider.test.js checks provider pinning across a preflight selection change and refusal of fallback in both strict modes.

• 712 regressions in both policy and handler: 15 minutes + 1 ms and

  22/647/2796-minute gaps produce the same neutral reply. Policy checks fresh timestamps, cleared loading state and immutable inputs; handler checks HTTP 200, stable session, exactly one additional wake, no replay, the inclusive 15-minute boundary and a fresh cooldown after the new wake.

• src/chat/wake-policy.test.js — table over D1: first wake, in-window wait with minute arithmetic (window edge inclusive), re-wake beyond the window, UNREADABLE, plain-text replies. D7/D9: modelLoading → wait with and without a wake on the clock, the loading clock stamped and seeded from the wake, the 20-minute ceiling (edge inclusive) flipping the reply to stuck, a non-loading not-ready clearing the clock, state never mutated, never wake even past the window. #684: describeWakeResult — timeout → in flight (log), 200/202 → accepted (log), 404/409/503 → refused (error), transport failure or no result → failed (error); the three non-accepted lines are distinct.

• src/agent/providers/endpoint-resolver.test.js — notReadyError shape for both variants (asleep byte-for-byte; modelLoading present only when true) and their two distinct userMessages (D11); a resolution served from cache during a conductor outage is tagged fromCache without mutating the cached object (D8) with wake() absent from JSON.stringify; startPod URL/method/headers/body/timeout signal, 200/202 vs 404/409/503; #684: TimeoutError/AbortError (and a real AbortSignal.timeout firing) → requested: true, timedOut: true, ECONNREFUSED → requested: false with error: 'TypeError (ECONNREFUSED)', never throws; the not-ready error carries a callable wake() for the right pod that is absent from JSON.stringify.
• src/agent.test.js — wake() survives processWithProviders' final re-throw.
• src/agent/providers/openai-api.test.js — a hanging completion is aborted by llm.timeoutMs, re-resolves once, and throws providerNotReady with wake() (D6). D7: a 502 and a transport failure that survive the re-resolve while the pod is RUNNING → modelLoading: true; the same 502 whose re-resolve says EXITED → the asleep error, unchanged; a 500 after the re-resolve → the plain provider error; a literal-baseUrl 502 → unchanged. D8/D10: a 502 whose re-resolve could not reach the conductor → the raw 502, not modelLoading; a 200 whose body cannot be read → httpStatus: 200, no re-resolve, no re-sent payload; an aborted completion that survives the re-resolve while RUNNING → not modelLoading.
• src/chat.test.js — first not-ready → 200 waking reply + one wake; second within the window → "still waking", no wake; after >15 min → wake again; UNREADABLE → no wake; retryable: false → 502 unchanged; plain error → 502 byte-for-byte; in-flight slot released and a rejecting wake() contained; modelLoading → 200 loading reply with no wake and the distinguishing …, model loading) → wait log line; a pod still loading 25 minutes on → the stuck reply on console.error (D9).