Skip to content

Operator task lifecycle walk

Tracking: rig-conductor#2228, gap 1 of the end-to-end plan written after #2104 shipped.

What was missing

The Rig-E operator-task feature shipped with 23 route-level e2e cases (CockpitTaskEndpointsE2eTests), 39 MergeGate cases and a cross-repo contract harness. Between them they cover every transition — and every one of them builds its own state and asserts one transition. Nothing drove one task through the whole pipeline, so:

  • an ordering or state-machine defect between two stages was invisible;
  • the 15-key status contract was asserted per stage, not on every read of one task's life;
  • "a stage reached stays in the timeline" was never asserted across more than two stages;
  • nothing proved a refused release leaves the task exactly as releasable as it was.

What the walk adds

CockpitTaskLifecycleWalkE2eTests — three walks over the real Program.cs pipeline and Testcontainers Postgres, each one task, each read asserting all 15 keys of CockpitTaskStatus:

Step Driven Asserted
1 signed propose proposed; issue/pr/release/deploy/verify all null; no GitHub call before Submitted
2 a second status unchanged, still no call (the card polls this route)
3 confirm, then a replayed confirm one issue POSTed as the App, the fixed template, a marker naming this request; the replay is existing with no second POST
4 — queued, "Waiting for Planner-E"
5 Planner-E's TaskSpec comment + tier:t1 queued, "Planned (tier:t1), waiting for agent-ready"
6 the agent-ready label and its event implementing, dated from the label event, not from the read
7 a draft PR, then ready for review implementing (a draft is not in review) → in_review; release null while nothing is held
8 Review-E approves the head; the check run completes MergeGate itself holds the task PR instead of merging → approval_needed, release.heldSha at the head
9 a release signed for another head 409 head_moved, nothing merged, nothing approved — and the task is still approval_needed
10 the release at the approved head merged pinned to that head, one audit comment naming the request, and the approval's MAC verifies under the task key
11-13 the deploy run, then the new href merged → deployed → verified, terminal, whole timeline in order
14 two more status reads terminal: GitHub and the probe are never called again

Two further walks cover the branches that only mean something on a task with real history: a release deferred by a pending check (the approval survives; the status retries the pinned merge 2 min later), and a create whose outcome is unknown (never retried, reconciled from the one issue the App filed, and the task then walks on to in_review).

The fake is driven, not replayed

WalkWorld serves GitHub, Valkey and dashecorp.com from mutable state that the test moves the way Planner-E, Dev-E and Review-E would: the issue body every read serves is the body the confirm route posted, and the merge PUT actually marks the PR merged and closes the issue. That is what makes the order load-bearing — a replayed fixture would answer the same thing whatever order the reads came in.

Two things this cost, worth knowing before extending it:

  • MergeGate records its own conditions with the single-field HashSetAsync overload. Substituting only the HashEntry[] overload makes the gate not see the ci-passed it just wrote, take the "review approved, CI silent" branch, and hold nothing — a green-looking drive that asserts nothing.
  • A pending check defers the merge before the hold is decided, so a walk that wants approval_needed must reach it with the check green and only then make one pending.

Proof it is not vacuous

Each of these mutations of shipped code was applied, run and reverted; every one fails the walk:

Mutation Result
Terminal renamed on the status record (the wire contract drifts) 3 of 3 walks fail
a draft PR projected as in_review (a stage-machine defect) the walk fails
releasing listed before approval_needed in the timeline 2 of 3 walks fail

Running it

dotnet test tests/ConductorE.Api.Tests --filter FullyQualifiedName~CockpitTaskLifecycleWalk

CI needs no new workflow: test.yml already runs the whole Api project on every PR and on main. Locally, pin LC_ALL=en_US.UTF-8 for the full suite — six formatting assertions elsewhere fail without it and look like a regression.

Still open

The walk closes gap 1 of five. The rest, in the order they are worth doing: the cross-repo contract harness is in no repo and no CI; the Worker leg is replayed rather than driven; nothing runs check-rig-e-persona.py in rig-gitops; there is no scheduled read-only live smoke; and the feature has never once run for real — that last one is operator-gated, because it wakes the GPU pod and files a real issue.