Skip to content

The delivery is recorded

rig-conductor#2271, in two PRs: the Core fact and its policy (#2289), then the store method and the route that writes it.

The bug this closes

verified did not stay verified. Nothing recorded it: the probe read the page, the projector derived the stage from that read, and the only thing holding the answer was the per-process status memo (CockpitTaskStatusRead.MemoFor, 30 s, "In process only: a restart costs one extra read").

So after a conductor restart — an image pin, a node move, anything — a merged task was probed again. If the page still served the template's to href it read Verified again, and nobody noticed the derivation. If the page had changed since, the probe read it long after the 15-minute deadline and the projector answered:

Failed — Not live 15 min after the merge

Failed is terminal and memoised, so it stuck. A task that shipped hours earlier was shown to the operator as a failure, and the thing that changed the page was usually re-arming the same task for another run.

That is why the production homepage is deliberately not re-armed, and why the loop was moved to the sandbox toggle pair: the pair's two templates each verify against their own expected state, so neither one's page change can be mistaken for the other's failure. The pair made the loop repeatable; it did not make a delivered task durable. This does.

The rule

A terminal success is a fact about the past, not a property of the present. "The page served this task's expected href at time T, after the merge" is true forever once it happens. Anything that has to be asked again is not that fact — it is a new observation of a page that is allowed to change.

So the delivery joins the task's own stream:

public sealed record OperatorTaskVerified(string Href, long CheckedAt);
  • Once. The fold accepts it only after Submitted and only while VerifiedAt is null. A second one throws, because re-recording the fact is the bug in a different costume.
  • Without moving state. The task stays Submitted; the stage is still projected, now from a record instead of from today's page. No status key and no state name changes, so the Worker's signed contract and the frozen intent digests are untouched.
  • Never overwritten. OperatorTaskPolicy.MarkVerified answers Existing for a replay — which is what every later status read of a delivered task is — and keeps the original href and time. It answers Existing even at a stale revision: a delivered task is delivered whatever the caller last read.
  • Whole or absent. A time with no href would make the projector claim a terminal success it cannot name, so half a record fails Valid(request) rather than reaching the projector.

What the read path does now

before after
verification re-derived from the live page on every read after a restart read from the record, forever
probe on a delivered task ran until the anchor was live or a probe read the page past the deadline never runs again
a failed deploy run read later outranked the live anchor → Failed cannot un-deliver a recorded delivery
deployed / verified timeline dates the read that happened to see it the read that did see it, from the record
GitHub reads once per memo window unchanged — the card keeps its issue and PR links

The record replaces the probe's read outright in CockpitTaskStatusRead.ReadPartsAsync. That is also what stops the probing: a recorded delivery satisfies NeedsProbe's own live test, so there is one mechanism here, not two. An explicit second gate was written first and then removed, because it could not be made to fail under mutation — a guard that cannot fail is not a guard.

In the projector, the record is ranked above every read below it, including the rc#2253 rule that a completed deploy run which did not succeed outranks a live anchor. That rule is about a page read now possibly describing someone else's deploy; it was never about a delivery already recorded.

What is deliberately not recorded

deployed. It is an intermediate stage, and a wrong intermediate stage self-corrects on the next read — the operator sees a card move forward, not a delivered task turn into a failure. Only the terminal success needs to survive, so only it is written. If deployed ever needs to be durable the same shape applies, one more event, no new mechanism.

Who writes it

The status route, on the first read that projects verified — best effort, exactly like the C10b reconcile beside it: the status answers what it read whatever the write does, and the next read tries again. A lost race is Conflict carrying the row the winner wrote, which is the same record; this is the one write in the feature where two callers want the identical outcome.

Whether a read may record is a pure Core decision, CockpitTaskStatusRead.Delivered, so it is testable without a host:

this read offers no delivery when because
no probe has read the page yet there is nothing to record
the page does not satisfy the template that is not this task's post-state
a completed deploy run did not succeed rc#2253 ranks it above a live anchor — the page may look right because of someone else's deploy
the pull request is not merged nothing shipped yet
the task already carries a record a card polling for hours writes once, not once per poll

The store side is IOperatorTaskStore.MarkVerifiedAsync, the same FetchForWriting transition as every other outcome, with reconfirmsScope: false: recording an outcome authorizes nothing, so a repo remap or a resolver outage must not block it (the existing rule for MarkUnknown and Reconcile).

Two things the end-to-end test caught

The live-aggregation view must fold the new event. OperatorTaskStreamView has one Apply per event, and Marten aggregates it live rather than as a registered projection. Without Apply(OperatorTaskVerified) the view SKIPS the event, so the folded Revision falls behind mt_streams.version and the next fetch throws the poisoned-stream exception — a recorded delivery would have bricked the task it was recorded on. For the same reason ReadAsync's tear bound moves from six fetches to seven: a stream can now hold six events.

The record is epoch milliseconds. The read that wrote it still had the probe's own snapshot, with sub-millisecond precision the record cannot carry, so verify.checkedAt changed value between that read and the next one — for a fact that had not changed. The view answers from the record too, once there is one.

Proof

Each guard is proved by mutation, not by a green run — the guard is gutted, the test that names it must fail, and it is reverted:

Guard gutted Test that must fail
the projector ranking the record first A_recorded_delivery_outranks_a_page_that_changed_again_and_a_run_that_failed_after_it
the read path substituting the record for the probe A_recorded_delivery_is_read_from_the_record_and_never_probed_again
the fold accepting exactly one delivery Fold_rejects_every_sequence_the_policy_cannot_produce
half a delivery failing validity A_task_carrying_half_a_delivery_is_not_a_valid_request

All four fail when gutted. The projector test is the bug itself: a task delivered at minute 33, read at minute 600 against a page that now serves the old link, is verified — and was Failed — Not live 15 min after the merge before this change.

The write half is proved the same way, against the walk the bug lived in (A_recorded_delivery_survives_a_restart_and_the_page_changing_back: one task to verified, then a SECOND host over the same database with its own empty memo, served the OLD href):

Guard gutted Test that must fail
the status route recording the delivery the restart walk
the Marten view folding the delivery the restart walk
the view answering from the record the restart walk
Delivered requiring a verified projection Only_a_read_that_projects_verified_offers_a_delivery_to_record
Delivered offering nothing once recorded Only_a_read_that_projects_verified_offers_a_delivery_to_record

All five fail when gutted. The restart walk asserts the second host answers verified with the byte-identical verify block and timeline entry, and reads the site not once — before this change that same read answered failed.