The delivery is recorded¶
rig-conductor#2271, in two PRs: the Core fact and its policy (#2289), then the store method and the route that writes it.
The bug this closes¶
verified did not stay verified. Nothing recorded it: the probe read the page, the projector derived the stage from that read,
and the only thing holding the answer was the per-process status memo (CockpitTaskStatusRead.MemoFor, 30 s, "In process only:
a restart costs one extra read").
So after a conductor restart — an image pin, a node move, anything — a merged task was probed again. If the page still served the
template's to href it read Verified again, and nobody noticed the derivation. If the page had changed since, the probe read it
long after the 15-minute deadline and the projector answered:
Failed is terminal and memoised, so it stuck. A task that shipped hours earlier was shown to the operator as a failure, and
the thing that changed the page was usually re-arming the same task for another run.
That is why the production homepage is deliberately not re-armed, and why the loop was moved to the sandbox toggle pair: the pair's two templates each verify against their own expected state, so neither one's page change can be mistaken for the other's failure. The pair made the loop repeatable; it did not make a delivered task durable. This does.
The rule¶
A terminal success is a fact about the past, not a property of the present. "The page served this task's expected href at time T, after the merge" is true forever once it happens. Anything that has to be asked again is not that fact — it is a new observation of a page that is allowed to change.
So the delivery joins the task's own stream:
- Once. The fold accepts it only after
Submittedand only whileVerifiedAtis null. A second one throws, because re-recording the fact is the bug in a different costume. - Without moving state. The task stays
Submitted; the stage is still projected, now from a record instead of from today's page. No status key and no state name changes, so the Worker's signed contract and the frozen intent digests are untouched. - Never overwritten.
OperatorTaskPolicy.MarkVerifiedanswersExistingfor a replay — which is what every later status read of a delivered task is — and keeps the original href and time. It answersExistingeven at a stale revision: a delivered task is delivered whatever the caller last read. - Whole or absent. A time with no href would make the projector claim a terminal success it cannot name, so half a record
fails
Valid(request)rather than reaching the projector.
What the read path does now¶
| before | after | |
|---|---|---|
| verification | re-derived from the live page on every read after a restart | read from the record, forever |
| probe on a delivered task | ran until the anchor was live or a probe read the page past the deadline | never runs again |
| a failed deploy run read later | outranked the live anchor → Failed |
cannot un-deliver a recorded delivery |
deployed / verified timeline dates |
the read that happened to see it | the read that did see it, from the record |
| GitHub reads | once per memo window | unchanged — the card keeps its issue and PR links |
The record replaces the probe's read outright in CockpitTaskStatusRead.ReadPartsAsync. That is also what stops the probing: a
recorded delivery satisfies NeedsProbe's own live test, so there is one mechanism here, not two. An explicit second gate was
written first and then removed, because it could not be made to fail under mutation — a guard that cannot fail is not a guard.
In the projector, the record is ranked above every read below it, including the rc#2253 rule that a completed deploy run which did not succeed outranks a live anchor. That rule is about a page read now possibly describing someone else's deploy; it was never about a delivery already recorded.
What is deliberately not recorded¶
deployed. It is an intermediate stage, and a wrong intermediate stage self-corrects on the next read — the operator sees a card
move forward, not a delivered task turn into a failure. Only the terminal success needs to survive, so only it is written. If
deployed ever needs to be durable the same shape applies, one more event, no new mechanism.
Who writes it¶
The status route, on the first read that projects verified — best effort, exactly like the C10b reconcile beside it: the
status answers what it read whatever the write does, and the next read tries again. A lost race is Conflict carrying the row the
winner wrote, which is the same record; this is the one write in the feature where two callers want the identical outcome.
Whether a read may record is a pure Core decision, CockpitTaskStatusRead.Delivered, so it is testable without a host:
| this read offers no delivery when | because |
|---|---|
| no probe has read the page yet | there is nothing to record |
| the page does not satisfy the template | that is not this task's post-state |
| a completed deploy run did not succeed | rc#2253 ranks it above a live anchor — the page may look right because of someone else's deploy |
| the pull request is not merged | nothing shipped yet |
| the task already carries a record | a card polling for hours writes once, not once per poll |
The store side is IOperatorTaskStore.MarkVerifiedAsync, the same FetchForWriting transition as every other outcome, with
reconfirmsScope: false: recording an outcome authorizes nothing, so a repo remap or a resolver outage must not block it (the
existing rule for MarkUnknown and Reconcile).
Two things the end-to-end test caught¶
The live-aggregation view must fold the new event. OperatorTaskStreamView has one Apply per event, and Marten aggregates it
live rather than as a registered projection. Without Apply(OperatorTaskVerified) the view SKIPS the event, so the folded
Revision falls behind mt_streams.version and the next fetch throws the poisoned-stream exception — a recorded delivery would
have bricked the task it was recorded on. For the same reason ReadAsync's tear bound moves from six fetches to seven: a stream
can now hold six events.
The record is epoch milliseconds. The read that wrote it still had the probe's own snapshot, with sub-millisecond precision the
record cannot carry, so verify.checkedAt changed value between that read and the next one — for a fact that had not changed. The
view answers from the record too, once there is one.
Proof¶
Each guard is proved by mutation, not by a green run — the guard is gutted, the test that names it must fail, and it is reverted:
| Guard gutted | Test that must fail |
|---|---|
| the projector ranking the record first | A_recorded_delivery_outranks_a_page_that_changed_again_and_a_run_that_failed_after_it |
| the read path substituting the record for the probe | A_recorded_delivery_is_read_from_the_record_and_never_probed_again |
| the fold accepting exactly one delivery | Fold_rejects_every_sequence_the_policy_cannot_produce |
| half a delivery failing validity | A_task_carrying_half_a_delivery_is_not_a_valid_request |
All four fail when gutted. The projector test is the bug itself: a task delivered at minute 33, read at minute 600 against a page
that now serves the old link, is verified — and was Failed — Not live 15 min after the merge before this change.
The write half is proved the same way, against the walk the bug lived in
(A_recorded_delivery_survives_a_restart_and_the_page_changing_back: one task to verified, then a SECOND host over the same
database with its own empty memo, served the OLD href):
| Guard gutted | Test that must fail |
|---|---|
| the status route recording the delivery | the restart walk |
| the Marten view folding the delivery | the restart walk |
| the view answering from the record | the restart walk |
Delivered requiring a verified projection |
Only_a_read_that_projects_verified_offers_a_delivery_to_record |
Delivered offering nothing once recorded |
Only_a_read_that_projects_verified_offers_a_delivery_to_record |
All five fail when gutted. The restart walk asserts the second host answers verified with the byte-identical verify block and
timeline entry, and reads the site not once — before this change that same read answered failed.