Promotion commands (rc#2760)

The plan is in dashecorp/invotek, projects/invotek-prosjekt/proposals/2026-10-22-releases-and-promotion.md (invotek#79). The operator requires that promotion to production be possible from prosjekt.invotek.no, for customers too (2026-10-22), and asked to automate it so that they can promote from there (2026-10-23). Steps 2 to 4 are built: the release App dashecorp-release, the conductor's release stream and GET /api/promotions in shadow (Release Promotion), and «Versjoner» on the project's card in the portal's /_admin (invotek-portal docs/2026-10-29-versjoner.md). On 2026-10-30 the operator decided that /_admin signs in with GitHub through Access, and that passkeys come with step 9.

This note settles step 5: the commands, the conductor's GitHub writes, the job's check, and the scratch repo they are tested on. invotek-no is not touched until step 6.

Decisions

1. The portal signs each command with ES256 (invotek-portal-command-v1), not with a key it shares with the conductor.
2. Two commands in step 5, promote and rollback, and only with the role operator. confirm, the per-repo policy and the role kundeadmin come with step 9, when a customer first promotes. The envelope carries role from the start.
3. The conductor writes to GitHub only as the release App: it approves at the run's callback URL, posts a status while a release waits, cancels a superseded run, and dispatches rollback.yml.
4. The job asks before it deploys: POST /api/promotions/check with its OIDC token. One approval admits one run attempt.
5. Rollback claims the repo once per command, cancels the releases that wait, dispatches rollback.yml, and approves that run's first deployment only. It goes back one deployment, as rollback.yml does with an empty input. A command sent again never rolls back again.
6. Every approval is for one deployment. GitHub gives a re-run a new deployment, so a re-run waits for a new tap.
7. No personal data goes to GitHub. The approval's comment names the role and the command, never the person.
8. A scratch repo first. dashecorp/release-scratch, private, with its own production environment and the App as its rule, carries the end-to-end test.

1. The command: invotek-portal-command-v1

Why ES256 and not HMAC. The plan said random_password, the way COCKPIT_TASK_SIGNING_KEY is made (Operator Task Signature). With a key pair: - the conductor holds no secret for it. Its public keys are plain values in deploy/k8s/deployment.yaml, so there is no SOPS Secret and no credential-ops run, and a leaked config or memory dump cannot forge a command; - rotation is one PR on each side.

The cost: ECDSA signatures are randomized, so shared test vectors pin verification cases (fixed signatures under a public test key), and the signer is tested by signing and then verifying.

The keys. - Made in infra: tls_private_key (ECDSA, P256). - The private key goes to invotek-portal's production environment as a secret. The gated deploy puts it into the Worker as CONDUCTOR_COMMAND_SIGNING_KEY (PKCS#8 PEM). It joins WORKER_SECRETS, so the first deploy that needs it waits for the operator's yes deploy. - The public key is an output, copied by PR into the conductor's INVOTEK_PORTAL_COMMAND_KEYS. - INVOTEK_PORTAL_COMMAND_KEYS holds one or two base64 SPKI DER keys, comma separated, each P-256. Two keys allow a rotation without lockstep: add the new one, switch the portal, drop the old one. - The key id is the first 16 lowercase hex characters of SHA-256 over a key's SPKI DER. - The conductor refuses to start commands if a key is missing, malformed, not P-256, or the public test key: every command then answers 503 not_configured, as TaskSigningKeyConfig does. So no key means no commands.

The envelope. After rig-cockpit-task-v1, with these differences: an ECDSA signature instead of an HMAC, a key id, and no context header (the body carries everything).

Header	Value
X-Invotek-Command-Timestamp	Epoch milliseconds, ^[1-9][0-9]{0,15}$, at most 2^53−1; fresh within ±30 000 ms, both ends inclusive
X-Invotek-Command-Key	The key id, 16 lowercase hex
X-Invotek-Command-Signature	The 64-byte IEEE P1363 r‖s (what WebCrypto's sign returns and .NET's ECDsa.VerifyData reads by default), as 86 characters of unpadded base64url

Signed bytes: UTF8("invotek-portal-command-v1\n" + METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + KEYID + "\n") ++ RAW_BODY_BYTES.

The routes, frozen. Both take a canonical JSON body: no whitespace, the exact key order, no extra or duplicate keys, at most 4096 bytes. The query must be empty.

Route	Body
POST /api/promotions/promote	{"commandId","repo","runId","sha","role","actor","auditHash"}
POST /api/promotions/rollback	{"commandId","repo","role","actor","auditHash"}

The fields: - commandId: a lowercase UUID, not all zeros. - repo: a lowercase GitHub owner/name, by ReleaseWebhookPayload.NormalizeRepo's pattern. - runId: a positive safe integer in canonical decimal. - sha: 40 lowercase hex. - role: operator. kundeadmin is well formed but answers 403 role_not_enabled until step 9. - actor: the e-mail Access gave the portal for the person who tapped, as the portal lowercases it: 3 to 254 printable ASCII characters with no space and no capital letter, and exactly one @ with something on each side. It stays in the conductor's stream. - auditHash: 64 lowercase hex, the hash of the portal's audit event for the tap (promote.requested or rollback.requested).

The verifier's order (first failure wins, never throws): 1. the key configuration (not_configured); 2. the clock (invalid_clock); 3. method, path and empty query (unsupported_route); 4. a known key id (unknown_key); 5. the signature's format (invalid_signature); 6. the timestamp's format (invalid_timestamp) and freshness (stale_timestamp); 7. the body's size (invalid_body); 8. the ECDSA verification (signature_mismatch); 9. the body's canonical form and fields (invalid_body).

Built (rig-conductor#2762, not yet wired to a route): PortalCommandSignature and PortalCommandKeyConfig in Core, with the frozen vector file tests/ConductorE.Core.Tests/Fixtures/portal-command-vectors.json (4 signing and 77 verification cases, SHA-256 6dff3783…ceb2). scripts/portal-command-vectors.mjs wrote it with WebCrypto, the API the portal signs with. Running it again writes a new test key and new signatures, so a new file that both repos must then pin. The config refuses the file's two public test keys (cae6f4015ade18eb, 9809e4fb674f47ba).

The body is read only after the signature verified. The reader works as SignedTaskRequestReader does: one pass over the body, RawTarget for the path, and a repeated header counts as missing.

What a valid signature does not prove. It proves that the holder of the portal's private key signed these bytes, and nothing about whether a person tapped: the portal decides that, under its own login (the plan's residual risk, §4). A replay inside the window verifies again, so durable state decides every effect (below). ECDSA's malleability (s and n − s both verify) gives a replay nothing more, since nothing keys on the signature's bytes.

2. What the conductor does

The command routes sit under /api/promotions, behind the Access app that admits the operator and the portal's service token (infra#736). Access admits the portal and the signature proves the command. The conductor resolves the repo's tenant itself, as the webhook does, and never takes one from the caller.

promote. A pure Core decision over the run's stream release:<repo>:<runId>:

Case	Answer
The commandId is already on this stream	The same answer as the first time. If GitHub never acknowledged the approval, it is sent again
No such release, or it is not waiting	409 not_waiting, with the state
The sha is not the release's	409 sha_mismatch
role is not operator	403 role_not_enabled
The release's current deployment is already approved	200 already_approved, no second call to GitHub
Otherwise	PROMOTE_REQUESTED and RELEASE_APPROVED (bound to the deployment id GitHub asked about), then the approval; APPROVAL_DELIVERED once GitHub answers 204. 200 approved

If GitHub fails, the answer is 502 github_failed, and the same commandId may be sent again. Every approval, by a promote or for a dispatched rollback, is for the one deployment id GitHub asked about. GitHub keeps a run's id on a re-run but gives it a new deployment, which waits again and needs a new tap.

rollback. The claim is a stream per repo, rollback:<repo>. Marten appends at the version it read, so of two commands racing, only one claims. 1. The claim. - If the commandId is already on the stream, the answer is the recorded one (dispatched with its run id, unknown, or failed), and nothing is cancelled or dispatched again, also long after that rollback finished. - If another command's claim is still open, the answer is 409 rollback_in_flight. - Otherwise ROLLBACK_CLAIMED (command, actor, role, audit hash) is appended. An append that loses the race re-reads and gets one of the two answers above. 2. Cancel the waiting releases first. A waiting promote holds the repo's production concurrency group, so a rollback dispatched behind it would wait without ever reaching the gate (the same reason supersession cancels, in Release Promotion). So the conductor cancels each waiting run of the repo. They end cancelled, and the portal shows «Avbrutt». A cancelled version can be sent again by re-running its job in GitHub, which makes it wait again. 3. Dispatch rollback.yml on main with an empty deployment_id: the deployment before the current one. return_run_details: true makes GitHub answer with workflow_run_id. ROLLBACK_DISPATCHED goes on the claim and on that run's stream, and the answer is 202 dispatched with the run id. - A refusal (4xx) is ROLLBACK_FAILED, 502 github_failed, and the claim closes. - An unknown outcome (a timeout, a lost answer) is 502 dispatch_unknown, and the claim stays open. The conductor reconciles by listing rollback.yml's workflow_dispatch runs created since the claim whose triggering actor is the release App. One binds as if dispatched. None after five minutes is ROLLBACK_FAILED, the claim closes, and a new tap may try again. More than one binds none, and each waits unapproved (fail closed). 4. Approve that run's first deployment, once. The conductor approves when a protection-rule request arrives for the dispatched run and the run's stream holds no earlier deployment. The request may come before or after the dispatch is recorded, so whichever lands second decides. RELEASE_APPROVED binds to that deployment id, as a promote's does. - A re-run of the rollback run is a new deployment and is not approved by the claim. It waits, and needs a promote tap like any release. - A rollback.yml run the conductor did not dispatch is never approved by a claim: one dispatched in GitHub waits like any release. 5. The claim closes when the run ends (workflow_run completed, whatever the conclusion): ROLLBACK_FINISHED.

GET /api/promotions grows two fields: - kind: release, or rollback for a run the conductor dispatched; - approvedAt: the time of RELEASE_APPROVED, or null.

state keeps its four values. An approved release stays waiting until its run ends, because the portal's copy (0021) has a CHECK on the four states, and a fifth would mean rebuilding that table. The portal ignores fields it does not know, so these can land before the portal reads them.

The Cockpit is pushed when a release is approved and when a rollback is dispatched, with the run link.

3. GitHub writes, as the release App

• The token provider. A second GitHubAppTokenProvider, registered for the release App and not dev-e-bot's. It reads RELEASE_APP_ID and RELEASE_APP_PRIVATE_KEY from the Secret rig-conductor-release-app (keys app-id, private-key), which rig-gitops already provisions, mapped with optional: true. Without them every write fails closed and the commands answer 503.
• The installation comes from GET /repos/{owner}/{repo}/installation with the App's JWT, cached per repo. It is never taken from a payload. Calls go through the github HttpClient, so LocationFenceGitHubHandler applies.

Write	Call	When
Approve	POST /repos/{o}/{r}/actions/runs/{run}/deployment_protection_rule, {environment_name: "production", state: "approved", comment}	promote; a dispatched rollback
Status while waiting	The same URL with comment and no state	Once per deployment that waits: «Waiting for «Send til produksjon» in prosjekt.invotek.no.» GitHub allows ten per deployment
Cancel	POST /repos/{o}/{r}/actions/runs/{run}/cancel	A release recorded RELEASE_SUPERSEDED, and the waiting releases before a rollback. This retires the orchestrator's manual cancel (#83)
Dispatch	POST /repos/{o}/{r}/actions/workflows/rollback.yml/dispatches, {ref: "main", inputs: {deployment_id: ""}, return_run_details: true}	rollback

The approval's comment: «Promoted from prosjekt.invotek.no by the operator (command 1f0c…).». GitHub allows an App to review only its own rule, and the App's permissions (Actions write, Deployments write) cover all four writes. The conductor never rejects a deployment.

4. The job's check: POST /api/promotions/check

The protection rule is in public preview, and an environment's rules can be changed. So the promote job's first step, and rollback.yml's, asks the conductor whether this run attempt may deploy: - The token is a GitHub Actions OIDC token for the audience rig-conductor-promotion, as Authorization: Bearer. GitHubOidcVerifier checks it as for the autonomy decision (RS256, GitHub's JWKS, iss, exp/nbf, repository_owner dashecorp, ref refs/heads/main), with the audience as a parameter. - A Core policy then requires: - environment production; - workflow_ref <repo>/.github/workflows/deploy.yml@refs/heads/main or …/rollback.yml@refs/heads/main; - a release for repository and run_id whose current deployment has RELEASE_APPROVED, and which no other run_attempt has already checked. An approval for an earlier deployment of the run admits nothing: a job cancelled after its approval and before its check, then re-run, is refused. - It records RELEASE_CHECKED with the attempt and the token's jti. The same jti again gets the same answer, so a retried request is safe. - The answer is 200 {"allowed": true, "sha"} or 403 {"allowed": false, "reason"}. The step fails the job unless allowed, and for deploy.yml unless sha is $GITHUB_SHA. - The step is a composite action in dashecorp/.github, actions/promotion-check, beside autonomy-decide. Workflows pin it by sha. - Access. The path gets a bypass app in infra, as /api/autonomy/decision has. The runner holds no Access credential, and the conductor checks every request itself.

A conductor that cannot answer fails the job before it touches production. A re-run then waits for a new tap.

5. The scratch repo

• dashecorp/release-scratch, private, declared in infra like invotek-no.
• Environments:
• staging, ungated;
• production: main only, can_admins_bypass = false, no reviewers.
• The App on its installation (176123484), by github_app_installation_repository.
• The App's rule on production, the REST call in infra BOOTSTRAP.md §26 (the provider has no resource: integrations/terraform-provider-github#2225). Nothing reads the rule back. Its drift check is the job's check, which refuses a run the conductor did not approve whether or not the rule is there.
• Its workflows copy invotek-no's shape, with no secrets and nothing hosted:
• deploy.yml: Build, then «Deploy to staging» (this exact job name, which supersession keys on), then «Promote to production», which runs the check and writes a summary;
• rollback.yml: dispatch only, the check, a summary;
• the same production concurrency group in both.
• A test project in /_admin gets it as its deploy repo, so «Versjoner» lists its runs.

The end-to-end test, all on the scratch repo: 1. A merge waits, and the card lists it. 2. The operator taps «Send til produksjon», once from the phone and once from the PC, on two merges. Each run is approved, checked and green. 3. A second merge supersedes a waiting one, and the conductor cancels the old run. 4. «Rull tilbake» cancels what waits, dispatches, approves and goes green. 5. A re-run of an approved job waits again and does not deploy without a new tap. 6. A rollback double-tapped, and the same command sent again after its run finished: one run, one rollback. 7. A rollback run cancelled after its approval and before its check, then re-run: it waits, unapproved, and its check refuses.

6. The portal (R2)

On the project's card in /_admin, behind Access: - «Send til produksjon» on the waiting release that has no approvedAt. An approved one says «Sendt til produksjon , kjører». - «Rull tilbake» below the list, with a folded confirmation: «invotek.no går tilbake til versjonen før den som er i produksjon nå. En versjon som venter blir avbrutt.» The name is the project's production address. - Each is an operator form route: 1. this origin, a form body, and the run and sha the card showed; 2. a commandId made when the card is drawn, so a double tap is one command; 3. the audit event (promote.requested or rollback.requested, with the repo, run and sha) goes in first; 4. then the signed command, through Access with the portal's service token; 5. the answer is shown in words. - The signed client is pinned by the same verification vectors as the conductor (SHA-256 asserted in both repos). - «Ny versjon venter» e-mails link to the project's card, not the list. - 0023 adds approved_at and kind to the copy. Additive (A1). - A phone-width check of the card.

7. Moved to step 9

POST /api/promotions/confirm, the promotion policy per repo (self_serve, operator_approves), its operator-gated /api/admin/ route, and the role kundeadmin. In steps 5 to 8 only the operator promotes, and an operator's tap needs no confirmation under either policy. When step 9 adds them, the envelope gains a route, as rig-cockpit-task-v1.1 added release.

8. Order

Each piece is its own issue and PR. The conductor's pieces are TDD, Core first (AGENTS.md).

1. rig-conductor: this note.
2. rig-conductor: the invotek-portal-command-v1 verifier and key configuration, pure Core, with the frozen vectors.
3. infra:
4. the scratch repo, its environments and the App installation, and the rule in BOOTSTRAP.md §26;
5. the command key pair, the private key to the portal's production environment;
6. the /api/promotions/check bypass.
7. rig-conductor: the release App's token provider and the four GitHub writes. Cancelling superseded runs goes live here.
8. rig-conductor: promote and rollback, the view's two fields, and the public key in the deploy config.
9. rig-conductor: POST /api/promotions/check.
10. dashecorp/.github: actions/promotion-check.
11. release-scratch: its two workflows.
12. invotek-portal: R2. Its deploy waits for the operator's yes deploy, for the new secret.
13. The end-to-end test above, then step 6 on invotek-no.

Sources

• The plan, §2 to §6, and its decision of 2026-10-30.
• Release Promotion, Operator Task Signature, and the autonomy decision route in 2026-10-04-autonomy-level.md.
• GitHub's REST reference:
• «Review custom deployment protection rules for a workflow run» (204; an App reviews only its own rule);
• «Cancel a workflow run» (202);
• «Create a workflow dispatch event» (200 with workflow_run_id, run_url, html_url; changelog 2026-02-19, «Workflow dispatch API now returns run IDs»).
• GitHub's «Creating custom deployment protection rules» (Actions read, Deployments write; status reports, ten per deployment) and «OpenID Connect» reference (run_id, run_attempt, environment, workflow_ref, a custom audience).
• invotek-no's deploy.yml and rollback.yml (the concurrency group, and previous-id for an empty input).
• invotek-portal src/conductor.js (unknown fields ignored, four states) and migrations/0021_releases.sql (the CHECK).