Skip to content

Agent Auto-Archive Policy

Context

On 2026-05-12 a rig audit found 14 records in GET /api/agents but only 11 correspond to running agents. Three are permanently stale:

Agent ID Reason stale
dev-e-node-1 KEDA ordinal suffix from a brief 2-replica scale window; reverted to maxReplicas=1 per feedback_dev_e_kedo_max_1_with_shared_pvc.md
dev-e Pre-rename ID — now dev-e-node; left over from the Apr/May rig migration
iclaw-e Retired per memory reference_openclaw_iclaw_status_unused.md

Without a cleanup and a forward policy, the list grows unbounded every time an agent is renamed, replaced, or a KEDA ordinal suffix is created and reverted.

Decision

Introduce AgentArchiveService — a daily background service that soft-deletes (archives) stale agent records rather than hard-deleting them, preserving the data for forensic queries.

One-time cleanup

The three IDs above are listed in AgentArchiveService.KnownStaleIds and are archived unconditionally on the first service startup — regardless of their lastHeartbeatAt value. This fires automatically when the new image is deployed.

Ongoing policy (≥ rig-conductor#800)

Any agent that has not sent a heartbeat for > 7 days is automatically archived on the next daily scan.

This used to claim that idle agents heartbeat every 60 s, so the threshold could only catch abandoned records. That is true for always-on agents, but not for the KEDA minReplicaCount: 0 fleets (review-e, dev-e-, rig-planner). They send nothing while scaled to zero, so one that sits idle for 7 days is archived as well. Archiving is therefore not proof that an agent is gone, and it is reversible (next section).

Threshold is configurable via AGENT_ARCHIVE_THRESHOLD_DAYS (integer; default 7).

Un-archive on heartbeat (rig-conductor#2376)

Until rig-conductor#2376, archiving was one-way. Both heartbeat writers copied ArchivedAt forward, and nothing cleared it. A returning agent stayed invisible to every agent reader. The live case was iBuild-E: archived 2026-05-28, heartbeating again on 2026-09-24, and absent from GET /api/agents. StaleClaimWatcher released every claim to it as stale-claim: agent ibuild-e no agent record, 8,211 times for dashecorp/rig-cockpit-ios#20 alone.

Now a heartbeat clears ArchivedAt when all of these hold (AgentArchivePolicy.ShouldUnarchive, pure, in ConductorE.Core):

  • the agent is archived;
  • the heartbeat is strictly newer than ArchivedAt, so a beat that predates the archive does not count;
  • the heartbeat is no more than 2 minutes old at the time it is written (the isOnline window of GET /api/agents);
  • the id is not one of the three KnownStaleIds and does not start with human-.

Excluded ids never come back:

Excluded Why
dev-e, dev-e-node-1, iclaw-e (KnownStaleIds) Retired. The archive scan archives exactly this set every time, so un-archive must exclude the same set or they flip daily. iclaw-e heartbeated on 2026-07-30, after its 05-19 archive. The set lives in AgentArchivePolicy and AgentArchiveService references it.
human-* The developer-hook presence signal, not a dispatchable agent. It always reports working plus the repo the developer is in, so an un-archive would list a private repo name on /public/agents and page the watchdog after every 10-minute pause.

Both heartbeat writers apply the rule. AgentStatusProjection.Apply(Heartbeat) covers normal beats. MartenAgentLivenessWriter covers the soft-drop path, which runs when a HEARTBEAT exhausts its concurrency retries (rc#1870). ibuild-e and review-e take that path often. Each writer does a full-document Store with last-writer-wins, so if only one of them cleared the archive, the other could write the archived copy back.

Other events (AGENT_STARTED, ISSUE_ASSIGNED, WORK_STARTED, TOKEN_USAGE, …) never un-archive. They do not move lastHeartbeat, so an agent they brought back would still look dead to StaleClaimWatcher's 30-minute stale path and to the watchdog.

No AGENT_UNARCHIVED event. The issue proposed emitting one from Apply(Heartbeat). A Marten projection Apply is a pure function and cannot append events or log. Emitting from /api/events would add a document read before every HEARTBEAT and a second append to the per-agent stream, which is the stream whose version contention causes the soft-drops. The archive itself is not an event either, so an un-archive event would add nothing on replay.

Replay. ArchivedAt is a document write, never derived from an event. A projection rebuild resets it to null for every agent, including KnownStaleIds, until the next archive scan (startup + 30 s, then every 24 h). That was already true before #2376. The un-archive rule never turns a null into a value, so it changes nothing under replay.

The cycle for scale-to-zero agents. A KEDA min-0 agent idle for 7+ days is archived, then un-archived on its next wake. That is expected. While it is archived, or while its heartbeat is over 30 minutes old, a new claim to it can still be released if a StaleClaimWatcher tick (every 5 min) lands before the woken pod's first heartbeat. rig-conductor#2375 caps that loop.

Alerting. Un-archiving puts review-e, dev-e-python, dev-e-python-codex and rig-planner-codex back in front of AgentHeartbeatWatchdog. Before they were archived, review-e alone fired about 9 AGENT_DOWN a day while idle. The watchdog therefore no longer treats a stale heartbeat from a scale-to-zero agent or a human-* hook as down. See Agent Heartbeat Watchdog.

API changes

GET /api/agents (default)

Returns only active (non-archived) agents — same semantics as before for all existing callers.

GET /api/agents?archived=true

Returns all agents including archived ones. Each archived record includes an archivedAt timestamp.

{
  "id": "dev-e-node-1",
  "status": "offline",
  "archivedAt": "2026-05-13T12:00:00Z",
  ...
}

No change to other callers

All internal services that call IAgentQuery.GetAllAsync() (dispatch, watchdog, reconciler, stale-claim watcher, etc.) use the default includeArchived: false and will never see archived agents. Archived agents are invisible to routing and alerting until a fresh heartbeat un-archives them. From then on every reader sees them again, with archivedAt: null.

Data model change

AgentStatus gains one nullable field:

/// Set by AgentArchiveService when the agent is soft-deleted.
/// Null for active agents.
public DateTimeOffset? ArchivedAt { get; init; }

The field is set via a direct Marten document store (not an event) by MartenAgentArchiveStore, and cleared by the two heartbeat writers (see "Un-archive on heartbeat"). Existing documents (without the field) read as null and are therefore treated as active — backward-compatible with all prior agent records.

Configuration

Env var Default Description
AGENT_ARCHIVE_THRESHOLD_DAYS 7 Days without a heartbeat before an agent is auto-archived

Multi-tenancy (PR-4)

AgentArchiveService runs its scan once per active tenant via ITenantWorkRunner.ForEachTenantAsync — each tenant in its own DI scope with ITenantContext.TenantId set, so the IAgentArchiveStore read/write binds to that tenant's database (each tenant archives only its own stale agents). Single-tenant today ⟹ one pass for invotek (behaviour-neutral). The service no longer opens its own DI scope; the runner supplies the tenant-scoped provider.

Implementation files

  • src/ConductorE.Core/Domain/AgentArchivePolicy.cs — pure un-archive rule and the shared KnownStaleIds (rig-conductor#2376)
  • src/ConductorE.Api/Adapters/MartenProjections.cs — Apply(Heartbeat) clears ArchivedAt
  • src/ConductorE.Api/Adapters/MartenEventStore.cs — MartenAgentLivenessWriter (soft-drop) clears ArchivedAt
  • tests/ConductorE.Core.Tests/Domain/AgentArchivePolicyTests.cs, tests/ConductorE.Api.Tests/AgentUnarchiveOnHeartbeatE2eTests.cs — policy and e2e tests
  • src/ConductorE.Core/Domain/ReadModels.cs — ArchivedAt field on AgentStatus
  • src/ConductorE.Core/Ports/IEventStore.cs — IAgentArchiveStore + updated IAgentQuery.GetAllAsync(includeArchived)
  • src/ConductorE.Api/Adapters/MartenEventStore.cs — MartenAgentQuery (filter) + MartenAgentArchiveStore (write)
  • src/ConductorE.Api/Services/AgentArchiveService.cs — background service
  • src/ConductorE.Api/Program.cs — service registration + ?archived=true query param on /api/agents
  • tests/ConductorE.Api.Tests/AgentArchiveServiceTests.cs — unit tests