Long-running work is a state machine

A build ran for nine minutes, completed four file edits, reached its cost limit, and stopped before verification. The user typed "continue." The agent began by listing the project and reading every file again because the previous process held its progress only in memory. The resume cost almost as much as the original attempt.

The system had durable files but no durable workflow state. Long-running work needs both.

State before orchestration

The diagram is useful only if each arrow corresponds to one database update with a guarded source state.

Terms

• State machine: A finite set of states and allowed transitions.
• Terminal state: A state with no automatic outgoing transition.
• Checkpoint: Bounded durable data needed to continue without rediscovery.
• Lease: Time-limited ownership of work by one worker.
• Occurrence: One scheduled firing of a recurring job.
• Compensation: A later action that mitigates an effect that cannot be rolled back.
• Isolated turn: A turn started by a schedule or event without a live user request.

Represent transitions in data

Avoid boolean collections such as is_running, is_paused, and is_done. They permit impossible combinations. Use one status plus timestamps and counters:

CREATE TABLE jobs (
  id              TEXT PRIMARY KEY,
  status          TEXT NOT NULL CHECK (status IN (
                    'pending', 'running', 'waiting_approval',
                    'paused', 'completed', 'failed', 'cancelled'
                  )),
  attempts        INTEGER NOT NULL DEFAULT 0,
  next_run_at     TIMESTAMPTZ,
  lease_until     TIMESTAMPTZ,
  checkpoint      JSONB,
  last_error_code TEXT,
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

Guard every transition:

UPDATE jobs
SET status = 'running',
    attempts = attempts + 1,
    lease_until = now() + interval '2 minutes',
    updated_at = now()
WHERE id = $1
  AND status = 'pending'
  AND (next_run_at IS NULL OR next_run_at <= now())
RETURNING *;

If no row returns, another worker won or the state changed. The caller should not continue.

Keep checkpoints small and useful

A checkpoint should answer "what must the next turn know to proceed?" It should not duplicate the full transcript.

For a multi-file build, useful checkpoint fields might include:

• current immutable revision;
• files changed in the previous turn;
• last successful verification;
• failed step index and stable error code;
• newest few tool results;
• budget consumed;
• pending approval ID.

The next turn can also receive deterministic project metadata such as file outlines and revision history. That reduces whole-file reads without relying on a model-written summary.

Bound checkpoint size. A recent-tool checkpoint that grows without limit moves the context problem from the provider request into the database.

Schedules are state machines too

A recurring task needs next_run_at, last_run_at, occurrence identity, cooldown, attempt count, and a policy for missed runs.

Do not infer state solely from cron text. Compute and persist the next occurrence after a successful claim or completion, according to the chosen semantics. Decide what happens after downtime:

• run every missed occurrence;
• run only the newest missed occurrence;
• skip missed occurrences and schedule the next future one.

For personal reminders and monitoring, running only the newest missed occurrence is often the least surprising. Financial or compliance jobs may require every occurrence.

Record one run row per occurrence. The stable key (job_id, scheduled_for) prevents duplicate firing after a crash.

Approval is a transition, not a sleeping process

Keeping a model request open while waiting for a human is fragile and expensive. Instead:

1. the tool creates a pending approval tied to exact input;
2. the current turn ends with waiting_approval;
3. the UI or channel records approve or deny;
4. approval creates a new durable request;
5. the new turn reissues the tool;
6. the wrapper finds the matching unexpired approval and executes.

This design reuses the ordinary outbox. It survives restarts and works across web and messaging clients. It does spend another model call. That cost is acceptable compared with holding an opaque provider session open.

Long model turns versus deterministic pipelines

Do not make the model orchestrate steps that code can determine. Verification, static checks, browser flows, and publication order are often deterministic. Wrap them in one task-shaped tool or job pipeline.

For example:

const pipeline = [
  { kind: "save_revision" },
  { kind: "static_check" },
  { kind: "browser_flow", timeoutMs: 30_000 },
  { kind: "publish_if_safe" },
] as const;

Persist the current step and each result. The model chooses files and acceptance intent. Code enforces the sequence and stopping rules.

This division lowers tool iterations and prevents the model from claiming that it verified work without running the verifier.

Decisions and rejected approaches

Prefer short claims to long database transactions. Claim, commit, perform remote work, then commit the result. Holding locks during model or browser calls harms concurrency and recovery.

Use leases only when several workers exist. A single worker can reset running work on startup. Leases add clock and renewal edge cases. Add them when process overlap is real.

Fail visibly after bounded retries. Infinite retry loops hide broken jobs and consume quota. Keep the last safe error code and post a user-visible result where appropriate.

Persist next action, not model reasoning. Chain-of-thought is unnecessary and sensitive. Revision, failed step, operation IDs, and compact outputs are enough to resume.

Use compensation for irreversible effects. A sent email cannot roll back. If a later step fails, record the partial result and offer a corrective action. Pretending the whole workflow is transactional is dishonest.

Failure modes

The process can die after the external effect but before the transition to completed. Use the operation key with the provider, then reconcile on retry. If reconciliation is impossible, enter an uncertain or failed state that requires human choice.

A lease can expire while a slow worker still runs. The worker must verify ownership before committing. Long steps should renew leases or use a lease longer than the measured upper bound.

A paused checkpoint can reference deleted state. Validate the revision on resume. If it no longer exists, explain the conflict instead of applying stale edits.

A schedule can remain running after restart. Startup recovery or lease expiry should return it to pending only when its occurrence key keeps the effect safe.

An event-triggered job can ingest a validly signed payload containing instructions. Pass the payload as labelled data, not as system authority.

Operations checklist

• One status column excludes impossible combinations.
• Every transition checks its expected source state.
• Remote work happens outside database transactions.
• Multi-worker claims have leases or row-lock ownership.
• Checkpoints are bounded and contain revision-level facts.
• Resume validates referenced revisions.
• Scheduled occurrences have stable identities.
• Missed-run policy is explicit.
• Approval ends the current turn and creates a new durable request.
• Deterministic steps run in code, with persisted step results.
• Retry caps produce visible terminal states.
• Uncertain external effects are represented rather than guessed.

Stable identities from Idempotency is a user feature make transitions safe to retry. Durable state now lets work stop and resume. The next constraint is how much of that state should enter each model request. Continue with Context is a budget.