An agent that does real work runs for minutes across dozens of model turns. That length is the point, and it's also what breaks it: the platforms we deploy on are built for short requests and are quietly hostile to long ones.

We spent a while fixing that one symptom at a time (a timeout here, a retry buffer there) until the patches started disagreeing with each other. This is the note I wrote to myself before throwing them out and rebuilding the runtime around three properties I actually wanted: persistent, resumable, fast.

• Persistent: a turn's work is durably recorded the moment it finishes.
• Resumable: a run continues on a different machine, exactly where it stopped.
• Fast: durable checkpointing costs about a millisecond per turn (still tiny next to a model call), scheduling stays in microseconds once the graph is hot, and independent work overlaps.

1. What the patch-by-patch version got wrong

The concrete failures, from an audit of the old runtime:

• The agent loop was one long HTTP request, and its state lived in RAM. The message list, tool results, and the reply being assembled existed only in the serving process. The reply was written to the database once, at the end.
• The gateway caps a single request at roughly two to four minutes. A genuinely long turn had its stream cut before it finished, reliably, not rarely.
• Replicas are disposable. A rolling deploy drains the old one; the autoscaler scales in when traffic dips; an out-of-memory kill takes the process with no warning at all. Any of these vaporizes an in-flight run.
• There was no resume. "Recovery" meant reloading the completed messages and starting the turn over from scratch.
• The limits were blunt. A wall-clock timeout and a step cap would end a run mid-turn, at no meaningful boundary, with nothing saved.

The through-line is that every fix lived where the symptom showed up (in a route handler, a client, a timeout constant).

2. Serialization technique is key

If you're going to record state after every turn, the recording has to be nearly free, or you can't afford to do it often enough to matter.

Two rules make it cheap:

Append the delta, not the state. Each completed turn appends one small record (its id, its parents, its structured output, what it spawned) to an append-only log. That's O(1) per turn. The obvious alternative, re-serializing the whole run each step, is O(n²) over the run. In a few hundred turns you're writing megabytes to record a kilobyte of new information.

Keep bytes out of the record. A record is a protobuf message: a fixed schema, compact on the network, and versionable as the run format evolves, which matters when you're reading back logs written by an older build. When a turn produces a file (a rendered document, an image), the bytes go to blob storage and the record holds only a pointer: a URL, a size, a hash. The log carries references. The object store carries payloads. That is the difference between a record that's a kilobyte and one that's a megabyte, and it's what keeps serialization itself in the microsecond range. The durable write is still a Postgres INSERT (~1.5 ms); small records just keep that write cheap enough to do after every turn.

We will use protobuf over JSON, because the encoded record is small, decoding thousands of them to rehydrate a run is fast. An old field that changed meaning is a compile-time conversation, not a silent parse of the wrong shape months later.

3. A graph you append to, one node at a time

You can't author the graph up front, because you don't know the steps. That's what the agent decides. So the graph is discovered at runtime. Each turn is a node -> running a node produces output and a list of successor nodes -> those run and produce more. The shape emerges from execution.

Two things make the graph better than a chain. First, a turn is I/O-bound. It spends its life waiting on a model or a tool, so a node routinely issues several tool calls at once, and the executor runs independent nodes concurrently. On a single thread with asyncio, forty independent I/O-bound calls that would take two seconds in series finish in about fifty milliseconds. There's no CPU contention because the executor's own work is negligible. Second, a sub-agent is just a node whose execution runs its own sub-graph. The parent doesn't need to know how the child decomposes; it only waits on the child's completion record.

Because the log is append-only and each record is written atomically, the log is always a clean prefix of the run, even when a turn is cut off. There's no half-written state to repair: either a node's record is in the log or it isn't. Recording is keyed by node id, so re-running is idempotent. A resumed run never double-applies a turn it already completed.

The graph log lives in Postgres, not in process memory and not in a cache. One table, one row per completed node. The primary key is (run_id, seq): run_id scopes the row to a specific agent run, and seq is a monotonically increasing counter that orders the appends. Everything else about the node (its id, parents, structured output, what it spawned, budget used) is packed into a protobuf blob in a BYTEA column. Appending is a single statement:

INSERT INTO run_log (run_id, seq, record)
VALUES ($1, $2, $3)
ON CONFLICT (run_id, seq) DO NOTHING;

If the write is retried after a crash, the conflict clause turns it into a no-op instead of a duplicate. That table is the source of truth.

A row after a tool-calling turn might look like this conceptually (the record column is binary on disk; shown decoded):

run_id | seq | record
-------+-----+----------------------------------------------------------
 a7f3… |  12 | NodeRecord {
             |   id: "turn_4",
             |   parents: ["turn_3"],
             |   kind: "llm_turn",
             |   output: { tool_calls: ["search", "fetch_doc"] },
             |   spawned: ["tool_search_0", "tool_fetch_1"],
             |   budget: { tokens: 1840, wall_ms: 620 }
             | }

We had briefly used a single-node cache to hold a run's result instead. That looks fine right up until the cache does its job and evicts.

4. Rehydration

Resuming a run is: read the log back from Postgres, replay it to rebuild the graph and the set of still-pending nodes, sum the per-node budgets, and continue. It's fast because the log is a flat sequence of small records. Replaying five thousand of them is about eight milliseconds.

The rehydrate path is a short loop over the ordered log:

def rehydrate(run_id: str) -> Graph:
    rows = db.fetchall(
        "SELECT seq, record FROM run_log WHERE run_id = %s ORDER BY seq",
        (run_id,),
    )
    graph = Graph.empty()
    for seq, blob in rows:
        node = NodeRecord.decode(blob)   # protobuf → struct
        graph.apply(node)                # attach node, spawn edges, mark done
        graph.budget.add(node.budget)
    graph.pending = {
        n for n in graph.nodes
        if n.parents_done and n.id not in graph.completed
    }
    return graph

Once that graph is in memory, the executor schedules from pending and never asks Postgres again to read the run. Postgres is for durability on the write path: each completed node still appends a row (~1.5 ms). You only run the fetch-and-replay above when the graph isn't already resident (a fresh replica after a crash, a deploy, or the first request that wakes a parked run). Keeping the live agent on the in-memory graph matters because every turn would otherwise pay that insert plus a round-trip read of the whole log. Staying hot means scheduling and fan-out stay in microseconds; the durable store is written every turn, and re-read only when process memory is gone.

A cache in front of the store buys nothing here, because rehydration is bound by decoding the five thousand records rather than fetching them, so reading them back from a cache lands in the same ~8 ms and only adds a component that can go stale.

5. SIGTERM before SIGKILL

Most of the ways a replica dies give you a warning. A rolling deploy, a scale-in, a graceful pod eviction: the platform sends SIGTERM first and waits (around thirty seconds by default) before SIGKILL.

On SIGTERM the executor stops launching new nodes, lets the in-flight ones finish and append their records, and returns a paused run that resume picks up later. It does not start work it can't finish in the grace window. The whole handler is a few lines:

drain = asyncio.Event()
loop.add_signal_handler(signal.SIGTERM, drain.set)
# ...in the scheduler loop:
if not draining:
    launch_ready_nodes()          # stop admitting work once draining
await first_completed(running)    # let in-flight nodes finish and checkpoint
# frontier is now a clean prefix; a fresh replica resumes from it

The nodes that don't finish in time aren't lost. They were never marked done, so resume re-runs them, idempotently. SIGTERM just shrinks the amount of re-work to whatever was genuinely in flight at the moment the platform reclaimed the box.

What it adds up to

The runtime is small and the properties fall out of the one structural choice rather than being bolted on:

• Persistent: every turn appends a durable record the instant it completes.
• Resumable: any replica rebuilds a run from its log and continues, exactly once per node, whether the run was cut by a deploy, a scale-in, an OOM, or a SIGTERM.
• Fast: ~1.5 ms to durably checkpoint a turn, microseconds for in-memory scheduling, milliseconds to replay thousands of nodes, and real concurrency across the I/O-bound work.