The Request Is Not the Unit Anymore

“Agents”, and the common product-shift towards long-running “personal agents” have fundamentally changed the client-server interaction. In the past we largely had 2 types of workloads: fast, request-based workloads, and slow, batch processing workloads. Agents are different. They are long-running processes with regular interrupts, and a (general) requirement for sequential message processing. While this last one is not always true, when an agent processes concurrent messages, the agent needs to make that decision (rather than an ingress layer).

Almost every backend I’ve worked on has rested on the same quiet assumption: the request is the unit of work. Something arrives, you authenticate it, you load whatever slice of the user’s state you need out of a database, you do the thing, you write the result back, and you throw the rest away. The process that served the request has no memory of it thirty milliseconds later, and that is very much the point.

That assumption starts to look shaky when the thing on the other end of the connection is an agent. What you’re talking to isn’t a row you rehydrate, it’s a process with a history, a working directory, and in-progress tool loops. You can already see the industry splitting on what to do about it. Some products have landed on a VM per user, warm and waiting behind an authenticated API. Plenty of other agentic systems are still loading the entire agent from scratch on every single request. Both of these can be valid, depending on the structure and requirements of the workload.

My read is that while the stateless request model has a lot of benefits, it is not the most appropriate model for hosted agentic workloads. However, fully isolated and persistent, individually-addressable VMs aren’t either.

What the Request Model Actually Bought Us#

It’s worth being specific about why we ended up here, because the reason was never aesthetic.

Stateless request handling is an operational choice. If any node can serve any request, then load balancing is round-robin, autoscaling is a number you turn up, a rolling deploy is killing pods in sequence, and crash recovery is “the request failed, retry it”. Every one of those gets meaningfully harder the moment a specific request has to land on a specific machine. We pushed durable state into a database because a database is a system that specializes in holding state safely, and an app tier is not.

None of this was ever truly stateless. We had sessions. We had sticky cookies. We had local caches that made a warm node measurably faster than a cold one. The trick was that we kept the stateful part small and kept it outside the application process, so that losing a process stayed cheap. That property is the one actually under threat here, not purity.

Statelessness gave us 2 things:

1. We did not have to concern ourselves with long-running state management
2. We did not have to worry about routing. Every server could service every request.¹

From an agent perspective, statelessness falls short in a couple of ways:

• Rehydration gets expensive: The necessary context to appropriately serve agent requests grows, and rehydration cost grows. This includes conversation context, environment context, and control context. The complexity of this grows when this context is in multiple locations (memory, disk, and persistent network connections)
• Work gets dropped: After the request returns, any ongoing action is effectively abandoned, if not offloaded to a durable execution system. As an example, an in-request agent action cannot watch an external system for 10 minutes, and report back on success.

What I keep coming back to is that context size, cold-start time, request frequency, and request distribution are not four independent dials. Context size and cold-start time are the same number: what one rehydration costs you. Frequency and distribution are also one number: how many messages you serve before you pay that cost again.

The request model only ever assumed the first number was small. It never had an opinion about the second, because until now the second was always one. Where this number was large, offline, batch-based processing implementations were used.

Sequential delivery is a separate problem. Messages should be delivered to agents sequentially. Multiple agents should not be operating on the same context concurrently. This one is not an economic argument, and it does not get better as rehydration gets cheaper. If concurrent messages against a single context are wrong, then something has to serialize them in an agent-implementation-specific way, so the ingress layer is the wrong place to decide it.

Session-Actors#

My proposition is a session-actor model: a session being implemented as a persistent actor on a target machine, with inbound messages intelligently routed based on the agent ID, and a Mailbox pattern implementation for each actor.

What we need for this:

1. A message ingestion and routing system
2. An actor persistence system

These are separate concerns, and I think that separation is the useful part. How a message finds its actor has nothing to do with what that actor is made of. You pick one from each column.

Getting a Message to the Right Actor#

This layer would be substantially similar to stateless systems: an API accepts messages, authenticates and authorizes those messages, and sends them for processing.

I see two methods for routing.

Online coordinator management. A coordinator tracks the destination machine for each individual agent. The API gateway makes a request to the coordinator for each request², and the coordinator responds with the scheduled location, and the lease information (i.e., how long that scheduling decision is valid for). If an agent is not scheduled, the coordinator picks a location (e.g., via a consistent hash across nodes), and returns it with a new lease. The gateway then sends the message (with the new lease) to the upstream machine/broker³⁴. Orleans virtual actors are an implementation of this option.

Partitions and shards. Agents are scheduled as part of shards, with each node/broker being assigned a set of shards. All requests for agents in a shard are assigned to a given node⁵. In this approach, a partitioning message broker (such as Kafka) can be utilized for inbound message processing, and no gateway API call is necessary. The topic uses a relatively high, fixed number of partitions, and each broker explicitly subscribes to the partitions allocated to it. Brokers hand messages to actor mailboxes asynchronously, so one long-running agent turn does not block the rest of its partition. Akka Cluster Sharding is a very similar approach.

What the Actor Is Made Of#

This can be implemented in multiple ways, largely based on the complexity of the agent and isolation required. Whichever you pick, each Actor implementation requires the capability to Snapshot, Shutdown, Lock (block acceptance of new messages), and Rehydrate. The mailbox is part of the Actor’s state, so it is either snapshotted along with the Actor, or stored durably.

Goroutine. My initial PoC of this utilized Actor goroutines for each agent. The Actor is started on a new goroutine, and listens on a chan for inbound messages. When a new message is received, the agent is looked up in a map, and the message passed into the Actor’s mailbox. The agent implementation reads the message, processes it, and writes to its output channel⁶.

gVisor and Firecracker. These are both options for more isolated agents, but the interface remains the same. The Actor is started and managed from an actor map, and messages forwarded through the Actor mailbox. Each implementation owns its own Snapshot and Rehydrate implementation. A Firecracker-based Actor, for example, needs to ensure hardware compatibility between the machine that snapshots it and the machine that rehydrates it.

With a consistent interface these implementations can be mixed. A simple in-house-built agent can live as a Goroutine, while something requiring further isolation can operate as a full VM (machine capability dependent, of course).

The next question is “well, how should I balance resource capacity? I do not have capacity to keep every agent alive at the same time”.

This is a fair question, and the crux of why this isn’t as big of a “stateless-v-stateful” agent serving difference as it might seem. Past the increased complexity of the message routing layer, the ttl of an Actor is fully dynamic. There is (technically) nothing stopping you from rehydrating an actor, serving a single request, and snapshotting⁷/shutting down again. My recommendation would be to find something between this one-request-per-load, and “keep it running forever”. Obviously, if a lease is applicable, the ttl should not exceed the coordinator-issued lease, but within that, you can balance with a sliding-ttl/expiration-time, or use eviction based on message prediction when you hit resource caps. Whichever policy you use, an actor with messages in its mailbox should not be selected for eviction. There are countless ways to balance capacity constraints with cold-start time/cost, and the right answer will be hyper-specific to the workload and context in question.

Operational Cost#

There are obviously some operational downsides of moving away from a purely-stateless architecture.

Deploys now have to drain or migrate live sessions, and some of those sessions are mid-tool-call. “Kill it and let the client retry” isn’t acceptable in this situation. The agent/process is not atomic; preempting it may leave external processes in invalid states, or you are discarding twenty minutes of accumulated work. Durable execution engines (such as Temporal, or DBOS) address this by reducing workflows to minimal units of work, and avoid interrupting those units.

I do not believe this is a blocker. Solutions for safely deploying updates with long-running connections (such as websockets) exist, and can be borrowed here. We can stage rollouts so that a broker is prohibited from refreshing a lease while it is in “terminating” mode, causing sessions to all reschedule to new machines within the maximum TTL configured. A terminating broker uses the Lock capability to stop accepting new messages for an actor, and only snapshots and shuts that actor down between turns, never mid-tool-call.

None of this is new. It’s the actor model, and Erlang/OTP, Akka, and Orleans all worked through these problems long before agents existed. If one of those fits your system, use it: if you’re on .NET, use Orleans, and if you’re already in the Akka ecosystem, use Akka Cluster Sharding. I’m not proposing anything that hasn’t been built before. The point is that building Agents makes this model applicable to a lot more of what we are building today.

Where I Think This Lands#

There are use cases where this approach will pay off, and where it will not. It’s worth saying plainly, and clearly, when none of this applies, because “agents are stateful now” risks being used to justify a lot of unnecessary infrastructure.

It gives sequential agent message-receiving out of the box, and prevents a hydration/snapshot on every request. Clearly if your agent (or process) already depends entirely on external state, or external coordination/mutexes, those are not benefits for you. If your agent is already implemented based on durable execution primitives (e.g., Temporal/DBOS), it’s likely not worth moving away from.

If your agent has no environment, meaning retrieval, classification, or a single call against an API you don’t own, then boot it per request, do the work, and throw it away. What you have is a request handler that happens to call a model. Adding a session layer buys you nothing and hands you a placement problem you didn’t have.

If traffic per user is low and the gaps between turns are long, a pinned sandbox is mostly idle cost. Someone who uses the thing twice a week doesn’t need a warm VM for the other six days. However, if those 2 times a week involve a high number of messages within those 2 use-sessions, this approach can still provide value. You can choose to keep the session warm for 1 hour, and then it goes away until next time. This is, for all intents and purposes, “scale to zero”, on a per-user basis.

Agents are quickly changing how people interact with hosted services. Stateless request processing is probably not the best option for many of these new use cases. Luckily, we have options.