We Accidentally Built Infrastructure for AI Agents

There's a question I keep getting asked at conferences, in Slack DMs, on GitHub threads: "How does rrweb fit into the AI agent world?"

And honestly? I think most people are asking the wrong version of that question. They're thinking about AI agents the way we thought about mobile in 2009 - as a feature you bolt onto what already exists. But AI agents aren't a feature. They're a completely new runtime. And like every new runtime before them, they're going to need infrastructure that doesn't exist yet.

Except some of it does. We just didn't know that's what we were building.

The recording problem nobody's talking about#

Here's what's happening right now: companies are deploying AI agents that browse the web, fill out forms, navigate workflows, and take actions on behalf of users. Some of these agents are doing thousands of sessions per hour. And when something goes wrong - and it will go wrong - someone asks: "What exactly did the agent do?"

Good luck answering that.

Most teams are logging API calls, maybe capturing screenshots at intervals, maybe dumping the DOM at certain checkpoints. It's the equivalent of debugging a car crash by looking at the odometer readings. You know something happened between mile 43 and mile 44, but you have no idea what.

What you actually need is a faithful recording of everything the agent saw and did. The full session. Not screenshots. Not logs. The real thing, reconstructable and replayable.

That's session replay. That's what rrweb does. But - and this is the part that gets interesting - the requirements for AI agent replay are fundamentally different from the requirements for traditional user session replay. And if we don't get ahead of those differences now, we're going to end up with infrastructure that doesn't scale.

Tokens are the new bandwidth#

In the human session replay world, storage is your main cost driver. You record a session, you compress it, you store it, and when someone wants to watch it, you stream it to a player. The consumer is a human with eyes. Humans are great at watching things.

AI agents aren't watching replays. They're reading them. And every byte they read costs tokens.

Think about what that means. When an AI agent needs to understand a recorded session - whether that's for debugging, for learning, for auditing, or for re-executing a workflow - it needs to parse that recording into something it can reason about. And the cost of that reasoning scales directly with the size of the recording.

So suddenly, the optimisation target shifts. It's not just "make the recording small for storage." It's "make the recording semantically dense for consumption." You need recordings that are token-efficient - that carry the maximum amount of meaning in the minimum number of tokens.

This is a different compression problem than we've ever dealt with. Traditional compression optimises for byte size. Token-optimised compression needs to optimise for information density relative to an LLM's context window. That means stripping out visual noise that a human would want but an AI doesn't need. It means preserving structural information - DOM mutations, interaction sequences, state changes - in formats that LLMs can parse cheaply. It means thinking about recordings not as video-like artefacts but as structured event logs that happen to be replayable.

We've been working on this. rrweb's architecture - event-sourced, mutation-based, structurally aware - turns out to be almost perfectly suited for it. We didn't design it for AI agents, but the same properties that make rrweb recordings compact and faithful also make them token-friendly. That's not a coincidence. Good infrastructure tends to find its moment.

Auditability isn't optional#

Let me paint a scenario that's going to become very common, very soon.

An AI agent, acting on behalf of a customer, navigates a procurement workflow. It fills out forms, selects options, confirms a purchase. The purchase is wrong. Maybe the agent misread a dropdown. Maybe it hallucinated a field value. Maybe the workflow changed since the agent last ran it and nobody updated the automation.

Who's accountable? The customer who deployed the agent? The vendor whose workflow the agent navigated? The platform that orchestrated it?

I don't know. Nobody knows yet. But I do know that whoever can prove what happened is in a much better position than whoever can't.

This is the auditability problem, and it's going to be enormous. Every AI agent action that touches real systems - financial transactions, data modifications, user-facing changes - needs an auditable, tamper-evident, replayable record. Not a log file. A full reconstruction of what the agent saw, what it did, and what the system looked like at every step.

This is session replay as a compliance layer. It's replay as a legal record. And it requires recordings that aren't just accurate but verifiable. You need to be able to prove that the recording faithfully represents what happened, that it hasn't been modified, and that it can be reconstructed deterministically.

rrweb's deterministic replay model - where the same event stream always produces the same visual output - gives you that foundation. But the surrounding infrastructure needs to be purpose-built: integrity hashing, chain-of-custody metadata, timestamp verification. We're building these layers because the market is going to demand them before most people realise it.

The storage wall#

Now let's talk about scale, because the numbers get uncomfortable fast.

A typical human user generates maybe 5-10 session recordings per day on a product they use regularly. An AI agent can generate thousands. If you're running a fleet of agents across workflows, you're looking at orders of magnitude more recording data than any traditional session replay use case.

At that volume, storage isn't just a cost - it's a wall. If your recording format isn't ruthlessly optimised, you simply cannot afford to record everything. And if you can't record everything, you're back to sampling, which means you're back to hoping that the session where things went wrong happened to be one you captured.

This is where rrweb's event-sourced architecture pays off in ways we always knew it would but couldn't fully justify until now. Because rrweb records mutations rather than states, the recording size scales with the amount of change, not the complexity of the page. A dense, complex application that sits mostly idle produces tiny recordings. An agent that clicks through a workflow generates a recording that's proportional to the actions taken, not the size of the DOM it was acting on.

But we need to go further. We're working on tiered recording strategies - full fidelity for critical workflows, compressed structural-only recording for routine operations, and ultra-light checkpoint formats for bulk agent sessions. The goal is to make the cost curve flat enough that recording every agent session is a line item, not a budget conversation.

From recording to recreation (and beyond it)#

Here's the piece that ties it all together, and it's the one I'm most excited about.

Right now, the AI agent ecosystem is fragmented. You've got agents built on Playwright, on Puppeteer, on custom browser automation stacks. You've got headless browsers, headed browsers, cloud browser farms. Everyone has their own way of driving a browser and their own way of capturing what happened.

What's missing is a reliable pipeline from recording to recreation. Take a recorded session - whether it was a human user or an AI agent - and deterministically recreate it in an automated environment. Not "replay it visually in a player." Actually re-execute it. Feed the recording into Playwright. Drive a real browser through the same sequence. Verify that the same outcomes occur.

Today, that means building bridges between rrweb's recording format and automation frameworks. rrweb recordings contain enough structural and temporal information to generate Playwright scripts, to drive Cypress tests, to serve as the ground truth for workflow verification. The event format was designed to be lossless and deterministic, which means you can go from recording to recreation without the "well, it works most of the time" caveat that plagues most record-and-replay tools.

We're building those bridges. They matter right now. But here's what I think happens next.

Playwright, Cypress, Puppeteer - these are translation layers. They exist because traditional automation needs explicit, step-by-step instructions: click this selector, wait for this element, type this string. The recording has to be converted into a script that a dumb executor can follow. That conversion is where fragility lives. Selectors break. Timing assumptions fail. The script is a lossy compression of the original intent.

AI agents don't need that translation.

An AI agent can consume an rrweb recording directly - the DOM structure, the mutation sequence, the interaction pattern - and understand the intent behind the workflow. It doesn't need a Playwright script that says page.click('#submit-btn'). It can look at the recording and understand "the user submitted the form." And then it can re-execute that intent in a live browser, adapting on the fly when a button has moved, a class name has changed, or a confirmation dialog has been added.

The recording becomes the program. Not a brittle script - a semantic description of what happened, rich enough for an intelligent agent to recreate it. The Playwright layer doesn't disappear overnight, but it becomes an intermediate format for teams that aren't ready to hand the keys to an AI. For everyone else, the recording is the automation.

This is why the recording format matters so much. If your recordings are just pixel data or screenshot sequences, an AI agent has to do expensive visual reasoning to figure out what happened. If your recordings are structured, event-sourced, DOM-aware - like rrweb's - then the agent gets a clean, parseable, token-efficient representation of exactly what occurred. The gap between "recording" and "instruction set" shrinks to almost nothing.

We're building for both worlds right now: robust Playwright and Cypress pipelines for teams that need deterministic script-based recreation today, and token-optimised recording formats that AI agents can consume directly tomorrow. My bet is that the second world arrives faster than most people expect.

Infrastructure, not analytics#

I've said this before and I'll keep saying it: rrweb is infrastructure, not analytics.

Analytics tools show you what happened. Infrastructure lets you build things on top of what happened. The AI agent world doesn't need another dashboard. It needs plumbing - reliable, scalable, boring plumbing that recordings can flow through, that automations can plug into, that compliance systems can trust.

That's what we're building. That's what we've been building, even before we knew this was where it was going.

The AI agent era isn't coming. It's here. And the teams building agents are about to discover what every software generation discovers eventually: the glamorous part is the intelligence. The hard part is the infrastructure.

We're here for the hard part.