Pixel-based agent
A pixel-based agent perceives interfaces through screenshots, deciding actions from rendered pixels rather than from the DOM or accessibility tree. The approach generalizes beyond the web to any software with a screen, at the cost of heavier inputs, coordinate-based actions, and losing the structure the page already had.
How it works
The pixel approach treats the interface as an image understanding problem: a vision-capable model receives screenshots, locates targets visually, and emits coordinate actions. Its strength is universality, since desktop applications, canvas-rendered interfaces, and hostile markup all look the same to a camera. Its costs are the mirror image: screenshots are token-expensive at every step, coordinates break when layouts shift, and structure the DOM states explicitly (this is a disabled button) must be inferred from appearance. Most production web agents therefore run hybrid perception, structure where available and pixels where not. For recording, the formats are not symmetric: a DOM event stream can render the pixels of any recorded moment, while screenshots cannot recover the structure, which is the practical argument for recording at the DOM layer even in pixel-first agent stacks.
Where you encounter it
Computer-use model releases, agent framework perception options, and cost analyses of agent operations.