Evals
Evals are structured evaluations of AI system behavior: defined tasks, graded outcomes, and comparisons across model versions, prompts, or configurations. For agents, evals score whole trajectories rather than single outputs, and the hard problems are defining "success" for multi-step tasks and diagnosing failures beyond a pass/fail bit.
How it works
An agent eval fixes a task set and environment, runs the agent, and grades results via programmatic checks (did the order exist afterward), model-based judges, or human review. The pass rate is the headline; the diagnostic value lives in the failures, which is where recorded trajectories earn their place: a failed run with a full interface recording can be classified (environment flake, perception error, reasoning error, action error) instead of merely counted. Interface recordings also make evals auditable, since a judge's grade can be checked against what actually happened on screen. Regression evals rerun the suite per change and diff trajectories between versions.
Where you encounter it
Agent development loops, model migration decisions, and release gates; trajectory-level tooling connects evaluations to the sessions that produced them.