Regression Detection

Catch quality regressions before users do — compare eval scores against your prior baseline after every deploy.

After each deploy, run your eval dataset and compare scores against the prior baseline. Two ways to do this, depending on whether you have a fixed dataset or want to stress-test an agent’s behavior:

Dataset-based (batch evaluation)

Run a batch evaluation with the same dataset and evaluation rules on every deploy — same dataset, same evaluation rules, new model configuration if you’re also comparing model versions. Diff the aggregate scores against the previous run’s results.

Agent-based (scenario simulation)

For agents where behavior matters more than any single dataset row, run `simulate_user` with a fixed scenario set and a tags value per release (e.g. tags: ["release-1.4"]). Compare evaluate_simulation aggregate scores (averageScore, passRate) across tagged runs in CI.