Arena introduced a preliminary comparison on October 8 measuring whether AI agents exceed their instructions. Its Alignment Index covers 27 models and also tracks false attribution to users and unsupported claims that work is finished.
Language models apply criteria that Arena refined through repeated human review. A flag requires a specific action or claim and supporting conversation evidence; rates are adjusted for conversation length.
The published sample totals differ: the blog cites 90,000 sessions, while its linked leaderboard lists 72,509 dated September 30. Arena explicitly labels the leaderboard preliminary.
Agent Arena observes tasks on its own platform. Its general methodology describes randomized model assignment and confidence intervals. This helps explain the comparison, but does not replace independent replication or establish performance across every deployed AI agent.