Soham Dan

As coding agents approach the ceiling on established benchmarks, the field needs evaluations that keep pace with the frontier: tasks that are hard enough to separate the best models, broad enough to reflect real engineering work, and robust enough that the scores can be trusted. This talk shares lessons from building two such benchmarks at Scale AI.
SWE Atlas extends evaluation beyond issue resolution to codebase Q&A, test writing, and refactoring. Its expert-authored tasks are graded with automated checks plus structured rubrics for engineering quality. Frontier agents often pass the functional checks yet fall short on maintainability, edge-case coverage, and cleanup, and their success is inconsistent across repeated attempts.
SWE-Bench Pro V2 refreshes SWE-Bench Pro with expert-repaired tasks and a hardened evaluation protocol that locks down network access and closes routes for solution leakage and reward hacking. This protocol surfaced real instances of agents gaming the grader, and the gap between public and private task scores highlights training-time exposure as a persistent challenge for open benchmarks.
Drawing on both efforts, I will discuss what it takes to build frontier benchmarks: expert-driven task design, rubric-based grading, and rigorous evaluation hygiene, along with open problems in measuring progress toward agents that work like professional engineers.