BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CAIL//Events//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:CAIL Events
BEGIN:VEVENT
UID:2026-10-27-soham-dan@cail.columbia.edu
DTSTAMP:20261025T183846Z
DTSTART:20261027T160000Z
DTEND:20261027T170000Z
SUMMARY:Agents Seminar: Soham Dan - Towards Building Frontier Coding Bench
 marks
LOCATION:453CSB
DESCRIPTION:As coding agents approach the ceiling on established benchmark
 s\, the field needs evaluations that keep pace with the frontier: tasks th
 at are hard enough to separate the best models\, broad enough to reflect r
 eal engineering work\, and robust enough that the scores can be trusted. T
 his talk shares lessons from building two such benchmarks at Scale AI.\nSW
 E Atlas extends evaluation beyond issue resolution to codebase Q&A\, test 
 writing\, and refactoring. Its expert-authored tasks are graded with autom
 ated checks plus structured rubrics for engineering quality. Frontier agen
 ts often pass the functional checks yet fall short on maintainability\, ed
 ge-case coverage\, and cleanup\, and their success is inconsistent across 
 repeated attempts.\nSWE-Bench Pro V2 refreshes SWE-Bench Pro with expert-r
 epaired tasks and a hardened evaluation protocol that locks down network a
 ccess and closes routes for solution leakage and reward hacking. This prot
 ocol surfaced real instances of agents gaming the grader\, and the gap bet
 ween public and private task scores highlights training-time exposure as a
  persistent challenge for open benchmarks.\nDrawing on both efforts\, I wi
 ll discuss what it takes to build frontier benchmarks: expert-driven task 
 design\, rubric-based grading\, and rigorous evaluation hygiene\, along wi
 th open problems in measuring progress toward agents that work like profes
 sional engineers.\n\nhttps://cail.columbia.edu/events/2026-10-27-soham-dan
URL:https://cail.columbia.edu/events/2026-10-27-soham-dan
END:VEVENT
END:VCALENDAR