BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CAIL//Events//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:CAIL Events
BEGIN:VEVENT
UID:2026-11-20-pavel-izmailov@cail.columbia.edu
DTSTAMP:20261118T190113Z
DTSTART:20261120T160000Z
DTEND:20261120T170000Z
SUMMARY:ML Seminar: Pavel Izmailov - Understanding Reasoning from Pretrain
 ing to Post-Training
LOCATION:SSW 903\, School of Social Work\, 1255 Amsterdam Ave\, New York\,
  NY 10027
GEO:40.81022;-73.958347
DESCRIPTION:Reinforcement learning (RL) has become central to improving la
 rge language models (LLMs) on complex reasoning tasks\, yet RL post-traini
 ng is largely studied in isolation from the pretraining that precedes it. 
 As a result\, two basic questions remain open: (1) how do pretraining choi
 ces (model size\, data) shape the returns to RL compute\, and (2) what doe
 s RL actually do to the model? These questions are difficult to study in t
 he standard LLM setting: pretraining corpora are vast and uncontrolled\, m
 aking it hard to attribute behaviors to pretraining versus RL\, and system
 atic compute sweeps across both stages are prohibitively expensive. To add
 ress these challenges\, we use chess as a controlled testbed for studying 
 reasoning across the full pretraining-to-post-training pipeline. We follow
  the standard LLM training pipeline by pretraining language models from 5M
  to 1B parameters on human chess games\, supervised fine-tuning on synthet
 ic reasoning traces\, and running RL on chess puzzles with verifiable rewa
 rds. Using this framework\, we find that the post-RL performance at given 
 RL compute level is well-predicted from the pretraining loss\, and slope o
 f the RL reward curves improves approximately linearly with the pretrainin
 g tokens. Beyond scaling\, we find that RL does not simply sharpen the SFT
  policy: on easy puzzles it amplifies correct moves the SFT policy already
  preferred\, while on hard puzzles it surfaces correct moves that were nea
 rly absent under SFT. We further test whether our findings transfer beyond
  chess by training a 1B language model on math-domain text\, where the sam
 e predictive pattern emerges: longer-pretrained checkpoints reach higher p
 ost-RL performance and improve faster under RL. In sum\, we provide a quan
 titative account of the pretraining-to-RL interface and a controlled testb
 ed for studying the science of reasoning across the full pretraining-to-po
 st-training pipeline.\n\nhttps://cail.columbia.edu/events/2026-11-20-pavel
 -izmailov
URL:https://cail.columbia.edu/events/2026-11-20-pavel-izmailov
END:VEVENT
END:VCALENDAR
