BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CAIL//Events//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:CAIL Events
BEGIN:VEVENT
UID:2026-11-05-deqing-fu@cail.columbia.edu
DTSTAMP:20261104T035112Z
DTSTART:20261105T210000Z
DTEND:20261105T220000Z
SUMMARY:NLP Seminar: Deqing Fu - Emergent Algorithms in Foundation Models:
  Theory\, Discovery\, and Design
LOCATION:CSB 453\, Computer Science Building\, 500 W 120th St\, New York\,
  NY 10027
GEO:40.809528;-73.960074
DESCRIPTION:Join on Zoom (Columbia sign-in): https://cail.columbia.edu/eve
 nts/2026-11-05-deqing-fu/zoom\n\nUnlike traditional computers\, which run 
 the programs we write\, foundation models built on the Transformer archite
 cture learn their programs from data. In this talk\, I show that on three 
 problems with textbook solutions\, Transformers learn algorithms that nobo
 dy programmed into them and that differ from the textbook ones\, and that 
 the training data decides whether they learn an algorithm at all. I first 
 present evidence that Transformers trained for in-context linear regressio
 n implement Iterative Newton’s method rather than gradient descent\, whi
 ch explains their second-order convergence rate and their robustness to il
 l-conditioned data. I then turn to graph connectivity\, where a simplified
  L-layer Transformer provably decides reachability by powering the adjacen
 cy matrix\, with a capacity of exactly 3^L in graph diameter. Training gra
 phs beyond this capacity add a shortcut based on node degrees alongside th
 e algorithm\, and surprisingly\, restricting the training data to within c
 apacity removes the shortcut\, so that even standard Transformers learn th
 e clean matrix-powering algorithm. Finally\, pretrained language models ad
 d numbers using Fourier features\, which emerge across architectures by co
 nvergent evolution. Building these features directly into number embedding
 s improves arithmetic\, and now encodes numbers in tabular foundation mode
 ls\, including Google’s TabFM\, which I co-developed. Together\, these f
 indings suggest that Transformers are best understood through the algorith
 ms they learn. Reverse-engineering these mechanisms opens new opportunitie
 s for algorithm discovery\, and for designing the data and representations
  that shape them.\n\nhttps://cail.columbia.edu/events/2026-11-05-deqing-fu
URL:https://cail.columbia.edu/events/2026-11-05-deqing-fu
END:VEVENT
END:VCALENDAR