Aviral Kumar

There has been remarkable progress in using LLMs to solve hard problems, even just over the past few months, driven by scaling both pretraining and post-training. Many of these problems involve a fundamental search or “exploration” bottleneck, as it is known in the reinforcement learning literature. But does recent progress mean that we have solved exploration? In other words, can we take a sufficiently powerful pretrained model, scale its test-time compute (TTC), and expect it to solve essentially any hard problem? In this talk, I will take a broader view of exploration in LLMs, give you a high-level view on some of our past work on this problem over the last 1 year, and discuss recent efforts from my group to push the limits of exploration using multi-agent systems and weight-update-based internalization. I will conclude with several open questions about what remains unsolved in my biased opinion and where progress will come from next.