Posts

TL;DR I made a new benchmark for coding agents based on 21 difficult Pokemon battles from the fanmade ROM hack Radical Red. Agents need to inspect data in a sandbox to build a team and battle an opponent. As they uncover more information about their opponent, they can update their team under a limited episode budget. In an initial eval, GPT 5.6 Luna wins ~83% of its battles within 10 episodes. Pretty strong, but we have levers to scale task difficulty up for future evaluations. Code is here if you want to run some evals yourself or contribute! Introduction Recently, Anthropic announced that their new model, Claude Fable 5, was able to beat Pokemon FireRed using vision only. Impressive! Like every other benchmark we’ve set for AI, Pokemon FireRed too has been saturated. So what now? ...

Recently I was helping train a VLM from scratch and I noticed something odd: during training, the output norms of the projectorThe projector is the component of a VLM that transforms the patch embeddings outputted by the vision encoder to token embeddings consumable by the LLM. exploded up to two orders of magnitude higher than the norms of text token embeddings: Projector mean output norm over the course of vision-language alignment training. The norm magnitude rises to roughly two orders of magnitude higher than the text token embedding norms. ...