Posts

TL;DR I made a new benchmark for coding agents based on 21 difficult Pokemon battles from the fanmade ROM hack Radical Red. Agents need to inspect data in a sandbox to build a team and battle an opponent. As they uncover more information about their opponent, they can update their team under a limited episode budget. In an initial eval, GPT 5.6 Luna wins ~83% of its battles within 10 episodes. Pretty strong, but we have levers to scale task difficulty up for future evaluations. Code is here if you want to run some evals yourself or contribute! Introduction Recently, Anthropic announced that their new model, Claude Fable 5, was able to beat Pokemon FireRed using vision only. Impressive! Like every other benchmark we’ve set for AI, Pokemon FireRed too has been saturated. So what now? ...

Compaction What is Compaction? When performing inference with a language model, eventually the context length will become too large, resulting in slower and more expensive decoding, along with potentially degraded performance (Hong et al. 2025). Applications such as Claude Code and Codex handle this issue by offering the user a “compaction” feature, which compresses the input tokens into a smaller, more manageable set via summarization. However, as I’m sure we’ve all seen by interacting with these tools, compaction via summarization can be incredibly lossy, causing model performance to tank. I often find myself scoping sessions to tasks that I know can be reasonably accomplished within a model’s effective context window and using /clear excessively rather than using /compact. ...

Recently I was helping train a VLM from scratch and I noticed something odd: during training, the output norms of the projectorThe projector is the component of a VLM that transforms the patch embeddings outputted by the vision encoder to token embeddings consumable by the LLM. exploded up to two orders of magnitude higher than the norms of text token embeddings: Projector mean output norm over the course of vision-language alignment training. The norm magnitude rises to roughly two orders of magnitude higher than the text token embedding norms. ...