GPT-6, Gemini and Claude lose track of chained edits in long texts

x equals 3. Then y takes the value of x. Then we add 5 to x. Then we swap x and y. What is y at the end? Over four lines, anyone gets it right. Over thousands of lines, the best AI models fall apart, according to a test published on November 3 by the Allen Institute for AI (Ai2).

This US nonprofit research institute introduced StateBench, a benchmark that measures a model's ability to track the state of data that changes over the course of a long text. The model reads a sequence of variable assignments and must give the final value of each one. Four difficulty levels differ only in the type of modification. At the first, a value is set once and for all. At the second, it is simply replaced. At the third, it depends on the previous one (“x = x + 5”). At the last, variables refer to one another or swap values (“x = y”).

Four leading models collapse on the hardest cases

Ai2 put four top commercial models through the test: Google's Gemini 3.8 Flash, Anthropic's Claude Opus 5.5, and OpenAI's GPT-6 Sol and GPT-6 Luna. All do well at the first two levels. At the last two, their accuracy drops sharply as the text gets longer. It falls to 13% correct answers, or less, on the longest series of references and swaps. The authors offer an explanation: in these exercises, the final answer depends on far more steps.

A different architecture does better, in the lab

The second part of the study compares architectures on small models trained specifically for these exercises. They learned on series of at most 256 assignments, then were tested up to 64,000. So-called recurrent models (Gated DeltaNet) and hybrid models generalize almost perfectly at the first three levels. “Transformers,” the architecture that dominates large language models today, learn the easy levels but degrade on long series, and never learn the hardest one. One recurrent variant stays nearly perfect at the hardest level up to about ten times the length seen during training.

One point to keep in mind: Ai2 itself develops hybrid models, such as Olmo Hybrid, whose architecture these results showcase. The institute also acknowledges a limitation. StateBench assumes a fixed number of variables, whereas in a real conversation, new elements appear along the way. And the exercises remain artificial.

What this means for how you use AI

The test does not use real documents, but it recalls common uses: tracking the successive amendments to a contract, an inventory with transfers between sites, accounting entries that cancel and correct each other. A few precautions:

• do not rely on the model's “memory” to know the final state after a long series of changes: have a spreadsheet or a script compute it, which many assistants can write and run;
• split long documents and have the intermediate steps checked;
• spot-check results that depend on many chained modifications.

StateBench's data and code are public, so other teams can repeat the measurement on upcoming models. It remains to be seen whether the next generation will do better on the two hardest levels.