StateBench: How LLMs Track State in Long Documents
See what StateBench reveals about long-context state tracking, why models miss chained updates, and what its results mean for AI evaluation and architecture.
StateBench Shows Where Long-Context Models Lose Track
A language model might retrieve a fact from a long document and still miss what that fact means after a few updates. If a document says a person picked up a key, passed it to someone else, and later put it in a drawer, answering "Where is the key?" takes more than finding the latest sentence. The model has to keep the whole chain of changes straight and apply them in order.
That matters in practical work, too - following variables through code, or keeping details consistent across a long conversation. It also matters when you compare model architectures. A system that handles simple updates may struggle when the next update depends on earlier state.
Ai2's StateBench is built to test those differences. It uses four levels of state-tracking difficulty and update sequences that go well beyond the lengths used for some training runs.
This article explains what StateBench measures, what its reported results show across pretrained and directly trained models, and what the benchmark does - and doesn't - tell you about deploying long-context AI.
What is state tracking, and why does it matter?
State tracking is the job of keeping an up-to-date view of entities as new information arrives. The entities might be people and objects in a story, variables in a program, or records across a series of edits. When you ask a question, the model has to identify which updates matter and figure out the current state.
Some questions are straightforward because the answer shows up directly in the latest update. Others require reconstructing a chain. If Alex puts a folder in a bag, gives the bag to Sam, and Sam leaves it in a car, the folder's location depends on the relationship between the folder and the bag - not on the last time "folder" appears. Miss one step and the answer changes.
This is different from "just handling long inputs." A model can fit a large document into its context window and still fail to work out what all those updates add up to. Context length tells you how much text the model can process; it doesn't automatically tell you how reliably it can maintain state across that text.
For product teams, the gap shows up quickly. A coding assistant might need to track how a variable changes across a file. A support tool might need to remember a customer's latest preferences throughout a conversation. A document-analysis system might need to reconcile multiple revisions. In all of these, a single missed update can produce an answer that sounds right but is out of date.
StateBench is designed to focus on the underlying operation, not only on natural-language scenarios. It uses sequences of assignments and then asks for the final values. That keeps the benchmark controlled and measurable, while still connecting to common state-tracking problems in software and everyday information.
It also connects to a research question about model design. Standard transformers process sequences through attention, while recurrent and hybrid models carry information forward in different ways. StateBench tests whether those architectural differences change which kinds of state updates a model can handle, and how results shift as sequences get longer.
How does StateBench measure state-tracking difficulty?
StateBench defines four levels, each one pushing the model into a more demanding type of update.
Four types of update
- R-trivial: A value is set once and never changes. The model just has to recover the original value, like reading a birth date or a constant in code.
- Aperiodic: A value can be reassigned directly. The latest assignment determines the answer, like setting a thermostat. Once a newer value appears, earlier values can be ignored.
- Soluble: An update can depend on the current value, like
x = x + 5. The model has to carry the running result forward, not just grab the latest assignment. - General finite: A variable can take the current value of another variable. These reference-based updates create dependencies across variables. A sequence of swaps between cups is the classic example: after each swap, the model has to track where each object went.
Success at one level doesn't guarantee success at the next. Retrieving the last direct assignment isn't the same as accumulating changes, and neither automatically shows that a model can follow references and swaps across a long chain.
Making comparisons fair
The benchmark builds easier tasks from a broader sequence of assignments. Those task versions keep the same number of variables and distinct values, and they place assignments at the same points in the sequence. What changes is the kind of assignment allowed.
That design helps separate update complexity from differences in input length or sequence structure. StateBench also tests multiple formats. This matters because the same underlying state problem can score very differently depending on whether it's presented as a familiar sequence or as spreadsheet edits. A result in one format shouldn't be treated as proof of broad state-tracking ability.
The four categories also connect to theoretical work on state-tracking problems, including distinctions associated with Krohn-Rhodes theory. For practitioners, the practical point isn't the label. It's that the benchmark treats state tracking as a progression of operations, not as a single yes-or-no skill.
What did StateBench find about current models?
Ai2 evaluated pretrained open models, frontier models, and smaller models trained directly on StateBench. These setups answer different questions. Pretrained evaluations reflect what models already learned. Direct training is meant to test what an architecture can learn when the task is made explicit.
Pretrained models and frontier systems
The open-model tests included Olmo 3, a transformer, and Olmo Hybrid, which adds Gated DeltaNet layers, alongside smaller models. All of the models were pretrained on the same data, and none had seen contexts longer than 64,000 tokens during training. Most were evaluated on inputs beyond that length.
Across the board, models did well on R-trivial tasks. As sequences grew, performance dropped on aperiodic tasks. On general finite tasks, models were generally near chance. Format also mattered: some smaller hybrid and Gated DeltaNet models did well on RULER-style R-trivial problems, but landed near chance when the same basic task was framed as spreadsheet edits.
Ai2 also tested Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. The reported results show strong performance on R-trivial and aperiodic problems, followed by a sharp decline on soluble and general finite problems as sequences got longer. On the longest general finite tasks, accuracy fell to 13% or lower.
That pattern lines up with what you'd expect from harder tasks. When the updates are more complex, more steps can affect the final answer - so one missed update can derail the result. High performance on simpler levels doesn't automatically mean the model can reliably follow long chains of dependent updates.
What changed when models were trained on the task?
Ai2 trained several 275-million-parameter models from scratch on StateBench, including transformer, hybrid, and Gated DeltaNet variants. Training sequences had up to 256 assignments, while testing went up to 64,000 assignments - 250 times longer than the longest training sequence.
The recurrent and hybrid models generalized almost perfectly on R-trivial, aperiodic, and soluble problems at the tested lengths. Transformers learned the easier levels, but degraded as sequences grew and didn't learn general finite problems. On the hardest level, the Gated DeltaNet variant with a more flexible memory update generalized the farthest, staying near perfect for roughly ten times the longest training length.
The gap between pretrained and directly trained results is useful. Some recurrent and hybrid models that struggled in the pretrained setting learned much more once they were trained specifically on StateBench. In that sense, training data and training objectives appear to play a big role. Transformer results, however, stayed weaker on general finite tracking even with targeted training.
What do the results mean for AI systems?
The main takeaway is simple: test the operation your system needs, not just how it handles long inputs. A benchmark that mainly checks whether a model can find a fixed fact doesn't tell you much about applying a chain of state changes. If your product depends on those chains, a generic long-context score won't be enough.
Build evaluations around failure cost
Start by listing what can change in your task and how each change affects the answer. Does the system need the latest value? Does it need to accumulate increments? Does it need to follow references across entities? Those map roughly to StateBench's increasing levels of difficulty. That mapping gives teams a clearer test plan than a single, broad "long-document understanding" score.
Next, vary the conditions that create operational risk:
- Increase the number of updates, not only the surrounding text.
- Include cases where early changes affect later ones.
- Test multiple presentation formats, especially the ones users actually encounter.
- Measure exact final-state accuracy, and check which update caused the error.
- Re-run tests when entities or relationships are introduced in less regular ways.
These steps make failures easier to diagnose. If your system handles direct reassignment but fails on cumulative updates, the issue isn't just "the document is too long." The underlying operation is different, and you may need a constrained workflow or a separate state store.
Treat architecture as one part of the system
StateBench suggests that architecture can matter, but it doesn't prove that one model family is automatically best for every application. The directly trained comparison is controlled and helpful for studying capability. Real products also depend on pretraining, instruction tuning, retrieval, tool use, latency, cost, and even how users structure their inputs.
For a business workflow, one safe option is to store changing facts explicitly in a database or application state, then have the model interpret or explain them. That adds engineering work and can limit flexibility, but it makes important state visible and testable. For lower-risk tasks, letting the model infer state directly from the conversation may be enough. The right choice depends on how costly a wrong answer is and how easy it is for users to catch it.
Either way, rollout should include both model evaluation and system-level checks. Test the model on sequences that look like what you'll see in the real world, then test the full product with realistic inputs, recovery paths, and clear handling for uncertainty. A benchmark result is evidence about performance under specific conditions - not a guarantee for every workflow built on top of the model.
What are StateBench's limitations?
StateBench isolates an important capability, but it doesn't recreate every challenge found in real documents or real conversations. Its core setup uses a fixed number of variables. In actual interactions, people often introduce new entities over time - new people, objects, or concepts - so a system also has to decide when something is new and start tracking it.
The benchmark's structured assignment sequences also let researchers dial in difficulty precisely. That's a strength for comparing models, but it leaves open the question of transfer. Doing well on these clean sequences doesn't automatically mean the model will do well on messy prose, ambiguous references, conflicting reports, or cases where the relevant update is implicit.
There are also unresolved theoretical questions about the architectures tested. Ai2 reports that a Gated DeltaNet with a more limited memory update can solve all aperiodic tasks and certain periodic tasks under the paper's assumptions about fixed-precision arithmetic and rounding. Exactly what it can do across the full hierarchy remains open. The impact of numerical precision, and what happens when this architecture is combined with attention layers, needs more study.
These caveats should guide how you use the results. StateBench is a diagnostic benchmark, not a complete readiness test for a product. It can show that a model struggles with a specific class of state updates under controlled conditions. It can't, by itself, establish safety, reliability, or business value across a broad deployment.
For teams, a practical response isn't to dismiss the benchmark as artificial or treat it as a universal ranking. Use it to identify which kinds of state operations might fail, then build application-specific tests with the entities, the language style, and the real-world consequences your users will actually see.
Conclusion
StateBench makes a key distinction: retrieving the latest mention, accumulating changes, and following dependencies are different capabilities. The evaluations show that strong performance on simple state updates doesn't guarantee reliable tracking through long, dependent sequences - and that both architecture and training conditions can shape the outcome.
If you're building with language models, the next step is to spell out which state changes your workflow requires, test those operations at realistic lengths, and decide whether explicit state storage is worth the extra complexity. Treat context length as an input limit, not a reliability guarantee.