AI Agent Architecture ⚡ LLM Context, Memory & Drift (2026)
Learn AI agent architecture: LLM context management, AI agent memory, and drift handling with spec-driven development for reliable multi-agent systems. Read now.
Why Reliable AI Agents Need Files, Not Just Memory
An AI agent can look busy while making no real progress. It might summarize what it just did, report metrics that seem fine, and keep running - only to repeat the same task or quietly drop an important requirement. The model isn't always the culprit. More often, it's the system wrapped around it: how it manages context, what it records as decisions, and what it does when instructions clash.
I ran into this while building Legion, a multi-agent framework. In one experiment, I forced context eviction during an agent's working session. The failures that followed turned into a practical lesson: conversation memory is great for carrying the current mission, but it's a shaky place to keep the only copy of something important. Files are a more durable reference - as long as the system can keep them current and handle disagreements.
The really interesting part came after I published an account of the experiment. Another developer, Yuhe He, shared a response with a surprisingly similar set of lessons. Two separate systems converged on similar design choices after hitting similar failure modes. It wasn't proof that one architecture is "best," but it did feel like a strong signal about what these workflows struggle with in practice.
This article digs into what that convergence suggests, where file-based state helps, and why a reliable agent needs more than a memory strategy. It needs a way to detect drift and resolve conflicts when the evidence doesn't line up.
When context management creates false progress
Long-running tasks put pressure on an LLM's context window. As the session grows, a system may compress or summarize earlier messages to make room for new work. That's a reasonable response to a real constraint.
But it also creates a failure mode that's easy to miss: the agent keeps going, yet the "compressed" history no longer preserves the distinctions it needs to complete the job.
In my original account of the experiment, Yuhe described a pattern that felt almost too familiar. History compression collapsed the transcript, and then deduplication treated a semantically new task like it had already been handled. Everything looked healthy on the surface - until the useful work quietly stopped.
This is worse than a clear crash. A process that fails loudly gets attention. A process that reports success while doing nothing can slip past monitoring, burn tokens, and leave you with incomplete work that no one realizes needs review. In a business workflow, that could mean a missed update, an incomplete code change, or a report that's just wrong enough to slip through.
The difference between a live loop and useful progress
A running agent isn't automatically a progressing agent. You have to separate operational signals (is the process active?) from task signals (did the requested change actually happen, and was it verified?).
For agent workflows, "useful" checks can look like this:
- State change: Did the relevant file, record, or artifact actually change?
- Acceptance criteria: Is there evidence the requirements are satisfied?
- Repeated work: Is the agent producing the same output or circling the same step?
- Traceability: Can the system show what it did and which instruction it followed?
These checks don't eliminate model errors. They make silent failure much easier to spot. A green status indicator should mean "the process is alive," not "the work is correct."
The lesson isn't "never summarize." Context windows are finite, and compression can be helpful. The lesson is to avoid making a compressed conversation the only record of a task's requirements, decisions, and current state.
Why files make better durable state
Legion and Yuhe's system landed on the same core idea independently: treat memory like a transport layer and a cache, and treat files like the durable source of truth.
Here's the split in plain terms. The agent's active context helps it stay focused and carry intent through the current loop. Durable records hold the decisions, artifacts, logs, and state changes that must survive context eviction, a restart, or an agent handoff. If the in-memory "story" and the recorded state disagree, you need a clear rule for which one wins - and it helps to record the disagreement as a possible sign of drift.
What belongs in memory, and what belongs on disk?
A useful division looks like this:
| Information | Working memory | Durable file or record |
|---|---|---|
| Current task and immediate next step | Yes | Yes, if needed to resume reliably |
| Long-term decisions and constraints | Brief reference | Yes |
| Generated artifact or code change | No substitute | Yes, in its canonical location |
| Task history and significant state changes | Summary may help | Yes, as an auditable record |
| Temporary reasoning | Often | Only when it affects decisions or handoff |
Exactly how you implement it depends on what you're building. A local agent might use version-controlled files; a hosted system might store durable state in a database or object store. "Files as the database" is a useful architectural principle, not a strict requirement to avoid databases.
The main point is simpler: durable state should live somewhere explicit, inspectable, and recoverable - not only in a model's changing context.
One caution: dumping every event to disk can also hurt. Excessive logging creates noise, makes it harder to find what matters, and can add storage and maintenance costs. You generally want to persist meaningful state transitions, decisions, and artifacts, with enough metadata to identify where they came from and when they happened. Don't turn every token of internal reasoning into a permanent record.
Durability does not guarantee correctness
A file can still be wrong, stale, or incomplete. It can also get overwritten by an agent that never saw the full picture. Durable storage fixes the "forgetting" problem, but it doesn't automatically fix the "conflicting or misleading information" problem.
For production use, file-based state needs the usual engineering controls: clear ownership, version history, safe writes, and permissions that match what the task requires. And if multiple agents can update the same specification, you need a way to detect concurrent changes. Without that, your "source of truth" can become another place where silent failure lives.
So yes - there's a trade-off. Explicit state costs engineering time. Relying on implicit conversation history costs reliability. For short, disposable tasks, a lightweight approach may be enough. For anything that spans sessions, involves multiple agents, or affects consequential changes, durable state usually pays for itself.
The NGINX failure and the case for intent files
Another incident made the limits of plain-text context feel concrete. While working on Legion, I gave an agent a live production NGINX configuration and asked it to make changes. It returned something that looked valid. But it had dropped several critical services that simply weren't present in the agent's local development container.
This wasn't a classic hallucination where the output invents facts out of thin air. The agent produced something plausible based on what it could see. The problem was different: it treated an incomplete view as if it represented the full system.
A configuration can be syntactically correct and still be operationally destructive if key services, ports, or endpoints disappear from the result. A small omission can matter more than a paragraph of correctly summarized context.
Make requirements inspectable and reusable
In Legion, the fix was to use structured intent files as part of Spec-Driven Development.
During requirements gathering, the system creates a document that captures the task in a more constrained form. It can include baseline requirements, business rules, and an inventory of structures that need special care - services, ports, endpoints, and so on.
Then the specification is used as a canonical reference for the agents that implement the work. That doesn't mean an agent should blindly obey the file forever. It means the agent has a stable, inspectable statement of what was understood at the start, rather than relying on a compressed recollection of a chat.
A useful intent file should answer questions like:
- What outcome is required?
- Which constraints must remain true?
- Which parts of the existing system are in scope?
- Which parts must not be changed or removed?
- How will completion be checked?
- When should an agent stop and ask for clarification?
These details matter most when the work touches infrastructure, security, billing, or anything where omissions can have outsized consequences. They also make handoffs easier. A new agent or a human reviewer can inspect the same requirements without reconstructing an entire conversation.
Still, a specification isn't a substitute for reviewing the resulting change. It's a mechanism for making intent explicit and easier to test against. In the NGINX case, an inventory of expected services could have flagged missing entries before the configuration was accepted.
Resolving conflicting instructions with evidence
Durable specs introduce a new question: what happens when they go stale?
The user might change the request mid-task. The repository might have changed since the specification was written. Another agent might already have committed a patch. And the executor's live context might contain newer instructions than the file does.
Yuhe raised this directly in the exchange: if the file says one thing but the user changes course halfway through, how does the system avoid following outdated acceptance criteria?
That's a fair concern, because "the file is the source of truth" doesn't solve the problem when you have multiple plausible sources of truth.
Legion's approach is to triangulate the conflict instead of asking the executing agent to resolve it from memory. When an agent detects a discrepancy, it escalates to the orchestrator - ChatMaster - with raw evidence from three places:
- Recent user dialogue: the relevant excerpt showing what the user said.
- Task specification: the current requirements recorded in the intent file.
- Repository state: the relevant diff, function, or code as it exists on the active branch.
ChatMaster can then inspect the underlying material and determine whether the user changed the requirement, another agent changed the code, or the executor made a mistake while breaking the task down. Depending on what it finds, the system can update the specification, adjust the implementation, or ask for clarification.
Why raw evidence beats a confident summary
A summary is useful for navigation, but it's still an interpretation. If the system asks an agent to summarize a conflict and then treats that summary as the main input, it risks carrying the same misunderstanding into the next decision.
Providing the underlying source material gives the orchestrator a chance to check the claim against the record.
This doesn't make the orchestrator infallible. It can still misread evidence, choose the wrong priority, or miss a relevant change. The architecture improves decision quality, but it doesn't eliminate the need for tests, review, or human escalation when the stakes justify it.
The key design choice is to make ambiguity visible. An executor shouldn't silently reconcile incompatible instructions just to keep the workflow moving. When the cost of guessing is high, it should stop, show the conflict, and ask for a decision. That can add latency - but the alternative is often a fast, plausible, incorrect change.
What convergent architecture does - and does not - prove
The exchange with Yuhe was striking because two independent efforts hit similar failure modes and landed on similar patterns: keep durable state outside working memory, make intent explicit, and escalate conflicts rather than smoothing them over.
That's a useful signal. It suggests these are practical responses to recurring constraints in long-running agent workflows, not just design choices peculiar to one framework.
But convergence isn't proof that any one architecture is universally optimal. Two systems can reach the same solution independently because the constraints are the same - while still needing different implementation details. A small coding assistant doesn't need the same orchestration layer as a system coordinating multiple agents across a production repository.
The more useful question is: what failure modes does my system need to detect, and what's the cheapest reliable mechanism for detecting them?
A lightweight agent might only need a task file, a change log, and a clear stop condition. A multi-agent system might need versioned specifications, ownership rules, conflict detection, and an orchestrator that reviews evidence. The architecture should grow because it earned that complexity from real failures - not because a diagram looks more complete.
In Legion, these patterns came from failed runs, looping agents, lost configuration details, and situations where the system sounded confident while the underlying work was broken.
The framework is currently undergoing refactoring in preparation for a closed MVP release. That stage matters: an architecture formed through experiments still has to become maintainable, understandable, and genuinely useful to the people operating it.
Conclusion
Reliable AI agents need more than a large context window. They need a real boundary between temporary memory and durable state, structured records of intent, and a safe way to handle disagreement between instructions and the system they're changing.
A practical place to start is fairly modest: figure out what must survive a session, write it somewhere inspectable, and define what the agent should do when the record conflicts with current evidence. Add orchestration and stronger controls when real failures justify it - not before, and not after a costly mistake makes the gap impossible to ignore.
If you're building an agent workflow, look at the last few failures before you add capability. Did it forget? Act on incomplete context? Guess through a conflict? That tells you which part of the architecture needs work. And if your agents hit the same wall independently, share the failure that led you there.