LLMs can write useful code, explain unfamiliar frameworks, and even make broad changes across a codebase. But ask an AI agent to design a system that uses language models, and it may default to an elaborate rules engine instead of making another model call. The result can look impressive - and still miss the point.
That matters because the damage isn't limited to a few buggy lines. A shaky architectural choice can pull a team into days of implementation, debugging, and prompt tweaks before anyone stops and challenges the overall approach. The more autonomous the agent is, the farther it can wander into the wrong direction before a human notices.
This isn't just about models lacking knowledge of AI techniques. Often, they can explain the right approach if you press them. The real issue is that they may not apply that knowledge when it's time to choose an architecture. In this article, I'll walk through why that happens, how to evaluate an AI-generated design, and a practical way to keep teams focused on the simplest solution that actually works.
Why do AI agents choose the wrong architecture?
An AI agent often picks something that resembles conventional software engineering, even when the job really comes down to interpreting language. That mismatch can lead to a design that's complex, easy to test in narrow ways, and still incapable of handling the actual task.
Take a tool that stores short user memories and needs to flag contradictions. A conventional approach might try to turn each sentence into structured facts, then compare those facts with rules or a solver. On paper, that sounds disciplined. In practice, "do these two natural-language statements mean the same thing?" is a language-understanding problem. Rules don't naturally cover paraphrases, qualifiers, and the way people contradict themselves.
For example, "I moved to Chicago last year" and "I still live in Boston" can conflict - or they might not - depending on when the second memory was recorded and what "still" is doing in context. A rule-based system needs the right entities, time references, and relationships encoded before it can compare the claims. If those representations are wrong, the comparison can be internally consistent and still produce the wrong answer.
Sometimes, a model call is a better fit: ask a language model to compare two memories, or use a model to assess a batch of them. That still isn't "solved." You have to think about false positives, missing context, latency, cost, and how the user can review a disputed result. But it keeps the language-dependent work inside a component designed to handle language.
Complexity can hide a category error
The danger is that an agent can crank out a lot of code that looks plausible before anyone challenges the premise. It may add a schema, a parser, a rule engine, and a solver. Each piece feels like progress. The system might even pass tests that were built around the assumptions baked into the design. And yet it may never reliably identify what the original sentences actually mean.
That's a category error: treating an uncertain interpretation problem like it's deterministic bookkeeping. The opposite mistake happens too. Teams sometimes reach for an LLM where a regular function would do the job more cheaply and more predictably - like validating a required field or calculating a total.
So the design question isn't "Should we use AI?" It's "Which parts need flexible interpretation, and which parts are better handled with explicit deterministic rules?" In a solid system, those responsibilities can sit side by side. A model can extract or classify; ordinary code can enforce permissions, validate formats, and apply business rules.
If you want the practical takeaway, it's this: check whether the component you're building actually matches the kind of uncertainty in the problem. If the core uncertainty is meaning, a symbolic layer isn't automatically more reliable just because it's explicit.
Why do LLMs struggle to design LLM-powered systems?
There isn't one clear, proven reason an agent might avoid using an LLM inside an AI application. Usually it's a mix of factors, and it's best to treat them as hypotheses until you test them.
One plausible factor is what software patterns appear in training data. A lot of available code was written before model APIs became a routine building block. That can make non-AI patterns feel familiar and easy to reproduce. But this explanation doesn't fully hold up: modern LLMs still choose newer libraries and approaches in many areas. Age of the code alone doesn't explain why they sometimes refuse to use an LLM when language interpretation is the whole point.
Another possibility involves reinforcement learning and evaluation. In some setups, a model gets feedback from automated checks: generate code, run tests, score it. If the training environment restricts credentials or provides no access to model APIs, the model might not get much practice designing software that calls other models. Hosted model calls can also bring real-world friction - expense, latency, network failures, and security concerns - making it harder to evaluate everything in a clean sandbox.
Even that doesn't finish the story. Models can often write prompts for subagents and describe systems that use models. So the issue may be less about whether they can generate the code, and more about how they pick an approach when the instructions are ambiguous.
The design skill is separate from "knowing things"
A broader explanation is that current agents can be inconsistent at software design. They may write code without reliably picking the right boundaries between components, identifying the most likely failure modes, or settling on the simplest architecture that will work. A model might propose a complex consensus mechanism when tolerating slightly stale data would be fine - or add synchronous checks into a system that's built around asynchronous events.
Those are ordinary design mistakes. They aren't necessarily evidence of a special failure "around AI." The frustrating part is that AI systems can make those errors harder to spot: the model can justify them confidently and implement them quickly. Fluency can make a weak premise sound like a reasoned decision.
There's also a gap between explaining a fact and applying it at the right moment. An agent might admit that natural-language equivalence is hard - when you ask directly - then fail to notice that this fact undermines its own architecture. That explanation/judgment gap is where a lot of wasted work starts.
For engineering teams, a useful stance isn't "the model knows best" or "the model can't design." Treat its architecture as a proposal. Ask what assumptions it depends on, what simpler alternative it rejected, and which part of the problem each component actually solves. That turns a vague opinion into something you can review.
How should you evaluate an AI system design?
Evaluate the architecture against the real task, not against how sophisticated the diagram or code looks. Start by splitting the job into operations and identifying what kind of uncertainty each one involves.
A good first pass is to label each operation as one of three types:
- Deterministic: The answer follows from explicit inputs and rules. Examples include checking required fields, enforcing access permissions, or computing a subtotal.
- Interpretive: The answer depends on meaning, context, or ambiguity. Examples include judging whether two memories conflict, summarizing a support request, or assigning a free-text message to a category.
- Mixed: The operation needs interpretation and then a firm rule. For example, a model might extract a requested delivery date, while code checks whether that date falls within a valid range.
That classification helps you avoid two common traps: building elaborate rules around something that depends on meaning, or using a model where the outcome should be predictable code.
Test the risky assumption before building the system
For an interpretive step, run the smallest experiment that can tell you whether a model call is actually useful. Build a representative set of examples: straightforward cases, ambiguous cases, and cases that are likely to trigger mistakes. For a contradiction detector, that could include direct opposites, paraphrases, statements with different dates, and memories that look consistent until you add context.
Then compare at least three things: a simple model-based approach, a deterministic baseline where it makes sense, and human judgment on the same examples. The point isn't to prove the model is always right. The point is to learn whether it's good enough for the product - and what kinds of errors it makes.
Set the failure cost before you pick a threshold. A false positive in a personal memory tool might annoy a user or make them delete useful information. In a medical, legal, or financial workflow, a mistaken interpretation can have far higher consequences and may require human review. "The model gets most cases right" isn't enough unless you understand what the remaining errors mean operationally.
Also measure practical constraints: cost per request, response time, rate limits, data handling requirements, and how the system behaves when the model is unavailable. Those aren't afterthoughts you bolt on after a demo - they shape the design from day one.
A solid evaluation doesn't ask whether an LLM can solve the task in principle. It asks whether a specific model, prompt, context window, and fallback strategy meet the product's quality and operational needs on data that resembles real usage. Keep your test set separate from any examples you use to tune the prompt - or you'll end up measuring how well the system fits your sample instead of how it behaves in production.
How can teams keep AI agents out of dead ends?
The most effective safeguard is to do design review before large-scale implementation. An agent can still write code, but first it should surface its assumptions and alternatives. This is especially important when the work is open-ended, the task involves language interpretation, or the proposal introduces multiple new components.
Use a short architecture checkpoint
Before you ask an agent to build anything, ask for a brief design note with five items:
- Task decomposition: What are the system's separate operations, and which ones depend on interpreting language?
- Candidate approaches: What are the simplest plausible options, including a model call and a non-model baseline where relevant?
- Trade-offs: How do the options compare on quality, cost, latency, reliability, and maintenance?
- Key uncertainty: What fact would flip the design decision if it turned out differently?
- Smallest validation: What experiment can test that uncertainty before you commit to a bigger build?
This isn't busywork. It forces the agent to explain why each component exists and gives you a real chance to reject a shaky premise while the cost of changing direction is still low. If the agent can't describe what evidence would distinguish the options, it's probably not ready to implement the more complex one.
For the memory example, the key uncertainty might be whether a model can identify meaningful contradictions at an acceptable error rate. The smallest validation might be a batch comparison over a few dozen carefully chosen examples, reviewed by a person. Building a symbolic representation and solver before running that experiment is work built on an assumption - not on evidence.
Add stop conditions and human ownership
Be explicit about constraints. For example: don't introduce a parser or rule engine unless it satisfies a named requirement; don't expand the scope until after the first experiment's results are in; and stop if the plan relies on extracting information the system can't reliably observe.
These rules won't make the agent infallible, but they do reduce the odds of a silent "implementation spiral" where the system keeps going because it can.
You should also set a budget for autonomous exploration - time limits, token limits, or a requirement to report after each milestone. When the system's core assumption is still uncertain, shorter feedback loops are usually more valuable than letting the agent work for hours without interruption.
And keep architectural ownership with the team. Have an experienced engineer review component boundaries, likely failure modes, and operational costs - not just whether the code passes tests. Tests can confirm the implementation follows its specification. They can't prove the specification is the right way to solve the product problem.
Key takeaways
- Use models when the task genuinely needs flexible interpretation; use regular code for deterministic rules.
- Treat an agent's architecture as a proposal, not proof that the approach is correct.
- Test the riskiest assumption on representative examples before adding supporting infrastructure.
- Include error cost, latency, expense, data handling, and fallback behavior in the decision.
- Set checkpoints and stop conditions so a "polished dead end" doesn't become a big one.
Conclusion
AI agents can generate working software and still choose the wrong system design. That's not a contradiction: code generation, technical explanation, and architectural judgment are different skills, and today's models don't apply them consistently.
The practical response is to treat design as a sequence of testable decisions. Identify where interpretation is required, test uncertain assumptions early, and spell out costs and failure modes. Use LLMs where they add real value, use deterministic code where it's more reliable, and bring in human review when the consequences warrant it.
The goal isn't to stop agents from suggesting ideas. It's to make sure a plausible suggestion earns its place before you spend days building around it.