LLM Agent Topology: Models, Loops, and Tools
Learn how LLM agent topology—the model, loop, and tools—shapes cost, capability, UX, and security, with a practical framework for designing safer AI products.
Where Your AI Agent Runs Matters More Than You Think
When people describe an AI agent, they usually focus on what it can do: write code, answer questions, summarize documents, update a CRM. That's enough for a demo - but it's not enough to judge what you'll get in production.
To understand an agent where it actually lives, map its topology: where the model runs, where the agent loop runs, and where its tools run. Those choices don't just change the architecture. They affect operating costs, data exposure, user experience, reliability, and the size of the blast radius when something goes wrong.
This matters whether you're adding an agent to a SaaS product or selecting one for your own workflow. A model that never touches sensitive data might be the smarter option than a more capable hosted model. A flexible loop can outperform a polished interface that's too rigid. And if a tool can run arbitrary code, you need a different security boundary than you would for a read-only SQL query.
This guide breaks the agent into three parts, shows how location changes the trade-offs, and gives you a practical way to map the topology before you ship.
Think of an agent as a topology
A practical way to model an agent is to treat it as three parts:
- A model, which interprets context and decides what to do.
- An agent loop, which manages the conversation, calls the model, handles tool requests, and delivers results.
- A set of tools, which lets the agent interact with data and systems outside the model.
Often, the loop and tools get bundled into what people call the harness. The distinction still matters because each part can run in a different place. The model might be hosted by an AI provider. The loop might run in a browser. Tools might execute in a cloud virtual machine - or connect to internal company systems.
A chatbot UI can make this look like one tidy product. In reality, it can be multiple systems exchanging prompts, outputs, credentials, and tool results across network boundaries. The topology is where those boundaries land.
Why topology is more than an infrastructure diagram
Topology helps you answer concrete questions:
- What does the agent cost? A hosted model may bill per token or request. A self-hosted model shifts costs to hardware, operations, and capacity planning. Tool-using agents can also consume compute through tests, queries, or file processing.
- What can it actually do? A model can't inspect a repository, update a ticket, or test code unless the loop gives it tools that can perform those actions.
- What data can leave your environment? Prompts, conversation history, retrieved documents, and tool outputs may cross a boundary to reach a model provider or a remote service.
- What can go wrong? An agent that can access a browser, credentials, and untrusted content can have a much larger blast radius than one that can only query a locked-down, read-only database.
- What experience can you build? The loop controls how users provide context, review actions, give feedback, and recover from mistakes.
On paper, it's tempting to pick the model first and treat everything else as "just implementation." But those "details" often decide whether the product is affordable, safe, and genuinely useful. A capable model paired with the wrong harness can still produce a weak agent.
A good first step is to draw three boxes - model, loop, tools - and note where each runs. Then list the data and credentials that cross between them. That map frequently surfaces design risks before you turn a prototype into production infrastructure.
Where the model runs changes cost and trust
For the model, you're generally choosing between a hosted service and infrastructure you control, including local or self-hosted models. Neither choice is automatically safer or cheaper. The right answer depends on the sensitivity of the data, your expected workload, and what operational capacity you have.
A hosted model can reduce the burden of provisioning GPUs, keeping inference infrastructure up, and handling scaling. It may also give you access to stronger models and make iteration faster. The trade-off is that your application sends some information outside its own environment. What happens to that information depends on the provider, the service terms, your account configuration, and the privacy requirements that apply to you. Those details need to be checked - not assumed.
Running the model locally or inside your environment can keep inference data from going to an external model provider. That can be a solid fit for sensitive workloads, but it's not a universal privacy guarantee. Your application may still transmit data through telemetry, logging, backups, or remote tools. You also inherit the work of model updates, hardware capacity, availability, and performance tuning.
Match the model location to the workload
Take a company that needs to OCR and classify scanned bills. If the workload can run overnight, latency might matter less than how data is handled. A local model or an approved private deployment could be better than sending every document to a general hosted endpoint. On the other hand, a customer-facing assistant that has to respond quickly under heavy load may justify a hosted model - assuming the data flow and service terms actually meet your requirements.
This isn't really "private versus public" as a principle. It's about the specific data and the consequences of mishandling it. A single rare identifier, customer record, or confidential document can still be sensitive even if most prompts look ordinary. And self-hosting for low-risk tasks can become just extra operational overhead without meaningful benefit.
Model integrity is another part of trust. A model is a dependency, and its behavior depends on more than the provider name or the model family. You should think about vendor controls, version changes, evaluation results, and how updates are rolled out. Moving inference in-house doesn't magically solve trust - it just shifts that problem into a broader supply-chain and operational security setup.
Make token costs visible
Agents change the cost structure of software. In many conventional SaaS products, a user's action maps to a fairly predictable set of database operations. With an agent, a short request can trigger multiple model calls, repeated tool use, a code build, and a long-running test. Users see one sentence; the system sees a workload.
If you're building an AI feature, decide who pays for inference and other compute. For example:
- Include a defined level of usage in the subscription.
- Charge for usage above a clear allowance.
- Let customers bring their own model account or subscription.
- Restrict expensive actions through quotas, budgets, or approval steps.
"Bring your own model" can align costs with how much users actually demand, but it adds setup friction and support work. Bundling usage can feel smoother, but you need to model what heavy users do - not just price based on average usage. The key is to set the economic boundary before the workload grows.
The loop determines the product experience
The agent loop is the control flow around the model. A simplified loop keeps message history, sends it to the model, receives a response, handles any tool calls, and repeats until the agent has a final answer - or needs more input.
This "plumbing" is also where a lot of the product experience ends up. The loop determines how context is assembled, how tool calls are managed, what the user sees, and whether the agent can recover from an error. In a hosted chatbot, the loop may run on the provider's servers. In a coding agent, it might run on a developer's laptop. It can also run in a browser or in a service your product team controls.
Choose the loop for the job
A general chatbot and a coding agent don't need the same interface. A chatbot might focus on conversation and file attachments. A coding agent might need repository access, a terminal, diffs, test results, and explicit controls for approving changes. A support agent might need a clear history of what happened inside a ticketing system.
That's why relying on a vendor's built-in agent loop can be limiting. A vendor's harness might work well inside their product. But your users may need an agent that coordinates across a codebase, monitoring tools, and deployment systems. And on the flip side, building a custom loop for a narrow task can create maintenance work without improving the experience.
The right choice is a product decision, not a universal rule. Ask what context the agent needs, what actions users must be able to inspect, and where the workflow currently breaks. Then decide whether an existing loop fits - or whether your product needs its own.
Treat conversation history as application data
The loop needs to maintain context across model calls. That history can include user input, model responses, tool requests, and tool results. It has to live somewhere, and the storage choice impacts privacy, reliability, and debugging.
A simple local application might store conversation data in SQLite. A multi-user hosted service might require a database with access controls, retention policies, and backup procedures. Either way, conversation history isn't "disposable" just because it came from an AI interaction. It can include customer data, code, internal decisions, or sensitive tool output.
You also need to decide what happens when context grows beyond the model's usable window. Truncation, summarization, retrieval, and starting a fresh task all fail in different ways. A summary can drop an important constraint; keeping everything can increase cost and expose more information than you need. Make the behavior explicit and test it with real workflows.
Finally, the loop owns interaction details that are easy to ignore in a demo: Can users attach files or images? Can they inspect or cancel a tool call? Can they correct an intermediate result? Can they share the conversation safely? Those choices determine whether the agent works outside a controlled showcase.
Tools define capability and blast radius
A model can generate a plan, but tools let the agent act. Tools might read files, run tests, query a database, search the web, or update a business system. Without tools, the agent produces text. With tools, it can change the world - and the risk profile changes with that.
Tool execution also has a practical cost. Compiling a project, processing video, running a test suite, or analyzing a large dataset takes compute and time. It's not enough to ask whether the model "knows" how to do something. You need to know where the action runs, what resources it can use, and what it can access.
Set the execution boundary deliberately
For arbitrary - or potentially unsafe - code, virtual machines can provide a solid isolation boundary when configured correctly. They can separate tasks from a user's machine, enforce resource limits, allow disposable environments, and restrict network access. But "VM" isn't automatically secure. Credentials, network permissions, storage, and VM lifecycle still require careful design.
Some tasks don't need a full machine. A browser sandbox, WebAssembly runtime, or a restricted Python environment may fit bounded workloads better. A read-only SQL connection can support analysis without giving the agent permission to change records. The rule isn't "always use a VM." It's: give each tool the smallest execution environment and permissions that still make the task work.
A simple permission ladder might look like this:
- Read from explicitly approved data.
- Write to a draft or staging area.
- Make a reversible change after user approval.
- Perform higher-impact actions only with stronger controls and a clear audit trail.
Not every agent needs every step. A reporting assistant shouldn't inherit deployment credentials just because a coding agent somewhere else uses them.
Account for untrusted input and credentials
Agents run into untrusted content in documents, web pages, code, and messages. That content can include instructions meant to steer the model. If the agent also has secrets or can perform sensitive actions, the combined risk is bigger than either capability alone.
Remote tools add another boundary. A tool might run on a different machine, or it might be provided through a connector or protocol such as MCP. In those setups, the loop might be hosted in one place while the tool and its data live elsewhere. You need to map where authentication tokens are stored, which service uses them, and what those permissions allow. A "convenient connector" is still an access path into a system.
For each tool, document:
- What data it can read and change.
- What credentials it uses and where they are stored.
- Whether it can reach the public internet or internal services.
- What happens if the model requests an invalid or destructive action.
- How users and operators can review, revoke, or audit access.
And don't treat the agent as the only security boundary. Use the controls provided by the systems under it - scoped credentials, network restrictions, read-only access, timeouts, resource limits, and approval flows. They aren't as exciting as a clever prompt, but they hold up under pressure.
Map the topology before choosing a design
There isn't a single "best" agent architecture that fits every product. The question isn't "Which stack is best?" It's "Which arrangement supports this workflow at an acceptable cost and risk?"
Use this sequence to make the decision concrete:
1. Define the task and its failure cost
Write down what you expect the agent to do, and what a bad outcome would mean. Misclassifying a low-risk internal document isn't the same as making an incorrect payment, leaking a source file, or deploying unapproved changes. Higher-impact tasks need tighter permissions, more review, and better observability.
2. Classify the data
List what the model sees, what the tools can access, and what the loop stores. Flag sensitive or regulated data and follow it across each boundary. Don't stop at the prompt - tool results and conversation history can be just as revealing.
3. Choose the loop and model together
Check whether an existing loop supports the workflow you actually need. Then pick a model deployment based on quality, latency, cost, and your data requirements. Test the full flow instead of comparing model answers in isolation. The loop affects what context reaches the model and how many calls a task ends up requiring.
4. Give tools minimum necessary access
Start with the smallest permission set that still makes the task useful. Prefer read-only access when you can. Isolate code execution, scope credentials, and require confirmation for actions that are hard to undo. Expand access only when real usage proves the limitation is worth addressing.
5. Measure cost and behavior in production
Track model usage, tool calls, duration, failures, and user corrections. These measures tell you whether the agent delivers enough value to justify its cost - and whether users trust what it does. They also uncover workflows where tool calls loop repeatedly or where the agent struggles with a predictable class of input.
A topology map makes those decisions visible. For each component, record its location, owner, data access, credentials, cost model, and failure boundary. Update the map when you add a connector, change model providers, or give the agent a new capability. Even if the user-facing feature looks the same, the risk can change.
The broader product lesson is simple: more software will include agent loops, but not every product should use the same one. A product is easier to build for when its capabilities and boundaries are clear - stable APIs, explicit permissions, structured outputs, and predictable errors. That doesn't mean replacing every interface with an agent. It means designing systems people and software can both use safely.
Conclusion: design the boundaries, not just the demo
An agent isn't just a model with a prompt. It's a model, a loop, and a set of tools - each with its own location, costs, and permissions. Together, those choices determine whether the agent can do useful work without creating an unacceptable operational or security burden.
Before you add an agent to a product, map its topology. Follow the data, credentials, and actions across each boundary. Then make the smallest set of changes needed to support the workflow safely - and measure what happens once real users start using it.
If you're evaluating an AI feature right now, start with that map. You'll get a better design discussion than debating models based on a benchmark or a polished demo.
Source perspective: The Topology of an LLM Agent Loop With Tool Use.