Local AI Agent: Build with FastAPI and ChromaDB
Build a private local AI agent with FastAPI, ChromaDB and LlamaIndex. Get the architecture, code and production trade-offs before you deploy.
Build a Production-Ready Local AI Agent Without API Fees
You can get a local AI proof of concept running fast with a cloud-hosted model. But the economics and the day-to-day reality change once people start depending on it. Usage-based API fees rise with traffic, sensitive prompts leave your environment, and response times depend on services you don't control. For teams building around internal knowledge, those aren't "nice-to-have" infrastructure details - they're product and operational constraints.
A local retrieval-augmented generation (RAG) agent is one way to reduce that dependency. FastAPI can expose the service over HTTP, ChromaDB can store and retrieve document embeddings, LlamaIndex can wire retrieval into generation, and a local runtime like Ollama can run an open-weight model on your own hardware.
That said, this doesn't make inference free. You still pay for compute, power, storage, engineering, and maintenance. What you avoid are per-token API charges, and you gain more control over where data is processed. This guide walks through a practical architecture, a minimal baseline implementation, what you should harden before production, and how to present it credibly to developers.
Why Consider a Local AI Agent?
A hosted model API is often the right choice for an early experiment. You don't have to manage model hardware, you can test different models quickly, and you can focus on whether the feature solves a real problem. The trap is treating that convenience as a permanent architecture without checking the economics, where your data goes, and how the system fails.
The costs are broader than tokens
API pricing is straightforward on paper: count input and output tokens, apply the provider's rates, and forecast usage. In practice, agentic workflows can make that forecast messy. Retrieval, retries, tool calls, and multi-step reasoning can turn one "user interaction" into multiple model requests.
Going local changes the cost profile, not the fact that cost exists. You may need a GPU, a bigger server, or accept slower responses on CPU. On top of that, you'll spend time on deployment, monitoring, model updates, and capacity planning. The comparison that matters is the total cost per successful task - not "API bill versus zero."
Local inference starts to look better when usage is predictable and sustained, when workloads are sensitive, or when avoiding external dependencies has real business value. For low or bursty traffic, a cloud API can still win - especially if you don't already have suitable hardware.
Data control is useful, but it doesn't equal compliance
Keeping prompts and documents inside your infrastructure can reduce exposure to third-party processors. That can matter for internal policies and data-residency requirements, including rules tied to GDPR or sector-specific regulations.
But "runs locally" isn't a compliance plan. You still need access control, retention rules, encryption, audit trails, backups, and data subject rights. You also have to ask whether your local model or embedding pipeline stores anything in logs or caches. Local deployment describes where the service runs, not whether your whole application meets privacy obligations.
Reliability moves inside your boundary
A local service avoids provider rate limits and external network jitter, but it introduces different failure modes: the model can run out of memory, hardware can saturate, and one machine can become a bottleneck. The trade-off is real - you gain control, and you inherit responsibility for fixing what breaks.
For many teams, that's worth it when you have the operational capacity. If your workload is low-volume and you don't have time to maintain model infrastructure, a managed API might be the safer production choice.
How the Architecture Fits Together
A clean first version has four separate responsibilities. Keeping them distinct makes the system easier to test, swap, and scale without turning your codebase into one big Python module.
1. FastAPI handles the application boundary
FastAPI gives you a typed HTTP interface, request validation with Pydantic, and an OpenAPI spec. It works well for an internal agent service, but async def doesn't magically make synchronous model inference non-blocking.
If you run a blocking query call directly inside the event loop, concurrent requests can get stuck waiting. For a simple deployment, use a synchronous endpoint for synchronous inference or move blocking work into a thread pool. For higher concurrency, use a queue or a model-serving layer that supports parallel inference. Base the choice on measured load, not on whether the handler is marked async.
2. ChromaDB stores vectors and document metadata
The ingestion pipeline chunks documents, generates an embedding for each chunk, and stores vectors plus useful metadata. At query time, ChromaDB returns semantically similar chunks for the user's question.
A persistent local ChromaDB collection is handy for prototypes and single-node setups. It is not automatically a horizontally scalable, multi-writer database. If multiple app replicas write to the same local filesystem, you can run into operational and consistency headaches. If you need multiple writers, high availability, or independent scaling, consider a server deployment or a different storage design.
3. LlamaIndex connects retrieval and generation
LlamaIndex provides building blocks for loading documents, creating indexes, retrieving relevant nodes, and sending context to an LLM. It reduces integration work, but it doesn't decide what "relevant enough" means for your product.
Chunk size, metadata filters, top_k retrieval, embedding choice, and prompt design all affect answer quality. Treat the framework as orchestration, not a quality guarantee. Test retrieval with representative questions and known answers. If it pulls the wrong passages fluently, that's still a failed system.
4. A local runtime performs inference
Ollama is a practical starting point for local model serving and a simple interface for development. vLLM is often a better fit when you care about throughput and batching, and your hardware and deployment setup can support it. The right runtime depends on your model, GPU, concurrency needs, and operational constraints.
Model weights also come with licenses and hardware requirements. Check both before deploying. Quantization can reduce memory usage, but it may affect quality and speed. A model that fits on a developer laptop isn't automatically suitable for a production workload.
You only avoid per-token provider bills if generation and embeddings run locally. You still pay for hardware, electricity, and operations.
Build a Working FastAPI and ChromaDB Baseline
In a RAG example, the part that's easy to skip is the boring one: you need a vector store filled with documents before retrieval can return useful context. The baseline below separates ingestion from serving and uses a local embedding model plus Ollama for generation.
Install the packages in a virtual environment, then confirm Ollama is running and the model is available. Package versions change - pin and test versions in your project rather than relying on an "install whatever" setup in production.
pip install fastapi uvicorn chromadb llama-index-core \
llama-index-vector-stores-chroma llama-index-embeddings-huggingface \
llama-index-llms-ollama
ollama pull llama3.1:8b
This example assumes a data/ directory with the documents you want to index. Run ingestion when documents change, not every time the API process starts.
# ingest.py
import chromadb
from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
Settings.embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
)
client = chromadb.PersistentClient(path="./chroma_storage")
collection = client.get_or_create_collection("enterprise_knowledge")
vector_store = ChromaVectorStore(chroma_collection=collection)
documents = SimpleDirectoryReader("./data").load_data()
VectorStoreIndex.from_documents(
documents,
vector_store=vector_store,
show_progress=True,
)
Then add a query endpoint. This uses a synchronous route because the LlamaIndex query operation shown here is synchronous. Runtime configuration happens once at startup, and errors are returned without leaking internal exception details to the caller.
# app.py
import logging
import chromadb
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from llama_index.core import Settings, VectorStoreIndex
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.vector_stores.chroma import ChromaVectorStore
logger = logging.getLogger(__name__)
app = FastAPI(title="Local AI Agent Gateway", version="1.0.0")
Settings.llm = Ollama(model="llama3.1:8b", request_timeout=60.0)
Settings.embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
)
client = chromadb.PersistentClient(path="./chroma_storage")
collection = client.get_or_create_collection("enterprise_knowledge")
vector_store = ChromaVectorStore(chroma_collection=collection)
index = VectorStoreIndex.from_vector_store(vector_store)
query_engine = index.as_query_engine(similarity_top_k=3)
class AgentQuery(BaseModel):
prompt: str = Field(min_length=1, max_length=4000)
class AgentResponse(BaseModel):
response: str
status: str
@app.post("/api/v1/query", response_model=AgentResponse)
def execute_agent(payload: AgentQuery) -> AgentResponse:
try:
result = query_engine.query(payload.prompt)
return AgentResponse(response=str(result), status="success")
except Exception:
logger.exception("Local agent query failed")
raise HTTPException(status_code=500, detail="Query processing failed")
Run it locally with:
uvicorn app:app --host 127.0.0.1 --port 8000This is a baseline, not a full production deployment. It doesn't include authentication, rate limiting, an ingestion API, document-level access controls, streaming responses, request tracing, or a model queue. Those omissions are intentional: what you add depends on who uses the service and what data they're allowed to access.
What Production Readiness Actually Requires
"Production-ready" isn't something you get from choosing FastAPI or adding Docker. It means the system behaves acceptably under real workloads, real failures, version upgrades, and misuse. Decide what "acceptable" means before you call the design production-ready.
Establish a quality and performance baseline
Start with a small evaluation set based on real tasks: the kinds of questions users ask, the relevant source documents, and the facts a good answer should include. Measure retrieval separately from generation. If the right passage isn't retrieved, switching the model won't magically fix it.
Track at least:
- Retrieval relevance and how often the needed source material is missing.
- Answer correctness against reviewed examples, including when answers are unsupported.
- Time to first token and total response time at expected concurrency.
- Memory and GPU utilization under representative workloads.
- Failure rates for model timeouts, malformed input, and missing dependencies.
The "right" numbers depend on the product. A private research assistant can handle a slower response than a customer-facing support workflow. Set targets based on user impact, then test on the hardware you plan to run.
Protect the API and the knowledge base
An internal endpoint can become externally reachable through a proxy misconfiguration or a changed network rule. Add authentication and authorization before you connect sensitive content to the service.
Enforce request-size limits, rate limits, and timeouts. Don't return stack traces or raw infrastructure errors to clients.
RAG also creates a permissions problem: you shouldn't return a passage just because it matches semantically. Store document ownership or access metadata and apply authorization filters at retrieval time. Protect the vector database and its persistence directory, and decide how documents, prompts, and logs are retained.
Plan for operations and growth
Keep the model server separate from the API process when their resource needs differ. Your API worker shouldn't accidentally load a large model multiple times and run out of memory.
For longer jobs or unpredictable traffic, use a queue to provide backpressure and clear retry behavior. Instrument the full request path: the API layer, retrieval, model inference, and the final response. Record timings and error categories, but avoid logging raw prompts or retrieved private content unless you truly have a controlled reason to do so.
Back up source documents and configuration. Decide whether the vector index can be rebuilt, or whether it needs its own recovery plan.
Scale only after you identify the bottleneck. Adding more API replicas won't improve throughput if one GPU is already saturated. Doing that once is manageable. Doing it across a fleet without clear ownership, model versioning, and capacity limits turns into an operational problem.
Conclusion
A local AI agent built with FastAPI, ChromaDB, LlamaIndex, and a local inference runtime can reduce API dependence and keep data processing inside the infrastructure you control. It's a sensible option when usage patterns, privacy needs, or operational independence justify the cost of running it yourself.
This isn't "cloud is bad" versus "local is free." It's a trade-off across cost, control, quality, latency, and whether your team can operate the system. Start with a small evaluation set, measure the full workload on real hardware, and harden the parts that affect your users. Then publish the code and assumptions so other developers can reproduce your results. For the next step, run the ingestion and query baseline on your own documents and record quality and latency before you decide on a production deployment.