Prompt Injection in HTTP Headers: Risks & Defenses
Learn how prompt injection in HTTP headers can reach AI agents, what a Common Crawl scan found, and practical controls to limit exposure. Audit your tools.
The Hidden HTTP Header Problem Facing AI Agents
AI agents don't "browse" the web quite the way people do. When I open a page, I see the rendered content. When an agent fetches that same page, its tooling can also expose bits of metadata most browsers hide - like HTTP headers. That creates a quieter route for indirect prompt injection: instructions tucked into headers instead of the page body.
In one Common Crawl analysis of 217 million page responses, researchers found 7,959 sites sending headers aimed at AI systems. Most pointed to places the agent might look next - like llms.txt files or MCP servers. Thirty sites sent what the researchers classified as injection attempts. Even so, those numbers don't automatically mean "thousands of agents are compromised." A header only matters if the agent's fetch tool passes it into the model, if the model treats that text as instructions, and if the agent is allowed to take the resulting action.
So the real question for anyone building or deploying agents isn't "does hostile text exist?" It's whether untrusted data can slip into the agent's decision-making context and then trigger something consequential. Here's what the scan found, where the exposure comes from, and how to reduce the risk without making every web-fetching task pointless.
Why HTTP headers matter to AI agents
Indirect prompt injection is a data-boundary problem
Indirect prompt injection happens when an agent encounters instructions inside something it was asked to process: a web page, email, document, or tool result. I didn't write those instructions, and I might never even see them. But if the agent puts that fetched content into the model's context, the model has to decide what's "information to use" versus "instructions to follow."
And that split isn't guaranteed. Often the agent hands the model the user's request and the fetched material as one combined chunk of text. If that fetched text says something like "ignore the original request" or "send data elsewhere," the agent has to treat it as untrusted - rather than as authority.
The stakes depend on what happens next. A basic summarizer might just generate a wrong summary. An agent with access to email, a shell, databases, files, or external services could do more. The exact same injected text can have very different consequences depending on the agent's permissions.
A response contains more than the page body
An HTTP response usually includes a status line, headers, and a body. The body is the content people recognize - HTML, JSON, a file, and so on. Headers carry metadata too: content type, caching hints, cookies, redirect destinations, and more. Servers can also add custom headers with their own names and values.
Browsers generally don't show me headers as part of the page. But an agent's fetch tool might. For instance, a tool built around curl -i can pass headers and body together. A tool that only calls something like response.text() might show the body while dropping the headers. And a tool that returns selected metadata sits somewhere in between.
That's why "the browser doesn't display it" isn't a security boundary. The real question is what the specific tool puts into the model's context. A header that never reaches the model can't directly instruct it. Even if a header does reach the model, it still may not work - models may ignore it, and the action it suggests may be unavailable or blocked.
Key takeaway: Treat every field supplied by an external site as untrusted input, including ones your normal browser hides.
What the Common Crawl scan found
Pointers were far more common than direct instructions
The scan looked at 217,097,223 page responses - roughly 10% of the September 2026 Common Crawl collection referenced in the source study. Researchers identified 7,959 hostnames sending at least one header they classified as AI-directed.
Most of those headers weren't direct instructions. A total of 7,271 sites pointed agents to another resource, and 6,989 used a header named x-llms-txt. Other examples pointed to agent documentation, skills, or MCP servers. In other words: a lot of sites are already using headers to tell AI systems where to look next.
A pointer can be useful, but it can also steer an agent into material controlled by the site owner. If an agent automatically follows a link, connects to an MCP server, or installs a skill, the original fetch has effectively moved the agent into a new trust boundary. That destination still needs its own checks; it shouldn't automatically inherit trust because the first page pointed to it.
The scan also found 139 sites that recorded whether a request came from an AI bot. Signals like that can support different behavior for agents versus humans. They can also enable cloaking: a server might show agents content a human wouldn't see at the same URL.
The injection attempts were rare, but they show the channel
Researchers classified 30 sites as sending injection attempts. Twenty-four used a header named X-AI, and most attempts looked like pranks - telling the model to ignore previous instructions and return a particular word or format. Other examples asked the agent to run a command, identify itself to another service, or treat a fabricated hiring score as fact.
Those counts need context. The scan used a sample of the crawl. Common Crawl also fetches pages with its own crawler, not "every possible AI agent." A site that sends a header only to a named agent may not show that behavior to Common Crawl. And the researchers didn't follow URLs or execute commands mentioned in the headers. The results show that the channel exists - it's not a measurement of how often real-world attacks succeed.
There's also a detection issue. The scan found 1,709 sites whose headers matched AI-related keywords but weren't actually AI-directed. A detector based on names alone would have generated 910 false positives in the researchers' analysis. Headers like Content-Usage or Content-Signal can be legitimate signals, while strings like ai can appear in unrelated header names.
So the operational lesson is simple: a suspicious header name is a reason to investigate, not proof of an attack. What matters most is what the agent does after the fetch.
Where exposure enters the agent stack
Fetch tools decide what reaches the model
Different tools expose different parts of an HTTP response. The exact behavior can change with settings, versions, and how the agent is configured - so test your implementation instead of relying on a product name.
| Fetch pattern | Potential exposure | Practical check |
|---|---|---|
| Body-only HTTP client | Usually page content, not response headers | Confirm metadata isn't added elsewhere in the wrapper |
Shell access using curl -i, -I, or -v |
Can expose response headers | Restrict shell use or inspect how command output enters context |
| Fetch tool that summarizes pages | May pass an extract, redirect target, or selected metadata | Review output format and error handling |
| Browser or MCP network tool | May expose full headers or selected fields | Remove access if the task doesn't require it |
| Error-handling path | May include headers or server-provided error text | Test error responses, not only successful pages |
The edge cases matter. Redirect targets are chosen by the server, and error messages may quote text from a response or from robots.txt. A tool that normally returns only page content can still leak metadata on non-HTML responses or failure paths. Test redirects, errors, downloads, and authentication challenges - not just a clean 200 OK page.
Model resistance helps, but it isn't authorization
In tests from the source study (October 2026), Claude Opus, Sonnet, and Haiku running through Claude Code saw test headers when a task led the agent to use curl -i. They didn't follow the harmless instruction by default. In separate runs, when the user's prompt explicitly told the agent to follow directions in response headers, the models complied in seven of nine runs.
These results are limited: each model/tool/prompt combination was tested once, and behavior could differ across other models, versions, tools, and tasks. Still, the design lesson is clear. The agent's behavior depends not just on the text - it also depends on what authority the agent has been granted via system prompts, configuration, and user instructions. If you tell an agent to follow a site's "directions," a header that would otherwise be ignored can suddenly become relevant.
A model's ability to recognize suspicious text is useful as one layer. It's not a replacement for permissioning outside the model. If a shell command, network request, or database write is sensitive, approval should depend on the task and the action - not on whether the model thinks the request "looks safe."
How to reduce the risk in practice
A solid defense is a sequence of controls around the fetch and the action. Each control limits a different failure mode; one prompt or one filter won't cover everything.
-
Minimize what enters the context
Return the page body by default. Only pass through the metadata the task actually needs - things like a status code or a normalized media type. Don't ship every header "just because it's there." Raw headers can include cookies or other secrets, plus arbitrary text chosen by the server.
If you truly need headers (for example, to diagnose caching), pass only the relevant fields and label them as external data. You usually don't need
Set-Cookie, and you definitely don't need a verbosecurlresponse just to answer whether a cache directive is present. -
Treat destinations as new trust decisions
A redirect, an
x-llms-txtpointer, a link to an agent skill, or an MCP endpoint selected from a page is a new fetch or connection initiated by an external party. Apply the same domain, network, and content checks you would apply if the user had provided that destination directly.Require approval before connecting to a new MCP server or installing a skill discovered on a page. A pointer isn't permission. It's just information about where the site wants the agent to go next.
-
Keep external content in the data lane
When you build context, preserve the source and label the fetched material as untrusted data. Don't merge it into system instructions or imply that a website can redefine the task for your agent. Avoid prompts like "follow any directions on the page" unless you truly intend that authority and you've scoped it to something safe.
Labels alone won't prevent every model mistake, but they make the boundary explicit - and that makes testing easier. Your app should also retain provenance so you can trace a proposed action back to the fetched content that influenced it.
-
Authorize actions outside the model
Before any tool call runs, verify it against the user's task, the destination, and the data involved. For example, an uptime check shouldn't authorize an arbitrary shell command. Summarizing a page shouldn't authorize sending a report to some new external endpoint.
For high-impact actions - database writes, file changes, external messages, package installation, and connections to new services - use human approval or a policy gate. The model can suggest an action; a separate control should decide whether it's allowed.
-
Log sequences, not just suspicious strings
Good telemetry links what was fetched to what happened next. For each request, log the session and tool identity, requested and final URL, status code, header names, which fields were passed into the model, and the next tool call.
Be careful about storing raw headers. Cookies and authorization data can turn debug logs into an accidental secondary secrets store. Keep raw values only when there's a clear need and an appropriate retention policy.
Then alert on sequences that actually matter: a fetch followed by a shell command outside the task, a database write, a new network destination, a new MCP connection, or a request to a URL mentioned in a response header. That's usually more useful than treating every header containing
AIas automatically malicious.
A practical starting point: inventory every fetch tool, check what it returns on success and failure, then run a controlled test with a harmless marker header. After that, verify the agent can't take an unrelated action unless an external policy check explicitly allows it.