AI On-Call Agents: Graph-Based Incident Response
AI on-call agents can connect alerts, dependencies, code changes and ownership. See how graph-based incident response finds causes, learns and handles limits.
Eight Alerts, One Cause: Building AI On-Call Agents on a Graph
At 2 a.m., your phone starts buzzing: checkout is failing. So are the frontend, the payment service, a proxy, and a few systems you barely touched. Eight alerts roll in - but that doesn't automatically mean eight separate failures. Often, one dependency is the real culprit, and everything else is just reacting.
The hard part of incident response isn't just getting an alert. It's stitching together enough context to answer: what's actually failing, what's only getting caught in the blast, and who should respond. Those answers usually live in different places - service maps, pull requests, team ownership files, on-call schedules, runbooks, and past incident notes.
A project called Blast Radius takes a straightforward approach to bringing that context together: model the relationships as a graph, then let a small set of AI agents use a handful of pre-tested graph queries as tools. The graph both supplies context and records what the agents did - evidence, hand-offs, and checkpoints. This article walks through how the design works, what its evaluation shows, and what teams should watch for before using something similar in production.
Why multiple alerts often point to one cause
An alert is evidence that something crossed a threshold. It's not an explanation. When a shared dependency fails, every downstream service can end up reporting symptoms. Responders then have to figure out which service is just showing the error and which component actually gives the best explanation for what's happening.
That distinction matters quickly. Page the wrong team and you burn time while the real failure keeps producing noise. If an automated agent files an issue against the wrong component - or sends a noisy notification - you don't get faster recovery, you get more work for everyone.
The useful question isn't just, "Which service is alerting?" It's: "Which component could account for the biggest chunk of these alerts, and what evidence supports that?"
Blast Radius frames this as a relationship problem. Its investigator needs to connect alerting components to dependencies, recent code changes, ownership, on-call rotations, and prior incidents. A service map helps, a pull request list helps, and ownership data helps - but the value is in being able to move between them during the same investigation.
The questions behind an incident
The project organizes investigation around a few concrete questions:
| Incident question | Relationship the agent needs to inspect |
|---|---|
| What might fail if this component goes down? | Dependencies traversed in reverse |
| What component could explain several alerts? | Shared paths from alerting services |
| What changed near the likely cause? | Recent pull requests connected to a component |
| Who owns the component, and who is on call? | Component to team to rotation to person |
| Have we seen a similar incident? | Incident similarity combined with graph relationships |
In a relational database, some of those questions turn into recursive queries or multi-step lookups. With a graph database, you can express the traversals directly - then package them as named query tools for the agent, and as inspectable evidence for the human reviewing the output.
One design choice matters here: it's not "use a graph because graphs are AI." It's "use a graph to represent the relationships responders already have to reason about." If the data is incomplete or stale, traversal can still produce confident-sounding but weak conclusions. A graph makes the connections easier to query; it doesn't magically make the underlying data correct.
Why a graph is useful incident context
Blast Radius uses FalkorDB and creates a separate graph for each organization within the same database instance. The tenant-specific graph naming pattern keeps one company's incident context separate from another's. That's helpful as an isolation boundary, though a real deployment would still need to validate access controls, backups, retention, and overall operational security around the database.
The model covers more than service-to-service dependencies. Components can represent services, data stores, Kafka topics, or user-facing flows like checkout and browsing. Edges describe relationships such as CALLS, READS_WRITES, PUBLISHES, and CONSUMES. Teams own components, people belong to teams, and pull requests modify components - while people author those pull requests. Runbooks, design decisions, incidents, alerts, actions, and lessons are also connected to the systems they relate to.
That breadth is what makes an investigation feel less like guesswork. It can move from "checkout is failing" to "these alerting services all consume the same topic; a recent change touched the producer; this team owns it." Without those connecting relationships, an LLM would either have to infer links from scattered text or depend on brittle integrations that don't capture the full picture.
Evidence and agent activity belong in the model
Two design details are especially practical.
First, agent state lives alongside the operational context. Each run and step is represented in the graph, and a step records which nodes it touched. That means you can ask what the agent actually examined, instead of reconstructing its actions from separate logs.
Second, evidence is modeled as relationships, not just prose. If an agent claims that a pull request supports a finding, that finding gets a SUPPORTED_BY edge to that pull request. A UI can show that connection, and the system can't cite evidence that isn't present in the graph. This doesn't prove the evidence is correct - but it improves traceability and makes unsupported claims easier to spot.
The project also stores timestamps as numbers, then presents them to the model as human-readable terms like "2h ago." That's a small implementation choice with an obvious payoff: smaller models can struggle with elapsed-time arithmetic from epoch values. Let the app do the math deterministically, and give the model a readable time instead.
A shared-dependency query
The root-cause query looks for components reachable from multiple alerting services, then ranks candidates based on whether the candidate is itself alerting, whether its dependencies are alerting, and whether recent pull requests modified it. In simplified form, the key traversal looks like this:
UNWIND $alerting AS alerting_name
MATCH (a:Component {name: alerting_name})
-[:CALLS|READS_WRITES|PUBLISHES|CONSUMES*0..6]->(c:Component)
WITH c, count(DISTINCT a) AS covers
The *0..6 range is intentional. Allowing zero hops means an alerting component can be considered as a possible cause itself, while keeping the path length bounded avoids scanning an unbounded topology. The full query ranks candidates using signals like:
- Coverage: how many alerting components depend on the candidate. Higher coverage suggests the candidate may explain more symptoms.
- Whether it is alerting: if the candidate is also reporting errors, it deserves attention - but it doesn't automatically mean it's the root cause.
- Alerting dependencies: if the candidate depends on other alerting services, it might be the victim rather than the cause.
- Recent changes: a recently merged change is useful context, not automatic proof.
This ranking is meant to support a decision - it's not a formal causal inference engine. A recent pull request can be coincidental. A dependency map can be incomplete. The design works best when the result is treated as a ranked hypothesis with inspectable evidence, not as a verdict that skips engineering judgment.
One example in the project uses the OpenTelemetry Demo's kafkaQueueProblems fault. Accounting and fraud detection don't call each other directly. They both consume the Kafka topic orders, and that shared dependency doesn't alert on its own. A search limited to the loudest service - or only direct service-to-service calls - could miss it. The graph finds it because both services consume the same topic.
How the agents investigate, coordinate, and recover
Blast Radius splits work across four agents, instead of asking one model to do everything at once. Triage groups alerts into an incident and assigns severity using graph logic, without an LLM call. The investigator searches for a likely cause and takes configured actions. A scribe writes incident memory after a human resolves the event. A "company brain" answers questions about system context, like why a component publishes to Kafka or who documented a decision.
The agents don't call each other directly. They coordinate through Task nodes in the graph. Triage creates an investigation task; a worker claims the oldest open task and marks it claimed using a single query. When a human resolves the incident, a record task hands off to the scribe. If a worker stops unexpectedly, the task stays visible along with its status and the worker responsible.
This is a more durable coordination pattern than relying entirely on in-memory conversation. There's also real operational overhead: task state, retries, stale claims, and worker recovery need clear rules. The upside is that work can survive process restarts and be audited by people maintaining the system.
Keep the model inside a controlled tool boundary
The LLM chooses which tools to call and produces a final finding, but it isn't allowed to invent arbitrary database queries. Instead, the project provides a set of named, tested graph tools. That keeps maintainers in control of what the model can retrieve, makes queries explainable, and lets the UI show what each investigation step did.
It also helps with a common failure mode: an LLM can run a useful query, get a strong result, and then ignore it when writing its final answer. In early testing, the investigator sometimes latched onto the loudest alerting service rather than the graph-backed candidate. Blast Radius addresses that by re-checking the submitted root cause against the graph.
If another component explains more of the alerting services, the system sends the model back with a comparison - for example, "Candidate A explains 2 of 4 alerts; candidate B is a dependency of all 4." After that rejection, the next answer is accepted. The graph acts like a second opinion, not an absolute veto, because the model might have relevant information not captured in the current graph.
There's a balance here. If you accept every model answer, the system becomes hard to trust. If you treat the graph as complete truth, you can also get stuck. In practice, the data model, ranking rules, and the agent's outputs should stay visible to the responder.
Incident memory and rate-limit recovery
After resolution, the scribe creates a postmortem and a lesson tied to the incident's root cause and fix. For later incidents, the investigator combines vector search with graph-based signals. The similarity score includes text similarity, whether the root-cause component matches, and overlap among impacted services.
That combination helps separate incidents that sound similar but have different causes. "Payment errors" and "payment latency" can look close in text while pointing to different failure patterns. Graph relationships add context that embeddings alone don't capture.
The project uses local bge-small-en-v1.5 embeddings via fastembed, avoiding an external embedding API call and its rate limits. One detail noted by the author: FalkorDB's vector query returns cosine distance, so the score is converted to similarity as 1 - score.
Free-tier model limits are treated like normal operating conditions. Blast Radius tries a provider chain, starting with Groq and then falling back to other configured providers or a local Ollama model. Errors get classified, and providers get cooldowns instead of being hammered by repeated retries.
Every model response and tool result is saved as a Step node and linked to the next step. If configured providers are exhausted, the run pauses and can resume later from the stored conversation and tool results. It doesn't need to repeat completed queries. A testing issue surfaced another constraint: after a long pause, a resumed model could keep calling tools until it ran out of budget without ever producing an answer. The fix was to allow only the final-answer tool once the budget is reached. A bounded agent needs a clear way to stop and summarize what it already knows.
What the evaluation proves - and what it does not
The project evaluates the investigator using the eight fault flags in the OpenTelemetry Demo. Each fault is injected into a fresh graph, the real investigator runs, and its answer is scored against hidden ground truth. The incident is then resolved, and the same fault is injected again to test whether the system can use incident memory.
| Measure | First occurrence | Same fault again |
|---|---|---|
| Correct root cause | 8/8 | 8/8 |
| Cited the incident it had just learned from | - | 7/8 |
| Average LLM calls / graph queries | 5.6 / 4.5 | 5.0 / 4.8 |
During evaluation, free-tier limits triggered 24 model fallbacks and 34 full pauses. Those paused runs resumed from their checkpoints and finished. That's genuinely useful: it suggests the implementation can handle the faults it tested, preserve progress across provider interruptions, and use stored incidents when the same fault shows up again.
What it doesn't prove is equally important. The author describes the evaluation as small: eight simulated faults on a realistic demo topology. Real incidents can involve partial failures, overlapping causes, missing telemetry, stale ownership data, and multiple independent changes. A system can perform perfectly on a fixed set of known fault patterns and still struggle outside that set.
Integrations, permissions, and operational safeguards
Blast Radius can populate context from a GitHub repository and accept Alertmanager-compatible webhooks (including alerts sent by Grafana). The GitHub ingestion path derives components and dependencies from Compose files, ownership from CODEOWNERS, and changes from merged pull requests. On the OpenTelemetry Demo repository, the project found 26 components and 43 dependencies. GitHub logins are stored as stable pseudonyms by default.
Alerts are matched to known components using labels like service, app, or pod name. Alerts that don't match are reported instead of being forced into a guessed component. That's a good default for automation: if there's uncertainty, it should stay visible rather than hidden behind a plausible mapping.
For actions, the project can create a GitHub issue with the likely cause, evidence, and a recommended fix, and it can send updates to Slack or Discord. The documented GitHub token is limited to one repository and Issues permission. Actions also use a fixed identifier per incident so retries or resumed runs don't create duplicate issues or re-send the same page twice.
For a real deployment, the same principle should apply to every side effect. Start with read-only investigation and human-reviewed recommendations. Add issue creation or notifications only after validating deduplication, permissions, and failure handling. A rollback or feature-flag change has a bigger blast radius than an issue, so it should require explicit approval and an auditable record. The project's planned direction - human approval before applying fixes - seems like a more credible path than giving an agent unrestricted production access.
The full source, design document, queries, and evaluation results are in the Blast Radius GitHub repository. The original project write-up includes additional implementation context.