An engineer describes a monitoring agent that writes its own alert rules

A team shipping a config change at 3am used to know which three charts a human on-call should refresh for the next hour. Ben Congdon, writing on his personal blog, says the same job is now better done by an agent that reads the service's code first, picks the metrics itself, and investigates when they move.

Congdon has worked on change-point monitoring (tight monitoring around binary releases and config pushes) since 2023 and writes that it has been hard to get traction on with human-only methods. His post reached 128 points and 25 comments on Hacker News. He links the practice to a short paragraph in a recent Anthropic engineering post that described "superfan" Claudes, each of them dedicated to the health of a single component.

What the prototype does

Congdon describes four parts, from his own work. One agent reads the service's source and proposes a monitoring plan after each change. A scaffold executes that plan. A second agent investigates when a monitor trips, and escalates to a human on an unresolved thread. "Even in this early prototype," he writes, "I've been able to detect regressions in systems I have zero understanding of with precision and latency that would have been impossible a year ago."

Congdon writes that the design that worked best for him splits the work between an agent and a deterministic executor. The agent produces a written specification of what to monitor, and the executor runs that specification. Congdon says the split "produces significantly cleaner signals with dramatically lower token usage." The parts worth tuning, he writes, are the noise-reduction rules, the prompts that get the agent to write a good specification, and a labelled eval set large enough to tell whether a change is improving things or not.

The numbers he quotes

Three figures in the post are Congdon's own, from his own work. He says a few sessions of what he calls Astra and Opus class models, pointed at a shared internal library that his colleagues had already spent effort optimising, found a 50 percent heap saving and a 30 percent latency reduction. He says a moderately large tech company could watch its whole fleet with monitoring agents for around $100 a day, and a startup fleet for a fraction of that. In a footnote he adds that observability work his team recently finished saved several million dollars in annualised cost, and that this was itself a fraction of what his team alone spends on observability.

He also names three security-side products in passing: OpenAI's Daybreak Blue, Anthropic's Project Glasswing and Google's Fairwind, all of them agent-driven cyber defence. He argues that offensive and defensive work are already fielding fleets of agents and that site reliability is the next part of software operations where the same pattern applies.

What the method asks of a team

An agentic monitoring prototype at Congdon's bar takes real work. The team has to connect the agent to the live production codebase, to the observability stack, and to an escalation path, and then build a labelled eval set so the noise-reduction rules can be tuned honestly. His description places most of the value in the harness around the model. A team that treats the model itself as the product and skips the harness will get back the signal-to-noise problem that human-written alerts already have.

Congdon ends on a position some teams will reject. He writes that every production team should be running monitoring agents soon, and that teams who start now will do better than their competitors while the rest of the industry catches up.

Source

Agentic Monitoring, Ben Congdon.