What to instrument when nobody typed the code

An incident used to have a quick first step: find the person who wrote it. They remembered the assumption, the unusual case they had worried about, and the reason for the strange part in the middle. That step helps much less now. The code was generated in a session that no longer exists, from a prompt nobody kept, and the engineer who accepted it read it once.

So the system has to record what it is doing. That is what observability means in this context, and it is why we treat it as part of the change rather than as platform work that happens separately.

Decide it before the code exists

The instrumentation decision belongs in the definition of done, written by the person who understands the requirement. Two reasons.

First, the useful signals come from the requirement, not from the code. What should happen, how often, for whom, and what would it look like if it silently stopped. Someone reading a finished implementation will instrument what the implementation does, which is a different and less useful set of measurements.

Second, deciding afterwards means deciding during an incident, and the thing you want at that moment is data from before the incident started. Telemetry added at two in the morning tells you about the next occurrence, not this one.

The practical form is three lines in the change description: the metric that says this feature is working, the metric that says it is failing, and the log or trace a responder will need. If those three lines cannot be written, the requirement is not well enough understood to build, which brings you back to what done means when an agent writes the code.

Instrument the requirement, not the function

The most common instrumentation mistake is measuring code paths. Counters on functions and timers on methods produce a lot of data that answers questions nobody asks.

Measure the outcome instead. For a payment flow: attempts, successes, failures by reason, and the time from start to confirmation. For an export: jobs started, completed, rows written, and the age of the oldest unfinished job. Those numbers describe whether the product is doing its job, and they stay meaningful when an agent refactors the implementation underneath them, which it will.

This is also what makes the signals durable. Metrics tied to function names stop working after the next refactor. Metrics tied to business outcomes keep working, which matters a lot in a codebase where the implementation changes weekly.

Zero errors does not mean it is working

Generated code has a habit worth knowing about: it avoids reporting failure. It catches exceptions and returns a default. It falls back to an empty list. It logs a warning and continues. Each of those choices is defensible in isolation, and together they produce a system that fails without saying so.

The consequence is that absence of errors is a weak signal. You need at least one positive signal per important flow: something that goes up when the feature is working, so that a flat line means it is broken. A payment error rate of zero is ambiguous. A payment success count of zero is not.

This is the practical version of the argument in our post on why observability matters more with generated code. It is not that generated code is worse. It is that it hides its failures in a specific and consistent way.

Four signals per user-facing flow

The set we ask for, per flow that matters to the business:

Volume. How many times it happened. Detects a flow that stops with no error, which is the failure mode dashboards usually miss.

Success rate. Of those, how many finished correctly, with failures grouped by reason rather than by exception type. Reasons tell you what to do; stack traces take long study to understand.

Latency of the slowest requests. The median hides the slow cases. The slowest few percent is where the timeouts, the retries and the unusual data appear, and it is where generated code most often behaves differently from what the test of normal use showed.

One business number. Orders placed, messages delivered, reports generated. This is the one that tells you a technically healthy system is not producing the outcome, which is a failure class no technical metric catches.

Four signals per flow, on flows that matter, beats a hundred metrics per service. The cost of the second approach is not only money, it is that nobody can find the important number when it matters.

Traces are worth more than they used to be

When nobody understands the whole system, the request trace becomes the fastest way to understand it under pressure. A responder who can see the path a request took, with timings, does not need the author's memory to work out where it went wrong.

Two things make traces useful rather than decorative. Propagate context across boundaries, including queues and background jobs, since the interesting failures are usually the ones that cross a boundary. And attach the identifiers the business uses, such as the order or account, so a responder can go from a customer report to a trace without a translation step.

Do not pay for what nobody reads

Telemetry costs money and attention, and in a fast-moving codebase it accumulates. Two rules keep it under control.

Every dashboard has an owner and a question it answers, written on it. If neither exists, delete it. Old dashboards built during past incidents only add noise that slows down the next responder.

Alerts must be actionable and rare. An alert nobody acts on trains the team to ignore alerts, which is the same failure as an ignorable gate. If an alert has fired twenty times and nobody has ever done anything, either fix what it is telling you or delete it.

Logs should let you rebuild what happened

Generated code produces a lot of logging, and most of it is the wrong kind: a line at the start of a function, a line at the end, and the value of a variable that seemed interesting at generation time. That output is expensive to store and nearly useless during an incident, because it describes execution rather than decisions.

The logs worth keeping answer one question: why did this request end up here. That means logging the branch taken and the reason, not the fact that a function was entered. It means logging the identifier that lets you follow one customer's request across services. And it means logging failures with the input that caused them, redacted, because a failure you cannot reproduce is a failure you will be guessing about.

Two rules keep the volume manageable. Log at a level that lets you turn detail up for one customer or one route without turning it up for everyone, so an investigation does not require a deploy. And be deliberate about cardinality: attaching a unique identifier to a metric is how a monitoring bill triples overnight, and it is an easy thing for a generated change to do without anyone noticing until the invoice arrives.

The practical test is the same as for documentation. Take a past incident, open only the logs, and see whether you could reconstruct what happened. If you could not, the gap is the thing to fix, and it is usually one decision point that was never recorded.

Alerts should describe symptoms, not causes

One more habit that matters more when the code is unfamiliar: alert on what the user is experiencing, not on what you guess the cause will be.

Cause-based alerts are guesses written into configuration. A rule that fires when a queue depth exceeds a number encodes a belief about which failure will happen and how it will look, and generated code fails in ways nobody predicted, which is exactly the case those rules miss. Symptom-based alerts, on the success rate of a flow or the latency a customer sees, catch the failure regardless of its mechanism.

The practical rule is that every alert should be able to complete the sentence "customers are currently unable to ...". If it cannot, it is a diagnostic signal rather than an alert, and it belongs on a dashboard where someone looks during an investigation instead of on a pager at three in the morning.

Where this connects

Instrumentation is what makes progressive delivery a decision rather than a wait, and it is the input for incident response for agent-written code. It also feeds the delivery numbers in measure delivery, not generated output. The main guide sets out how the pieces fit, and this is a standard part of how we run an AI development engagement.