AI agent in production

A demo covers the questions the person building it already expects. Real users ask questions nobody planned for, and a demo has no evaluation set behind it to catch a wrong answer before it reaches a customer. The difference between the two is safety checks, a backup response, and measurement. Those only get built once someone treats real customer use as the actual requirement.

It is a set of real questions with agreed correct answers, written before any agent code exists. Writing it first forces the team to settle what counts as a correct answer and who decides, which is normally the argument that teams skip, and that then comes back after launch as a support complaint. Every later change to the agent is measured against that same set instead of judged by whether it feels right.

No. At launch, by default, a person checks each answer before it reaches a customer. We remove that review step only once the evaluation set shows the agent's answers are correct without it, which is a measured decision rather than a launch-date deadline.

We plan the backup response from the start, instead of adding it later after an incident. That means a wrong or low-confidence answer goes to a safe response or a human, rather than reaching the customer as if it were correct. Retrieval also has to stay current, since an agent working from out-of-date information will confidently state something that used to be true.

Six to ten weeks is the timeline we scope to, not a measured average, since this is engagement pattern guidance rather than a record of past projects. The factor that changes that number is how well-defined the correct answers already are: an agent replacing a documented process is built faster than one where the team is still arguing about what a right answer looks like.

It is retrieval that stays current as source data changes, an evaluation set you can run on every code change, safety checks that stop a bad answer before it is sent, and a backup response for when the model gets it wrong. A working demo covers the other 80% in a weekend; those four pieces are what turn a demo into something that keeps working with real customer traffic.

The people who are responsible for the result, rather than the engineers building the agent, and that conversation happens while the evaluation set is being written, before any agent code exists. Deferring it to after launch means the first real disagreement about a wrong answer happens in front of a customer instead of in a planning session.

Yes, the same evaluation-first approach applies, and the safety checks and human review can usually be reduced sooner for internal use, since the cost of a wrong answer is normally lower than it is for a customer. The evaluation set still has to exist first: an internal agent with no measured standard for a correct answer fails in the same way, only for fewer people.