Image: Raindrop via The New Stack
Why it mattersA team can see how a prompt or model swap changes agent behaviour before it reaches production. The check runs on every pull request.
Changing a prompt in an AI agent can break things that unit tests never see. Raindrop opened its Simulations platform in public beta on Thursday. It shows what a change does to agent behaviour before the code merges.
Raindrop rebuilds the agent's production environment from its own traces, says the company. The agent's real code runs in a sandbox. It talks to synthetic versions of its tools, APIs and databases. For each tool call, Raindrop writes a reply from production traces. It then checks that reply against the types the code expects. The simulated environment stays consistent for the whole run.
What the beta covers
Ben Hylak, Raindrop's co-founder and CTO, told The New Stack that predefined tests miss failures a developer did not anticipate. "Every time agents run, they go different paths," he said. A prompt can depend on feature flags, on account details and on the user's plan, the company says. A static test cannot cover every path.
Raindrop's product page shows a run that finished 20 simulations in 3 minutes 42 seconds. The example reports the cost of each run moving from $0.042 to $0.051, a 21.4 percent rise. The run names 2 regressions and 1 warning. The simulated environment in that example came from 1,284 production traces across 5 services and 11 tools.
Filling gaps and flagging them
A new tool the agent calls brings data Raindrop may never have seen. The company can fill that gap with synthetic data, Hylak said. It marks the simulation as less faithful when it does. Each run carries a fidelity score. The lowest-scoring parts are flagged. Customers can connect a data source through the Model Context Protocol and rerun the test.
A missing feature-flag value can also leave the simulator without context the prompt depended on. Hylak said Raindrop can suggest pull requests that improve the agent's telemetry. The value is then recorded next time.
Models can notice that a test is synthetic, Hylak said. They can act differently when they do. The team tries to measure this testing awareness in each run. To account for normal variation, Raindrop runs each simulation several times. The results are compared with production history.
Early use and price
Customers and Raindrop's own team ran more than 33,000 simulations during early access, according to the company. Those simulations covered more than 1,700 pull requests. Raindrop first described Simulations in September. The Series A announced then brought its total funding to $50 million.
Pricing is per pull request, Hylak said. Raindrop charges for rebuilding the environment and for running the simulations. The agent's own model usage is billed through Raindrop during the beta. The final bill depends in part on how long the agent runs. The product page says general-availability pricing is not yet set.
A team that already has written evals keeps them. What Simulations adds is a check on cases the team never wrote down. The reader sees a prompt edit, a model swap or a new tool play out across many paths. The check runs before anyone merges the change.
Source
The New Stack by Frederic Lardinois. Raindrop's own Simulations page and its Series A post.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
