AI NewsModels & agentsAnnouncement
ORO AI launches a benchmark that puts frontier models in charge of a produce wholesaler
ORO AI's SMB Bench puts 16 frontier models in charge of a produce wholesaler for 90 days. Only three beat a simple buying rule.

Why it mattersA new benchmark scores AI agents on profit over 90 days of a real-feeling business. Buyers of an AI agent now have a public number to look at. The number is small.
ORO AI, a YC Fall 2026 startup, launched on Y Combinator on 18 November. It had 26 votes on the YC launch page at the time of writing. Its product is SMB Bench. The simulation puts an AI model in charge of a produce wholesaler for 90 simulated days. ORO AI published the full technical report on 6 November. It was written by Shardul Bansal, Seth Schilbe and Jarrod Barnes.
What the benchmark measures
Each day the model decides what produce to order. It has to supply walk-in customers and restaurant accounts. It has to manage stock that goes bad. Customers do not always say why they stopped ordering. Suppliers do not always deliver on time. The score is mean profit per quarter.
The report runs each business in three variants: a normal quarter, a quarter where shorted accounts cut future orders, and a quarter where suppliers run late. ORO AI compares every model against two baselines. A fixed buying rule earns $6,412 a quarter. A scripted expert earns $10,795.
What the models earned
ORO AI tested 16 frontier models across 485 runs on 18 tasks. Three models beat the fixed buying rule. GPT-6 Astra earned $7,749, Gemini 4 Carbon earned $7,703, and Claude Fable 5.2 earned $7,275. Nine models lost money. Claude Haiku 5.5 lost $7,440 a quarter. Muse Wonder 1.0 lost $8,291. GPT-6 Luna lost $8,667.
Every model under-bought against the fixed rule. The gap ran from 5 percent to 42 percent fewer purchases by volume. ORO AI then replayed the runs using each model's own stated orders. Net value rose by 1.3 to 2.7 percent of quarter sales. The report calls this "a gap between observation and action".
Where the models failed
The report calls out learning from delayed consequences. ORO AI ran 323 cases where shorted accounts cut future orders. A model connected the falling orders to its own shortfalls in writing only 69 times. Only 12 of those runs then protected every account within three days.
Model summaries were often wrong. ORO AI read 208,012 reasoning traces and found that 0.35 percent described work outside the task. In 91 stocking decisions, provider-generated summaries left the reason out. One model, Gemini 4 Argon, lost its notes for more than 40 days in 15 of 54 runs. Those were also the runs with lower scores.
A buyer who was planning to hand a working budget to a general-purpose agent has a public number to check. The best score is still $3,046 a quarter under a scripted buyer. ORO AI's limits apply here too. One benchmark on one kind of business is a starting point. Run the same model on your own task before letting it spend real money.
Source
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.

