Replit's ViBench 2.0 benchmark asks AI agents to build whole products

Giving an AI agent a short, well-shaped task is one thing. Asking it to start from an empty folder and build an entire product is another. Replit said on Wednesday that it has released ViBench 2.0. It tests the second kind of work. The post is by Michele Catasta. ViBench 2.0 joins the Specialized Intelligence Index from Fireworks. That index groups benchmarks that compare open, closed and specialised models on real tasks.

What the benchmark asks

ViBench 2.0 has 17 products, 241 feature requests and 544 test plans, Replit said. Each product is modelled on something a real user would know, like Uber, GitHub or Google Docs. The agent starts with nothing. It builds one feature at a time. Once the last feature is in, a grading agent runs the test plans in a real browser. It clicks, types and checks the result.

The requests describe what a user expects, not how to build it. Replit shared one example from a design tool's undo feature. Alice and Bob take turns editing a rectangle. Alice presses undo. The plan checks that only Alice's own edit is rolled back. It checks that Bob's rename stays. It checks that the shape ends up at the right position and colour. Replit said the three strongest models fail this plan in eight of their twelve builds.

The numbers Replit measured

Fireworks ran the official grades. Each model built every product four times, on the same harness and settings.

GPT-6.1 Sol led at 65 percent pass@1, Replit said. GPT-6 Astra followed at 60 percent. Opus 5.5 came third at 49 percent. All three pass more than 90 percent of their test plans. Most failed builds miss by only a few checks. Sol costs about $12 per build. Astra costs about $90. DeepSeek V4.1 Flash passes 77 percent of its test plans at about $2.50 per build, Replit said. No open-weight model yet passes a full build.

Opus 5.5 finishes a build in about 70 minutes on average. That is roughly 2.5 times faster than Sol, Replit said. Opus produces only about 6 percent fewer tokens, so most of the gap is the API provider's speed.

Replit also ran older GPT models on the eight public products. Pass@1 rose from zero at GPT-5.4 to 31 percent at GPT-6 Sol and 53 percent at GPT-6.1 Sol. Plan pass rate rose from 42 to 95 percent. The newer GPT-6.1 Terra passes 47 percent of builds at about half the cost of Sol, Replit said.

Where the top models trip

Replit said two failure modes stand out. In about one in 40 of the top models' builds, the product never starts on the empty database. Later schema code names a table before the statement that creates it. More often, builds fail where features meet. Nearly three-quarters of the failed builds from the top models fail only at these feature-to-feature joins.

The grader is Opus 5.5. Replit said it chose Opus because it made the fewest grading mistakes in testing. Replit planted 34 bugs one at a time in working builds. Opus never let a plan pass while a bug was visible. GPT-6.1 Sol re-graded 477 of Opus's passes. 99.2 percent were confirmed correct, Replit said.

A team running agents on real codebases now has a cheaper number than its own eval suite to point at. ViBench says the hard part is not one feature. The hard part is the second feature not breaking the first. A pass rate on short tasks tells a buyer very little about that risk. A pass rate on whole builds, with the method open, does.

Source

• ViBench 2.0: A Benchmark for Long-Horizon Software Building, Replit