AI NewsProductivityAnnouncement

Stripe published evals for 14 coding agents on real payment scenarios

Stripe ran 276 evals against 14 coding agents on real payment integration tasks. Headline pass rate was 78.1 percent. Strict pass rate was 29.4 percent.

AI News

Editorial2 min read

LinkedInX
Reveneau hero illustration for a coding agent benchmark story

Why it mattersA pass rate on partial credit hides what a payment integration needs. Read strict pass rates before picking a coding agent for a payment task.

A coding agent writes a payment integration that looks correct and still leaves a bug that costs money. Stripe put a number on that gap. On the strict pass rate, the agents it tested dropped from 78.1 percent on average to 29.4 percent.

Stripe published its integration evals on November 19. The authors are Kiran Prasad, Yiing Chau Mak and Manish Kurse of Stripe. The post says the evals measure how well coding agents build high-quality integrations. A second agent plays the role of the developer asking for the work.

What they measured

Stripe wrote 69 scenarios across three product areas. There are 24 for payments, 26 for subscriptions and 19 for Connect. It tested 14 model-harness pairs. Each pair ran every scenario twice, with and without Stripe's coding plugin. That is 276 runs per agent.

The three harnesses are Claude Code, Codex and OpenCode. Models tested include Opus 5.5, Sonnet 5.5, Fable 5.2 and GPT-6 Astra. The set also covers GPT-6.1 Sol, Gemini 4 Argon, Grok 4.7, Kimi K3 and GLM 5.3.

Stripe says agents often reach for older APIs and miss key features. The strict pass rate requires every mandatory criterion in a scenario. The 78.1 percent to 29.4 percent gap is the cost of partial credit.

Which agents scored best

GPT-6 Astra on Codex led with 85.5 percent, rising to 90.0 percent with the plugin. GPT-6.1 Sol on Codex reached 85.6 percent, 89.6 percent with the plugin. Opus 5.5 on Claude Code reached 83.5 percent, 86.2 percent with the plugin. Fable 5.2 on Claude Code reached 81.1 percent, 85.2 percent with the plugin.

Gemini 3.8 Flash on OpenCode trailed at 64.8 percent, 72.7 percent with the plugin. GLM 5.3 on OpenCode read 68.1 percent, 72.6 percent with the plugin.

The plugin raised every score. Scores moved by 3 to 8 points on average. Web searches dropped 96 percent, from 1.26 to 0.05 per eval. Page fetches fell 63 percent. The rate of outdated API parameters fell from 7 percent to 1.3 percent.

What a run costs

Costs ran from $0.05 to $5.76 per run without the plugin. With the plugin they ran $0.07 to $6.13. Time per eval ran from 5.1 to 28.2 minutes without, 5.3 to 27.9 with.

Stripe says it is also "publishing our Stripe benchmark and eval framework" on GitHub, as a project called Forge. It plans to maintain the harness and update the results. For a team running a coding agent against any payment provider, a headline score is not enough. The strict pass rate is the number that matters, and the method behind it belongs in the choice.

Source

Stripe evals: Measuring real-world coding agent performance, Stripe.

SourceStripe

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project