Armature measured 16,893 coding agent sessions to see which tools they pick

Armature has published a study of 16,893 sandboxed sessions in which Claude Code, Codex, and Cursor were asked to pick and install a third-party service inside 75 real repositories. The company said the three agents chose the same tool in only 42 percent of the categories tested, and picked a different tool for the same job depending on the codebase's language.

Armature said upfront that it sells growth services to developer-tool vendors and that this study is part of that work, so the study exists to support a sales argument. The methodology and the raw traces are public, which is what makes it worth reading anyway.

How the runs were built

Armature said it started from statistics across thousands of public GitHub repositories and rebalanced the distribution to reflect the real developer population. It ended up with 75 repositories in 10 languages, each with fake company names, fake git histories, fake API keys, and real lockfiles verified against npm. Each experiment asked one of four personas, from a casual coder who mostly prompts AI to an engineer at a large enterprise, to complete a real task inside one of those repositories. Prompts totalled 1,163 variations. Every run happened in an ephemeral sandbox, rotated across E2B, Blaxel, and Daytona. A second Gemini 3.7 Flash instance acted as a judge, keeping only sessions that met validity criteria. From the 16,893 runs, 5,292 sessions across 51 codebases and 18 sectors were kept for this first release.

Where the agents disagreed

Armature reported that Cursor uses the web for two thirds of its decisions, Codex uses web search in 94 percent of runs and often with site: operators pointed at trusted domains, and Claude Code relies on what it learned in training and searches the web in about 30 percent of cases. When Claude Code did search, it browsed three times more pages than Codex. In the voice agents category, Claude Code picked Twilio, Codex picked the OpenAI Realtime API, and Cursor picked Vapi. Claude Code also built the feature in-house nearly twice as often as the other two, at 19 percent versus 10 percent.

The codebase language changes the choice

Given the same task on four codebases in four different languages, four different email providers were chosen: Resend on TypeScript, SendGrid on Python, Postmark on Go, and Azure Communication Services on Java. Vercel was the top choice for TypeScript deployments and was never recommended on Python repositories, where Render was the top choice.

Mentions rarely led to picks

Armature reported that being brought up in the conversation almost never mattered. PayPal was cited 139 times in payment-provider sessions and picked zero times, with Stripe picked in 124 of those. LangChain was the most mentioned framework at 194 mentions and was picked four times. Supabase was the most mentioned database at 242 mentions and was still picked less often than Neon. The report said Supabase's bundled features did not help when the task needed only a database.

Vendor choice changes with language and with tool

If an engineer lets a coding agent pick a third-party service, three engineers on the same team will end up on three different vendors, and the same engineer will end up on different vendors depending on which language the file is in. The study also said small copy details on a vendor's docs page, a retention window or a bundled feature list, made agents switch to a competitor. A team that cares which set of tools it runs has to say so in the prompt or in a project rule, because the agent's default answer is not stable across tools or across repositories.

Source

Primary source: Which tools do Claude Code, Codex and Cursor choose? We measured 16,893 sessions to find out.