AI NewsProductivityAnnouncement
Surge AI tests agents on 50 tax jobs, top passes 11 percent
Surge AI's DAYJOB Tax benchmark tests coding agents on 50 real tax assignments with 1,789 workspace files. Claude Sonnet 5.5 passes 10.8 percent of tasks.
Image: Surge AI
Why it mattersA buyer looking at tax AI should ask the vendor to score here. The benchmark is public and the grading rubric is strict, so a one-shot demo tells you very little.
A software company that sells an AI agent to accountants can claim almost any accuracy number it likes. On 27 October, Surge AI published a benchmark that puts those claims against real tax work. DAYJOB Tax measures how coding agents handle 50 professional tax assignments across eight areas of practice. Claude Sonnet 5.5 leads the first run with a 10.8 percent pass rate at a 95 percent rubric threshold. Claude Fable 5.1 is next at 9.6 percent. Claude Opus 5.5 follows at 9.2 percent.
Surge AI runs the DAYJOB family of benchmarks. Earlier releases covered healthcare and finance. Each one asks an agent to take ownership of a professional's work. The agent reads an initial client request and the records that come with it. It then produces the finished tax return, the workpaper, the technical review or the client advice.
What is in the test
The benchmark holds 50 environments. The median environment carries 30 files. The largest holds 123. There are 1,789 files in all. Of those, 1,487 are PDFs. 117 are Excel workbooks. 38 are Word documents. The rest are emails, images, CSVs, JSON records and text files. The workspaces include filed and draft tax returns, K-1s, W-2s, IRS notices, general ledgers, payroll records, contracts and correspondence.
One assignment includes a photograph of a handwritten receipt. The receipt is for merchandise bought for resale. The agent has to read the item descriptions and amounts. It then matches them with spending records. From there it works out how the purchases affect inventory and cost of goods sold.
Another assignment involves an industrial silo-services company. The client thinks the company operates only in Alabama. The field dispatch log shows crews in five states. The agent has to spot the multistate tax exposure.
The eight practice areas are individual and family tax, small-business tax, partnerships, and property and investments. The other four are estates and trusts, international tax, corporate transactions, and tax review and disputes.
Grading
A run passes when it meets at least 95 percent of the task's criteria. The criteria cover calculations, supporting evidence, consequential issues and the finished deliverable. Surge AI reports pass@1 at that threshold. The company says every task was built from a real professional workflow and passed a three-layer expert review for correctness.
Surge AI shows a worked miss. A client named Anna was selling her condo. She asked whether closing in October 2026 or January 2027 would be better for taxes. Both dates satisfy the residence test for a $250,000 exclusion. But Anna moved out in August 2024, so her eligibility runs out around August 2027. Across five attempts, one agent named Astra never flagged the deadline. Opus flagged it in all five. Both agents evaluated the proposed dates correctly. Only one would stop Anna losing the exclusion by moving the sale.
The leaderboard is live on Hugging Face. Surge AI says a tax adviser has to spot issues a client did not know to ask about. That is the harder test. A buyer of AI for tax work can send a vendor to this score before signing anything. A 10.8 percent pass rate also tells a buyer not to let the agent sign off alone.
Source
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
