AI NewsProductivityAnnouncement
Atlan says context moves an analytics agent from 32 to 83 percent
Atlan AI published a playbook with its own benchmark on 107 real analyst questions. Adding context moved one model from 31.8 percent to 83.2 percent. Switching models moved it less.
Image: GitHub
Why it mattersA team building an analytics agent has been told to try more models and reach for retrieval. Atlan's own numbers say the model matters less than the context, and past SQL beats documentation.
A team building an analytics agent runs demos that look good. Then the agent fails on the tenth real question. On 27 October Atlan AI published a playbook and a benchmark with numbers on where that work actually goes. Atlan says context moved the same model from 31.8 percent to 83.2 percent on its own go-to-market warehouse. Switching models moved the score far less. The playbook is open source under CC BY 4.0 for the text and MIT for the code.
Atlan AI ran the benchmark on 107 real questions its analysts had asked. No person wrote the context behind the result. Atlan's systems built every metric, past query and term from the query log, the catalog and existing documentation. The grading was done by code.
What moves accuracy, in Atlan's words
The playbook lists five findings. Each one is tied to the benchmark run.
- Context moved the score more than the model did. Adding context took one model from 31.8 percent to 83.2 percent on the same questions. Switching models moved it less.
- Past SQL beats documentation. Queries analysts already ran carry the business rules. More documentation added a few points for about seven times the context. A join graph added nothing.
- SQL beats a sentence. On multi-table questions, a metric served as SQL beat the same metric written as prose by 16 points.
- Retrieve per question. Do not load everything. Served per question, the metrics layer scored 83.2 percent. Loaded whole into every prompt, it scored 52.3 percent. It cost more than twice as much per right answer.
- Provenance settles most conflicts. When two definitions disagree, the one most analysts ran recently, and that matches the warehouse, usually wins. The owner settles the rest.
The method
Atlan's method works one domain at a time. First the team scopes the domain and seals a golden set of at least 20 real questions. Then it generates metrics from past SQL and BI output, resolves conflicts and gets owners to approve them. Then it hill-climbs from traces, finds the smallest fix for each miss, and releases.
The design has one context call per question. The call is served live over MCP from one store. Two lanes decide how strict an answer has to be. The governed lane serves metrics generated from past SQL, checked against the warehouse and approved by owners. No metric, no answer. The exploratory lane lets the agent write SQL grounded in trusted past queries and shows its sources.
What the numbers mean, and what they do not
Atlan is honest about the limits. The playbook says the method transfers, and that another team's numbers will be its own. Atlan expects the same order to hold. Past SQL beats documentation. Metrics add to both. The company sells a product that runs this method, so the context behind the benchmark was built by Atlan's systems. The playbook says everything in it can be built without that product.
A team evaluating analytics agents can read this before buying one. The headline number is not 83.2 percent. It is that one model moved from 31.8 to 83.2 by adding context, with the model held the same. A buyer who swaps models to chase the last ten points is working on the wrong knob.
Source
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.