Letta's FrontierMemory test says agent harness matters as much as the model

A team choosing between agent products on the model inside them has been looking at the smaller of two differences. The same Claude Opus 5.5 model, running the same 15-task benchmark, scored 73.3 percent in one agent system and 33.3 percent in another, and the thing that changed is how each system remembers what it did before.

Letta published FrontierMemory 1.0, the benchmark that produced those numbers, and The New Stack carried the write-up. The benchmark adapts coding and knowledge-work tasks from public tests so each one depends on something the agent should have learned from an earlier related task, rather than giving the agent all the context in the task prompt. The 15 tasks are run once each, at the harnesses' top effort setting.

What the numbers say

Across five agent systems, all running Claude Opus 5.5 for both memory and tasks, Letta Dream scored 73.3 percent, Prime Agent 60 percent, OpenAI Agents SDK 40 percent, Claude Managed Agents 33.3 percent, and Hermes Agent 26.7 percent. The gap between the top and bottom of that list is wider than the gap Letta measured between frontier models in the same harness. For a team already running a frontier model, the next place to look for better results is in the agent system around it.

A separate run compared seven models all generating memory inside Letta Dream, with Opus 5.5 running the task. Opus 5.5 and GLM-5.3 tied at 73.3 percent, Kimi K3 reached 66.7 percent, GPT-6.1 Sol and Gemini 4 Argon both reached 60 percent, Muse Wonder 46.7 percent, and Grok 4.7 33.3 percent. Letta also reports task cost: GPT-6.1 Sol cost $0.77 per task against $2.80 for Opus 5.5 and $3.58 for GLM-5.3, so a cheaper model generating memory for a stronger model to execute is one combination worth measuring.

What the benchmark does

Each task gives the agent transcripts of earlier related work, including failures, and asks it to write memory the agent will later use on a harder task. The verifiers were rewritten after Letta found that 53 percent of the sourced tasks had broken verifiers in the original benchmarks, so the pass or fail reflects whether the memory the agent kept helped with the real work.

Memory is split into three kinds, each with its own test. Task memory is a contract the first task did not spell out, such as skipping files named in an omit list. User memory is a repeated preference the user corrected earlier, such as a one-line changelog format. Environment memory is a non-obvious property of the workspace, such as where domain files live and which rules apply to them.

Limits in the paper

Letta states the limits on the figures. With 15 tasks and one attempt each, one task flipping outcome moves the score by 6.7 percentage points. The benchmark measures agents learning from past work rather than updating memory as they go, and Letta says a later version will add longer-running tasks and online updates. These are Letta's own figures on its own benchmark, so a team should treat them as a well-documented first run rather than a settled ranking.

For a team building a product on top of an agent, the useful move is to collect ten repeated tasks from its own backlog, run them through two candidate agent systems with the model held fixed, and measure the second attempt. The benchmark says the gap will show.

Source

FrontierMemory 1.0 by Letta, covered on The New Stack by Amanda Caswell.