What is a context window and why does it matter for agent tasks?

A context window is the total amount of text a model can read and work with at one time, measured in tokens. It matters because the model can only act on what fits inside it. Instructions, examples, prior steps, and the work itself all share the same space, and when the window fills, the model starts to lose earlier parts of the conversation, which changes what it does next.

How does context length affect the quality of an agent's output?

When a task is long and the agent reaches the far end of the context window, it may lose the specific instructions you gave at the start. The model continues generating, but the constraints from the beginning of the prompt are no longer in the part it can see. The result looks coherent but may not follow the original requirements, because those requirements are no longer in the context the model is working from.

What should go first in a task prompt for an agent?

The most important constraints and the precise output format. Models attend to recent text more reliably than text far back in a long prompt, so if the last thing the agent reads before it starts writing is the format requirement, it is more likely to follow it than if the format was stated at the top of a 10,000-token document. Put the fixed rules close to the point where the agent begins producing output.

How does a long system prompt affect what an agent can do?

A long system prompt uses context space that the agent would otherwise use for the actual work. If a system prompt runs to 2,000 tokens and the task requires reading a 4,000-token document, those 6,000 tokens together consume a meaningful fraction of the available window. Keep system prompts short by stating the role and the constraints without examples when the agent already knows the format, and pass examples as separate context only when the agent genuinely needs them.

What is the difference between stuffing context and structuring it?

Stuffing is adding every potentially relevant piece of information to the prompt in case it helps. Structuring is deciding what the agent needs to know at the moment it produces each piece of output, and providing only that. Stuffing fills the window with material the agent will not use on the specific step it is running, which crowds out the material it will use. Structuring asks: what does the agent need to know right before it writes this output?

How should reference material be handled when it is too long to fit?

Retrieve only the parts of the reference material that are relevant to the specific step the agent is running, rather than passing the whole document. A retrieval step that pulls the two or three relevant sections from a long document costs one extra call and keeps the working context small enough that the agent can reason about what it is reading rather than processing it in chunks too large to attend to fully.

Does the order of content in a prompt affect what the agent does?

Yes. Models attend to the beginning and end of a prompt more reliably than to the middle, and to recent text more than to text far back in a long conversation. For a task that requires following specific rules, repeat the most important rule near the point in the prompt where the agent will start generating the constrained output. For a long conversation, re-state the core constraint in the message where you need it honored.

When is it worth using a model with a larger context window versus splitting the task?

Use a larger window when the task genuinely requires holding a large body of information in view at once, for example a refactoring step that needs to read the whole file before rewriting any of it. Split the task when the steps are separable: when the agent can produce the first output without needing the full set of inputs, run the first step, check the output, then pass it into the second step with fresh context. Splitting is cheaper and usually produces more reliable output on each step.

Why context windows change how you structure a task for an agent: Reveneau

Q: When should a task be split across multiple agent calls rather than one long call?

When the total input, including the task description, any reference material, and the expected output, is longer than about half the model's context window. At that point, the agent is spending a large share of its available attention on reading rather than reasoning, and earlier parts of the input are starting to drop from reliable attention. Two shorter calls with focused context will usually produce better output than one long call that fills the window.

Q: How do you know if context is hurting your agent's performance?

Run the same task twice: once with the full prompt as written, and once with a shorter version that cuts material you think is redundant. If the shorter version produces the same or better output, the extra material was using context space without contributing to the result. The cleaner test is to look at whether the agent follows late instructions. If it follows them consistently in a short prompt but inconsistently in a long one, the context window is a factor.

There is a constraint on every task you give an AI agent that most prompts ignore: the agent reads everything you give it, in order, and by the time it starts producing output, the beginning of your instructions may be far back in its context. The model does not forget, exactly, but it does not give the same weight to text at different positions in a long document. The structure of a task determines what the agent still knows when it matters.

This is not a hypothetical problem. Teams that write detailed system prompts with careful instructions find that the agent follows them reliably on short tasks and inconsistently on long ones. The instructions have not changed. The agent has not changed. The task length changed, and that changed what the agent was effectively working from when it generated the output.

How a context window works in practice

A context window is the total amount of text a model can read and work with at one time, measured in tokens, where a token is roughly four characters of English text. Everything the agent reads is in this window: the system prompt, the task description, any reference material you passed, the conversation history, and the work in progress. All of it competes for the same space.

When the window is full, the model does not give equal attention to all of it. Research from multiple sources, including work from Anthropic on how models use long contexts, shows that models attend more reliably to text near the beginning and end of a prompt than to text in the middle. For a task where the most important constraint is stated at the start of a long document, by the time the model reaches the generation step, that constraint may be in the part of the prompt it is least likely to follow precisely.

This matters most for specific, non-obvious requirements: a particular output format, a constraint on what the agent should not include, an instruction to stop at a certain step and ask for confirmation. These are the instructions that feel clear when you write them and get dropped when the agent runs.

What to put first and what to put last

The practical implication is straightforward: put the most important constraints and the exact output format close to the point where the agent begins producing output.

If you are asking the agent to generate a structured report, the section that describes the structure belongs just before the generation step, not in the system prompt at the top. If you are running a tool-using agent and you need it to stop before taking an irreversible action, that instruction should appear in the last message the agent reads before it reaches the step where the action is available.

The inverse is also useful: put background material and reference context early, before the instructions that will constrain the output. The agent reads the background, processes it, and by the time it reaches the specific instructions, those instructions are the most recent thing in its context. This is not how most prompts are written, which is why most prompts underperform on specific, constrained tasks.

When to split a task into separate calls

There is a point at which structuring the prompt is not enough and the task needs to be split into separate model calls. The clearest signal is when the total input, including everything the agent needs to read to produce the output, runs to more than about half the available context window.

At that point, the agent is spending a large share of its available attention on reading rather than reasoning. The working context, the part the agent actively draws on when generating output, is crowded by background material that is relevant to the task in general but not to the specific step the agent is running.

The fix is to split the task at natural seams. If the first step is research and the second is writing, run the research, check the output, and pass only the relevant findings into the writing step with a fresh context. The writing step now has the full context window available for the actual writing, rather than half of it occupied by research material it has already processed.

This costs one extra API call. It is almost always worth it. Two calls with focused context will produce more reliable output than one call that fills the window, for the same reason that two focused conversations produce more useful results than one conversation that covers too much at once.

Retrieval changes the equation

One of the most common reasons prompts run long is reference material: a spec document, a codebase excerpt, a policy document the agent needs to follow. The instinct is to pass the whole document so the agent has everything it might need.

The problem is that "everything it might need" and "everything that fits in the context" are different sizes. A retrieval step that pulls the two or three sections of the document relevant to the specific step the agent is running keeps the context small. The agent reads the relevant parts, not the whole document, and the space it saves is available for the work itself.

This requires deciding in advance what the agent needs for each step, which is itself a useful design exercise. If you cannot name what parts of the reference material the agent needs for a specific step, the step is probably underspecified, and adding more context will not fix that.

The practical test

There is a simple check for whether context structure is affecting your agent's reliability. Run the same task twice: once with the full prompt, and once with a shorter version that cuts material you believe is redundant. If the shorter version produces the same quality of output, the removed material was using context space without contributing to the result.

The more targeted test is to check whether the agent follows late instructions reliably. Write a specific constraint, place it at the end of a short prompt, and check whether the agent follows it. Then take the same constraint, place it at the top of a prompt long enough to push it far from the generation step, and check again. The difference in compliance rate between the two is the context structure tax you are paying on every run.

The goal is not to minimize prompt length as an end in itself. It is to make sure that at the moment the agent starts writing, the most important constraints are in the part of the context it is working from.

Sources

Anthropic. Claude's approach to long context. Anthropic research blog.
Liu, Nelson F., et al. "Lost in the Middle: How Language Models Use Long Contexts." arXiv, 2023. Research on how model attention varies with position in a long context.

Why context windows change how you structure a task for an agent