The New Stack walks through a hybrid response cache for LLM apps, with exact-match and semantic tiers

The engineer Abhilash Rao Mesala published a walk-through on The New Stack this morning on how to add a response cache in front of an LLM call so a workload does not pay the model for repeat questions. The piece is bylined, dated 14 September 2026, and files under the site's AI Engineering, AI Infrastructure, and FinOps categories.

Two tiers, then both together

The exact-match tier normalises the model request body, hashes it with SHA-256, and looks the hash up in a store such as Redis. Mesala writes that this fits workloads where requests are bounded and predictable: most batch pipelines, CI runs, and boilerplate summarisation.

The semantic tier runs the query through an embedding model, stores the vector in a vector database, and looks for close matches by cosine similarity. He gives a starting threshold in the 0.90 to 0.95 range for the general case, tightens code-like queries to about 0.95 or higher, and loosens conversational ones to 0.85 to 0.90, and says every number is a starting point to test against real traffic. A looser threshold, he warns, risks answering the wrong question: "What is the weather in my town?" cannot be safely reused for a different town just because the vectors are close.

The hybrid tier runs both in sequence. On a semantic hit, the response is promoted back into the exact-match store under the hash of the new query, so the paraphrase becomes a first-tier hit next time.

The keys carry more than the query

Both tiers key on the query text, the context and documents in the prompt, the model and its settings, the version of any retrieved source, and the caller's access scope. Two identical questions asked against different documents, or by users with different permissions, must not share a cache entry. Skip caching entirely for personal or account-specific data, for creative tasks that should differ each run, and for real-time data such as stock prices or live inventory.

The worked example

Mesala's illustrative arithmetic: a workload of 1,000,000 LLM calls per month at $0.006 per call is $6,000 with no caching. A hybrid cache landing a 60 percent hit rate avoids 600,000 model calls; embedding and vector-store costs come to about $150. Monthly spend closes at about $2,550, a 57.5 percent reduction, plus the latency saving of returning a stored answer without waiting on the model. He labels these numbers as an illustration and adds one line in italics: measure the hit rate before projecting any savings.

Response caching does a different job from the native prompt caching that OpenAI, Anthropic, and Google now bill at reduced rates. Prompt caching reuses cached prompt computation and still charges for the output tokens. Response caching aims to skip the call entirely when a valid stored answer exists. For a team where a chunk of the token bill is the same question paraphrased twice, the exact-match tier alone catches the CI and batch traffic for the cost of a Redis lookup, and the semantic tier only runs on the misses. The trap Mesala keeps returning to is the freshness policy: a market answer stales in a minute, an HR policy answer in a month, and the wrong TTL for either shows up as a stale price or a wasted embedding call rather than a broken build.

Source

The New Stack: Why an old caching trick is your secret to lower LLM costs by Abhilash Rao Mesala, 14 September 2026.