AI NewsProductivityReported

PostHog skips the LLM 98.7 percent of calls with a decision gate

PostHog's Jina Yoon reports a small yes/no model in front of its LLM call. The pipeline now skips the LLM 98.7 percent of the time.

AI News

Editorial2 min read

LinkedInX
PostHog skips the LLM 98.7 percent of calls with a decision gate

Why it mattersA decision-model gate is a cheap lever a team can pull this week. It turns most model calls into near-free classification. The slow LLM is kept for the hard cases.

Most LLM-backed features ask one model to do everything, and pay for everything. PostHog's Jina Yoon argues the real lever is deciding which calls need an LLM at all. On PostHog's build mode blog, she describes a product pipeline. A small decision model answers two yes/no questions before the big model ever sees the input. 98.7 percent of production traffic never reaches the LLM, Yoon writes.

The pipeline is Replay Vision. It is PostHog's feature for removing personal information out of session recordings. A session summary first passes a decision model. The model answers two questions. Does the text contain personal information, and does it need rewriting? Only the sessions that answer yes to both reach the LLM. Yoon states the gate runs on Jev, a decision model built by TypeSafe AI. It also runs on Jeeves, a hybrid model PostHog's in-house team trained. Jeeves is now the default for decision calls that do not pick a model.

What the numbers are and where they come from

The 98.7 percent skip rate is PostHog's own measurement in production, Yoon writes. For Jev, she quotes 444.6 times cheaper and 193.6 times faster against frontier models. Those come from TypeSafe AI's own blog, which Yoon links. The 25 percent search-quality lift came from a separate experiment. The team reranked search results in Replay Vision across 20 test queries, using a decision model instead of an LLM.

Where the technique stops working

Yoon is clear about the limit. Decision models judge what is in front of them in one pass. Anything that needs multi-step reasoning still belongs with an LLM. PostHog tried a decision model to rank their internal pull-request review agent's findings. They used five scoring questions. The result was an AUC of 0.52, which Yoon calls inconclusive. The lesson is plain. A decision model replaces classification steps, and it does not replace reasoning steps.

How to use the pattern

Yoon names four places a team can plug a decision model in today. Gate expensive LLM calls behind a probability threshold. Route tasks to the right model with a small set of choice questions. Rerank results by asking a face-value match question. Replace brittle if-else rules with a decision call that generalises to new cases.

A team watching its token bill has a straightforward move after reading this. List every call in a feature that spends on LLMs. Mark each one as classification or reasoning. Put a cheap classifier in front of the classification ones. PostHog's numbers show the saving can turn an expensive feature into a nearly free one. The honest part is that the gate only helps where the question is a single pass. A reasoning task still needs the slow model, and you only find out which is which by measuring.

Source

Not everything needs an LLM, by Jina Yoon on PostHog Build Mode.

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.

Share
LinkedInX
Start a project