An engineer argues LLMs should be treated as feature extractors for a logistic regression, beats the competition winner on an irony dataset with 0.747 F1

An anonymous data scientist writing at Minimally Sufficient argues that using an LLM directly as a classifier is the wrong pattern, and that treating the LLM as a feature extractor for a standard machine learning model is a better one. The post appeared on 17 September 2026 and reached 95 points on Hacker News. It walks through a worked benchmark on the SemEval-2018 irony detection dataset, and includes a coefficient plot of the trained model.

The argument

The author's problem with LLM-as-classifier is that it hides all the properties a team would want from a classifier: calibration, sensible confidence scores, a way to trade recall against precision, and the ability to reason about which population the model is trained on. Prompting to fix any of these is the author's phrase for "arcane" work: advice is plentiful but wisdom is scarce.

The proposed pattern wraps the LLM's verdict in a logistic regression:

p(y = 1 | x) = sigma(alpha + beta times LLM(x))

The two coefficients are then fitted from a small labelled training set. The author points out that in the special case where beta goes to infinity, the model collapses back to the LLM classifier. Fitted from data, it usually does not, because the fitted coefficients are the ones that best match observed behaviour on the training population.

The worked benchmark

The author runs the pattern on SemEval-2018 Task 3, a dataset of 4,618 tweets (3,834 train and 784 test) that expert annotators labelled for irony. A single-prompt LLM verdict reaches F1 0.747 on the test set, above the 2018 competition winner at 0.705. That is what the author calls the "one-shot" result: the model is asked once whether each tweet is ironic, and the answer is used directly.

Adding a logistic regression on top of the same verdict does not change the F1 score, because the ranking is the same, but it produces a calibrated probability instead of a bare yes-or-no. The author then adds extra LLM-extracted features, then rule-based deterministic features on top of those, and reports a further gain in F1 with each layer. The coefficient plot shows which features the fitted model actually leans on, with the size of each coefficient sorted by magnitude.

The caveats and the outlook

The author is careful to state the limits: this is one dataset, and beating a 2018 winner with a 2026 LLM is not by itself a strong claim. The overlapping-confidence-intervals comparison with the post-competition state of the art is what the author calls the interesting result, because it is achieved using only a logistic regression on top of LLM features.

Three recent papers are cited in the same direction: Han et al. (ICML 2024) on LLMs engineering features for few-shot tabular learning; Balek et al. (2024) on LLM-based feature generation for interpretable machine learning; and Malberg, Mosca and Groh (ECML PKDD 2024) on FELIX.

The reason a team building software would look at this is that it changes what to do when an LLM classifier stalls. Instead of prompt iteration, the team fits a small model on top of the LLM output and reads the coefficients, which is the same loop teams already run for any other statistical model.

Source

LLM Classification Is Feature Engineering at Minimally Sufficient. Dataset citation: Van Hee, Lefever and Hoste, "SemEval-2018 Task 3: Irony Detection in English Tweets" (S18-1005). Hacker News discussion at item 49742437.