Using LLMs directly as classifiers presents significant challenges: they produce hard labels without reliable calibration, struggle to incorporate structured data alongside unstructured text, and lack interpretability regarding which prompt elements influence decisions. LLMs also carry baked-in priors that may not match the target population's class distribution.

The solution is to treat LLM outputs as features within a traditional machine learning framework. Wrapping an LLM verdict with logistic regression — p(y=1|x) = σ(α + β·LLM(x)) — recovers calibrated probabilities, enabling principled threshold control for precision-recall tradeoffs. Training on labeled data adapts the model to the specific population baseline and allows incorporation of additional structured covariates. Reweighting examples can further target different distributions of interest.

This approach preserves the LLM's power on unstructured data while restoring the desirable properties of standard ML algorithms: calibration, flexible feature integration, and clearer interpretability of how the LLM signal contributes to final predictions. The LLM becomes a powerful feature extractor rather than an unreliable end-to-end classifier.