Before reaching for an LLM API on every classification problem, it is worth knowing what a decades-old baseline can already do with the labeled data you have and exactly how much more data buys you. A classical baseline like TF-IDF plus a linear classifier costs nothing to run once trained, requiring no API calls or network round-trips, with sub-millisecond inference. The question worth answering before defaulting to an LLM is not which method is better in the abstract, but how much labeled data is actually available and how good the free, instant option gets with it.

An experiment measured accuracy scaling as labeled examples increase from barely any to modest amounts for routing support tickets into five categories. Results show accuracy climbs steeply at first, then flattens. Going from 2 to 5 examples per category bought +15 points of accuracy, the single highest-leverage move in the experiment.

Going from 5 to 10 bought only +5 more, following the classic diminishing-returns curve of a linear classifier on a small vocabulary. Even at 60% accuracy, deploying the model is not "walk away" territory, as a router misclassifying 2 in 5 tickets is a real product problem. The interesting part is where the curve sits relative to what your product can tolerate.

A full runnable script, including dataset, train/test split, and evaluation loop, is linked at the end of the original post.