A new class of tabular foundation models, often termed "tabular LLMs," are emerging that can predict missing columns in spreadsheets zero-shot, much like language models complete text. On the Tab Arena benchmark, these models now outperform fully tuned gradient-boosted trees. This post introduces how these models work, verifies the strongest open-weight model, Tab ICLv2, through independent reproduction, and identifies areas where traditional tree-based methods still hold an advantage.

Tabular foundation models are pretrained transformers designed to predict on any table without requiring task-specific training. They operate via in-context learning, where provided labeled examples guide predictions for unlabeled data. Unlike text LLMs, tabular models process raw table cells and columns, pretraining on synthetic data to understand feature relationships. Models like Tab ICLv2 are significantly smaller than text LLMs, with tens of millions of parameters.

Independent verification of Tab ICLv2 on the official Tab Arena benchmark demonstrated performance closely matching the leaderboard scores, with minimal differences attributable to standard computational variance. The reproduction involved running the model on 51 datasets using a single AWS A10G instance, costing just over $2. While these tabular LLMs represent a significant advancement, the article notes that gradient-boosted trees still win in certain scenarios, and text LLMs prompted with serialized tables perform considerably worse than native tabular models.