Machine learning on tabular data: the strengths of tree models and the challenge from TabPFN and LLMs

Why it was needed: tabular data is unfamiliar ground for deep learning

Most company data is in tables made of rows and columns. Each customer is one row, and values such as age, tenure, most recent payment amount, and region are attached as columns. This kind of data is called tabular data. Most business problems that use supervised learning (learning from examples with correct answers attached), such as churn prediction, fraud detection, and demand forecasting, start from tables like these.

Deep learning has been highly successful on images, audio, and text, and there have been many attempts to apply it directly to tables as well. A comparison study of deep learning for tabular data paper explains the motivation this way: there was the possibility of better performance, and it was also attractive to be able to train multimodal pipelines that handle tables together with images and text in one go. However, this field had no common benchmark like ImageNet for images. Each paper used different datasets and experimental setups, making it hard to judge which model was really better.

Meanwhile, in practice, gradient boosting (GBDT), which combines many decision trees, was the default choice. The same study paper noted that GBDT libraries such as XGBoost, LightGBM, and CatBoost are often the first choice in competitions. So why are trees strong, what do neural networks lack, and has a new approach emerged that could change this picture? These are the questions this article addresses.

Key idea: playing twenty questions many times over

A decision tree is like twenty questions. It asks a series of yes/no questions about one column at a time, such as "Was there a payment in the last 30 days?" or "Has the customer been subscribed for more than a year?", and when it reaches the last box (a leaf), it gives that box's value as the answer. Because each question looks at only one column, it does not matter if columns with different units are mixed together. Useless columns are rarely chosen for questions, so they are naturally ignored.

A single tree is crude. Gradient boosting adds trees one at a time: the second tree learns to correct what the first tree got wrong, that is, the remaining error, and the third tree corrects the error still left, and so on. It is like a collaboration where people take turns fixing only the previous person's mistakes. XGBoost paper defines the final prediction as the sum of the scores produced by each tree.

The thinking behind TabPFN and LLMs is completely different. They do not train a new model for each table. Instead, a large model that has already seen a huge number of table problems is told, "Look at the labeled rows in this table and predict the rest." It is like how someone who has solved many problems can get the hang of a new problem after seeing just a few examples. This is called in-context learning.

How it works: the two paths of boosting and TabPFN

Gradient boosting works like this. First, build an initial tree from the tabular data. For each row, compute the gap between the prediction and the correct answer (the remaining error), then build the next tree to reduce that error. Repeat this a set number of times, and at prediction time add up the scores of all the trees. XGBoost paper added a regularization term that penalizes the number of leaves in a tree and the size of the leaf scores, which keeps trees from becoming overly complex. This is meant to reduce overfitting, where a model fits only the training data and gets new data wrong.

TabPFN paper works in two stages. First, it is pretrained once, offline. This pretraining uses synthetic datasets drawn from a "prior" rather than real data. The prior broadly covers structural causal models, which represent causal structure, and is designed to sample simpler structures more often. Second, when a new table arrives, the training rows (with labels) and the rows to predict are fed into a transformer all at once. The model then produces answers for all the rows to predict in a single computation (forward pass). Because no parameters are updated, no hyperparameter tuning is needed either.

TabLLM paper turns each row of a table into a natural-language sentence (serialization). For example, it creates a sentence such as "Age is 45, occupation is teacher…" and feeds it to an LLM as a prompt together with a description of the classification task. Zero-shot use, asking directly without any labeled examples, is possible, and a few-shot setting is also used, where a small number of labeled examples are used to fine-tune the LLM. The paper compared several serialization methods, including templates, a table-to-text model and the LLM itself.

How the field developed: from the heyday of trees to new challengers

XGBoost 2016 paper turned tree boosting into a system that can run at scale. It combined a sparsity-aware algorithm for data with many missing values, a weighted quantile sketch for approximate splitting, a cache-aware block structure, and out-of-core computation for data that does not fit in memory. As a result, the authors reported that it runs more than 10 times faster than existing tools on a single machine and scales to billions of examples. According to the paper, 17 of the 29 winning solutions published on the Kaggle blog in 2015 used XGBoost, while 11 used deep neural networks. At KDDCup 2015, all of the top 10 teams used XGBoost.

The tabular deep learning comparison study 2021 paper re-compared a range of deep learning models under the same training and tuning rules. It showed that a simple ResNet-style model is a strong baseline that had been missing until then. It also reported that FT-Transformer, a transformer adapted for tables, performed best on most tasks. Even so, its conclusion was that there is still no "always-winning" side between GBDT and deep learning.

The tree-model advantage study 2022 paper set a benchmark using 45 datasets and a hyperparameter search of about 20,000 compute hours per learner. On medium-sized data of around 10,000 rows, tree models still performed best even when their speed advantage was set aside. The study attributed this to differences in the inductive bias of neural networks. It then set out three challenges that table-specific neural networks need to solve: not being thrown off by uninformative columns, preserving the orientation of the data (the distinct axis each column has), and easily learning irregular functions.

The same year brought challenges from other directions. TabPFN paper clearly outperformed boosted trees on 18 OpenML-CC18 datasets with up to 1,000 training rows, up to 100 numerical columns and up to 10 classes. It matched the performance of complex AutoML systems while being up to 230 times faster, and 5,700 times faster on a GPU. TabLLM paper, using only serialization and an LLM, outperformed existing tabular deep learning methods on several benchmarks. It was competitive even with gradient boosting, especially when there were very few labeled examples, and it achieved meaningful performance zero-shot as well. This is because the LLM can draw on knowledge it learned during pretraining.

Where things are heading: practical challenges for tabular foundation models

Models like TabPFN, which are pretrained on many table tasks and then predict on a new table through in-context learning, are now called tabular foundation models (TFMs). Research in 2026 has moved past the "accuracy is high" stage and addresses problems that come up in real use.

The first is cost. These models have to reread all the training rows as context for every prediction. The activation alignment study paper trained a lightweight linear transformation so that the intermediate activations of a "Student" that uses only part of the context resemble those of a "Teacher" that uses the full context. Both TabPFN-3 and TabFM improved on 38 classification datasets from TabArena. In the low-data regime, the method recovered nearly half of the performance gap with the Teacher. The transformation trains in a few seconds to a few minutes without a GPU.

The second is data that keeps arriving over time. According to the data stream study paper, TFMs adapt by changing their context rather than their parameters. A simple approach of keeping only recent examples was as effective as existing memory management techniques. After data drift (a shift in the data distribution), TFMs also recovered faster than existing streaming learners. The study also pointed out a limitation: serving costs are high because nearly the same context is re-encoded for every prediction.

The third is sensitive data. For tables containing personal information, such as in healthcare and finance, PrivTab paper built differential privacy (a mathematical guarantee that whether any one individual is included cannot be detected from the results) into the model architecture. It was pretrained on simulated data, and the authors reported that it cuts the time to fit a dataset by a factor of 10,000, finishing in a single computation.

Limitations and open questions: too early to declare a winner

First, look at the conditions under which the results hold. TabPFN's paper strong results came from small tables: up to 1,000 rows, up to 100 numerical columns, no missing values and up to 10 classes. The tree-model advantage study paper confirmed the strength of trees on medium-sized data of around 10,000 rows. The two results describe different regimes, so one result should not be used to judge the other regime.

The cost structures also differ. Once a boosted tree model is trained, prediction is cheap. TFMs, by contrast, skip the training step but process the entire context for every prediction. This is exactly what the data stream study paper means by high serving costs, and the context cost the activation alignment study paper tries to address. For a service with many rows and frequent prediction requests, this cost should be weighed alongside accuracy.

The LLM approach stands out when there are very few examples. TabLLM's paper claim of being competitive with gradient boosting also comes with the condition "especially when there are very few labeled examples." It should also be kept in mind that results vary depending on how rows are serialized into text. In short, the conclusion of the 2021 comparison study paper, that there is no single method that always wins, is still the starting point for practice.

Further reading

If you are new to the topic, start with the tree-model advantage study paper. It explains why trees are strong from three angles, uninformative columns, data orientation and irregular functions, and gives you a map of the topic. Then read Section 2 of XGBoost paper to understand how trees are added step by step and the regularized objective function.

For the deep learning side, the tabular deep learning comparison study paper is a good choice. It shows why fair comparison matters and what FT-Transformer is. For the new approaches, reading TabPFN paper and TabLLM paper together shows the difference between "a transformer trained on synthetic data" and "an LLM that uses language knowledge."

For the latest work, choose based on your practical concerns. If you are interested in operating costs, read the activation alignment study paper and the data stream study paper; if you work with data involving personal information, read PrivTab paper.