TabCUQ: Context Uncertainty Quantification for Tabular Foundation Models

Problem

Tabular foundation models such as TabPFN estimate conditional means and the conditional average treatment effect (CATE) at inference time using only a small labeled dataset (the context). Using these estimates for scientific decisions requires confidence intervals. However, the error has two sources: variation that arises when the context sample changes (context variation), and systematic bias that arises because the context is small (limited-context bias). The spread of the predictive distribution alone cannot capture both. The authors observed that the standard Bootstrap substantially overestimates the variation, while Subsampling underestimates it.

Method

TabCUQ keeps the feature values and the context size unchanged and injects Wild Bootstrap-style noise only into the observed outcomes, creating multiple perturbed contexts. The same foundation model is applied to each context to obtain perturbed estimates. The standard deviation of the perturbed estimates is used to estimate the size of the context variation. The maximum absolute deviation between the original estimate and the perturbed estimates serves as a 'bias cushion'. The final interval is a normal-approximation interval centered on the original point estimate, widened on both sides by this cushion. CATE is obtained as the difference from a T-learner, which applies a separate estimator to the treatment and control groups. The authors proved that pointwise coverage is asymptotically valid in a kernel-regression-like model where the error decomposes exactly into the two components.

Results

• In simulations (conditional means and CATE, three foundation models), intervals that ignored bias fell well short of the target coverage even when the standard errors were well calibrated. The authors reported that TabCUQ generally reached the nominal 95% level in the main settings.
• The average width of TabCUQ intervals was about twice that of oracle intervals that know the true bias and variance.
• On seven real-data tasks, including socioeconomic, environmental, behavioral, protein and single-cell data, average coverage relative to a reference function trained on a much larger sample rose from 0.614–0.842 for Wild Bootstrap intervals to 0.966–1.000 for TabCUQ.
• The standard Bootstrap tended to estimate standard errors larger than the actual standard deviation, while Subsampling tended to estimate them smaller.
• In a protein stability example (the 1G3P N139V mutation), applying TabPFN 2.6 to 20 contexts of 100 observations each produced 6 positive and 14 negative estimates. This shows that with point estimates alone, the conclusion can flip depending on how the context is composed.

Limitations

The theoretical coverage guarantee is proven only for a kernel-regression-like model; for actual neural-network-based foundation models it is supported only by experiments. Because of the bias cushion, the intervals are about twice as wide as the oracle and are therefore conservative. Since the true values are unknown for real data, a reference function trained on a large sample (e.g. LightGBM) was used as a proxy for the truth. CATE identification relies on the consistency, overlap and unconfoundedness assumptions.

Why it matters

Tabular foundation models are strong at prediction on small datasets, but using their estimates to guide experiments or judge treatment effects requires reliable confidence intervals. This paper shows that the commonly used Bootstrap may not work well here, and presents a practical method that builds intervals accounting for bias by repeating only inference, without retraining the model.

Key terms

In-context Learning (tabular)

An approach that keeps a pretrained model's parameters unchanged and predicts outcomes for new samples by feeding a small labeled dataset in as part of the input. TabPFN, TabICL and others use this approach.

Wild Bootstrap

A Bootstrap variant that keeps feature values fixed and creates resamples by multiplying or adding random signs or noise to the outcomes (residuals). It is robust to heteroscedastic errors.

CATE(Conditional Average Treatment Effect)

For a group with specific characteristics, the difference in average outcome between receiving a treatment and not receiving it.

Bias Cushion

The farthest distance by which a perturbed estimate deviates from the original estimate. It is used to widen the confidence interval to account for bias that arises from a small context.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Yuebei Shi, Molei Liu, Zijian Guo · Posted 2026-10-29 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.