Distillation of Tabular Foundation Models into Efficient Predictors

Excerpt

arXiv:2610.01435v1 Announce Type: new Abstract: Tabular foundation models (TFMs) achieve strong predictive performance through in-context learning, yet repeatedly conditioning on labeled data makes inference expensive. Knowledge distillation can reduce this cost by transferring their predictive ability to lightweight, dataset-specific students. However, the dependence of TFM predictions on both a labeled context and a query introduces two design questions: how to construct teacher supervision an