Summary of Retrieval & Fine-tuning For In-context Tabular Models, by Valentin Thomas et al.
Retrieval & Fine-Tuning for In-Context Tabular Models
by Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, Anthony Caterini
First submitted to arxiv on: 7 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper addresses the challenge of applying transformer-based deep learning to large and complex tabular datasets. Recent advancements in in-context learning have shown promise on smaller datasets, but struggle to scale to larger ones. The proposed approach combines retrieval and fine-tuning: nearest neighbours are collected from a local subset of the data, and then task-specific fine-tuning is performed using this retrieved set. This locally-calibrated PFN (LoCalPFN) model outperforms existing in-context models on 95 datasets curated by TabZilla from OpenML, achieving state-of-the-art results even with respect to tuned tree-based models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper tries to make deep learning work better for really big and complicated tables of data. Right now, it’s hard to use this kind of AI on very large datasets because they’re too complex. The solution is to take a smaller part of the dataset that looks similar to what you’re trying to predict, and then train the model using those examples. This new approach, called LoCalPFN, works really well even compared to other methods that are specifically designed for tables. |
Keywords
* Artificial intelligence * Deep learning * Fine tuning * Transformer
