Summary of A Data-centric Perspective on Evaluating Machine Learning Models For Tabular Data, by Andrej Tschalzev et al.
A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data
by Andrej Tschalzev, Sascha Marton, Stefan Lüdtke, Christian Bartelt, Heiner Stuckenschmidt
First submitted to arxiv on: 2 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a data-centric evaluation framework for supervised learning of tabular data, addressing the limitations of model-centric evaluations that rely on standardized preprocessing. The authors demonstrate that real-world modeling pipelines require dataset-specific preprocessing and feature engineering, which can significantly impact model performance rankings. They select 10 Kaggle datasets, implement expert-level preprocessing pipelines, and conduct experiments with different preprocessing pipelines and hyperparameter optimization (HPO) regimes to quantify the impact of various factors on model selection and performance. The findings suggest that recent models still benefit from manual feature engineering, and adapting to distribution shifts is important even in seemingly static data. The authors argue for a shift towards a data-centric perspective, acknowledging the need for feature engineering and temporal characteristics in tabular data. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how we evaluate machine learning models when they’re used on tables of data. Right now, most evaluations are model-focused, where people test different models with the same data preprocessing. But this doesn’t match what happens in real-world projects, where you need to tailor your approach to the specific dataset. The authors propose a new way to evaluate models that takes into account how they’re prepared for each dataset. They use 10 real-world datasets and show that even recent models can benefit from manual feature engineering. This matters because it means we should focus on understanding the data itself, rather than just trying out different models. |
Keywords
» Artificial intelligence » Feature engineering » Hyperparameter » Machine learning » Optimization » Supervised
