Summary of Cherry-picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine, by Luis Roque et al.
Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine
by Luis Roque, Carlos Soares, Vitor Cerqueira, Luis Torgo
First submitted to arxiv on: 19 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed approach tackles the issue of dataset selection bias in time series forecasting, which can lead to exaggerated performance claims. By analyzing a diverse set of benchmark datasets, researchers found that cherry-picking datasets can significantly distort the perceived performance of methods, often favoring certain approaches over others. Specifically, the study reveals that by selectively choosing just four datasets, 46% of methods could be deemed best in class, and 77% could rank within the top three. Furthermore, recent deep learning-based approaches show high sensitivity to dataset selection, whereas classical methods exhibit greater robustness. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making sure time series forecasting models are reliable and accurate. Right now, some researchers are picking and choosing which datasets they use to test their models, which can make it look like certain models are better than others when they’re not. The study shows that this “cherry-picking” can really distort the results, making some models seem way better than they actually are. It also finds that newer deep learning-based approaches are super sensitive to which datasets you use, while older methods are more consistent. |
Keywords
» Artificial intelligence » Deep learning » Time series
