Summary of Measuring Model Variability Using Robust Non-parametric Testing, by Sinjini Banerjee et al.
Measuring model variability using robust non-parametric testing
by Sinjini Banerjee, Tim Marrinan, Reilly Cannon, Tony Chiang, Anand D. Sarwate
First submitted to arxiv on: 12 Jun 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel deep neural network training method is introduced, which sheds light on the relationship between model quality and random seed initialization. The paper shows that stochastic optimization can lead to varying model performances despite similar accuracy metrics. A new summary statistic called the -trimming level is proposed to quantify network similarity, enabling reliable ensemble model creation. The authors demonstrate that this statistic outperforms individual performance metrics like validation accuracy, churn, or expected calibration error. The findings have implications for hyper-parameter optimization and random seed selection in deep learning applications. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Deep neural networks can produce different results even when trained with the same data. This is because the initial settings of the training process, such as the random seed, can affect the outcome. Researchers are trying to understand how these initial settings impact the model’s performance. A new way to measure how similar different models are has been developed. This method can help create a collection of models that works well together. The proposed statistic is more useful than just looking at accuracy or other metrics alone. It can even help choose the best random seed for a specific task. |
Keywords
» Artificial intelligence » Deep learning » Ensemble model » Neural network » Optimization
