Summary of Disentangle Sample Size and Initialization Effect on Perfect Generalization For Single-neuron Target, by Jiajie Zhao et al.
Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Target
by Jiajie Zhao, Zhiwei Bai, Yaoyu Zhang
First submitted to arxiv on: 22 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the intriguing phenomenon of overparameterized models like deep neural networks recovering target functions with fewer sampled data points than parameters. To gain insights into this phenomenon, the authors focus on a single-neuron target recovery scenario and examine how initialization and sample size influence the performance of two-layer neural networks. The experiments reveal that a smaller initialization scale is associated with improved generalization, and identify a critical quantity called the “initial imbalance ratio” that governs training dynamics and generalization under small initialization. Additionally, the authors empirically delineate two critical thresholds in sample size – the “optimistic sample size” and the “separation sample size” – which align with theoretical frameworks established by previous studies. The results indicate a transition in the model’s ability to recover the target function: below the optimistic sample size, recovery is unattainable; at the optimistic sample size, recovery becomes attainable albeit with a set of initialization of zero measure. Upon reaching the separation sample size, the set of initialization that can successfully recover the target function shifts from zero to positive measure. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper explores why overparameterized models like deep neural networks can sometimes learn target functions even when they have more parameters than training examples. The authors focus on a simple case where there is only one “neuron” or unit in the model, and examine how this affects what happens when the model is trained and tested. They find that making the model’s initial conditions smaller helps it generalize better, and identify a key ratio that determines whether the model can recover the target function. The authors also discover two important thresholds in the amount of training data: below one threshold, the model cannot learn the target function; at the other threshold, the model can learn it but only if its initial conditions are very specific. |
Keywords
» Artificial intelligence » Generalization
