Summary of Scaling and Renormalization in High-dimensional Regression, by Alexander Atanasov et al.
Scaling and renormalization in high-dimensional regression
by Alexander Atanasov, Jacob A. Zavatone-Veth, Cengiz Pehlevan
First submitted to arxiv on: 1 May 2024
Categories
- Main: Machine Learning (stat.ML)
- Secondary: Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a novel approach to analyzing the training and generalization performance of high-dimensional ridge regression models using random matrix theory and free probability. By leveraging these fundamental tools, the authors derive analytic formulas for the training and generalization errors in just a few lines of algebra. This allows for a clear identification of the sources of power-law scaling in model performance. The study also explores the generalization error of a broad class of random feature models and finds that the S-transform corresponds to the train-test generalization gap, yielding an analogue of the generalized-cross-validation estimator. Additionally, the authors derive fine-grained bias-variance decompositions for a general class of random feature models with structured covariates, uncovering a scaling regime where variance due to features limits performance in the overparameterized setting. The results provide new insights into neural scaling laws and shed light on the limitations of anisotropic weight structure in random feature models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper shows how math can help us understand how machine learning models work. It uses special tools from physics and deep learning to analyze how well these models perform when they’re trained and tested. The authors find some surprising patterns, like how certain types of features can make the model worse, not better. They also figure out a way to measure how good or bad the model is at making predictions. This research helps us understand why machine learning models sometimes work really well and sometimes don’t, which is important for building better AI systems. |
Keywords
» Artificial intelligence » Deep learning » Generalization » Machine learning » Probability » Regression » Scaling laws
