Summary of Bias in Motion: Theoretical Insights Into the Dynamics Of Bias in Sgd Training, by Anchit Jain et al.
Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
by Anchit Jain, Rozhin Nobahari, Aristide Baratin, Stefano Sarao Mannelli
First submitted to arxiv on: 28 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the evolution of bias in machine learning systems when trained on data with different sub-populations using a Gaussian-mixture model. It provides an analytical description of the stochastic gradient descent dynamics of a linear classifier in this setting, which is exact in high dimension. The analysis reveals how different properties of sub-populations influence bias at different timescales and shows that there is a shifting preference of the classifier during training. The findings are applied to fairness and robustness, demonstrating how and when heterogeneous data and spurious features can generate and amplify bias. Empirical validation is conducted using synthetic and real datasets, including CIFAR10, MNIST, and CelebA. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how machine learning systems get biased when they use certain parts of the data. It focuses on the middle part of the training process where biases are formed. The researchers find that different groups in the data affect the bias at different times during training. They show how these biases can be amplified by using certain features or datasets. This paper helps us understand how to make machine learning systems fairer and more robust. |
Keywords
* Artificial intelligence * Machine learning * Mixture model * Stochastic gradient descent
