Summary of Ted: Accelerate Model Training by Internal Generalization, By Jinying Xiao et al.
TED: Accelerate Model Training by Internal Generalization
by Jinying Xiao, Ping Li, Jie Nie
First submitted to arxiv on: 6 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract proposes TED pruning, a method to compress dataset sizes for large language models while achieving strong performance. The approach addresses overfitting by quantifying Internal Generalization (IG) and optimizing an objective based on Internal Generalization Distance (IGD). IGD measures changes in IG before and after pruning, allowing the model to achieve implicit regularization. The optimization objective is verified to result in the smallest upper bound on generalization error. The impact of mask fluctuations on IG is studied through masks and Taylor approximation, enabling fast estimation of IGD. Experiments on image classification, natural language understanding, and large language model fine-tuning show TED achieves lossless performance with 60-70% of the data. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are super smart computers that can understand human language, but they need a lot of data to learn. The problem is that collecting all that data is very expensive! To solve this issue, scientists came up with an idea called TED pruning. It helps reduce the amount of data needed without losing performance. They did it by creating a special way to measure how well the model works on less data and then adjusting its learning process accordingly. This new method is really good at making sure the model doesn’t overfit (get too stuck in one way of thinking) and can even learn from fewer examples than before! The results show that this technique works amazingly well, achieving excellent performance with only 60-70% of the original data. |
Keywords
» Artificial intelligence » Fine tuning » Generalization » Image classification » Language understanding » Large language model » Mask » Optimization » Overfitting » Pruning » Regularization
