Summary of Single Parent Family: a Spectrum Of Family Members From a Single Pre-trained Foundation Model, by Habib Hajimolahoseini et al.
Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model
by Habib Hajimolahoseini, Mohammad Hassanpour, Foozhan Ataiefard, Boxing Chen, Yang Liu
First submitted to arxiv on: 28 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty summary: This paper presents Progressive Low Rank Decomposition (PLRD), a novel method for compressing large language models (LLMs) while preserving their performance. PLRD leverages a pre-trained model and incrementally decompresses it to smaller sizes, reducing computational overhead and energy consumption. The approach optimizes the trade-off between model performance and resource usage by strategically decreasing tensor ranks. Experimental results show that models trained with PLRD on 1 billion tokens maintain comparable performance to traditionally trained models while using only 0.1% of the tokens. PLRD’s versatility is demonstrated by its ability to generate multiple model sizes from a single foundational model, adapting to varying computational and memory budgets. The findings suggest that PLRD could set a new standard for efficient LLM scaling, making advanced AI more feasible on diverse platforms. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty summary: This paper develops a way to make big language models smaller without losing their ability to understand text. The method, called Progressive Low Rank Decomposition (PLRD), starts with a well-trained model and breaks it down into smaller pieces that use less computer power and energy. PLRD is good at balancing the need for powerful models with the need to conserve resources. In tests, models made using PLRD performed just as well as bigger models while using much less data. This could help make advanced AI technology available on a wider range of devices. |
