Summary of Adaptwin: Low-cost Adaptive Compression Of Product Twins in Transformers, by Emil Biju et al.
AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers
by Emil Biju, Anirudh Sriram, Mert Pilanci
First submitted to arxiv on: 13 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Sound (cs.SD); Audio and Speech Processing (eess.AS)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a low-rank adaptive compression technique called AdaPTwin that reduces the size of transformer-based speech recognition models, making them more suitable for resource-constrained settings. The approach prioritizes performance on specific speakers while maintaining generalizability to new speakers and acoustic conditions. It requires only 8 hours of speech data for fine-tuning, which can be accomplished in under 20 minutes. The technique is demonstrated by compressing the Whisper and Distil-Whisper models, achieving up to 45% size reduction with less than a 2% increase in word error rate. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models have excelled at speaker-independent speech recognition, but their size and computational requirements make them difficult to use in resource-limited settings. This paper develops a way to shrink these models while keeping them effective. The new method, called AdaPTwin, works by compressing certain parts of the model that aren’t as important. This lets it focus on specific speakers while still understanding others. It only needs 8 hours of speech data to fine-tune itself, which is quick and cost-effective. |
Keywords
» Artificial intelligence » Fine tuning » Transformer
