Summary of Compressed Models Are Not Miniature Versions Of Large Models, by Rohit Raj Rai et al.
Compressed models are NOT miniature versions of large models
by Rohit Raj Rai, Rishant Pal, Amit Awekar
First submitted to arxiv on: 18 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Information Retrieval (cs.IR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract discusses the limitations of using compressed neural models, which are often assumed to be miniature versions of their larger counterparts. By comparing the BERT-large model with its five compressed versions on four key characteristics (prediction errors, data representation, data distribution, and vulnerability to adversarial attacks), the authors reveal significant differences between the compressed models themselves, as well as between these models and the original large model. These findings have major implications for the use of compressed models in practical applications. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large neural models are often compressed before deployment, but this paper questions whether they really are just smaller versions of their larger counterparts. The authors compare the BERT-large model with its five compressed versions on four important aspects: how well they predict, what data they represent, where that data comes from, and how vulnerable they are to fake attacks. What they found was surprising – not only did the compressed models differ from each other, but they also differed greatly from the original large model. |
Keywords
» Artificial intelligence » Bert
