Summary of Zero-space Cost Fault Tolerance For Transformer-based Language Models on Reram, by Bingbing Li et al.
Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM
by Bingbing Li, Geng Yuan, Zigeng Wang, Shaoyi Huang, Hongwu Peng, Payman Behnam, Wujie Wen, Hang Liu, Caiwen Ding
First submitted to arxiv on: 22 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to mitigate hardware failures in Resistive Random Access Memory (ReRAM) deep neural networks (DNNs). ReRAM’s parallel matrix-vector multiplication capability makes it an attractive platform for DNNs. However, stuck-at-fault defects can lead to significant prediction errors during model inference. The proposed method addresses this issue by pruning redundant structures, duplicating weights and voting for robust outputs, and embedding duplicated most significant bits (MSBs) into the model weight. Experimental results on nine tasks of the GLUE benchmark with the BERT model demonstrate its effectiveness. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary ReRAM is a new way to make computers work faster using special memory chips. But sometimes these chips can get stuck, which makes it hard for them to do certain calculations. To fix this problem, the researchers came up with a plan that uses three strategies: removing extra parts of the model, duplicating important weights, and adding extra information to the model. They tested their idea on nine different tasks using a popular AI model called BERT, and it worked really well. |
Keywords
* Artificial intelligence * Bert * Embedding * Inference * Pruning
