Summary of On Expressive Power Of Quantized Neural Networks Under Fixed-point Arithmetic, by Geonho Hwang et al.
On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic
by Geonho Hwang, Yeachan Park, Sejun Park
First submitted to arxiv on: 30 Aug 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the expressive power of neural networks under discrete fixed-point parameters and operations, considering the effects of rounding errors. It provides necessary and sufficient conditions on fixed-point arithmetic and popular activation functions like Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU for universal approximation by quantized networks. The study reveals that these activation functions satisfy the sufficient condition, making them capable of universal approximation. Additionally, it demonstrates that even quantized networks with binary weights can universally approximate using practical activation functions. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how well neural networks work when their parameters and operations are limited to a specific range of values. It finds some rules that help decide if these networks can learn any function they want (universal approximation). The researchers show that many popular ways of changing the input to the network (activation functions) follow these rules, making them good for learning anything. They also prove that even networks with very limited weights (just -1 and 1) can do this if the activation functions are right. |
Keywords
» Artificial intelligence » Relu » Sigmoid
