Summary of Swan: Sgd with Normalization and Whitening Enables Stateless Llm Training, by Chao Ma et al.
SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
by Chao Ma, Wenbo Gong, Meyer Scetbon, Edward Meeds
First submitted to arxiv on: 17 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a new stochastic optimizer called SWAN (SGD with Whitening And Normalization) that eliminates the need for maintaining optimizer states during training, achieving optimal memory efficiency. By pre-processing instantaneous stochastic gradients using normalization and whitening, SWAN stabilizes gradient distributions and counteracts local curvature of the loss landscape. Compared to Adam, SWAN reduces total end-to-end memory by approximately 50% while demonstrating comparable or better performance in language modeling tasks. Specifically, when pre-training the LLaMA model with 350M and 1.3B parameters, SWAN achieves a 2x speedup by reaching the same evaluation perplexity using half as many tokens. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a new way to make big language models work more efficiently. They introduce an optimizer called SWAN that doesn’t need to keep track of extra information during training, which makes it much faster and uses less memory. By doing some special calculations on the numbers used for training, SWAN makes sure the process goes smoothly and helps the model learn better. Compared to another popular optimizer called Adam, SWAN is faster and uses 50% less memory while giving similar or even better results. For example, when training a big language model with 350 million and 1.3 billion parameters, SWAN was able to finish in half the time it would have taken for Adam. |
Keywords
» Artificial intelligence » Language model » Llama » Perplexity
