Summary of Improving Generative Pre-training: An In-depth Study Of Masked Image Modeling and Denoising Models, by Hyesong Choi et al.
Improving Generative Pre-Training: An In-depth Study of Masked Image Modeling and Denoising Models
by Hyesong Choi, Daeun Kim, Sungmin Cha, Kwang Moo Yi, Dongbo Min
First submitted to arxiv on: 26 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper investigates the impact of additive noise on deep networks during pre-training. The authors explore why previous attempts at combining masked image modeling with latent denoising diffusion models have yielded only marginal improvements in recognition tasks. They identify three critical conditions for effective noise injection: corruption and restoration must occur within the encoder, noise should be introduced in the feature space, and explicit disentanglement between noised and masked tokens is necessary. By implementing these findings, the authors demonstrate improved pre-training performance across various recognition tasks, including those requiring fine-grained information. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how adding noise to deep networks during training can help them learn better. So far, people have tried combining this with another technique called masked image modeling, but it hasn’t made a big difference for recognizing things. The authors wanted to figure out why and found three important things: the noise has to be added inside the network’s first part, it should happen in the middle of the information being processed, and the noisy parts need to be separated from the masked parts. By following these rules, they showed that adding noise can really help deep networks learn better for a variety of tasks. |
Keywords
» Artificial intelligence » Diffusion » Encoder
