Summary of Ndp: Next Distribution Prediction As a More Broad Target, by Junhao Ruan et al.
NDP: Next Distribution Prediction as a More Broad Target
by Junhao Ruan, Abudukeyumu Abudula, Xinyu Liu, Bei Li, Yinqiao Li, Chenglong Wang, Yuchun Fan, Yuan Ge, Tong Xiao, Jingbo Zhu
First submitted to arxiv on: 30 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models (LLMs) trained on next-token prediction (NTP) have shown impressive capabilities, but existing NTP limitations hinder performance. Our research highlights these shortcomings and proposes Next Distribution Prediction (NDP), which uses n-gram distributions to replace one-hot targets, enhancing learning without additional training time. We demonstrated NDP’s effectiveness across translation, general tasks, language transfer, and medical domain adaptation experiments, achieving significant improvements compared to NTP. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are very powerful tools that can understand and generate human-like text. However, the way they’re trained currently has some limitations. Our research looks at what these limits are and proposes a new way of training called Next Distribution Prediction (NDP). We tested NDP on several tasks and found it can do better than the current method in some cases. |
Keywords
» Artificial intelligence » Domain adaptation » N gram » One hot » Token » Translation
