Summary of Learning to Plan For Language Modeling From Unlabeled Data, by Nathan Cornille et al.
Learning to Plan for Language Modeling from Unlabeled Data
by Nathan Cornille, Marie-Francine Moens, Florian Mai
First submitted to arxiv on: 31 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel approach for planning and generating coherent writing by training a self-supervised learning objective. By predicting future abstract writing actions, which correspond to centroids in a clustered text embedding space, the model extends language modeling performance to more abstract planning. The method improves language modeling performance, particularly with respect to text structure. The framework uses an unsupervised planner module that can be trained at large scale and shared within the community. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper trains a machine to write better by predicting what it will say next. Instead of just copying what was written before, this method helps the machine generate coherent writing and plan its future actions. This is useful because many tasks require planning, like writing an article. The model learns to predict what it will do next based on the context, which makes it better at writing in general. |
Keywords
» Artificial intelligence » Embedding space » Self supervised » Unsupervised
