Summary of Discrete Diffusion Language Model For Efficient Text Summarization, by Do Huu Dat et al.
Discrete Diffusion Language Model for Efficient Text Summarization
by Do Huu Dat, Do Duc Anh, Anh Tuan Luu, Wray Buntine
First submitted to arxiv on: 25 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper addresses the limitations of previous discrete diffusion models for conditional long-text generation, specifically in tasks like abstractive summarization. The authors propose a novel semantic-aware noising process that enables Transformer backbones to handle long sequences effectively. They also introduce CrossMamba, an adaptation of the Mamba model to the encoder-decoder paradigm, which integrates seamlessly with the random absorbing noising process. The proposed approaches achieve state-of-the-art performance on three benchmark summarization datasets: Gigaword, CNN/DailyMail, and Arxiv, outperforming existing discrete diffusion models on ROUGE metrics while possessing much faster inference speed compared to autoregressive models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research paper improves a type of computer model that can generate text. The current models are good at creating short texts, but they struggle when asked to write longer texts like summaries. The authors invent a new way to add noise to the model’s calculations, which helps it handle long sequences better. They also create a new version of an existing model called CrossMamba, which works well with their new noise addition method. The results show that their approach is better than previous models at generating good summaries and can do it much faster. |
Keywords
» Artificial intelligence » Autoregressive » Cnn » Diffusion » Encoder decoder » Inference » Rouge » Summarization » Text generation » Transformer
