Summary of Pdf-wukong: a Large Multimodal Model For Efficient Long Pdf Reading with End-to-end Sparse Sampling, by Xudong Xie et al.
PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling
by Xudong Xie, Hao Yan, Liang Yin, Yang Liu, Jing Ding, Minghui Liao, Yuliang Liu, Wei Chen, Xiang Bai
First submitted to arxiv on: 8 Oct 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes PDF-WuKong, a multimodal large language model (MLLM) designed to enhance multimodal question-answering (QA) for long PDF documents. The MLLM incorporates a sparse sampler that operates on both text and image representations, improving efficiency and capability. The sparse sampler selects paragraphs or diagrams most pertinent to user queries, processed by the language model. To train and evaluate the model, the authors construct PaperPDF, a dataset of English and Chinese academic papers, generating 1.1 million QA pairs with evidence sources. Experimental results demonstrate the superiority and high efficiency of PDF-WuKong over other models on long multimodal document understanding, surpassing proprietary products by an average of 8.6% on F1. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary PDF-WuKong is a new way to understand documents that have text and images mixed together. The model uses a special technique called a sparse sampler that helps it find the most important parts of the document to answer questions. This makes it better at understanding long PDF documents, like academic papers. To test this model, the authors created a big dataset of these types of papers and generated many questions with answers. The results show that PDF-WuKong is really good at understanding documents and can even do better than some other models. |
Keywords
» Artificial intelligence » Language model » Large language model » Question answering
