Summary of Raft: Adapting Language Model to Domain Specific Rag, by Tianjun Zhang et al.
RAFT: Adapting Language Model to Domain Specific RAG
by Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, Joseph E. Gonzalez
First submitted to arxiv on: 15 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to pretraining large language models (LLMs) on textual data is presented in this paper. The authors propose Retrieval Augmented FineTuning (RAFT), a training recipe that enhances the model’s ability to answer questions in “open-book” in-domain settings. RAFT achieves this by ignoring distractor documents and citing verbatim relevant sequences from retrieved documents, which helps improve reasoning capabilities. The proposed method is tested on PubMed, HotpotQA, and Gorilla datasets, demonstrating consistent performance improvements for pre-trained LLMs. This post-training recipe can be used to fine-tune pre-trained models for in-domain applications. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper presents a new way to make large language models better at answering questions about specific topics. The researchers created a training method called Retrieval Augmented FineTuning, or RAFT. RAFT helps the model focus on the most important information and ignore things that aren’t relevant. This makes the model better at understanding and answering questions in its area of expertise. The team tested RAFT with three different datasets and found that it worked well for all of them. |
Keywords
» Artificial intelligence » Pretraining
