Summary of Long-context Llms Meet Rag: Overcoming Challenges For Long Inputs in Rag, by Bowen Jin et al.
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
by Bowen Jin, Jinsung Yoon, Jiawei Han, Sercan O. Arik
First submitted to arxiv on: 8 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the phenomenon where retrieval-augmented generation (RAG) of large language models (LLMs) initially improves output quality but then declines as the number of retrieved passages increases. This is attributed to the detrimental impact of “hard negatives” in the retrieval set. To mitigate this and enhance robustness, the authors propose training-free and training-based approaches. They demonstrate the effectiveness of retrieval reordering and explore training-based methods such as LLM fine-tuning and intermediate reasoning for RAG-specific fine-tuning. The paper also analyzes design choices for these methods, including data distribution, retriever selection, and training context length. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how big language models can get better by using more information from the internet. It seems that when they use too much information, it actually makes their answers worse. The authors found out what’s going wrong and came up with ways to fix it. They showed that rearranging the information they find helps make them better, and they also tested some new training methods. They even looked at different ways of choosing what information to use and how long to train. |
Keywords
» Artificial intelligence » Context length » Fine tuning » Rag » Retrieval augmented generation
