Summary of Reasons: a Benchmark For Retrieval and Automated Citations Of Scientific Sentences Using Public and Proprietary Llms, by Deepa Tilwani et al.
REASONS: A benchmark for REtrieval and Automated citationS Of scieNtific Sentences using Public and Proprietary LLMs
by Deepa Tilwani, Yash Saxena, Ali Mohammadi, Edward Raff, Amit Sheth, Srinivasan Parthasarathy, Manas Gaur
First submitted to arxiv on: 3 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates whether large language models (LLMs) can generate references based on two types of sentence queries: direct and indirect. The authors introduce a dataset called REASONS, comprising abstracts from 12 popular domains of scientific research on arXiv. They test various LLMs, including state-of-the-art models GPT-4 and GPT-3.5, and find that augmenting relevant metadata improves performance and reduces hallucination rates. The authors also introduce an advance retrieval-augmented generation (RAG) model called Mistral, which demonstrates consistent citation support and matches the performance of GPT-3.5 and GPT-4. The study contributes valuable insights into the reliability of RAG for automated citation generation tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are being tested to see if they can automatically generate references for sentences in documents or reports. This is important for people who need to keep track of where ideas come from, like intelligence analysts and researchers. The authors created a big dataset with abstracts from scientific papers and tested different models to see how well they do. They found that some models are better than others at getting the right references, especially when they have more information to work with. They also developed a new model called Mistral that does a good job of generating references. |
Keywords
* Artificial intelligence * Gpt * Hallucination * Rag * Retrieval augmented generation
