Summary of Evaluating and Enhancing Large Language Models For Novelty Assessment in Scholarly Publications, by Ethan Lin et al.
Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications
by Ethan Lin, Zhiyuan Peng, Yi Fang
First submitted to arxiv on: 25 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces a novel benchmark, SchNovel, to evaluate large language models’ (LLMs) ability to assess creativity and novelty in scholarly publications. The benchmark consists of 15,000 pairs of papers across six fields, sampled from the arXiv dataset with publication dates spanning two to ten years apart. Each pair assumes the more recently published paper is more novel. To simulate the review process taken by human reviewers, the authors propose RAG-Novelty, which leverages the retrieval of similar papers to assess novelty. Extensive experiments demonstrate that RAG-Novelty outperforms recent baseline models in assessing novelty. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us understand how well computers can tell if a new scientific idea is truly innovative or just slightly improved from what’s already known. To do this, they created a big test set of 15,000 pairs of scientific papers with varying ages. They also developed a way for the computer to “read” and compare these papers like a human would. The results show that some computers are better than others at identifying new and innovative ideas. |
Keywords
» Artificial intelligence » Rag
