Summary of Relevai-reviewer: a Benchmark on Ai Reviewers For Survey Paper Relevance, by Paulo Henrique Couto et al.
RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance
by Paulo Henrique Couto, Quang Phuoc Ho, Nageeta Kumari, Benedictus Kent Rachmat, Thanh Gia Hieu Khuong, Ihsan Ullah, Lisheng Sun-Hosoya
First submitted to arxiv on: 13 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Recent advancements in Artificial Intelligence (AI), particularly the widespread adoption of Large Language Models (LLMs), have significantly enhanced text analysis capabilities, promising considerable potential for automating scientific paper review. This technological evolution offers significant promise for maintaining research quality by accelerating the propagation of scientific knowledge. We propose RelevAI-Reviewer, an automatic system that conceptualizes survey paper review as a classification problem, assessing relevance in relation to a specified prompt. We introduce a novel dataset comprising 25,164 instances, each containing one prompt and four candidate papers varying in relevance. The objective is to develop a machine learning (ML) model capable of determining the relevance of each paper and identifying the most pertinent one. We explore various baseline approaches, including traditional ML classifiers like Support Vector Machine (SVM) and advanced language models such as BERT. Preliminary findings indicate that the BERT-based end-to-end classifier surpasses other conventional ML methods in performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Automating scientific paper review could revolutionize how we share knowledge. Researchers usually read and evaluate papers to ensure they meet quality standards, but this process is slow and can be biased. A new system called RelevAI-Reviewer aims to speed up the process by classifying papers based on their relevance to a specific topic. The system uses machine learning algorithms to analyze large amounts of text data and identify the most important papers. The goal is to develop an AI model that can quickly and accurately assess the relevance of each paper. |
Keywords
» Artificial intelligence » Bert » Classification » Machine learning » Prompt » Support vector machine
