Summary of Monitoring Ai-modified Content at Scale: a Case Study on the Impact Of Chatgpt on Ai Conference Peer Reviews, by Weixin Liang et al.
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
by Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, James Y. Zou
First submitted to arxiv on: 11 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Social and Information Networks (cs.SI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach to estimate the proportion of texts in a large corpus that are likely to be substantially modified or generated by a large language model (LLM). The maximum likelihood model utilizes expert-written and AI-generated reference texts to accurately examine real-world LLM use at the corpus level. The authors apply this approach to a case study of scientific peer review in AI conferences, analyzing submissions from ICLR 2024, NeurIPS 2023, CoRL 2023, and EMNLP 2023. The results suggest that between 6.5% and 16.9% of text submitted as peer reviews could have been substantially modified by LLMs, going beyond minor writing updates or spell-checking. The study also reveals trends in generated text that may be too subtle to detect at the individual level, highlighting the need for future interdisciplinary work to examine how LLM use is changing information and knowledge practices. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how often people use large language models (LLMs) to write or modify texts. The researchers created a special model that uses both expert-written and AI-generated texts to understand how LLMs are used in real-life situations. They tested this approach on peer reviews from four important AI conferences. The results show that around 6-17% of these reviews might have been written or heavily edited by LLMs. This study also found that the way people use LLMs can depend on factors like how confident they are, how much time they have left to submit their review, and whether they usually respond to authors’ rebuttals. |
Keywords
* Artificial intelligence * Large language model * Likelihood
