Summary of Remodetect: Reward Models Recognize Aligned Llm’s Generations, by Hyunseok Lee et al.
ReMoDetect: Reward Models Recognize Aligned LLM’s Generations
by Hyunseok Lee, Jihoon Tack, Jinwoo Shin
First submitted to arxiv on: 27 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The remarkable capabilities of large language models (LLMs) have increased societal risks, necessitating the development of LLM-generated text detection methods. Our research identifies a common feature among recent powerful LLMs: alignment training, which maximizes human preferences. We find that aligned LLMs generate texts with higher estimated preferences than human-written texts, making them easily detectable using a reward model trained to model human preference distribution. To further improve detection ability, we propose two training schemes: continual preference fine-tuning and reward modeling of Human/LLM mixed texts. Our method demonstrates state-of-the-art results across six text domains and twelve aligned LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are very good at generating text, but they can also be used to create fake news. To stop this from happening, we need a way to detect when the model is being used to generate fake text. Our research shows that there’s something special about how these powerful models work – they’re trained to make text sound like it was written by humans. We find that texts generated by these models are actually preferred more than human-written texts! This means we can use a special kind of model, called the reward model, to detect when the language model is being used to generate fake text. We also come up with two new ways to make this detection even better. |
Keywords
» Artificial intelligence » Alignment » Fine tuning » Language model
