Summary of Rlvf: Learning From Verbal Feedback Without Overgeneralization, by Moritz Stephan et al.
RLVF: Learning from Verbal Feedback without Overgeneralization
by Moritz Stephan, Alexander Khazatsky, Eric Mitchell, Annie S Chen, Sheryl Hsu, Archit Sharma, Chelsea Finn
First submitted to arxiv on: 16 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models (LLMs) are used in various settings, requiring flexibility in their default behaviors. Researchers propose a method called Contextualized Critiques with Constrained Preference Optimization (C3PO), which incorporates verbal feedback without overgeneralization. The approach generates synthetic preference data based on high-level feedback and fine-tunes the model to adhere to the given feedback while minimizing divergence from the original model for irrelevant scenarios. Experimental results show that C3PO effectively applies verbal feedback in relevant contexts, reducing overgeneralization by 30%. This method can be used in various applications where nuanced requirements need to be incorporated. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are really good at understanding human language, but sometimes they don’t know what we mean. For example, if you tell them not to use emojis when writing an email to your boss, they might still use emojis in other emails. The new method called C3PO helps solve this problem by using a small piece of feedback to create a set of rules for how the feedback should be used. This way, the model only uses the feedback where it’s needed and doesn’t get too carried away. |
Keywords
* Artificial intelligence * Optimization
