Summary of Dr. Sow: Density Ratio Of Strong-over-weak Llms For Reducing the Cost Of Human Annotation in Preference Tuning, by Guangxuan Xu et al.
Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning
by Guangxuan Xu, Kai Xu, Shivchander Sudalairaj, Hao Wang, Akash Srivastava
First submitted to arxiv on: 4 Nov 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a cost-effective method to eliminate human annotation in preference tuning, called Density Ratio of Strong over Weak (Dr.SoW). Dr.SoW leverages off-the-shelf Large Language Models (LLMs) for preference data annotation. It uses the log-density ratio between better-aligned and less-aligned LLMs as a reward signal. The authors evaluate Dr.SoW across 221 different LLM pairs, finding a strong correlation between the performance gap of paired models and the quality of the reward signal. This insight provides a practical guideline for selecting LLMs for data annotation. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us find better ways to use computers to make choices. Right now, we need people to tell us which option is better, but that takes time and money. The authors created a new method called Dr.SoW that uses special computer models (LLMs) to help choose between options. They tested this method with many different computer models and found out what makes it work well. |
