Summary of A Framework For Fine-tuning Llms Using Heterogeneous Feedback, by Ryan Aponte (1) et al.
A Framework for Fine-Tuning LLMs using Heterogeneous Feedback
by Ryan Aponte, Ryan A. Rossi, Shunan Guo, Franck Dernoncourt, Tong Yu, Xiang Chen, Subrata Mitra, Nedim Lipka
First submitted to arxiv on: 5 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel framework for fine-tuning large language models (LLMs) utilizing heterogeneous feedback. This framework combines various feedback formats into a unified supervision format compatible with methods like supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). The authors also introduce a method to extract a high-quality and diverse subset of the data, which can potentially lead to performance increases exceeding that of the full dataset. Experimental results demonstrate the effectiveness of this framework in improving LLMs’ performance in multiple areas simultaneously, such as instruction following and bias reduction. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) are very smart computer programs that can understand and generate human-like text. They’re used for tasks like summarizing articles or chatting with people. To make them better, researchers have been fine-tuning these models using different types of feedback from humans. However, collecting this feedback can be tricky because it’s hard to get high-quality data and the formats might vary greatly. The authors of this paper suggest a new way to fine-tune LLMs by combining all the different feedback into one format that’s easy to work with. They also show how to pick out the best parts of the data to improve performance even more. By using their framework, researchers can make LLMs better at following instructions and reducing bias in their responses. |
Keywords
» Artificial intelligence » Fine tuning » Reinforcement learning from human feedback » Rlhf » Supervised
