Summary of Minor Sft Loss For Llm Fine-tune to Increase Performance and Reduce Model Deviation, by Shiming Xie et al.
Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation
by Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, Xiuyu Wu
First submitted to arxiv on: 20 Aug 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paradigm in large-scale language models (LLMs) aims to align them with human preferences using supervised fine-tuning and reinforcement learning from human feedback. This approach is also applied in downstream scenarios to adapt LLMs to specific corpora and applications. Notably, the focus on RLHF has led to the development of various algorithms, such as PPO, DPO, IPO, KTO, MinorDPO, and others. In contrast, efforts for supervised fine-tuning (SFT) have primarily focused on collecting, filtering, and mixing high-quality data. Building upon insights from DPO and MinorDPO, this study introduces a training metric to measure the discrepancy between optimized and original models, as well as a loss function called MinorSFT that enhances training effectiveness and reduces discrepancies. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are trying to understand human preferences better. To do this, they use two techniques: fine-tuning with help from humans and learning from feedback. This approach works not only for the big model itself but also for smaller models in specific areas like text or images. There’s a lot of research on making language models more like humans, and some methods are better than others at collecting good data. A new idea is to measure how much a model changes when it’s trained with human help, which can make training more effective and the results more similar to what humans want. |
Keywords
» Artificial intelligence » Fine tuning » Loss function » Reinforcement learning from human feedback » Rlhf » Supervised
