Summary of Token-level Direct Preference Optimization, by Yongcheng Zeng et al.
Token-level Direct Preference Optimization
by Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, Jun Wang
First submitted to arxiv on: 18 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel approach to fine-tuning pre-trained Large Language Models (LLMs) by optimizing policy at the token level. The method, called Token-level Direct Preference Optimization (TDPO), aims to align LLMs with human preferences and values. Unlike previous methods, TDPO incorporates forward KL divergence constraints for each token, improving alignment and diversity. The approach utilizes a token-based reward system based on the Bradley-Terry model and preserves simplicity without explicit reward modeling. Experimental results across various text tasks demonstrate TDPO’s superior performance in balancing alignment with generation diversity. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper introduces a new way to adjust pre-trained language models to match human ideas and intentions. The method, called Token-level Direct Preference Optimization (TDPO), is designed to make the models more aligned with what people think is right. Unlike other methods, TDPO uses special rules for each word in the text, which helps improve how well it matches human preferences. |
Keywords
» Artificial intelligence » Alignment » Fine tuning » Optimization » Token
