Summary of Advancing Llm Reasoning Generalists with Preference Trees, by Lifan Yuan et al.
Advancing LLM Reasoning Generalists with Preference Trees
by Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, Maosong Sun
First submitted to arxiv on: 2 Apr 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary We introduce Eurus, a suite of large language models optimized for reasoning, fine-tuned from Mistral-7B and CodeLlama-70B. Eurus models achieve state-of-the-art results on various benchmarks in mathematics, code generation, and logical reasoning problems, outperforming existing open-source models by margins over 13.3%. Notably, Eurus-70B beats GPT-3.5 Turbo on a comprehensive benchmarking across 12 tests covering five tasks. The strong performance is primarily attributed to UltraInteract, a newly-curated large-scale alignment dataset designed for complex reasoning tasks. UltraInteract allows preference learning and includes pairwise data to facilitate this process. Our investigation reveals that some well-established preference learning algorithms may be less suitable for reasoning tasks. We derive a novel reward modeling objective using UltraInteract, leading to a strong reward model. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper introduces Eurus, a new kind of language model that can reason and solve problems. It’s like a super smart computer that can understand and respond to complex math problems, code, and logical puzzles. The researchers tested Eurus on many different types of tasks and found it performed better than other similar models. They also created a special dataset called UltraInteract that helps the model learn how to make good decisions. By using this dataset, they were able to figure out why some methods work well for general conversations but not for complex reasoning tasks. |
Keywords
» Artificial intelligence » Alignment » Gpt » Language model
