Summary of Profuser: Progressive Fusion Of Large Language Models, by Tianyuan Shi et al.
ProFuser: Progressive Fusion of Large Language Models
by Tianyuan Shi, Fanqi Wan, Canbin Huang, Xiaojun Quan, Chenliang Li, Ming Yan, Ji Zhang
First submitted to arxiv on: 9 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper introduces a novel approach to fusing the capacities of various large language models (LLMs) by incorporating both training and inference modes. The method, called ProFuser, evaluates model advantage not only through cross entropy during training but also by considering inference outputs. This comprehensive assessment is achieved by progressively transitioning from inference mode to training mode. To validate ProFuser’s effectiveness, the authors fuse three models (vicuna-7b-v1.5, Llama-2-7b-chat, and mpt-7b-8k-chat) and demonstrate improved performance in knowledge, reasoning, and safety compared to baseline methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper introduces a new way to combine different language models by using both training and testing modes. This helps to get a better understanding of which model is the most helpful. The authors test this method with three models and show that it works better than other approaches. |
Keywords
» Artificial intelligence » Cross entropy » Inference » Llama
