Summary of Map: Low-compute Model Merging with Amortized Pareto Fronts Via Quadratic Approximation, by Lu Li et al.
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation
by Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, Yoshua Bengio
First submitted to arxiv on: 11 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to model merging is introduced in this paper, which aims to combine multiple single-task models into a multitask model. The existing methods focus on enhancing average task accuracy, but neglect the trade-offs that can occur between different tasks’ objectives. This paper proposes an efficient algorithm called Model Merging with Amortized Pareto Front (MAP), which identifies a set of scaling coefficients for merging multiple models, reflecting these trade-offs. MAP uses quadratic approximation surrogate models to amortize the computational cost of evaluating the Pareto front. Experimental results on vision and natural language processing tasks demonstrate that MAP accurately identifies the Pareto front, providing practitioners with flexible solutions to balance competing task objectives. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper introduces a new way to combine multiple single-task models into one multitask model. Right now, when we merge models, we usually just take an average of their parameters without any extra training. But sometimes this can cause problems because different tasks might have different goals that conflict with each other. This can make it hard to choose the best solution. The authors propose a new algorithm called MAP (Model Merging with Amortized Pareto Front) that helps find the best trade-offs between these conflicting task objectives. It does this by using pre-computed models to quickly estimate the best solutions. The authors tested MAP on computer vision and language processing tasks and showed that it works well. |
Keywords
» Artificial intelligence » Natural language processing
