Summary of Towards Incremental Learning in Large Language Models: a Critical Review, by Mladjan Jovanovic and Peter Voss
Towards Incremental Learning in Large Language Models: A Critical Reviewby Mladjan Jovanovic, Peter VossFirst submitted…
Summary of Xft: Unlocking the Power Of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-experts, By Yifeng Ding et al.
XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Expertsby Yifeng Ding,…
Summary of Multi-head Mixture-of-experts, by Xun Wu et al.
Multi-Head Mixture-of-Expertsby Xun Wu, Shaohan Huang, Wenhui Wang, Furu WeiFirst submitted to arxiv on: 23…
Summary of Intuition-aware Mixture-of-rank-1-experts For Parameter Efficient Finetuning, by Yijiang Liu et al.
Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuningby Yijiang Liu, Rongyu Zhang, Huanrui Yang, Kurt Keutzer, Yuan…
Summary of Dense Training, Sparse Inference: Rethinking Training Of Mixture-of-experts Language Models, by Bowen Pan et al.
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelsby Bowen Pan, Yikang Shen, Haokun…
Summary of Seer-moe: Sparse Expert Efficiency Through Regularization For Mixture-of-experts, by Alexandre Muzio et al.
SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertsby Alexandre Muzio, Alex Sun, Churan HeFirst submitted…
Summary of Half-space Feature Learning in Neural Networks, by Mahesh Lorik Yadav et al.
Half-Space Feature Learning in Neural Networksby Mahesh Lorik Yadav, Harish Guruprasad Ramaswamy, Chandrashekar LakshminarayananFirst submitted…
Summary of Jamba: a Hybrid Transformer-mamba Language Model, by Opher Lieber et al.
Jamba: A Hybrid Transformer-Mamba Language Modelby Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan…
Summary of Xmoe: Sparse Models with Fine-grained and Adaptive Expert Selection, by Yuanhang Yang et al.
XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectionby Yuanhang Yang, Shiyi Qi, Wenchao Gu,…
Summary of Generalization Error Analysis For Sparse Mixture-of-experts: a Preliminary Study, by Jinze Zhao et al.
Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Studyby Jinze Zhao, Peihao Wang, Zhangyang WangFirst…