Summary of How to Train Long-context Language Models (effectively), by Tianyu Gao et al.
How to Train Long-Context Language Models (Effectively)by Tianyu Gao, Alexander Wettig, Howard Yen, Danqi ChenFirst…
Summary of Dynamic Gradient Alignment For Online Data Mixing, by Simin Fan et al.
Dynamic Gradient Alignment for Online Data Mixingby Simin Fan, David Grangier, Pierre AblinFirst submitted to…
Summary of Listening to the Wise Few: Select-and-copy Attention Heads For Multiple-choice Qa, by Eduard Tulchinskii et al.
Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QAby Eduard Tulchinskii, Laida Kushnareva,…
Summary of Menakbert — Hebrew Diacriticizer, by Ido Cohen et al.
MenakBERT – Hebrew Diacriticizerby Ido Cohen, Jacob Gidron, Idan PintoFirst submitted to arxiv on: 3…
Summary of Mitigating Memorization in Language Models, by Mansi Sakarvadia et al.
Mitigating Memorization In Language Modelsby Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Nathaniel Hudson, Caleb Geniesse,…
Summary of Llm-augmented Symbolic Reinforcement Learning with Landmark-based Task Decomposition, by Alireza Kheirandish et al.
LLM-Augmented Symbolic Reinforcement Learning with Landmark-Based Task Decompositionby Alireza Kheirandish, Duo Xu, Faramarz FekriFirst submitted…
Summary of Towards a Theoretical Understanding Of Synthetic Data in Llm Post-training: a Reverse-bottleneck Perspective, by Zeyu Gan et al.
Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspectiveby Zeyu Gan,…
Summary of Circuit Compositions: Exploring Modular Structures in Transformer-based Language Models, by Philipp Mondorf et al.
Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelsby Philipp Mondorf, Sondre Wold, Barbara PlankFirst…
Summary of In-context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks, By Dingzirui Wang et al.
In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasksby Dingzirui Wang, Xuanliang Zhang, Qiguang Chen,…
Summary of Investigating the Synergistic Effects Of Dropout and Residual Connections on Language Model Training, by Qingyang Li and Weimao Ke
Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Trainingby Qingyang Li,…