Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelsby Philipp Mondorf, Sondre Wold, Barbara PlankFirst…
TiVaT: A Transformer with a Single Unified Mechanism for Capturing Asynchronous Dependencies in Multivariate Time…
Attention layers provably solve single-location regressionby Pierre Marion, Raphaël Berthier, Gérard Biau, Claire BoyerFirst submitted…
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelsby Lucas Bandarkar, Benjamin Muller, Pritish…
FlashMask: Efficient and Rich Mask Extension of FlashAttentionby Guoxia Wang, Jinle Zeng, Xiyuan Xiao, Siming…
Towards Dynamic Graph Neural Networks with Provably High-Order Expressive Powerby Zhe Wang, Tianjian Zhao, Zhen…
Transformers Handle Endogeneity in In-Context Linear Regressionby Haodong Liang, Krishnakumar Balasubramanian, Lifeng LaiFirst submitted to…
nGPT: Normalized Transformer with Representation Learning on the Hypersphereby Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun,…
Replacing Paths with Connection-Biased Attention for Knowledge Graph Completionby Sharmishtha Dutta, Alex Gittens, Mohammed J.…
RisingBALLER: A player is a token, a match is a sentence, A path towards a…