Summary of Delay Embedding Theory Of Neural Sequence Models, by Mitchell Ostrow et al.
Delay Embedding Theory of Neural Sequence Models
by Mitchell Ostrow, Adam Eisen, Ila Fiete
First submitted to arxiv on: 17 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Neural and Evolutionary Computing (cs.NE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty summary: This paper investigates the capabilities of language models in reconstructing unobserved variables from observed data sequences. The authors draw inspiration from theories of delay embeddings in dynamical systems, which show that a few observations can suffice to infer unobserved states. They train one-layer transformer decoders and state-space sequence models on noisy time series data for next-step prediction tasks. Results demonstrate that each sequence layer learns a viable embedding of the underlying system. However, state-space models exhibit a stronger inductive bias than transformers, enabling more efficient parameterization and better performance on dynamics tasks. This work establishes a connection between dynamical systems and deep learning sequence models via delay embedding theory. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty summary: This research explores how language models can figure out missing information from the past. The authors think that this ability might be related to ideas about “delay embeddings” in math and science. They test different types of computer models on a task where they have to predict what will happen next based on some noisy data. The results show that each part of these models can learn to represent the underlying system. Interestingly, one type of model does this better than another, which means it needs less information and is more efficient. This study connects two different areas: math and computer science. |
Keywords
* Artificial intelligence * Deep learning * Embedding * Time series * Transformer
