Summary of Time-moe: Billion-scale Time Series Foundation Models with Mixture Of Experts, by Xiaoming Shi et al.
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts
by Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, Ming Jin
First submitted to arxiv on: 24 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Time-MoE, a scalable and unified architecture designed to pre-train larger, more capable forecasting foundation models while reducing inference costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE enhances computational efficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows Time-MoE to scale effectively without a corresponding increase in inference costs. The model is composed of decoder-only transformer models that operate in an auto-regressive manner and support flexible forecasting horizons with varying input context lengths. The authors pre-trained these models on their newly introduced large-scale data Time-300B, which spans over 9 domains and encompasses over 300 billion time points. For the first time, they scaled a time series foundation model up to 2.4 billion parameters, achieving significantly improved forecasting precision. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper introduces a new architecture called Time-MoE that helps with time series forecasting. It’s like a super smart calculator that can predict what will happen in the future. The authors made it so it uses less energy and is faster than other models, which makes it really useful for real-world applications. They tested it on a huge dataset and showed that it works much better than other models. |
Keywords
* Artificial intelligence * Decoder * Inference * Mixture of experts * Precision * Time series * Transformer
