Summary of Pretraining Billion-scale Geospatial Foundational Models on Frontier, by Aristeidis Tsaris et al.
Pretraining Billion-scale Geospatial Foundational Models on Frontier
by Aristeidis Tsaris, Philipe Ambrozio Dias, Abhishek Potnis, Junqi Yin, Feiyi Wang, Dalton Lunga
First submitted to arxiv on: 17 Apr 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel study explores the potential of Foundation Models (FMs) in geospatial applications, particularly with regards to scaling up model size and training methods. The authors investigate the feasibility of billion-scale FMs pre-trained on publicly available data, leveraging state-of-the-art hardware accelerators and high-performance computing resources like the Frontier supercomputer. By analyzing performance experiments with Vision Transformer architecture variants up to 15B parameters, this study provides insights on how to optimize parallelism configurations for geospatial imagery applications. The results show a significant improvement in top1 scene classification accuracy (up to 30%) when scaling up from smaller models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A team of researchers is working on using special types of artificial intelligence called Foundation Models for geographic tasks like analyzing satellite images. These models are trained using huge amounts of data and can be adapted for different jobs. The challenge is that these large models require very powerful computers to train, which isn’t easy or affordable. The study looks at how well billion-scale Foundation Models work on publicly available data and how we can make the most of these powerful computers. |
Keywords
» Artificial intelligence » Classification » Vision transformer
