Summary of Core Tokensets For Data-efficient Sequential Training Of Transformers, by Subarnaduti Paul et al.
Core Tokensets for Data-efficient Sequential Training of Transformers
by Subarnaduti Paul, Manuel Brack, Patrick Schramowski, Kristian Kersting, Martin Mundt
First submitted to arxiv on: 8 Oct 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper proposes a novel approach to deep learning, focusing on constructing a deeper-level data summary based on token-level information. The traditional method of retaining entire samples (coresets) is inadequate for recent transformer architectures that operate on tokens. Instead, the authors introduce core tokensets that select the most informative data points and leverage feature attribution to store only their most relevant features. This approach yields significant performance retention in incremental image classification, open-ended visual question answering, and continual image captioning with significantly reduced memory. In fact, a core tokenset of 1% of the data performs comparably to at least a twice as large and up to 10 times larger coreset. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making deep learning models work better in real-life situations where they need to learn from new data over time. Traditionally, these models retain old information by storing entire pieces of data, but this isn’t effective for newer architectures that process smaller chunks of data (tokens). The authors suggest a new way to summarize the data at the token level, which helps models remember important details while using less storage space. This approach works well in tasks like recognizing objects in images, answering questions about what’s in an image, and generating descriptions for images. In some cases, using just 1% of the data is as good as using much more. |
Keywords
» Artificial intelligence » Deep learning » Image captioning » Image classification » Question answering » Token » Transformer
