Summary of Feature Alignment: Rethinking Efficient Active Learning Via Proxy in the Context Of Pre-trained Models, by Ziting Wen et al.
Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Models
by Ziting Wen, Oscar Pizarro, Stefan Williams
First submitted to arxiv on: 2 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores the intersection of fine-tuning pre-trained models with active learning, a promising approach for reducing annotation costs. However, this combination can introduce significant computational costs due to the growing scale of pre-trained models. To address this issue, researchers have proposed proxy-based active learning, which pre-computes features to reduce computational costs. Yet, this approach often incurs a significant loss in active learning performance, sometimes outweighing the computational cost savings. The authors demonstrate that not all sample selection differences result in performance degradation and show that suitable training methods can mitigate the decline of active learning performance caused by certain selection discrepancies. Building upon this analysis, they propose a novel method, aligned selection via proxy (ASVP), which improves proxy-based active learning performance by updating pre-computed features and selecting a proper training method. Extensive experiments validate that ASVP improves the total cost of efficient active learning while maintaining computational efficiency. The code for this research is available on GitHub at https://github.com/ZiTingW/asvp. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Fine-tuning pre-trained models with active learning could reduce annotation costs, but it’s a complex process. This paper shows that some methods can make things worse, but they also propose a new method to improve the situation. It uses “proxy” features to reduce computational costs and select training data wisely. The results show that this approach is more efficient while still maintaining good performance. |
Keywords
* Artificial intelligence * Active learning * Fine tuning
