Summary of Improving Black-box Robustness with In-context Rewriting, by Kyle O’brien et al.
Improving Black-box Robustness with In-Context Rewriting
by Kyle O’Brien, Nathan Ng, Isha Puri, Jorge Mendez, Hamid Palangi, Yoon Kim, Marzyeh Ghassemi, Thomas Hartvigsen
First submitted to arxiv on: 13 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to improving the out-of-distribution (OOD) robustness of machine learning models for text classification. The existing techniques for OOD robustness are often not applicable when the model is effectively a black box, such as when retraining or modifying the weights is costly. In this work, the authors leverage test-time augmentation (TTA), which aggregates predictions across multiple augmentations of the test input. To generate effective natural language augmentations, they propose LLM-TTA, which uses large language model (LLM)-generated augmentations as TTA’s augmentation function. The results show that LLM-TTA outperforms conventional augmentation functions across sentiment, toxicity, and news classification tasks for BERT and T5 models. Additionally, the authors explore selectively augmenting inputs based on prediction entropy to reduce the rate of expensive LLM augmentations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps machines learn better from texts they haven’t seen before. When a machine is good at recognizing what text means, it often struggles when seeing new text that’s different from what it was trained on. To fix this, researchers used a technique called test-time augmentation, which looks at the same text in many different ways and makes a prediction based on those different views. They also proposed using a large language model to generate these different views, which worked even better! This new approach works with any machine learning model for text classification and doesn’t require special labels or lots of data. |
Keywords
* Artificial intelligence * Bert * Classification * Large language model * Machine learning * T5 * Text classification
