Summary of Data Contamination Can Cross Language Barriers, by Feng Yao et al.
Data Contamination Can Cross Language Barriers
by Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, Jingbo Shang
First submitted to arxiv on: 19 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper investigates the opacity in developing large language models (LLMs) and the potential contamination of public benchmarks in pre-training data. The authors highlight the limitations of existing contamination detection methods, which are typically based on text overlap between training and evaluation data. They present a cross-lingual form of contamination that inflates LLMs’ performance while evading current detection methods, deliberately injected by overfitting LLMs on translated versions of benchmark test sets. To address this issue, the authors propose generalization-based approaches to unmask deeply concealed contamination. Specifically, they examine the LLM’s performance change after modifying the original benchmark by replacing false answer choices with correct ones from other questions. Experimental results demonstrate that cross-lingual contamination can easily fool existing detection methods, but not the proposed approach. The paper also discusses the potential utilization of cross-lingual contamination in interpreting LLMs’ working mechanisms and in post-training LLMs for enhanced multilingual capabilities. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper is about finding ways to prevent large language models from being tricked or “contaminated” when they’re trained on certain data. This can happen when the same information is repeated in different languages, making it seem like the model is more skilled than it really is. The authors of this paper are trying to figure out how to detect and stop this kind of contamination from happening. |
Keywords
» Artificial intelligence » Generalization » Overfitting
