Summary of Purifying Large Language Models by Ensembling a Small Language Model, By Tianlin Li et al.
Purifying Large Language Models by Ensembling a Small Language Model
by Tianlin Li, Qian Liu, Tianyu Pang, Chao Du, Qing Guo, Yang Liu, Min Lin
First submitted to arxiv on: 19 Feb 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research proposes a method for purifying large language models (LLMs) contaminated by uncurated training data. The issue is that current LLMs heavily rely on external sources, which can lead to copyright infringement, data poisoning, and privacy violations. The proposed solution involves ensembling LLMs with small language models (SLMs), which provides both theoretical guarantees and empirical evidence of its effectiveness in preserving the performance of LLMs while mitigating these issues. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you have a super smart computer program that can understand and generate human-like text. But, to make it work well, this program needs lots of data to learn from. The problem is that some of this data might be copied without permission, or even contain bad information. To solve this issue, the researchers suggest combining this large language model with a smaller one that’s “clean”. This way, the large model still learns and improves, but it’s less likely to cause problems. |
Keywords
* Artificial intelligence * Large language model
