Summary of Llm Stability: a Detailed Analysis with Some Surprises, by Berk Atil et al.
LLM Stability: A detailed analysis with some surprises
by Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, Breck Baldwin
First submitted to arxiv on: 6 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The study examines the stability of large language models (LLMs) when generating outputs for various tasks. Despite deterministic configurations, the researchers observe accuracy variations up to 10% across different runs on eight common tasks. Additionally, no single LLM consistently produces repeatable accuracy across all tasks. The study highlights the need for stability metrics in evaluating LLM performance and proposes two new metrics: TARr@N (total agreement rate at N runs over raw output) and TARa@N (total agreement over parsed-out answers). These metrics can be integrated into leader boards and research results to better quantify LLM stability. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are very smart computers that can do many tasks, like answering questions or translating languages. But did you know that these models don’t always give the same answer when asked the same question? This study looked at how consistent these models are across different tasks and found that they can vary a lot – sometimes up to 10%! The researchers also discovered that no single model is always right, even when doing the same task. They think this variation is important to consider when measuring how well language models do. |
