Summary of Simplestrat: Diversifying Language Model Generation with Stratification, by Justin Wong et al.
SimpleStrat: Diversifying Language Model Generation with Stratification
by Justin Wong, Yury Orlovskiy, Michael Luo, Sanjit A. Seshia, Joseph E. Gonzalez
First submitted to arxiv on: 11 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed approach, SimpleStrat, generates diverse responses from large language models (LLMs) by partitioning the space into strata and selecting a random stratum at inference. This alternative method outperforms traditional temperature-based approaches in terms of quality and diversity. The authors introduce CoverageQA, a dataset of underspecified questions with multiple plausible answers, to measure diversity using KL Divergence between the output distribution and uniform distribution over valid ground truth answers. The evaluation shows that SimpleStrat achieves higher recall by 0.05 compared to GPT-4o and 0.36 average reduction in KL Divergence compared to Llama 3. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models can generate many different responses, which is important for tasks like planning and creating fake data. Most approaches try to increase diversity by making the model more uncertain. However, researchers found that this method doesn’t work well because it relies on the model’s predictions being similar to the true answers. The new approach, called SimpleStrat, uses the language model itself to divide the possible responses into groups or “strata”. Then, it picks a random stratum and generates multiple responses within that group. This method is better than traditional approaches at generating diverse responses. |
Keywords
» Artificial intelligence » Gpt » Inference » Language model » Llama » Recall » Temperature
