Summary of Are Large Language Models Consistent Over Value-laden Questions?, by Jared Moore et al.
Are Large Language Models Consistent over Value-laden Questions?
by Jared Moore, Tanvi Deshpande, Diyi Yang
First submitted to arxiv on: 3 Jul 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models (LLMs) have been shown to exhibit biases in their survey answers. Researchers investigated the value consistency of LLMs, defined as the similarity of answers across paraphrases, related questions, multiple-choice and open-ended use-cases, and multilingual translations. The study applied this measure to small and large, open LLMs, including llama-3 and gpt-4o, using 8,000 questions spanning over 300 topics. The findings revealed that models are relatively consistent across paraphrases, use-cases, translations, and within a topic, but still exhibit some inconsistencies. Notably, consistency increased for uncontroversial topics like “Thanksgiving” compared to controversial ones like “euthanasia”. Interestingly, base models were more consistent than fine-tuned models, which showed greater inconsistency on certain topics like “euthanasia”. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you ask a large language model the same question in different ways. Do you get similar answers? Researchers tested this by asking thousands of questions and seeing how the answer changed when the question was asked differently. They found that the model’s answers were pretty consistent, but not perfect. The models did better on easy topics like Thanksgiving than harder ones like euthanasia. This study looked at whether these language models are biased and if they can be used to predict human opinions. |
Keywords
» Artificial intelligence » Gpt » Large language model » Llama
