Summary of Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models, by Matthew Dahl et al.
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
by Matthew Dahl, Varun Magesh, Mirac Suzgun, Daniel E. Ho
First submitted to arxiv on: 2 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models (LLMs) are being increasingly used to augment legal practice, education, and research. However, their potential is threatened by the presence of hallucinations – textual output that is not consistent with legal facts. Our paper presents systematic evidence of these hallucinations, documenting LLMs’ varying performance across jurisdictions, courts, time periods, and cases. We find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2 when asked specific questions about random federal court cases. We develop a typology of legal hallucinations, providing a conceptual framework for future research in this area. Our findings also illustrate that LLMs often fail to correct a user’s incorrect legal assumptions in a contra-factual question setup and cannot always predict or know when they are producing legal hallucinations. Overall, our paper cautions against the rapid and unsupervised integration of popular LLMs into legal tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) might seem like super smart computers that can answer any question about the law. But did you know that sometimes these models make mistakes? They might tell us something that isn’t true! This is called a “hallucination.” Our research shows that these hallucinations happen a lot when we ask LLMs questions about the law. We looked at how well different LLMs do at answering questions and found that some are really good at making mistakes! We also discovered that even if someone asks an LLM a question that is totally wrong, it might not correct them. This is important because people who don’t have access to lawyers or are trying to figure out the law on their own might rely too heavily on these models. |
Keywords
» Artificial intelligence » Hallucination » Llama » Unsupervised
