Summary of Alignment Between the Decision-making Logic Of Llms and Human Cognition: a Case Study on Legal Llms, by Lu Chen et al.
Alignment Between the Decision-Making Logic of LLMs and Human Cognition: A Case Study on Legal LLMs
by Lu Chen, Yuxuan Huang, Yixing Li, Yaohui Jin, Shuai Zhao, Zilong Zheng, Quanshi Zhang
First submitted to arxiv on: 6 Oct 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach is proposed in this paper to assess the alignment between Large Language Models’ (LLMs’) decision-making processes and human cognition, focusing on legal LLMs. Unlike traditional evaluations, which concentrate on language generation results, this study evaluates the correctness of an LLM’s underlying decision-making logic, a crucial step towards earning human trust. By quantifying interactions encoded by the LLM as primitive decision-making logic, the authors design metrics to evaluate the detailed decision-making logic of LLMs. Experimental results indicate that even when language generation outputs appear correct, significant issues exist in the internal inference logic. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about a new way to check if Large Language Models (LLMs) are making good decisions like humans do. The authors want to make sure these models can be trusted, so they came up with a special method to look at what’s going on inside the model’s “brain”. They compare how the model makes decisions with how people think, and found that even when the model produces good answers, its internal thinking is often wrong. |
Keywords
* Artificial intelligence * Alignment * Inference
