Summary of Towards Unifying Evaluation Of Counterfactual Explanations: Leveraging Large Language Models For Human-centric Assessments, by Marharyta Domnich et al.
Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments
by Marharyta Domnich, Julius Valja, Rasmus Moorits Veski, Giacomo Magnifico, Kadi Tulver, Eduard Barbu, Raul Vicente
First submitted to arxiv on: 28 Oct 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach is proposed to evaluate the transparency of machine learning models by developing a diverse set of counterfactual scenarios and collecting ratings from 206 respondents across 8 evaluation metrics. This method allows Large Language Models (LLMs) to predict average or individual human judgment, achieving an accuracy of up to 63% in zero-shot evaluations and 85% with fine-tuning. The study’s findings demonstrate the potential for LLMs to improve comparability and scalability in evaluating different counterfactual explanation frameworks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Machine learning models are getting smarter, but it’s hard to understand how they make decisions. One way to fix this is by using “what if” scenarios that show what would have happened if something had been done differently. This helps people make better choices. Right now, there aren’t good ways to test these scenarios, so the researchers created 30 different ones and asked a group of people how they felt about each one. They then used special computers (LLMs) to predict what most people would think about each scenario. The results show that these LLMs can be very accurate in predicting human judgment. |
Keywords
» Artificial intelligence » Fine tuning » Machine learning » Zero shot
