Summary of White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in Llms, by Yixin Wan et al.
White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in LLMs
by Yixin Wan, Kai-Wei Chang
First submitted to arxiv on: 16 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This study investigates social biases in Large Language Model (LLM)-generated content, specifically focusing on language agency. While previous research has examined agency-related bias in human-written language, this area remains understudied in LLMs. The authors introduce the novel Language Agency Bias Evaluation (LABE) benchmark to comprehensively evaluate biases in LLMs by analyzing agency levels attributed to different demographic groups. LABE leverages 5,400 template-based prompts, an accurate agency classifier, and corresponding bias metrics to test for gender, racial, and intersectional language agency biases on three text generation tasks. The authors also contribute the Language Agency Classification (LAC) dataset, consisting of 3,724 agentic and communal sentences. Using LABE, they unveil language agency social biases in three recent LLMs: ChatGPT, Llama3, and Mistral. The study finds that LLM generations tend to demonstrate greater gender bias than human-written texts, with intersectional bias being the most prominent aspect. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how Large Language Models (LLMs) create language and if they have biases towards certain groups of people. Most research on this topic has been done for text written by humans, not computers. The authors created a special tool called LABE to measure these biases in LLMs. They used 5,400 prompts, a special classifier, and some metrics to see how well different models did at writing texts about men and women, people of different races, and people who are both male or female and from a certain racial group. They also made a new dataset with 3,724 sentences that were either agentic (like “he can”) or communal (like “we love”). The results show that LLMs tend to write about men more than women, and there’s even more bias when it comes to people who are both male and from a certain racial group. |
Keywords
» Artificial intelligence » Classification » Large language model » Text generation
