Summary of Anna Karenina Strikes Again: Pre-trained Llm Embeddings May Favor High-performing Learners, by Abigail Gurin Schleifer et al.
Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners
by Abigail Gurin Schleifer, Beata Beigman Klebanov, Moriah Ariely, Giora Alexandron
First submitted to arxiv on: 6 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC); Information Retrieval (cs.IR); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Unsupervised clustering of student responses to open-ended biology questions into behavioral and cognitive profiles using pre-trained Large Language Model (LLM) embeddings is a promising technique, but its ability to capture pedagogically meaningful information remains unexplored. Our study investigates the discoverability of theory-driven Knowledge Profiles (KPs) in student responses by comparing expert-identified KPs with data-driven clustering techniques. We find that most KPs are poorly discovered, except for those including correct answers, which is attributed to the representations of KPs in the pre-trained LLM embeddings space. This “discoverability bias” has implications for the development and evaluation of AI-powered educational tools. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Researchers are trying to figure out if a new way to group student answers to open-ended biology questions using special computer language models can help us understand what students know. They compared this method with how experts already group student answers, but found that most groups were not discovered correctly. The only groups that were correctly found were the ones that had correct answers. This means we need to be careful when using these language models in educational tools. |
Keywords
» Artificial intelligence » Clustering » Large language model » Unsupervised
