Summary of Limits to Predicting Online Speech Using Large Language Models, by Mina Remeli et al.
Limits to Predicting Online Speech Using Large Language Models
by Mina Remeli, Moritz Hardt, Robert C. Williamson
First submitted to arxiv on: 8 Jul 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Computers and Society (cs.CY); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the predictability of online speech on social media, focusing on whether information outside a user’s own posts improves predictability. The study builds upon recent theoretical results suggesting that social circle posts are as predictive of future posts as past posts. To test this hypothesis, the authors collect over 16 million tweets and analyze four large language models with varying sizes (1.5 billion to 70 billion parameters). They define predictability as a measure of model uncertainty (negative log-likelihood on future tokens given context) and find that predicting user posts from peers’ posts performs poorly. In contrast, the value of a user’s own posts for prediction is consistently higher than that of their peers. The authors also explore what’s learned in-context and the robustness of their findings, discovering that base models learn to correctly predict @-mentions and hashtags. The paper contributes to the understanding of large language model capabilities and limitations in predicting online speech. The results have implications for applications such as user profiling, content generation, and social media analytics. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how well we can predict what people will say on social media based on information they’ve shared before. Researchers wanted to know if knowing what other people are saying helps or hurts our ability to predict what someone else might say. They analyzed millions of tweets and used special computer models to test their ideas. The results showed that trying to predict what someone will say based on what others have said doesn’t work very well. Instead, the best way to make predictions is still by looking at what the person has said before. The researchers also found out that these computer models can learn to recognize certain patterns in language, like special symbols and hashtags. |
Keywords
» Artificial intelligence » Large language model » Log likelihood
