Summary of A Systematic Analysis on the Temporal Generalization Of Language Models in Social Media, by Asahi Ushio et al.
A Systematic Analysis on the Temporal Generalization of Language Models in Social Media
by Asahi Ushio, Jose Camacho-Collados
First submitted to arxiv on: 15 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed unified evaluation scheme assesses the performance of language models under temporal shifts on standard social media tasks. The paper focuses on Twitter, testing five diverse NLP tasks under different temporal settings. Results show that entity-focused tasks like named entity recognition and hate speech detection exhibit consistent performance drops under temporal shift, while topic and sentiment classification tasks do not. Additionally, continuous pre-training on the test period does not improve a model’s adaptability to temporal shifts. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how language models work when they’re trained on old social media data but tested on newer content. They found that some types of tasks, like recognizing people or detecting hate speech, get worse when there’s a time gap between training and testing. But other tasks, like classifying topics or sentiments, don’t change much. The study also showed that even if you keep training the model on new data, it won’t help it adapt to changes over time. |
Keywords
» Artificial intelligence » Classification » Named entity recognition » Nlp
