Summary of Zero-shot Classification Of Crisis Tweets Using Instruction-finetuned Large Language Models, by Emma Mcdaniel et al.
Zero-Shot Classification of Crisis Tweets Using Instruction-Finetuned Large Language Models
by Emma McDaniel, Samuel Scheele, Jeff Liu
First submitted to arxiv on: 30 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper assesses the capabilities of three commercial large language models (LLMs) in zero-shot classification of short social media posts related to humanitarian crises. The LLMs, OpenAI GPT-4o, Gemini 1.5-flash-001, and Anthropic Claude-3-5 Sonnet, are tasked with classifying crisis tweets into informative or non-informative categories and ranking them according to their relevance to 16 possible humanitarian classes. The performance of the models is evaluated using macro, weighted, and binary F1-scores. Interestingly, the results show that providing extra information about the event during which the tweet was mined improves the accuracy of the humanitarian label classification task. Furthermore, the study highlights significant variations in model performance across different datasets, raising questions about dataset quality. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at three special computers called large language models (LLMs) and how well they can understand short messages on social media that are related to disasters. The LLMs try to decide if a message is helpful or not, and also sort the messages into categories like “food” or “shelter”. The researchers used a big collection of tweets about different disasters to test the models’ abilities. They found that giving the models more information about what’s happening in the world helps them make better decisions. However, they also discovered that each LLM works differently depending on the type of messages it sees. |
Keywords
» Artificial intelligence » Classification » Claude » Gemini » Gpt » Zero shot
