Summary of Evaluating Large Language Models For Public Health Classification and Extraction Tasks, by Joshua Harris et al.
Evaluating Large Language Models for Public Health Classification and Extraction Tasks
by Joshua Harris, Timothy Laurence, Leo Loman, Fan Grayson, Toby Nonnenmacher, Harry Long, Loes WalsGriffith, Amy Douglas, Holly Fountain, Stelios Georgiou, Jo Hardstaff, Kathryn Hopkins, Y-Ling Chi, Galena Kuyumdzhieva, Lesley Larkin, Samuel Collins, Hamish Mohammed, Thomas Finnie, Luke Hounsome, Michael Borowitz, Steven Riley
First submitted to arxiv on: 23 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents an evaluation of Large Language Models (LLMs) for public health tasks involving text classification and extraction. Eleven open-weight LLMs are tested on 16 tasks using zero-shot in-context learning, with the highest performing model being Llama-3.3-70B-Instruct, achieving excellent results on 8/16 tasks. The models show significant variation across tasks, with some performing well on GI Illness Classification and others struggling with Contact Classification. GPT-4 and GPT-4o series models also perform similarly to the best-performing LLM. This research suggests that LLMs may be valuable tools for public health experts to extract information from various text sources, supporting surveillance, research, and interventions. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how well big language models can help health experts understand and use free text information. It tests 11 different models on 16 tasks, like classifying health problems or finding important facts in texts. The best model did really well on most of the tasks, but some models struggled with certain types of information. This research shows that these language models might be helpful for health experts to make sense of lots of different text sources and do their jobs better. |
Keywords
» Artificial intelligence » Classification » Gpt » Llama » Text classification » Zero shot
