Summary of Evaluating the Performance and Robustness Of Llms in Materials Science Q&a and Property Predictions, by Hongchen Wang et al.
Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions
by Hongchen Wang, Kangming Li, Scott Ramsay, Yao Fehlis, Edward Kim, Jason Hattrick-Simpers
First submitted to arxiv on: 22 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Materials Science (cond-mat.mtrl-sci); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large Language Models (LLMs) have the potential to revolutionize scientific research, yet their robustness and reliability in domain-specific applications remain insufficiently explored. In this study, we evaluate the performance and robustness of LLMs for materials science, focusing on domain-specific question answering and materials property prediction across diverse real-world and adversarial conditions. We use three distinct datasets: undergraduate-level materials science questions, steel compositions, and band gap values. The performance of LLMs is assessed using various prompting strategies, including zero-shot chain-of-thought, expert prompting, and few-shot in-context learning. Robustness testing involves realistic and intentionally manipulated noise to evaluate their resilience and reliability under real-world conditions. We also observe unique phenomena like mode collapse behavior when prompt examples are altered and performance recovery from train/test mismatch. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large Language Models (LLMs) can help scientists with research, but we don’t know if they’re reliable in specific areas like materials science. This study tests LLMs to see how well they do on tasks like answering questions and predicting material properties. We used three different datasets: questions from a university course, information about different types of steel, and details about crystal structures. The models performed differently depending on how we asked them questions. We also tested how well the models worked when we added “noise” to the data, which can represent real-world problems or intentional attempts to make it harder for the models to work. This study shows that LLMs are not always reliable and can have strange behaviors. It’s meant to help us understand the limitations of these models so we can use them better in the future. |
Keywords
» Artificial intelligence » Few shot » Prompt » Prompting » Question answering » Zero shot
