Summary of Temporally Consistent Factuality Probing For Large Language Models, by Ashutosh Bajpai et al.
Temporally Consistent Factuality Probing for Large Language Models
by Ashutosh Bajpai, Aaryan Goyal, Atif Anwer, Tanmoy Chakraborty
First submitted to arxiv on: 21 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper proposes a novel framework, TeCFaP, to evaluate Large Language Models’ (LLMs) ability to provide factually consistent responses across different temporal contexts. To achieve this, the authors introduce TEMP-COFAC, a high-quality dataset of paraphrased English queries with temporally varying consistency requirements. The study also extends existing metrics to capture LLMs’ performance on TeCFaP and explores the effectiveness of a novel learning framework, CoTSeLF, which combines multi-task instruction tuning and consistent-time-sensitive reinforcement learning to improve temporal factuality in LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making sure language models can provide correct answers even when they’re talking about different times. Right now, these models are not very good at this because they don’t think about time very well. The researchers created a new way to test the models and found that most of them didn’t do very well. They then came up with a new idea, CoTSeLF, which is a combination of two other methods that helps language models be better at providing correct answers across different times. |
Keywords
» Artificial intelligence » Instruction tuning » Multi task » Reinforcement learning
