Summary of Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time, by David Herel et al.
Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time
by David Herel, Vojtech Bartek, Jiri Jirak, Tomas Mikolov
First submitted to arxiv on: 20 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a novel framework and dataset for evaluating large language models (LLMs) on their ability to reason about temporal context. The dataset spans over 8,000 events from 2018 to 2024, with annotations at day-level granularity across various domains. The TimeShift evaluation method is introduced to assess LLMs’ temporal reasoning capabilities, showing that base models often outperform instruction-tuned and synthetic-trained counterparts on time-sensitive recall. Additionally, the paper highlights the brittleness of even large-scale models in handling paraphrased facts, underscoring unresolved challenges in temporal consistency. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper is about how language models can understand when things happened in the past. It’s like asking “Who is the US President?” and the answer changing depending on when you ask. The model has to consider time too! They created a special dataset with over 8,000 events from 2018 to 2024, so it can learn how to think about time better. The results show that some models are good at remembering things that happened in the past, but others struggle with words that mean the same thing but are used differently. |
Keywords
» Artificial intelligence » Recall
