Summary of Long-context Llms Struggle with Long In-context Learning, by Tianle Li et al.
Long-context LLMs Struggle with Long In-context Learning
by Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, Wenhu Chen
First submitted to arxiv on: 2 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces a new benchmark, LongICLBench, to evaluate Large Language Models (LLMs) in handling long sequences and complex classification tasks. The benchmark consists of six datasets with 28-174 classes and input lengths from 2K to 50K tokens. It assesses the models’ ability to comprehend entire input sequences and recognize massive label spaces to make correct predictions. The authors evaluate 15 LLMs on LongICLBench and find that they perform well in less challenging classification tasks but struggle with more complex tasks like Discovery, which has 174 labels. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. This study highlights the challenges of long context understanding and reasoning for existing LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research paper creates a new test to see how well computers can understand very long pieces of text. The computer models are called Large Language Models (LLMs), and they’re really good at handling short texts. But when it comes to longer texts, they struggle. The researchers created a special set of tests with many different labels to see how the LLMs do. They found that the models can handle some tasks easily, but others are much harder. They also discovered that the models tend to understand things better if they’re near the end of the text rather than at the beginning. This research shows that computers still have a lot to learn about understanding very long pieces of text. |
Keywords
» Artificial intelligence » Classification
