Summary of Meta-cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models, by Zhuoqun Li et al.
Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models
by Zhuoqun Li, Hongyu Lin, Yaojie Lu, Hao Xiang, Xianpei Han, Le Sun
First submitted to arxiv on: 14 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates the importance of declarative knowledge (DK) and procedural knowledge (PK) in pre-training and inference of Large Language Models (LLMs). The authors note that existing meta-cognitive theory emphasizes both types of knowledge, but a comprehensive comparison is lacking due to challenges in definition, probing, and quantitative assessment. To address this gap, the study provides ground-truth knowledge for LLMs and evaluates their effective scores. Through extensive experiments with widely-used datasets and models, the authors find that (1) declarative knowledge generally outperforms procedural knowledge, (2) procedural knowledge excels only in reasoning tasks with simple logic, and (3) as pre-training progresses and size increases, model ability to utilize both types of knowledge improves at different speeds. The study’s findings provide primary guidance for evaluating and enhancing large language models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how big language models learn and use two kinds of knowledge: what we know (declarative) and how to do things (procedural). Researchers wanted to compare these two types of knowledge to see which one is more important. They found that in most cases, the model does better with declarative knowledge. However, when the task involves simple logic, procedural knowledge is better. The study also shows that as the model gets bigger and better trained, it can use both kinds of knowledge better. This research helps us understand how to make language models even better. |
Keywords
» Artificial intelligence » Inference
