Summary of Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance?, by Fumiya Uchiyama et al.
Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance?
by Fumiya Uchiyama, Takeshi Kojima, Andrew Gambardella, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo
First submitted to arxiv on: 9 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The recent advancements in large language models (LLMs) have shown impressive generalization capabilities in mathematical and logical reasoning tasks. Researchers have found that LLMs pre-trained with programming language data exhibit high mathematical and reasoning abilities, but a causal relationship has not been thoroughly investigated. This study aims to verify which programming languages and features during pre-training impact logical inference performance. The authors pre-trained decoder-based language models from scratch using datasets from ten programming languages (e.g., Python, C, Java) and three natural language datasets under identical conditions. The trained models were then evaluated in a few-shot in-context learning setting on logical reasoning tasks: FLD and bAbi, which do not require commonsense or world knowledge. The results show that nearly all models trained with programming languages consistently outperform those trained with natural languages, indicating that programming languages contain factors that elicit logic inference performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models have shown amazing skills in math and logical thinking. Scientists wanted to see if training these models on computer code makes them better at solving problems. They took popular programming languages like Python and Java, as well as books and websites, and used them to train the models. Then, they tested how well the models could solve tricky math and logic puzzles without needing any outside help. The results showed that models trained on programming languages were way better at solving these puzzles than those trained on natural language. This means that there’s something special about computer code that helps machines get better at logical thinking. |
Keywords
» Artificial intelligence » Decoder » Few shot » Generalization » Inference
