Summary of Codi: Conversational Distillation For Grounded Question Answering, by Patrick Huber et al.
CoDi: Conversational Distillation for Grounded Question Answering
by Patrick Huber, Arash Einolghozati, Rylan Conway, Kanika Narang, Matt Smith, Waqar Nayyar, Adithya Sagar, Ahmed Aly, Akshat Shrivastava
First submitted to arxiv on: 20 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper introduces CoDi, a novel data distillation framework that enables the synthesis of large-scale, assistant-style datasets for Small Language Models (SLMs) with approximately 1 billion parameters. This framework addresses challenges such as limited model capacity and scarce high-quality conversational datasets. The authors explore and evaluate CoDi’s potential on the task of conversational grounded reasoning for question answering, demonstrating comparable performance to models trained on human-annotated data in standard metrics. Additionally, when generating larger datasets from web data, SLMs trained with CoDi outperform instruction-tuned models in zero-shot conversational grounded reasoning tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper introduces a new way to make small language models better at having conversations. These models have limited capacity and need more training data. The authors created a framework called CoDi that can take existing data and make it bigger and more diverse, so the model can learn from it. They tested this on a specific task of answering questions based on context, and the results were promising. The small language models performed as well as larger models trained on human-annotated data. |
Keywords
» Artificial intelligence » Distillation » Question answering » Zero shot
