Summary of Chinese Tiny Llm: Pretraining a Chinese-centric Large Language Model, by Xinrun Du et al.
Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
by Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Wenhu Chen, Ge Zhang
First submitted to arxiv on: 5 Apr 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A medium-difficulty summary of this research paper introduces CT-LLM, a large language model prioritizing Chinese language development. This unique approach diverges from conventional methods by primarily incorporating 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens into its training. The model’s exceptional proficiency in understanding Chinese is further enhanced through alignment techniques. CT-LLM demonstrates remarkable performance on the CHC-Bench, excelling in Chinese language tasks and showcasing adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and adapting them to other languages, broadening horizons for LLM training methodologies. The paper open-sources the full process of training a Chinese LLM, including data processing procedure, CHC-Bench, and CT-LLM, fostering further exploration and innovation in academia and industry. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary CT-LLM is a big language model that’s very good at understanding Chinese. It’s different because it was trained on lots of Chinese text, not just English like most other models. This helps it understand Chinese really well and do tasks related to Chinese language. The model also does okay with English too! The research shows that we don’t have to train models mainly on English and then adapt them for other languages. Instead, we can start with the language we want to focus on and get better results. The team is sharing all their work so others can build on it. |
Keywords
» Artificial intelligence » Alignment » Language model » Large language model
