Summary of Codeact: Code Adaptive Compute-efficient Tuning Framework For Code Llms, by Weijie Lv et al.
CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs
by Weijie Lv, Xuan Xia, Sheng-Jun Huang
First submitted to arxiv on: 5 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Code Adaptive Compute-efficient Tuning (CodeACT) framework aims to bridge the performance gap between open-source and closed-source large language models (LLMs) in code-related tasks. CodeACT introduces Complexity and Diversity Aware Sampling (CDAS) to select high-quality training data and Dynamic Pack padding strategy to reduce computational resource usage. Experimental results show that CodeACT, fine-tuned on 40% of the EVOL-Instruct data, achieves an 8.6% performance increase on HumanEval, reduces training time by 78%, and decreases peak GPU memory usage by 27%. These findings demonstrate CodeACT’s ability to enhance open-source LLMs’ performance and efficiency. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper proposes a new way to train large language models (LLMs) called Code Adaptive Compute-efficient Tuning or CodeACT. This method helps make the training process more efficient by selecting only the most important data and reducing unnecessary computer power usage. The experiment shows that this method can improve the performance of LLMs by 8.6% while also reducing the time it takes to train them by 78%. This is an important step towards making language models more accessible and useful for everyone. |
