Summary of Parameter-efficient Fine-tuning Via Selective Discrete Cosine Transform, by Yixian Shen et al.
Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform
by Yixian Shen, Qi Bi, Jia-Hong Huang, Hongyi Zhu, Anuj Pathania
First submitted to arxiv on: 9 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Selective Discrete Cosine Transformation (sDCTFT) fine-tuning scheme leverages the energy compaction and decorrelation properties of Discrete Cosine Transform (DCT) to improve model efficiency and accuracy in large language models. This approach projects weight changes from low-rank adaptations into DCT space, partitioning them across different frequency levels and selecting critical components. Compared to prior arts like LoRA and FourierFT, sDCTFT achieves better performance while reducing computational cost and storage requirements. For instance, it outperforms LoRA with just 0.05M trainable parameters compared to LoRA’s 38.2M. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A new way of making big language models smaller and more efficient is proposed in this paper. The idea is to use a special kind of transformation called Discrete Cosine Transform (DCT) to make the model more compact while still being able to learn and understand things. This approach helps reduce the amount of calculations needed and the storage space required, making it more suitable for large models. The results show that this method performs better than others in similar situations. |
Keywords
» Artificial intelligence » Fine tuning » Lora
