Summary of L3tc: Leveraging Rwkv For Learned Lossless Low-complexity Text Compression, by Junxuan Zhang et al.
L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression
by Junxuan Zhang, Zhengxue Cheng, Yan Zhao, Shihao Wang, Dajiang Zhou, Guo Lu, Li Song
First submitted to arxiv on: 21 Dec 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Information Theory (cs.IT); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Learned Lossless Low-complexity Text Compression method (L3TC) combines learning-based probabilistic models with an entropy coder for data compression. The approach focuses on a low-complexity design while maintaining compression performance, making it suitable for text compressors. L3TC leverages RWKV models as the backbone and introduces an outlier-aware tokenizer to handle frequent tokens and outliers. Additionally, a novel high-rank reparameterization strategy enhances learning capability during training without increasing complexity during inference. Experimental results demonstrate that L3TC achieves 48% bit saving compared to gzip compressor, with compression performance comparable to other learned compressors, but with significantly reduced model parameters (50x). Moreover, L3TC offers real-time decoding speeds up to megabytes per second, making it the fastest among all learned compressors. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary L3TC is a new way to compress text data. It uses a combination of machine learning and coding techniques to make data smaller and faster to decode. The method starts by using RWKV models as the backbone and then introduces special tokens for common words and handles rare words differently. This helps improve compression performance while keeping things simple. L3TC also includes a new way to learn from training data without increasing complexity during decoding. Results show that L3TC can compress text data 48% better than a commonly used method, with similar quality but much faster decoding speeds. |
Keywords
» Artificial intelligence » Inference » Machine learning » Tokenizer
