Summary of Training Compute-optimal Protein Language Models, by Xingyi Cheng et al.
Training Compute-Optimal Protein Language Models
by Xingyi Cheng, Bo Chen, Pan Li, Jing Gong, Jie Tang, Le Song
First submitted to arxiv on: 4 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Quantitative Methods (q-bio.QM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the best practices for training protein language models, an area with limited guidance. Most existing approaches focus on increasing model sizes rather than optimizing compute resources to balance performance and budget. The study uses a massive dataset of 939 million protein sequences and trains over 300 models with different parameters and objectives. It observes diminishing returns for the Causal Language Model (CLM) and overfitting for the Masked Language Model (MLM). To address this, it includes metagenomic protein sequences to increase diversity and avoid plateaus or overfitting. The study also finds scaling laws for CLM and MLM on Transformer, tailored to protein sequence data, and observes transfer scaling from CLM to MLM. Finally, it validates these laws by comparing large-scale versions of ESM-2 and PROGEN2 on downstream tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how to train language models best for proteins. Most people do this by making the model bigger, but that doesn’t always help. The study used a huge dataset with 939 million protein sequences and trained many different models. It found that some models got less good as they got bigger (this is called diminishing returns) and others got too good and stopped improving (this is called overfitting). To fix this, the researchers added more kinds of proteins to the training set. They also figured out rules for how well different-sized models work on protein data and found that some models can learn from each other. Finally, they tested these ideas by comparing big versions of two different models. |
Keywords
» Artificial intelligence » Causal language model » Masked language model » Overfitting » Scaling laws » Transformer
