Summary of The 2024 Foundation Model Transparency Index, by Rishi Bommasani et al.
The 2024 Foundation Model Transparency Index
by Rishi Bommasani, Kevin Klyman, Sayash Kapoor, Shayne Longpre, Betty Xiong, Nestor Maslej, Percy Liang
First submitted to arxiv on: 17 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents an analysis of foundation models’ transparency using the Foundation Model Transparency Index (FMTI). FMTI 2023 assessed 10 major developers on 100 indicators, revealing limited public disclosure with an average score of 37. The follow-up study, FMTI 2024, scored 14 developers against the same indicators and found a 21-point improvement to an average score of 58. This increase is attributed to developers disclosing new information during the process. The study identified sustained opacity in areas such as copyright status, data access, and downstream impact. Transparency reports for each developer are published, showcasing consolidations of disclosed information. The FMTI’s findings demonstrate that transparency can be improved in this ecosystem. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how well foundation model developers share information with the public. It uses a scorecard to rate 10 top developers on factors like how much they pay data workers and whether they reveal their copyright policies. The first test got an average score of only 37 out of 100. Six months later, the same test was run again, and this time the developers scored better – an average of 58 out of 100! This improvement is because some developers shared new information for the second test. The study also found that some areas are still very secretive, like what happens to data once it’s used. |
