The 2024 Foundation Model Transparency Index
Abstract: Foundation models are increasingly consequential yet extremely opaque. To characterize the status quo, the Foundation Model Transparency Index was launched in October 2023 to measure the transparency of leading foundation model developers. The October 2023 Index (v1.0) assessed 10 major foundation model developers (e.g. OpenAI, Google) on 100 transparency indicators (e.g. does the developer disclose the wages it pays for data labor?). At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study (v1.1) after 6 months: we score 14 developers against the same 100 indicators. While in v1.0 we searched for publicly available information, in v1.1 developers submit reports on the 100 transparency indicators, potentially including information that was not previously public. We find that developers now score 58 out of 100 on average, a 21 point improvement over v1.0. Much of this increase is driven by developers disclosing information during the v1.1 process: on average, developers disclosed information related to 16.6 indicators that was not previously public. We observe regions of sustained (i.e. across v1.0 and v1.1) and systemic (i.e. across most or all developers) opacity such as on copyright status, data access, data labor, and downstream impact. We publish transparency reports for each developer that consolidate information disclosures: these reports are based on the information disclosed to us via developers. Our findings demonstrate that transparency can be improved in this nascent ecosystem, the Foundation Model Transparency Index likely contributes to these improvements, and policymakers should consider interventions in areas where transparency has not improved.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: We thank the AE and reviewers for their feedback on this work. In response, here is the updated camera ready.
Change log:
1. Deanonymized and switched to camera ready format. (This also includes links to the GitHub/website that were anonymized previously, which should help with providing links to the long-form definitions for all 100 indicators)
2. In line with the AE comment, we have added discussion of independent scorers (see Section 3.4) directly in line with what we provided in author response.
3. In line with the AE comment, we have added discussion of why information is not disclosed (see Section 5.1) directly in line with what we provided in author response.
4. In line with the AE comment, we have added discussion of the indicators including moving the list of 100 indicators up to the main text as Figure 1 (see Section 3.1) directly in line with what we provided in author response.
Overall, we believe this addresses all changes stated by the AE, as well as comments that came up in individual reviews.
Please let us know if anything else is needed and thanks once again for engaging with our work!
Code: https://www.github.com/stanford-crfm/fmti
Assigned Action Editor: ~Nihar_B_Shah1
Submission Number: 3021
Loading