ACL 2022 Workshop BigScience Submissions

What Language Model to Train if You Have One Million GPU Hours?
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, Iz Beltagy
- Published: 11 Apr 2022, Last Modified: 22 Oct 2023
- BigScience#5
- Readers: Everyone
Adapting BigScience Multilingual Model to Unseen Languages
Zheng Xin Yong, Vassilina Nikoulina
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned
Sameera Horawalavithana, Ellyn Ayton, Shivam Sharma, Scott Howland, Megha Subramanian, Scott Vasquez, Robin Cosbey, Maria Glenski, Svitlana Volkova
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Emergent Structures and Training Dynamics in Large Language Models
Ryan Teehan, Miruna Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, Aaron Gokaslan
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Dataset Debt in Biomedical Language Modeling
Jason Alan Fries, Natasha Seelam, Gabriel Altay, Leon Weber, Myungsun Kang, Debajyoti Datta, Ruisi Su, Samuele Garda, Bo Wang, Simon Ott, Matthias Samwald, Wojciech Kusa
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Language Model Accidents Waiting to Happen?
Anonymous
- 09 Mar 2022 (modified: 05 May 2023)
- ACL 2022 Workshop BigScience Blind Submission
- Readers: Everyone
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Gregory Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Martin Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, Samuel Weinbach
- Published: 09 Apr 2022, Last Modified: 22 Oct 2023
- BigScience#5
- Readers: Everyone
Fine-tuned Language Models can be Continual Learners
Tuhin Chakrabarty, Thomas Scialom, Smaranda Muresan
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5 ednonarchival
- Readers: Everyone
A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
Imad LAKIM, Ebtesam Almazrouei, Ibrahim Abu Alhaol, Merouane Debbah, Julien Launay
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0
Francesco De Toni, Christopher Akiki, Javier de la Rosa, Clémentine Fourrier, Enrique Manjavacas, Stefan Schweter, Daniel Van Strien
- Published: 09 Apr 2022, Last Modified: 22 Oct 2023
- BigScience#5
- Readers: Everyone
Scalable Training of Language Models using JAX pjit and TPUv4
Anonymous
- 09 Mar 2022 (modified: 05 May 2023)
- ACL 2022 Workshop BigScience Blind Submission
- Readers: Everyone
Pipelines for Social Bias Testing of Large Language Models
Debora Nozza, Federico Bianchi, Dirk Hovy
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xiaochang Peng, Xiang Ren, Hamed Firooz
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, Kentaro Inui
- Published: 09 Apr 2022, Last Modified: 22 Oct 2023
- BigScience#5
- Readers: Everyone
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, Oskar van der Wal
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Predicting Fine-Tuning Performance with Probing
Zining Zhu, Soroosh Shahtalebi, Frank Rudzicz
- Published: 09 Apr 2022, Last Modified: 22 Oct 2023
- BigScience#5
- Readers: Everyone
Using ASR-Generated Text for Spoken Language Modeling
Nicolas Hervé, Valentin Pelloin, Benoit Favre, Franck Dary, Antoine LAURENT, Sylvain Meignier, laurent besacier
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren
- Published: 09 Apr 2022, Last Modified: 05 May 2023
- BigScience#5
- Readers: Everyone