NeurIPS 2024 Workshop OPT Submissions

Tensor-GaLore: Memory-Efficient Training via Gradient Tensor Decomposition
Robert Joseph George, David Pitt, Jiawei Zhao, Jean Kossaifi, Cheng Luo, Yuandong Tian, Anima Anandkumar
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Solving hidden monotone variational inequalities with surrogate losses
Ryan D'Orazio, Danilo Vucetic, Zichu Liu, Junhyung Lyle Kim, Ioannis Mitliagkas, Gauthier Gidel
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
SOAP: Improving and Stabilizing Shampoo using Adam
Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, Sham M. Kakade
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Neural Entropic Multimarginal Optimal Transport
Dor Tsur, Ziv Goldfeld, Kristjan Greenewald, Haim H. Permuter
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Discrete-Continuous Variational Optimization with Local Gradients
Jonathan H Warrell, Francesco Alesiani, Cameron Smith, Anja Mösch, Martin Renqiang Min
- Published: 10 Oct 2024, Last Modified: 12 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Graph Neural Networks for Hyperparameter Inference in Ising Solvers
Edward Jiang, Sam Reifenstein, Milin Doppalapudi, Timothee Leleu
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Dimensionality Reduction Techniques for Global Bayesian Optimisation
Luo Long, Coralia Cartis, Paz Fink Shustin
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Stochastic Proximal Point Methods for Monotone Inclusions under Expected Similarity
Abdurakhmon Sadiev, Laurent Condat, Peter Richtárik
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Role of Parametrization in Learning Dynamics of Recurrent Neural Networks
Adwait Datar, Chinmay Datar, Zahra Monfared, Felix Dietrich
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Scalable Second-Order Optimization Algorithms for Minimizing Low-rank Functions
Edward Tansley, Coralia Cartis
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Deconstructing What Makes a Good Optimizer for Language Models
Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, Sham M. Kakade
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Communication-Efficient Loss Minimization over Heterogeneous Data with Federated Hierarchical Ensemble Aggregation via Distillation
Sayantan Chowdhury, Ben Liang, Ali Tizghadam, Ilijc Albanese
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Remove Symmetries to Control Model Expressivity and Improve Optimization
Liu Ziyin, Yizhou Xu, Isaac L. Chuang
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
A Unified Convergence Theory for Large Language Model Efficient Fine-tuning
Zhanhong Jiang, Nastaran Saadati, Aditya Balu, Minh Pham, Joshua Russell Waite, Nasla Saleem, Chinmay Hegde, Soumik Sarkar
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
ACCO: Accumulate while you Communicate, Hiding Communications in Distributed LLM Training
Adel Nabli, Louis Fournier, Pierre ERBACHER, Louis Serrano, Eugene Belilovsky, Edouard Oyallon
- Published: 10 Oct 2024, Last Modified: 13 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
From Gradient Clipping to Normalization for Heavy Tailed SGD
Florian Hübler, Ilyas Fatkhullin, Niao He
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Don't Be So Positive: Negative Step Sizes in Second-Order Methods
Betty Shea, Mark Schmidt
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Glocal Smoothness: Line Search can really help!
Curtis Fox, Mark Schmidt
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Intuitive Analysis of the Quantization based Optimization : From establishing a SDE to Quantum Mechanical Perspective
Jinwuk Seok, Changsik Cho
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models
Zeman Li, Xinwei Zhang, Peilin Zhong, Yuan Deng, Meisam Razaviyayn, Vahab Mirrokni
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks
Louis Fournier, Edouard Oyallon
- Published: 10 Oct 2024, Last Modified: 12 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
Louis Fournier, Adel Nabli, Masih Aminbeidokhti, Marco Pedersoli, Eugene Belilovsky, Edouard Oyallon
- Published: 10 Oct 2024, Last Modified: 12 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
Memory-Efficient Large Language Model (LLM) Training and Fine-Tuning via Gradient Subspace Tracking
Sahar Rajabi, Sirisha Rambhatla
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
On the Convergence of DP-SGD with Adaptive Clipping
Egor Shulgin, Peter Richtárik
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone
On the Crucial Role of Initialization for Matrix Factorization
Bingcong Li, Liang Zhang, Aryan Mokhtari, Niao He
- Published: 10 Oct 2024, Last Modified: 07 Dec 2024
- NeurIPS 2024 Workshop
- Readers: Everyone