ICML 2024 Workshop TF2M Submissions

Transformer Designs for In-Context Learning in Foundation Models for Time Series Forecasting with Covariates
Afrin Dange, Vaibhav Raj, Praneeth Netrapalli, Sunita Sarawagi
- Published: 18 Jun 2024, Last Modified: 20 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Implicit Optimization Bias of Next-token Prediction in Linear Models
Christos Thrampoulidis
- Published: 18 Jun 2024, Last Modified: 03 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Attention Is All You Need But You Don’t Need All Of It For Inference of Large Language Models
Georgy Tyukin, Gbetondji Jean-Sebastien Dovonon, Jean Kaddour, Pasquale Minervini
- Published: 18 Jun 2024, Last Modified: 20 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
ImportanceWeighted Multi-Draft Speculative Sampling
Ashish J Khisti, Arash Behravesh, Hassan Dbouk, Arash Behboodi, Roland Memisevic, Christos Louizos
- Published: 18 Jun 2024, Last Modified: 03 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
On the Power of Convolution Augmented Transformer
Mingchen Li, Xuechen Zhang, Yixiao Huang, Samet Oymak
- Published: 18 Jun 2024, Last Modified: 18 May 2025
- TF2M 2024 Poster
- Readers: Everyone
Detrimental Memories in Transfer Learning
Amal Alnouri, Timothy J Wroge, Bilal Alsallakh
- Published: 18 Jun 2024, Last Modified: 19 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
How Do Nonlinear Transformers Acquire Generalization-Guaranteed CoT Ability?
Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, Pin-Yu Chen
- Published: 18 Jun 2024, Last Modified: 26 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
MSAMamba: Adapting Subquadratic Models To Long-Context DNA MSA Analysis
Vishrut Thoutam, Dina Ellsworth
- Published: 18 Jun 2024, Last Modified: 07 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
Siyu Chen, Heejune Sheen, Tianhao Wang, Zhuoran Yang
- Published: 18 Jun 2024, Last Modified: 03 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
SAIL: Self-improving Efficient Online Alignment of Large Language Models
Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Amrit Bedi, Furong Huang
- Published: 18 Jun 2024, Last Modified: 03 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers
Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam
- Published: 18 Jun 2024, Last Modified: 14 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
A deeper look at depth pruning of LLMs
Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, Pavlo Molchanov
- Published: 18 Jun 2024, Last Modified: 19 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Local to Global: Learning Dynamics and Effect of Initialization for Transformers
Ashok Vardhan Makkuva, Marco Bondaschi, Chanakya Ekbote, Adway Girish, Alliot Nagle, Hyeji Kim, Michael Gastpar
- Published: 18 Jun 2024, Last Modified: 21 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Hallmarks of Optimization Trajectories in Neural Networks and LLMs: Directional Exploration and Redundancy
Sidak Pal Singh, Bobby He, Thomas Hofmann, Bernhard Schölkopf
- Published: 18 Jun 2024, Last Modified: 18 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Fast Machine Unlearning via Robust Training
Youssef Allouah, Joshua Kazdan, Rachid Guerraoui, Sanmi Koyejo
- Published: 18 Jun 2024, Last Modified: 20 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Meta-optimization for Deep Learning via Nonstochastic Control
Xinyi Chen, Evan Dogariu, Zhou Lu, Elad Hazan
- Published: 18 Jun 2024, Last Modified: 03 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
On Provable Length and Compositional Generalization
Kartik Ahuja, Amin Mansouri
- Published: 18 Jun 2024, Last Modified: 17 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
Adway Girish, Alliot Nagle, Ashok Vardhan Makkuva, Marco Bondaschi, Michael Gastpar, Hyeji Kim
- Published: 18 Jun 2024, Last Modified: 22 Jul 2024
- TF2M 2024 Oral
- Readers: Everyone
Transformer Efficiently Learns Low-dimensional Target Functions In-context
Yujin Song, Denny Wu, Kazusato Oko, Taiji Suzuki
- Published: 18 Jun 2024, Last Modified: 25 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Rethinking Invariance in In-context Learning
Lizhe Fang, Yifei Wang, Khashayar Gatmiry, Lei Fang, Yisen Wang
- Published: 18 Jun 2024, Last Modified: 19 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Unavoidable Learning Constraints Alter the Foundations of Direct Preference Optimization
David Wipf
- Published: 18 Jun 2024, Last Modified: 17 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Understanding and Minimising Outlier Features in Neural Network Training
Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, Thomas Hofmann
- Published: 18 Jun 2024, Last Modified: 19 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
State Space Models are Comparable to Transformers in Estimating Functions with Dynamic Smoothness
Naoki Nishikawa, Taiji Suzuki
- Published: 18 Jun 2024, Last Modified: 09 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Progressive distillation improves feature learning via implicit curriculum
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, Surbhi Goel
- Published: 18 Jun 2024, Last Modified: 03 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone
Active Preference Optimization for Sample Efficient RLHF
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, Sayak Ray Chowdhury
- Published: 18 Jun 2024, Last Modified: 11 Jul 2024
- TF2M 2024 Poster
- Readers: Everyone