ICML 2025 Workshop ES-FoMo-III Submissions

Foreign Sparse Attention: Effective Distillation into Sparse Attention
Vijaykaarti Sundarapandiyan, Tom Goldstein, Ashwinee Panda
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Large Reasoning Models Know How to Think Efficiently
Zeyu XING, Xing Li, Huiling Zhen, Xianzhi Yu, Mingxuan Yuan, Sinno Jialin Pan
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Byzantine-Resilient Zero-Order Optimization for Scalable Federated Fine-Tuning of Large Language Models
Maximilian Egger, Mayank Bakshi, Rawad Bitar
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach
Ayeong Lee, Ethan Che, Tianyi Peng
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
SageAttention2++: A More Efficient Implementation of SageAttention2
Jintao Zhang, Xiaoming Xu, Jia wei, Haofeng Huang, Pengle Zhang, Chendong Xiang, Jun Zhu, Jianfei Chen
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
Jiecheng Zhou, Ding Tang, Rong Fu, Boni Hu, Haoran Xu, Yi Wang, Suzhongling, Liang Liu, PeiZhilin, Hengjie Li, Xingcheng Zhang, Weiming Zhang
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
ConMeZO: Adaptive Directional Sampling for Gradient-Free Finetuning of Language Models
Lejs Deen Behric, Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Private Zeroth-Order Optimization with Public Data
Xuchen Gong, Tian Li
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
SpecCoT: Accelerating Chain-of-Thought Reasoning through Speculative Exploration
Junhan Shi, Yijia Zhu, Zhenning Shi, Dan Zhao, Qing Li, Yong Jiang
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Accelerating Linear Attention Design by Unifying Forward & Backward Propagation
Zhen Qin, Xuyang Shen, Dong Li, Yiran Zhong
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Resource-efficient Inference with Foundation Model Programs
Lunyiu Nie, Zhimin Ding, Kevin Yu, Marco Cheung, Chris Jermaine, Swarat Chaudhuri
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III Spotlight
- Readers: Everyone
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Thinking
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
FrugalRAG: Learning to retrieve and reason for multi-hop QA
Abhinav Java, Srivathsan Koundinyan, Nagarajan Natarajan, Amit Sharma
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Adaptive Backbone Selection for Efficient and Real-Time Vision Inference
Syed Amir Hamza, Alexander Jesser
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas
Austin Silveria, Soham V. Govande, Daniel Y Fu
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III Spotlight
- Readers: Everyone
The Road Not Taken: Hindsight Exploration for LLMs in Multi-Turn RL
Huaxiaoyue Wang, Sanjiban Choudhury
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
Yehjin Shin, Seojin Kim, Noseong Park
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention
Zhihao Zhan, Jianan Zhao, Zhaocheng Zhu, Jian Tang
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
InterLoRA: An Adaptive LoRA Structure Based on The Mechanistic Interpretability of Transformer
Jihao Gu, Zelin Wang, Yibo Zhang, Ping Gong, Zhisong Bie
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis
Gholamali Aminian, Idan Shenfeld, Amir R. Asadi, Ahmad Beirami, Youssef Mroueh
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Demystifying Language Model Forgetting with Low-rank Example Associations
Xisen Jin, Xiang Ren
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
SD$^2$: Self-Distilled Sparse Drafters
Mike Lasby, Nish Sinnadurai, Valavan Manohararajah, Sean Lie, Yani Ioannou, Vithursan Thangarasa
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
zip2zip: Inference-Time Adaptive Vocabularies for Language Models via Token Compression
Saibo Geng, Nathan Ranchin, Yunzhen Yao, Maxime Peyrard, Chris Wendler, Michael Gastpar, Robert West
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III Oral
- Readers: Everyone
A Minimalist Optimizer Design for LLM Pretraining
Athanasios Glentis, Jiaxiang Li, Andi Han, Mingyi Hong
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone
Iterative Amortized Inference: Unifying In-Context Learning and Learned Optimizers
Sarthak Mittal, Divyat Mahajan, Guillaume Lajoie, Mohammad Pezeshki
- Published: 11 Jun 2025, Last Modified: 10 Jul 2025
- ES-FoMo III
- Readers: Everyone