ICLR 2025 Workshop SLLM Submissions

Exploring the dual lottery ticket hypothesis in finetuning through specialised sparsification
Sampreeth R S, Arindam Biswas, Pabitra Mitra, BISWAJIT BASU
- Published: 05 Mar 2025, Last Modified: 07 Apr 2025
- SLLM
- Readers: Everyone
How Can Representation Dimension Dominate Structurally Pruned LLMs?
Mingxue Xu, Lisa Alazraki, Danilo Mandic
- Published: 05 Mar 2025, Last Modified: 22 Apr 2025
- SLLM
- Readers: Everyone
SPEX: Scaling Feature Interaction Explanations for LLMs
Justin Singh Kang, Landon Butler, Abhineet Agarwal, Yigit Efe Erginbas, Ramtin Pedarsani, Bin Yu, Kannan Ramchandran
- Published: 05 Mar 2025, Last Modified: 02 Apr 2025
- SLLM
- Readers: Everyone
CAMEx: Curvature-aware Merging of Experts
Dung Viet Nguyen, Minh Hoang Nguyen, Luc Nguyen, Rachel S.Y. Teo, Tan Minh Nguyen, Linh Duy Tran
- Published: 05 Mar 2025, Last Modified: 04 Apr 2025
- SLLM
- Readers: Everyone
2SSP: A Two-Stage Framework for Structured Pruning of LLMs
Fabrizio Sandri, Elia Cunegatti, Giovanni Iacca
- Published: 05 Mar 2025, Last Modified: 29 Mar 2025
- SLLM
- Readers: Everyone
Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training
Elia Cunegatti, Leonardo Lucio Custode, Giovanni Iacca
- Published: 05 Mar 2025, Last Modified: 29 Mar 2025
- SLLM
- Readers: Everyone
TASP: Preserving Training Dynamics in Transformers via NTK-Aware Structured Pruning
Mengting Ai, Tianxin Wei, Jingrui He
- Published: 05 Mar 2025, Last Modified: 24 Apr 2025
- SLLM
- Readers: Everyone
Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
Zifei Xu, Sayeh Sharify, Wanzin Yazar, Tristan J Webb, Xin Wang
- Published: 05 Mar 2025, Last Modified: 17 Apr 2025
- SLLM
- Readers: Everyone
S2-ATTENTION: HARDWARE-AWARE CONTEXT SHARDING AMONG ATTENTION HEADS
Xihui Lin, Yunan Zhang, Suyu Ge, Liliang Ren, Barun Patra, Vishrav Chaudhary, Hao Peng, Xia Song
- Published: 05 Mar 2025, Last Modified: 05 Mar 2025
- SLLM
- Readers: Everyone
SpargeAttn: Training-Free Sparse Attention Accelerating Any Model Inference
Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia wei, Haocheng Xi, Jun Zhu, Jianfei Chen
- Published: 05 Mar 2025, Last Modified: 02 Apr 2025
- SLLM
- Readers: Everyone
LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation
CHEN Han, Zicong Jiang, Zining Zhang, Bingsheng He, Luo Pingyi, Mian Lu, Yuqiang Chen
- Published: 05 Mar 2025, Last Modified: 27 Apr 2025
- SLLM
- Readers: Everyone
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski, Michał Krutul, Jan Małaśnicki, Maciej Stefaniak, Piotr Sankowski, Marek Cygan, Kamil Adamczewski, Piotr Miłoś, Sebastian Jaszczur
- Published: 05 Mar 2025, Last Modified: 21 Apr 2025
- SLLM
- Readers: Everyone
PRUNING AS A DEFENSE: REDUCING MEMORIZATION IN LARGE LANGUAGE MODELS
Mansi Gupta, Nikhar Waghela, Sarthak Gupta, Shourya Goel, Sanjif Shanmugavelu
- Published: 05 Mar 2025, Last Modified: 03 Apr 2025
- SLLM
- Readers: Everyone
LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
Guangtao Wang, Shubhangi Upasani, Chen Wu, Darshan Gandhi, Jonathan Lingjie Li, Changran Hu, Bo Li, Urmish Thakker
- Published: 05 Mar 2025, Last Modified: 09 Apr 2025
- SLLM
- Readers: Everyone
Symmetric Pruning for Large Language Models
Kai Yi, Peter Richtárik
- Published: 05 Mar 2025, Last Modified: 05 Mar 2025
- SLLM
- Readers: Everyone
Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learning
Chengsong Huang, Langlin Huang, Jiaxin Huang
- Published: 05 Mar 2025, Last Modified: 31 Mar 2025
- SLLM
- Readers: Everyone
Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity
Weixin Liang, Junhong Shen, Genghan Zhang, Ning Dong, Luke Zettlemoyer, LILI YU
- Published: 05 Mar 2025, Last Modified: 30 Mar 2025
- SLLM
- Readers: Everyone
Recovery-on-the-line: Linear trends in post-quantization performance recovery
Shashata Sawmya, Shuvom Sadhuka, Ragulan Sivakumar, Nir N Shavit, Dan Alistarh, Bonnie Berger
- Published: 05 Mar 2025, Last Modified: 25 Apr 2025
- SLLM
- Readers: Everyone
Sparse Gradient Compression for Fine-Tuning Large Language Models
David H. Yang, Mohammad Mohammadi Amiri, Tejaswini Pedapati, Subhajit Chaudhury, Pin-Yu Chen
- Published: 05 Mar 2025, Last Modified: 03 Apr 2025
- SLLM
- Readers: Everyone
MobiLlama: Towards Accurate & Lightweight Fully Transparent GPT
Omkar Chakradhar Thawakar, Ashmal Vayani, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Michael Felsberg, Timothy Baldwin, Eric P. Xing, Fahad Shahbaz Khan
- Published: 05 Mar 2025, Last Modified: 20 Apr 2025
- SLLM
- Readers: Everyone