ICLR 2025 Workshop SLLM Submissions

LoRA Without Forgetting: Freezing and Sparse Masking for Low-Rank Adaptation
Juzheng Zhang, Jiacheng You, Ashwinee Panda, Tom Goldstein
- Published: 05 Mar 2025, Last Modified: 11 Apr 2025
- SLLM
- Readers: Everyone
Scaling Sparse Feature Circuits For Studying In-Context Learning
Dmitrii Kharlapenko, Stepan Shabalin, Arthur Conmy, Neel Nanda
- Published: 05 Mar 2025, Last Modified: 23 Apr 2025
- SLLM
- Readers: Everyone
Faster, Cheaper, Just as Good: Cost- and Latency-Constrained Routing for LLMs
Javid Lakha, Minlan Yu, Rana Shahout
- Published: 05 Mar 2025, Last Modified: 02 Apr 2025
- SLLM
- Readers: Everyone
ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Utkarsh Saxena, Sayeh Sharify, Kaushik Roy, Xin Wang
- Published: 05 Mar 2025, Last Modified: 21 Apr 2025
- SLLM
- Readers: Everyone
Robustly identifying concepts introduced during chat fine-tuning using crosscoders
Julian Minder, Clément Dumas, Bilal Chughtai, Neel Nanda
- Published: 05 Mar 2025, Last Modified: 21 Apr 2025
- SLLM
- Readers: Everyone
Low-rank Adapting Models for Sparse Autoencoders
Matthew Chen, Joshua Engels, Max Tegmark
- Published: 05 Mar 2025, Last Modified: 21 Apr 2025
- SLLM
- Readers: Everyone
From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs
Kumari Nishu, Sachin Mehta, Samira Abnar, Mehrdad Farajtabar, Maxwell Horton, Mahyar Najibi, Moin Nabi, Minsik Cho, Devang Naik
- Published: 05 Mar 2025, Last Modified: 17 Apr 2025
- SLLM
- Readers: Everyone
On multi-token prediction for efficient LLM inference
Somesh Mehra, Javier Alonso Garcia, Lukas Mauch
- Published: 05 Mar 2025, Last Modified: 23 Apr 2025
- SLLM
- Readers: Everyone
Wanda++: Pruning Large Language Models via Regional Gradients
Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M. Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, Nathan Susanj, Zheng Zhang, Jack FitzGerald, Abhishek Kumar
- Published: 05 Mar 2025, Last Modified: 14 Apr 2025
- SLLM
- Readers: Everyone
Differentiable Attention Sparsity via Structured $D$-Gating
Chris Kolb, Bernd Bischl, David Rügamer
- Published: 05 Mar 2025, Last Modified: 02 Apr 2025
- SLLM
- Readers: Everyone
Accelerating Transformer Inference and Training with 2:4 Activation Sparsity
Daniel HAZIZA, Timothy Chou, Dhruv Choudhary, Jesse Cai, Luca Wehrstedt, Francisco Massa, Jiecao Yu, Geonhwa Jeong, Supriya Rao, Patrick Labatut
- Published: 05 Mar 2025, Last Modified: 18 Apr 2025
- SLLM
- Readers: Everyone
Antipodal Pairing and Mechanistic Signals in Dense SAE Latents
Alessandro Stolfo, Ben Peng Wu, Mrinmaya Sachan
- Published: 05 Mar 2025, Last Modified: 08 Apr 2025
- SLLM
- Readers: Everyone
Low-Rank is Required for Pruning LLMs
Stephen Zhang, Vardan Papyan
- Published: 05 Mar 2025, Last Modified: 16 Apr 2025
- SLLM
- Readers: Everyone
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
Amir Zandieh, Majid Daliri, Insu Han
- Published: 05 Mar 2025, Last Modified: 09 Apr 2025
- SLLM
- Readers: Everyone
EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Oliver Sieberling, Denis Kuznedelev, Dan Alistarh
- Published: 05 Mar 2025, Last Modified: 21 Apr 2025
- SLLM
- Readers: Everyone
Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification
Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, Kartik Ahuja
- Published: 05 Mar 2025, Last Modified: 31 Mar 2025
- SLLM
- Readers: Everyone
QuEST: Training Accurate LLMs over Highly-Compressed Weights and Activation
Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L. Castro, Mahdi Nikdan, Dan Alistarh
- Published: 05 Mar 2025, Last Modified: 04 Apr 2025
- SLLM
- Readers: Everyone
InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer
Tony Zhang, Rickard Brannvall
- Published: 05 Mar 2025, Last Modified: 31 Mar 2025
- SLLM
- Readers: Everyone
DeltaMoE: Memory-Efficient Inference for Merged Mixture of Experts with Delta Compression
Boyko Borisov, Xiaozhe Yao, Nezihe Merve Gürel, Ana Klimovic
- Published: 05 Mar 2025, Last Modified: 22 Apr 2025
- SLLM
- Readers: Everyone
The Surprising Effectiveness of Randomness in LLM Pruning
Shuyao Xu, Liu Jiayao, Zhenfeng He, Cheng Peng, Weidi Xu
- Published: 05 Mar 2025, Last Modified: 03 Apr 2025
- SLLM
- Readers: Everyone
LEWIS (LayEr WIse Sparsity) - A Training Free Guided Model Merging Approach
Hetarth Chopra, Vidhi Rambhia, Vikram S. Adve
- Published: 05 Mar 2025, Last Modified: 07 Apr 2025
- SLLM
- Readers: Everyone
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin El-Nouby, Joshua M. Susskind, Vimal Thilak
- Published: 05 Mar 2025, Last Modified: 30 Mar 2025
- SLLM
- Readers: Everyone
ChamaleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters
Kamer Ali Yuksel, Hassan Sawaf
- Published: 05 Mar 2025, Last Modified: 29 Mar 2025
- SLLM
- Readers: Everyone
How Sparse Attention Approximates Exact Attention?Your Attention is Naturally $n^C$-Sparse
Zhao Song, Jing Xiong, Chiwun Yang
- Published: 05 Mar 2025, Last Modified: 05 Mar 2025
- SLLM
- Readers: Everyone
Matryoshka Quantization
Pranav Ajit Nair, Puranjay Datta, Jeff Dean, Prateek Jain, Aditya Kusupati
- Published: 05 Mar 2025, Last Modified: 21 Apr 2025
- SLLM
- Readers: Everyone