Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment

TMLR Paper6872 Authors

Abstract:

Large Language Models (LLMs) enable advanced natural language processing but face deploy-
ment challenges on resource-constrained edge devices due to high computational, memory,
and energy demands. Optimizing these models requires addressing three key challenges: ac-
quiring task-specific data, fine-tuning for performance, and compressing models to accelerate
inference while reducing resource demands. We propose an integrated framework combining GPTQ-based
quantization, low-rank adaptation (LoRA), and a specialized data distillation process to
significantly reduce model size and complexity while preserving or enhancing task-specific
performance. By leveraging data distillation, knowledge distillation via Kullback-Leibler
divergence, Bayesian hyperparameter optimization, and the Muon optimizer, we
achieve up to 2× memory compression (e.g., reducing a 6GB model to 3GB) and enables
efficient inference for specialized tasks. Empirical results demonstrate the superior
performance on standard LLM benchmarks compared to GPTQ quantization alone, with the
Muon optimizer notably enhancing fine-tuned models’ resistance to accuracy decay during
quantization.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Changyou_Chen1

Submission Number: 6872