Advancing Sequential Numerical Prediction in Autoregressive Models

ACL ARR 2025 February Submission2113 Authors

Abstract: Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces \textit{\textbf{N}umerical \textbf{T}oken \textbf{I}ntegrity \textbf{Loss} (NTIL)} to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.

Paper Type: Short

Research Area: Generation

Research Area Keywords: LLM,analysis,Autoregressive Models

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 2113