Investigating Token-Level Supervision of Multi-Dimensional Attribute Combinations

Ziyao Xu, Jianxun Shi, Cong Wang, Tianhuang Su, Haonan Lu, Houfeng Wang

Keywords: token-level supervision, multi-dimensional attribute combinations, large language models

Abstract: In multi-dimensional attribute combination training for LLMs, the dimension conflict is an unavoidable issue. Since each token can have different influences on different dimensions, applying token-level supervision across multiple dimensions is a potential method to mitigate dimension conflicts. However, the difficulty in obtaining token-level supervision signals across multiple dimensions through annotation has hindered further investigation into supervision methods. In this work, we experimentally validate the impact of dimension conflicts on LLM training and propose a method for applying token-level supervision for multi-dimensional attribute combination training. This method establishes token-level connections between the trained model and attribute models using token sequences generated by the trained model for optimization, and controls the optimization process through entropy-based weight calculation, without requiring any additional token-level annotations or external models. This method effectively improves multi-dimensional performance and provides new insights into the investigation of token-level supervision for multi-dimensional attribute combinations.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 8603