XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters

Xuanyu Zhang, Qing Yang

Abstract: Recently, with the popularity of ChatGPT, large-scale language models have experienced rapid development. However, there is a scarcity of open-sourced chat models specifically designed for the Chinese language, especially in the field of Chinese finance, at the scale of hundreds of billions. To address this gap, we introduce XuanYuan 2.0, the largest Chinese chat model to date, built upon the BLOOM-176B architecture. Additionally, we propose a novel training method called hybrid-tuning to mitigate catastrophic forgetting. By integrating general and domain-specific knowledge, as well as combining the stages of pre-training and fine-tuning, XuanYuan 2.0 is capable of providing accurate and contextually appropriate responses in the Chinese financial domain.