FinMegatron: Large Financial Domain Language Models

Xianchao Wu

Abstract:

General domain pretrained large-scale language models, such as BERT and GPT3, have
achieved state-of-the-art results among numerous NLP classification and generation applications. This pretraining technology is also
willing to be used in vertical domains, such
as finance. The downstream applications include financial event extraction from news,
summarization, and causal inferencing. In
this paper, we propose large-scale pretrained
BERT models for financial domain in English and Japanese languages. The original datasets come from professional financial
news. We empirically study the factors of
sub-word vocabulary set, model size and their
impacts to the downstream financial NLP applications. The code and pretrained models
are released from https://github.com/
NVIDIA/Megatron-LM.