Analysing The Impact of Sequence Composition on Language Model Pre-Training

Anonymous

Abstract: Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use \emph{causal masking} to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and efficiency. However, to this day, the influence of the pre-training sequence composition strategy on the generalisation properties of the model remains under-explored. In this work, we find that applying causal masking can lead to the inclusion of distracting information from previous documents during pre-training, which negatively impacts the performance of the models on language modelling and downstream tasks. In \emph{intra-document causal masking}, the likelihood of each token is only conditioned on the previous tokens in the same document, which eliminates potential distracting information from previous documents and significantly improves the performance. Furthermore, we find that concatenating related documents can reduce some potential distractions during pre-training, and our proposed efficient retrieval-based sequence construction method, \textsc{Bm25}Chunk, can improve in-context learning (+11.6\%), knowledge memorisation (+9.8\%), and context utilisation (+7.2\%) abilities of language models without sacrificing efficiency.

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency

Languages Studied: English

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A2: n/a

A3: yes

B: yes

B1: n/a

B2: n/a

B3: n/a

B4: n/a

B5: n/a

B6: n/a

C: yes

D: no

E: yes