## 2.1 General Pre-training Corpora

The general pre-training corpora are large-scale datasets composed of extensive text from diverse domains and sources. Their primary characteristic is that the text content is not confined to a single domain, making them more suitable for training general foundational models. As illustrated in Figure 3, the data types can be categorized into eight major classes: Webpages , Language Texts , Books , Academic Materials , Code , Parallel Corpus , Social Media , and Encyclopedia . The collected and organized information about general pre-training corpora is presented in Table 1 and Table 2.