Testing Cross-Lingual Text Comprehension in LLMs Using Next Sentence Prediction.

ICLR 2026 Conference Submission13402 Authors

Keywords: Next Sentence Prediction, Large Language Models, Cross-Lingual NLP, Text Comprehension, Low-Resource Languages, Chain-of-Thought Prompting, Semantic Similarity, Perplexity, Educational NLP, Benchmarking

Abstract: While large language models are trained on massive datasets, this data is heavily skewed towards English. Does their impressive performance reflect genuine ability or just this data advantage? To find out, we tested them in a setting where they couldn’t rely on data abundance: low-resource languages. Building on prior work (Agarwal et al., 2025) that used Next Sentence Prediction (NSP) as a test, we created a large-scale benchmark with 10,000 questions each for English (a high-resource language), Swahili (medium-resource), and Hausa (low-resource). We then tested several top models, including GPT-4 Turbo, Gemini 1.5 Flash, and LLaMA 3 70B, to see how their performance holds up. The results painted a clear picture of how levels of language resources impact outcomes. While all models excelled in English, their accuracy dropped in Swahili and fell sharply in Hausa, with LLaMA 3 struggling the most. The story became even more interesting when we introduced Chain-of-Thought (CoT) prompting. For the struggling LLaMA 3, CoT acted as a helpful guide, significantly boosting its accuracy. However, for the more capable GPT-4 and Gemini, the same technique often backfired, leading to a kind of “overthinking” that hurt their results in the cross-lingual context. This reveals that Chain-of-Thought is not a universal solution; its effectiveness depends heavily on the model’s baseline capability and the specific context of the task. Our framework pinpoints LLM weaknesses, highlights when CoT helps or hinders cross-lingual NSP performance, and factors influencing their decisions.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 13402