Automated Evaluation of the Linguistic Difficulty of Short Texts for LLM Applications

Anonymous

Abstract: There is an unmet need to evaluate the language difficulty of short passages of text, particularly for training and filtering Large Language Models (LLMs). Existing datasets fail to train models for this task, so we introduce ShortDiff, a new dataset with 890 short text passages in English together with their level of text difficulty. We experiment with a variety of models on ShortDiff, including finetuning Transformer-based models and prompting LLMs. Our best model achieves accuracy surpassing human experts and has latency appropriate to production environments. Finally, we release the ShortDiff dataset to the public for further research and development.

Paper Type: long

Research Area: Machine Learning for NLP

Contribution Types: Data resources, Data analysis

Languages Studied: English