Building Japanese Creativity Benchmarks and Applying them to Enhance LLM Creativity

So Fukuda, Hayato Ogawa, Kaito Horio, Daisuke Kawahara, Tomohide Shibata

Keywords: LLM, creativity

TL;DR: This paper introduces three Japanese creativity benchmarks for LLMs - a comprehensive one (JCQ) and two efficient ones (DAT and SAT) - and investigates whether training with the DAT benchmark can enhance LLM creativity.

Abstract: To evaluate the creativity of large language models (LLMs) in Japanese, we construct three benchmarks: Japanese Creativity Questions (JCQ), Divergent Association Task (DAT), and Story Alteration Task (SAT). JCQ comprehensively evaluates creativity using LLMs. Meanwhile, DAT and SAT measure specific aspects of creative ability using embeddings. We also analyze correlations between JCQ and DAT, JCQ and SAT, and DAT and SAT. While JCQ provides comprehensive evaluation, it is relatively time and resource intensive. In contrast, DAT and SAT offer lower comprehensiveness but enable quick, low-cost assessment. Additionally, we investigate whether training with DAT contributes to enhancing LLM creativity.

Archival Status: Archival

Paper Length: Long Paper (up to 8 pages of content)

Submission Number: 257