CLAWS:Creativity detection for LLM-generated solutions using Attention Window of Sections

Keuntae Kim, Eunhye Jeong, Sehyeon Lee, Seohee Yoon, Yong Suk Choi

Abstract: Recent research on the increasing of Large Language Model (LLM)'s reasoning ability has been very successful. LLMs trained with Reinforcement Learning (RL) for reasoning ability show strong performance in challenging tasks like math and coding, even at small model sizes. However, despite the remarkable increase in accuracy of task, the assessment of creativity of LLM's generations has been overlooked in the reasoning task, unlike in the writing task. The main reason why creativity assessment research has not been actively conducted in the reasoning task is that, firstly, it was difficult to define the ‘range of creativity’ and secondly, human evaluation was essential in the process of assessment creativity. To overcome these difficulties, we proposed CLAWS, a novel method that can classify mathematical solutions into ‘Typical, Creative, and Hallucinated’ solutions without human evaluation by using Attention weight by prompt section. CLAWS showed superior performance in five 7-8B math RL models (Deepseek, QWEN, Mathstral, Openmath, OREAL) than five existing white-box detection methods (Perplexity, Logit Entropy, Window Entropy, Hidden Score, Attention Score). We validated this on 4545 math problems from 181 math contests (A(J)HSME, AMC, AIME). The code will be released on github after publication.