Go to ICML 2025 Conference homepageA Reasoning-Based Approach to Cryptic Crossword Clue Solving
TL;DR: Our LLM prover/verifier for the reasoning steps in cryptic crossword clues obtains SoTA results
Abstract: Cryptic crossword clues are challenging language tasks for which new test sets are released daily by major newspapers on a global basis. Each cryptic clue contains both the definition of the answer to be placed in the crossword grid (in common with regular crosswords), and ‘wordplay’ that *proves* that the answer is correct (i.e. a human solver can be confident that an answer is correct without needing crossing words as confirmation). This work describes an LLM-based reasoning system built from open-licensed components that solves cryptic clues by (i) hypothesising answers; (ii) proposing wordplay explanations; and (iii) using a verifier system that operates on codified reasoning steps. Overall, this system establishes a new state-of-the-art performance on the challenging Cryptonite dataset of clues from The Times and The Telegraph newspapers in the UK. Because each proved solution is expressed in Python, interpretable wordplay reasoning for proven answers is available for inspection.
Lay Summary: We're interested in a type of puzzle common in major newspapers (in the UK, and elsewhere) : Cryptic Crosswords. Each cryptic clue hints to its unique answer in two ways : a 'regular crossword definition' and 'wordplay'. Because the wordplay and definition must have the same answer, solvers know whether they've got the answer correct (even without having other answers in the crossword grid).
Our work uses a large language model (LLM) to guess possible answers, and then justify the reasoning, finally delivering its solution in the Python programming language. By getting the reasoning as a small computer program, we can easily tell whether the LLM has got the reasoning correct, and this enables our method to beat ChatGPT and other models.
Although we focused on Cryptic Crosswords, our ideas could also be applied to other linguistic tasks, opening them up to methods that are more commonly used for mathematics and programming problems.
Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.
Link To Code: https://github.com/mdda/cryptic-crossword-reasoning-verifier
Primary Area: Applications->Language, Speech and Dialog
Keywords: NLP, Cryptic Crosswords, Reasoning, Proof/Verification
Submission Number: 15105
Loading