LeetPrompt: Leveraging Collective Human Intelligence to Study LLMs

Sebastin Santy, Ayana Bharadwaj, Sahith Dambekodi, Alex Albert, Cathy Lang Yuan, Ranjay Krishna

Keywords: evaluation of foundation models, collective intelligence

TL;DR: Leveraging collective human intelligence to solve complex reasoning problems using large language models

Abstract: With the advent of pre-trained large language models (LLMs), natural language prompts are now becoming a de-facto method for interacting with language models. However, prompting as a technique is an esoteric art, involving cumbersome manual processes by individuals to search different strategies that make language models work for the intended task. We introduce LeetPrompt, a citizen science platform that leverages on collective human creativity with prompting to solve reasoning questions across various domains. Users of \leetprompt attempt questions by writing prompts that solve all the hidden test cases. To measure the efficacy of LeetPrompt, we conduct a study $10$ questions across $5$ domains (biology, physics, math, programming, and general knowledge) with $20$ human subjects. We gather a total of $1173$ GPT-4 prompts with the following observations: First, problems deemed unsolvable by question setters were successfully solved. Second, diverse prompting strategies were used by the different participants. Third, the more difficult problems also had a high number of prompt submissions enabling better debugging of the LLM behaviour for that problem. These observations support various downstream implications in robust approaches to prompt interpretability and model evaluation, high quality data collection, human-AI alignment and real-world usage of LLMs.

Primary Area: infrastructure, software libraries, hardware, etc.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4511