Socratic Human Feedback (SoHF): Understanding Socratic Feedback Based Steering Strategies Used by Expert Programmers for Code-generation with LLMs
Abstract: Large Language Models (LLMs) are increasingly used for generating code solutions, empowered by features like self-debugging and self-reflection. However, LLMs often struggle with complex programming problems without human guidance. This paper investigates the strategies employed by expert programmers to steer code-generating LLMs toward successful outcomes. Through a study involving experts using natural language to guide GPT-4, Gemini Ultra, and Claude Opus on highly difficult programming challenges, we frame our analysis using the "Socratic Feedback" paradigm for understanding effective steering strategies. By analyzing 30 conversational transcripts across all three models, we map observed feedback strategies to five stages of Socratic Questioning: Definition, Elenhus, Maieutic, Dialectic, and Counter-factual reasoning. We find evidence that by employing a combination of different Socratic feedback strategies across multiple turns, programmers successfully guided the models to solve 58\% of the problems that the models initially failed to solve on their own.
Paper Type: Long
Research Area: Dialogue and Interactive Systems
Research Area Keywords: Steering Strategy; Socratic Feedback; LLM
Contribution Types: NLP engineering experiment
Languages Studied: Python
Submission Number: 1584
Loading