Unlocking Chain of Thought in Base Language Models by Heuristic Instruction

Ren Zhuang, Ben Wang, Shuifa Sun, Yanlong Wang, Zhipeng Ding, Wenbo Liu

Abstract: Chain of thought (CoT) prompting drives complex reasoning in large language models (LLMs), but remains scarcely explored for smaller Base Language Models (BLMs). We pioneer the Heuristic Chain of Thought (HCoT) approach for BLMs simply via "Let’s use knowledge" prompts. Further, we bridge the lack of guidance in HCoT by innovating SPIRE, a template providing specificity, purpose, information, role & capacity and expression to efficiently apply knowledge. Experiments show that combining HCoT with the SPIRE format significantly improves BLMs performance on question answering and translation tasks after minimal tuning. For example, on SQuADv1.1, our method increases the character F1 by 4.12% over zero-shot CoT using a 41.7M parameter BERT model.