Pushing the Limits of ChatGPT on NLP Tasks

Anonymous

Abstract:

Despite the success of ChatGPT, its performances on most NLP tasks are still well below the supervised baselines. In this work, we looked into the causes, and discovered that its subpar performance was caused by the following factors:
(1) mismatch between the generation nature of ChatGPT and NLP tasks;
(2) token limit in the prompt does not allow for the full utilization of the supervised datasets;
(3) insufficient utilization of the reasoning power of ChatGPT.
(4) intrinsic pitfalls of LLMs models, e.g., hallucination, overly focus on certain keywords, etc.

In this work, we propose a collection of general modules to address these issues, in an attempt to push the limits of ChatGPT on NLP tasks:
(1) proper task formalization to better align with the generation nature of LLMs;
(2) one-input-multiple-prompts strategy to overcome token limitations and maximize training data utilization;
(3) demonstration retrieval using fine-tuned model for k-nearest neighbor~($k$NN) search to improve the selection of semantically relevant demonstrations;
(4) chain-of-Thoughts reasoning that are tailored to addressing the task-specific complexity;
(5) self-verification to address the hallucination issue of LLMs;
(6) paraphrase voting to improve the robustness of model predictions.

We conduct experiments on 21 datasets of 10 representative NLP tasks.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: Model analysis & interpretability

Languages Studied: English, Chinese