Don’t Say No: Jailbreaking LLM by Suppressing Refusal

Anonymous

Abstract: Ensuring the safety alignment of Large Language Models (LLMs) is crucial to generating responses consistent with human values. Despite their ability to recognize and avoid harmful queries, LLMs are vulnerable to "jailbreaking" attacks, where carefully crafted prompts elicit them to produce toxic content. One category of jailbreak attacks is reformulating the task as adversarial attacks by eliciting the LLM to generate an affirmative response. However, the typical attack in this category GCG has very limited attack success rate. In this study, to better study the jailbreak attack, we introduce the \textit{DSN} (Don't Say No) attack, which prompts LLMs to not only generate affirmative responses but also novelly enhance the objective to suppress refusals. In addition, another challenge lies in jailbreak attacks is the evaluation, as it is difficult to directly and accurately assess the harmfulness of the attack. The existing evaluation such as refusal keyword matching has its own limitation as it reveals numerous false positive and false negative instances. To overcome this challenge, we propose an ensemble evaluation pipeline incorporating Natural Language Inference (NLI) contradiction assessment, external LLM evaluation, as well as refusal matching. Extensive experiments demonstrate the potency of the \textit{DSN} and the effectiveness of ensemble evaluation compared to baseline methods.

Paper Type: long

Research Area: Generation

Languages Studied: English

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: 6

A2: yes

A2 Elaboration For Yes Or No: 7

A3: yes

A3 Elaboration For Yes Or No: 1

B: no

B1: n/a

B2: n/a

B3: n/a

B4: n/a

B5: n/a

B6: n/a

C: yes

C1: n/a

C2: n/a

C3: n/a

C4: n/a

D: yes

D1: n/a

D2: n/a

D3: n/a

D4: n/a

D5: n/a

E: no

E1: n/a