PrivacyGLUE: A Benchmark Dataset for General Language Understanding in Privacy Policies

Anonymous

Abstract: Benchmarks for general language understanding have been rapidly developing in recent years of NLP research, with well-known examples such as GLUE and SuperGLUE. While benchmarks have been proposed in the legal language domain, virtually no such benchmarks exist for privacy policies despite their increasing importance in modern digital life. This could be explained by privacy policies falling under the legal language domain, but we find evidence to the contrary that motivates a separate benchmark for privacy policies. Consequently, we propose PrivacyGLUE as the first comprehensive benchmark of relevant and high-quality privacy tasks for measuring general language understanding in the privacy language domain. Furthermore, we release performances from the BERT, RoBERTa, Legal-BERT, Legal-RoBERTa and PrivBERT transformer language models and perform model-pair agreement analysis to detect PrivacyGLUE task examples where models benefited from domain specialization. Our findings show PrivBERT outperforms other models by an average of 2-3% over all PrivacyGLUE tasks, shedding light on the importance of in-domain pretraining for privacy policies. We believe PrivacyGLUE can accelerate NLP research and improve general language understanding for humans and AI algorithms in the privacy language domain.

Paper Type: long

Research Area: Resources and Evaluation