Privacy Awareness for Information-Sharing Assistants: A Case-study on Form-filling with Contextual Integrity
Abstract: Advanced AI assistants combine frontier LLMs and tool access to autonomously perform complex tasks on behalf of users. While the helpfulness of such assistants can increase dramatically with access to user information including emails and documents, this raises privacy concerns about assistants sharing inappropriate information with third parties without user supervision. To steer information-sharing assistants to behave in accordance with privacy expectations, we propose to operationalize the design of privacy-conscious assistants that conform with *contextual integrity* (CI), a framework that equates privacy with the appropriate flow of information in a given context. In particular, we design and evaluate a number of strategies to steer assistants' information-sharing actions to be CI compliant. Our evaluation is based on a novel form filling benchmark composed of human annotations of common webform applications, and it reveals that prompting frontier LLMs to perform CI-based reasoning yields strong results.
Certifications: Reproducibility Certification
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Changes include:
* Adjusted the title/introduction to emphasize form-filling as a case study.
* Added paragraphs in synthetic form and personas generation to stress its limitations compared tor real data, how we ensured realism and why we opted against using real-world data
* Added paragraph in beginning of the related work on comparison to differential privacy and training-data leakage
* Added final paragraph on susceptibility to jailbreaking as a limitation and future work.
Assigned Action Editor: ~Li_Erran_Li1
Submission Number: 3891
Loading