Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism
Abstract: Neural Information Retrieval (NIR) has significantly improved upon heuristic-based Infor- mation Retrieval (IR) systems. Yet, failures remain frequent, the models used often being unable to retrieve documents relevant to the user’s query. We address this challenge by proposing a lightweight abstention mechanism tailored for real-world constraints, with par- ticular emphasis placed on the reranking phase. We introduce a protocol for evaluating abstention strategies in black-box scenarios (typically encountered when relying on API services), demonstrating their efficacy, and propose a simple yet effective data-driven mech- anism. We provide open-source code for experiment replication and abstention implemen- tation, fostering wider adoption and application in diverse contexts.
Submission Length: Regular submission (no more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=tMNvfhA0PP&nesting=2&sort=date-desc
Changes Since Last Submission: Changed the font type to fit formatting requirements.
Assigned Action Editor: ~Pavel_Izmailov1
Submission Number: 2457
Loading