CROSS-JEM: Accurate and Efficient Cross-encoders for Short-text Ranking Tasks

Bhawna Paliwal, Deepak Saini, Mudit Dhawan, Siddarth Asokan, Nagarajan Natarajan, Surbhi Aggarwal, Pankaj Malhotra, Jian Jiao, Manik Varma

Keywords: information retrieval, efficient architectures, ranking algorithms, cross encoders, sponsored search, industrial applications, large-scale machine learning

TL;DR: A novel listwise ranking approach which establishes a new state-of-the-art for short-text ranking tasks and can support real-time inference for large-scale recommendation systems

Abstract: Ranking a set of items based on their relevance to a given query is a core problem in search and recommendation. Transformer-based ranking models are the state-of-the-art approaches for such tasks, but they score each query-item independently, ignoring the joint context of other relevant items. This leads to sub-optimal ranking accuracy and high computational costs. We propose Cross-encoders with Joint Efficient Modeling (CROSS-JEM), a novel ranking approach that enables transformer-based models to jointly score multiple items for a query, maximizing parameter utilization. CROSS-JEM leverages (a) redundancies and token overlaps to jointly score multiple items (short-text phrases in search and recommendations), and (b) a novel training objective that models ranking probabilities. CROSS-JEM achieves state-of-the-art accuracy on publicly available ranking benchmarks with over 4x-lower ranking latency compared to the baselines.

Submission Number: 49