Frequency-Augmented Mixture-of-Heterogeneous-Experts Framework for Sequential Recommendation

Junjie Zhang, Ruobing Xie, Hongyu Lu, Wenqi Sun, Xin Zhao, yu chen, Zhanhui Kang

Track: User modeling, personalization and recommendation

Keywords: Sequential Recommendation, Mixture-of-Heterogeneous-Experts

TL;DR: The paper proposes developing a novel MoE framework with heterogeneous experts, integrating their architectural biases personally to capture user diverse behavioral patterns.

Abstract:

Recently, many efforts have been devoted to building effective sequential recommenders. Despite their effectiveness, these methods typically develop a single model to serve all users. However, our empirical studies reveal that different sequential encoders have intrinsic architectural biases and tend to focus on specific behavioral patterns, \ie particular frequency range of user behavior sequences. For example, the Self-Attention module is essentially a low-pass filter, focusing on low-frequency information while neglecting the high-frequency details. This evidently limits their ability to capture diverse user patterns, leading to suboptimal recommendations.

To tackle this problem, we present FamouSRec, a Frequency-Augmented mixture-of-Heterogeneous-Experts Framework for personalized recommendations. Our approach builds an MoE-based recommender system, integrating the strengths of various experts to achieve diversified user modeling. For developing the MoE framework, as the key to our approach, we instantiate experts with various model architectures, aiming to leverage their inherent architectural biases and capture diverse behavioral patterns. For selecting appropriate experts to serve individuals, we introduce a frequency-augmented router. It first identifies frequency components in user behavior sequences that are suited for expert encoding, and then conducts customized routing based on the informativeness of these components. Building on this framework, we further propose two novel contrastive tasks to enhance expert specialization and alignment, thus further improving modeling efficacy and enabling robust recommendations. Extensive experiments on five real-world datasets demonstrate the effectiveness of our approach.

Submission Number: 1853