Personalized Transformers for Everyone

Anonymous

Abstract: Personalized Intelligence (PI) is the problem of providing customized AI experiences tailored to each individual user. A related problem is the compartmentalization of intelligence that maintains a partition between the personalized and the general models. Existing personalization approaches involve fine-tuning pre-trained models to create new customized models. However, these require a significant amount of computation to train, which scales with model size and the number of users, inhibiting PI to be realized widely. A compartmentalized approach enables a small model to be specialized for each individual user, which needs to be used together with a larger model to provide personalization. By separating personalized and general models, we enable higher accuracy, scalability, and stronger privacy guarantees. In this paper, we aim to design a compartmentalized personalization approach that can scale to millions of users and beyond. We investigate the landscape of model fine-tuning techniques and construct new design adaptations based on the requirements of PI. We then introduce Personalized Head (PH), a new model training/inference framework designed for scalable PI. We explore the design space of these techniques and evaluate their efficacy under various production-level constraints. Specifically, we break down the trade-off between accuracy, scalability and production deployment limitations. We present several production-ready personalization approaches suited for various production use case scenarios.

Paper Type: long