Function Basis Encoding of Numerical Features in Factorization Machines
Abstract: Factorization machine (FM) variants are widely used for large scale real-time content recommendation systems, since they offer an excellent balance between model accuracy and low computational costs for training and inference. These systems are trained on tabular data with both numerical and categorical columns. Incorporating numerical columns poses a challenge, and they are typically incorporated using a scalar transformation or binning, which can be either learned or arbitrarily chosen. In this work, we provide a systematic and theoretically-justified way to incorporate numerical features into FM variants by encoding them into a vector of function values for a set of functions of one's choice.
We view FMs as approximators of segmentized functions, namely, functions from a field's value to the real numbers, assuming the remaining fields are assigned some given constants, which we refer to as the segment. From this perspective, we show that our technique yields a model that learns segmentized functions of the numerical feature spanned by the set of functions of one's choice, namely, the spanning coefficients vary between segments. Hence, to improve model accuracy we advocate the use of functions known to have powerful approximation capabilities, and offer the B-Spline basis due to its well-known approximation power, widespread availability in software libraries and its efficiency in terms of computational resources and memory usage. Our technique preserves fast training and inference, and requires only a small modification of the computational graph of an FM model. Therefore, incorporating it into an existing system to improve its performance is easy. Finally, we back our claims with a set of experiments that include a synthetic experiment, performance evaluation on several data-sets, and an A/B test on a real online advertising system which shows improved performance. We have made the code to reproduce the experiments available at https://github.com/alexshtf/cont_features_paper.
Submission Length: Long submission (more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=NmY98pFBAo
Changes Since Last Submission: Conformity to TMLR style. Prev submission number: 3286
## Update
Revised to address reviewer comments:
- pulled FM background from the appendix, and created a 'background' section.
- formally defined what 'segmentized' means in the background section using a $\mathrm{seg}$ opeator, and use it throughout the paper.
- explained why binning is typically used with FM models in the background section.
- clarified that fitting a parametric distribution is just an efficiency enhancement over the well-known quantile transform
- in the experimental section, clarified why the data-sets were chosen, and that the task is regression.
- re-draw Figure 2 to show that x multiplies all rows of the matrix V that contains the embedding vectors.
## Camera ready update
- Addressed action editor requests, proofread, fixed citation styles.
Code: https://github.com/alexshtf/cont_features_paper
Assigned Action Editor: ~Andriy_Mnih1
Submission Number: 3297
Loading