A large-scale study of the effect of training set characteristics over learning-to-rank algorithms

Evangelos Kanoulas, Stefan Savev, Pavel Metrikov, Virgiliu Pavlu, Javed A. Aslam

Abstract: In this work we describe the results of a large-scale study on the effect of the distribution of labels across the different grades of relevance in the training set on the performance of trained ranking functions. In a controlled experiment we generate a large number of training datasets wih different label distributions and employ three learning to rank algo- rithms over these datasets. We investigate the effect of these distributions on the accuracy of obtained ranking functions to give an insight into the manner training sets should be constructed.