Coping with Big Data in Transfer Optimization

Mojtaba Shakeri, Abhishek Gupta, Yew-Soon Ong, Xu Chi, Allan Zhang NengSheng

Abstract: Transfer optimization is an emerging concept that promises to enhance productivity of planning and decision-making processes by allowing for the adaptive reuse of knowledge (data) drawn from various “source” problems in a related ongoing “target” task of interest. Despite the recent advances in transfer optimization, however, a continuing challenge is the scalability of associated algorithms given big data of source problem instances. This paper tackles the scaling problem of an online adaptive knowledge transfer framework under big source data. We propose an efficient source selection algorithm based on the theory of multi-armed bandits such that the most related source task to the target is chosen for knowledge transfer, as opposed to extracting knowledge from all sources simultaneously. For this purpose, we introduce a novel and principled reward measure to reflect the source-target similarities. The efficacy of our proposed approach is assessed on the well-known knapsack problem that has practical implications in optimization of supply chain and manufacturing processes. Extensive experiments are conducted under big data of source problem instances. The numerical results clearly reveal that the incorporation of the proposed source selection mechanism in the existing adaptive knowledge transfer framework makes it successfully feasible for fast/real-time decision-making in the big data source setting.