HyperSyn: Synthesizing Instance-wise Model by Fusing Blackbox Expert via Hypernetwork

Ken Cheong, William K. Cheung, Ivor Tsang

Keywords: Hypernetwork, Ensembling, deep learning

Abstract: Pretrained experts are now ubiquitous, encouraging their ensembling to achieve improved performance. However, in many scenarios, they are exposed only through prediction APIs, creating black-box settings where weights and internal representations are unavailable. Existing black-box ensembling methods often perform poorly under out-of-domain conditions, since they rely solely on expert outputs. To highlight this limitation, we identify three types of data regions and show how current methods fail in certain cases. To address these challenges, we propose HyperSyn, a deep learning framework that synthesizes an instance-specific model for each data point using expert outputs as input to a hypernetwork. HyperSyn naturally fits the black-box setting and provides greater expressiveness, particularly when existing experts fail on unseen test domains. Extensive experiments on both synthetic and real-world datasets demonstrate that HyperSyn outperforms commonly used ensemble techniques and achieves state-of-the-art performance when the data exhibit complex and unknown domain structure.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 10679