GenEval: A framework to evaluate feasibility of domain generalization

Ayan Banerjee, Sandeep Gupta

Keywords: domain generalization, necessary sufficient condition, violation detection

TL;DR: Given a set of source domains, a method for detection of feasibility of generalization to a target domain is proposed.

Abstract: This paper proposes a novel methodology for evaluating whether a learned hypothesis is generalizable to a new domain. EvalGen extracts underlying models that represent effects of causal factors on domain data and labels and uses a novel model divergence detection mechanism based on conformal inference to evaluate significant shift in causal factors in the new domain. As such, EvalGen can predict the performance of a learned hypothesis in the new domain without the need for execution in the new domain. We evaluate EvalGen on single, multi-dimensional time series applications as well as challenging medical imaging case studies on diabetic retinopathy in both single and multi-domain generalization experiments.

Primary Area: learning theory

Submission Number: 12336