Injecting Sensitivity Constraint Into Continual Learning Significantly Enhances Surrogate-Aided Optimization

Doaa Aboelyazeed, Leo Lonzarich, Abdolmehdi Behroozi, Chaopeng Shen

Keywords: Optimization, Surrogate models, Online continual Learning, Sensitivity Constraints

TL;DR: Here we propose a sensitivity-constrained online training framework that jointly improves parameter learning and prediction accuracy with high efficiency

Abstract:

A myriad of scientific and engineering optimization and learning tasks involve
running a numerical model to guide optimization directly or generate training
data for function mapping algorithms. Surrogate models can greatly accelerate
these tasks, but they often fail to capture the true input-output relationships (sensi-
tivities) so they lose the ability to guide high-dimensional and long-horizon op-
timization. Online continual learning (OCL) – iteratively obtaining numerical
results to continue training the surrogate – can mitigate this issue, but may still
be insufficient. Here we propose scheduled injection of sensitivity constraints
(SC, matching the Jacobian of the surrogate model with that of the true numer-
ical model) for the surrogate into OCL to enforce realistic output-parameter re-
lationships. We evaluate this approach across diverse datasets and optimization
frameworks where continual surrogate training is used: (1) multi-objective multi-
fidelity surrogate-assisted Bayesian optimization and Pareto front exploration; (2)
hybrid end-to-end training of coupled neural networks and process-based mod-
els; and (3) a modified unifying framework for generative parameter inversion
and surrogate training. Across all of these tasks, inserting SC accelerates the de-
scent to optimality and consistently improves the main optimization outcome, as
it critically improves the future trajectory of optimization. OCL improves data
relevance and SC ensures sensitivity fidelity, and they together produces an ef-
ficient surrogate model that almost achieves the same effect as the full physical
model, only achievable by OCL+SC. It consistently outperform pretrained-only
surrogate models with SC or OCL without SC, not to mention the pretrained-only
model without SC, so the benefits of two procedures reinforce each other. Even
infrequent surrogate finetuning with SC injection (once every 5 epochs) can in-
duce large benefits in optimization outcome. Together, these results demonstrate
the possibility to enable large-scale optimization of complex systems for big-data
learning and knowledge discovery

Primary Area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Submission Number: 15467