Are We Forgetting about Compositional Optimisers in Bayesian Optimisation?

Antoine Grosnit, Alexander Imani Cowen-Rivers, Rasul Tutunov, Ryan-Rhys Griffiths, Jun Wang, Haitham Bou Ammar

Link To Paper: https://jmlr.org/papers/v22/20-1422.html

Journal Of Paper: JMLR

Confirmed Open Access: Yes

Topics From Call For Papers: Bayesian Optimization for AutoML, Hyperparameter Optimization (HPO)

Broader Impact Statement On Ethical And Societal Implications:

In our work, we conduct an empirical study on acquisition function optimisation methods, and derive a new form of acquisition function amenable to compositional optimisation, which tends to accelerate the convergence of the Bayesian optimisation (BO) algorithm. This new compositional form is mostly agnostic to the specific BO application, and can be applied in a wide range of BO frameworks (batch-BO, multi-fidelity BO, latent space BO, etc.). Therefore, the societal and ethical impacts of our contribution are heavily dependent on the nature of the problems solved with BO.

We start by noting that beneficial applications of BO are thick on the ground, ranging from the discovery of new antiviral drugs, to the optimisation of data-center cooling system parameters to reduce energy consumption, or the design of new highly-resistant materials. Emergence of new pandemics -- which could be accelerated by the global warming that entails population displacements, and accelerates the melt of the permafrost threatening to unleash ancient viruses -- makes the ability to discover cures, such as antiviral drugs, in a short time more desirable. Therefore, accelerating convergence of BO, through more efficient acquisition function optimisation, could lead to faster drug discoveries favoring the containment of the disease. Another by-product of climate change is the need to make every system energetically efficient to massively curb the use of fossil fuels. The adoption of BO to tune the parameters of energy glutton systems such as data-centers cooling systems, has proven successful in real-world applications. Making BO even more efficient could further reduce the amount of energy spent on cooling data-centers, but could also reduce the time spent on the tuning of the hyperparameters of very large neural networks, also a very high energy-demanding process.

From the latter example we can perceive that, even though the better tuning of neural networks can make them more accurate or efficient to accomplish their tasks, the positive or negative impact of our work fully depends on the nature of the tasks considered. So, improving BO could lead to a more accurate tumor detection model, as it could allow the design of a more racially biased recognition system. However we hope that our contribution alone, as it is incremental, will not in itself encourage individuals to design new malicious models. The same goes for the drug discovery case, where instead of discovering new beneficial antigens in a faster way, one could accelerate the search for new dangerous molecules.
Nevertheless, we believe that considering the fundamental challenges we face, starting with the global warming calling for massive optimisation efforts as we showed, the overall impact of our work will, in the long run, be positive for the global economy and the well-being of the inhabitants of our planet.