Go to NeurIPS 2024 Conference homepageOn the Identifiability of Hybrid Deep Generative Models: Meta-Learning as a Solution
The interest in leveraging physics-based inductive bias in deep learning has resulted in recent development of hybrid deep generative models (hybrid-DGMs) that integrates known physics-based mathematical expressions in neural generative models. To identify these hybrid-DGMs requires inferring parameters of the physics-based component along with their neural component. The identifiability of these hybrid-DGMs, however, has not yet been theoretically probed or established. How does the existing theory of the un-identifiability of general DGMs apply to hybrid-DGMs? What may be an effective approach to consutrct a hybrid-DGM with theoretically-proven identifiability? This paper provides the first theoretical probe into the identifiability of hybrid-DGMs, and present meta-learning as a novel solution to construct identifiable hybrid-DGMs. On synthetic and real-data benchmarks, we provide strong empirical evidence for the un-identifiability of existing hybrid-DGMs using unconditional priors, and strong identifiability results of the presented meta-formulations of hybrid-DGMs.
Paper Decision
The manuscript discusses the identifiability of hybrid deep generative models, and provides theoretical and experimental evidence that hybrid DGMs with unconditional priors are unidentifiable, but that identifiable hybrid DGMs can be constructed via meta-learning.
All reviewers found the topic interesting, but found the focus on the identifiability of the data-driven component and the experimental focus on simple physical systems with few numbers of parameters limiting. Since the authors responded to these points adequately during the rebuttal, the reviewers accordingly raised their scores and eventually recommend acceptance. Since some of the reviewers also criticized the presentation of the paper, I strongly suggest to revise the text and structure for the camera ready submission.
Author Rebuttal by Authors
We clarify the reviewers’ main questions about the theoretical contribution of this work as follow:
- Identifiability of the physics-based component in hybrid-DGMs: The reviewers questioned our motivation for focusing on the un-identifiability of the neural (instead of the physics) component. We do not focus on the physics component because 1) existing works (e.g., [1-2,5] cited in the manuscript) have proposed promising solutions in accurate estimation of the physics parameters , and 2) the physics-component does not suffer from the type of theoretical un-identifiability as the neural-component does – as shown in Table 1 of the attached pdf, with or without the meta-formulation, the estimation of physics parameters saw good accuracy. In comparison, the identifiability of the neural component has a much more significant effect on the performance of the hybrid-DGM (Table 1/Figs 1-2 in the manuscript). Regardless, as a side benefit of improved identification of the neural component, our approach did result in a higher accuracy of the physics parameter estimation as well (Table 1).
- Un-identifiability of the neural component in hybrid-DGMs: We respectfully disagree with Reviewer HAHc on the (in)significance of the identifiability of the neural component. The goal of a hybrid-DGM is to use the data-driven part to learn the “missing physics” (those not explained by our prior knowledge) from data – to correctly identifying this missing component, therefore, is critical even if the model itself is a neural-net (for the same reason, there is an active body of research on the identifiability for non-hybrid DGMs even though the model is data-driven). Left unaddressed, many different solutions of a hybrid-DGM can fit the reconstruction accuracy equally well, but many – due to the non-identifiable neural component – will have minimum to no predictive power as a hybrid-DGM: in addition to our strong results in Table 1/Figs 1-2 in the manuscript, we add additional evidence (thanks to Reviewer 5WN4’s comments) to show that the un-identifiable neural component will significantly reduce the ability of the hybrid model to predict over longer time intervals (Fig 1 in the attached pdf) or to perform in OoD situations (Fig 2 in the attached pdf). We believe these results (with theoretical support) provide strong evidence about the significance of constructing identifiable neural components in a hybrid-DGM – a critical gap in the current literature.
- Meta-learning as a formulation to construct identifiable DGMs: We acknowledge that the presented theory heavily draws on [6]. As a seminal work in nonlinear ICA and identifiable DGMs, however, [6] serves as the theoretical foundation for a line of follow-up identifiability theorems [a-c] where, similar to the presented work, the focus is on how to construct conditionally independent generative models based on the theory in [6]: in most existing works, this conditioning leverages observed auxiliary variables such as class labels. In this regard, our work is the first to theoretically show that the meta-formulation of DGMs enables the construction of conditionally independent models, with theoretically-proven conditions for their identifiability (see Theorem 1). This connection, though may now appear intuitive given the Theorem presented in this manuscript, is a novel contribution to general non-hybrid DGMs that has never been constructed in the existing literature. In Table 2 of the attached pdf, we add identifiability results of the meta-formulation of a non-hybrid VAE compared to the identifiable-VAE in [6] that uses class label as the auxiliary variable. [a] Klindt et al, Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding, ICLR 2021 [b] Halva et al, Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA, NeurIPS 2021 [c] Yao et al, Temporally Disentangled Representation Learning, NeurIPS 2022
- The number of parameters to be identified: Reviewer 1aic questioned about the identifiability of the presented hybrid-DGMs as the number of parameters increases. We would like to clarify that 1) The number of parameters considered in our work followed the standard practice used in published hybrid-DGMs (see [1-2, 5] cited in the manuscript: the maximum number of unknown parameters to be identified was 3 in [1],4 in [2], and 3 in [5]), and 2) Theorem 1-iv) in the manuscript did already specify the identifiability condition for the presented model dependent on the number of parameters to be identified: for a d-dimensional parameter vector following exponential families of distributions with e-dimensional sufficient statistic, its theoretical identifiability is guaranteed as long as we have observations generated from a minimum of de+1 unique parameter distributions (regardless how large the value of d may be) – in Fig 3 of the attached pdf, we empirically verify this theoretical condition on the Pendulum system with 3 unknown parameters in the neural component. Additionally. In Table 3, we show that the performance of the presented hybrid-DGM is minimally affected by the increase in the number of parameters to be identified (on the Pendulum system). In Fig 4, we add results from a complex system of positron emission tomography (PET), where activity images x are generated from radiotracer kinetics governed by 2-tissue compartment model with pixel-wise kinetic parameters: the data-generating compartment model has 4 unknown parameters in each region of interest (ROI), with 5 ROIs in total. The meta-hybrid-DGM (using simplified 1-tissue compartment model as the prior physics) demonstrated strong identifiability results as measured by the MCC.
Official Review of Submission17125 by Reviewer 1aic
The original contribution for his manuscript is a metalearning framework to identify the physical parameters in a hybrid deep generative models.
This article starts with a good question about the identifiability of physical parameters in hybrid deep generative models (that include physics as an inductive bias). Solving this question could answer key questions in data-driven knowledge discovery. This article presents meta learning framework which could help identify the physical parameters statistically.
This article seems to be a work in progress. The text alone is not an end product. Symptomatically of this work in progress, the central concept of this article of identifiability is misspelled more than 10 times throughout. The contributions of this paper are not clearly stated and hard to find. It seems that the theory stated in this article was already published elsewhere, as illustrated by the reference to another paper for the proof of the only theorem of this article.
The proof-of-concept results on pendulum models and the reaction-diffusion equation have very few parameters to identify and do not inform that the method would be useful in more realistic or more complex models. Could you provide examples with more parameters to identify? Could you give theoretical support to how fast your method would converged to the identified quantities of interest as the number of parameters to identify increases?
Limitations are discussed in the text.
Rebuttal by Authors
We thank Reviewer 1aic for the critical comments. Please see the overall response 4 (and the corresponding results in the attached pdf) for clarification regarding the effect of the number of parameters to be identified, and the overall response 1-3 (and the corresponding results in the attached pdf) for the clarifications on the contribution of the presented work.
Replying to Rebuttal by Authors
Official Comment by Reviewer 1aic
The authors made a convincing case to answer the concerns about the theoretical contribution of this work. They also clearly addressed the question about the scaling of this method with the number of parameters to identify.
I updated the soundness and contribution scores to 3, and the rating to 5.
Replying to Official Comment by Reviewer 1aic
Official Comment by Authors
We're glad that we were able to resolve your concerns. We are thankful for your constructive feedback which has helped significantly improve the quality of our work.
Official Review of Submission17125 by Reviewer 1hrQ
A method and a theory of learning hybrid deep generative models (esp., VAEs) are proposed. The method is based on meta-learning, and the theory is about the identifiability of the neural component's latent variable. The method is also empirically compared to baseline methods of hybrid DGMs.
- In the introduction, the motivation for analyzing the identifiability of hybrid DGMs is clearly stated.
- The potential connection to the study around nonlinear ICAs sounds reasonable and interesting.
- The method is technically reasonable. To my knowledge, meta-learning has not clearly been applied in the context of hybrid modeling.
- Experiments are done with reasonable, yet mostly synthetic, datasets and with adequate baseline methods.
(1) it is unclear what is the theoretical contribution particularly by this work. The theory in Section 5 looks like a mere rehash of what was presented in [6]. Also, the theory in Section 5 focuses only on the neural network part of hybrid DGMs, and the presence of the physics part of the model does not seem to affect the discussion. This limits the significance of the theoretical contribution.
Moreover, as the authors pointed out, the un-identifiability of hybrid DGMs stems both from 1) the un-identifiability of the neural component and 2) the over-powering effect of the neural component. The authors say that they focus on 1), but then the discussion becomes simply about non-hybrid, general DGMs, which makes the authors' claim that they theoretically analyzed the identifiability of hybrid DGMs questionable.
Minor points:
- In Eq. (13), the prior of $z_P$ is missing.
- What makes the synthetic and real double pendulum data different? E.g., is there supposed to be any source of non-negligible noises?
Please elaborate on the point (1) in the Weaknesses section above.
Limitations nicely discussed.
Rebuttal by Authors
We thank Reviewer 1hrQ for the constructive comments.
- Contribution: Please see the overall response 1 for our rationale for not focusing on the identifiability of the physics-component, response 2 for the significance for the (un)identifiability of the neural component that we focused on, and response 3 for the theoretical contribution of our meta-formulation of identifiable-DGMs in general (applies to hybrid or non-hybrid DGMs). We have added new results in the attached pdf to support each of these responses.
- Difference between synthetic and real double pendulum data: In the synthetic pendulum data, we know that the source of error in the prior physics is due to missing the friction and external force terms. In the real double pendulum data, the true governing physics is unknown. The sources of unknown factors include the unknown masses of the two pendulums (assuming to be equal), presence of friction and its associated parameters, potential vibrations or errors in extracting the arms’ positions from the videos, as described in [5] as cited in the manuscript.
- Typo in Equation (13): Thanks for pointing this out. The prior term of $p(z_p)$ is missing. We will add it in the final manuscript.
Replying to Rebuttal by Authors
Official Comment by Authors
Dear Reviewer 1hrQ,
Thank you again for your constructive feedback. We hope that you have had time to go through our response in addressing your previous comments. As the author discussion period is closing soon, we would like to follow up to see if you have additional comments for further discussion.
Best regards,
Authors
Replying to Rebuttal by Authors
Official Comment by Reviewer 1hrQ
Thanks for the clarification. I think it helps a lot to strengthen the discussion. Please make sure to fully reflect the additional explanation given in the rebuttal. Assuming the paper will be updated accordingly I raised my score.
Official Review of Submission17125 by Reviewer HAHc
This paper discusses the identifiability of Hybrid Deep Generative Models and proposes a Meta-Learning-based approach to address the (un)identifiability issue of current DGMs. Theoretical discussions are provided, and experiments are conducted to verify the proposed method.
This paper adopts the result from [1] gives a discussion on the indetifiability issue of Hybrid Deep Generative Models's data driven part.
[1] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.
In "Augmenting Physical Models with Deep Networks," identifying the physical part and the data-driven part is important, as mentioned in previous work such as [1]. As mentioned by the authors, this paper focuses on the identifiability issue of the latent variables and the model parameters of the data-driven model. However, the motivation for identifying this part is unclear since it does not have any physical interpretation. The main goal of the data-driven part is more about approximation accuracy rather than interpretability (identifiability). Therefore, the reviewer is not fully convinced by the motivation.
The authors also agree that there are two sides to the identifiability issue: (1) physical parameters and parameters from the data-driven part, and (2) the identifiability of the parameters in the data-driven part. This paper focuses on the latter. However, assuming there is no identifiability issue with (1), the problem degrades to the same setting as [2]. Therefore, the theoretical discussion seems borrowed from paper [2], without any revisions.
The writing needs to be further improved, including the definition of notations.
[1] Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124012, 2021.
[2] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.
In Definition 2, what is $T(\cdot)$? Is that the sufficient statistic? Please define it explicitly.
In Theorem 1, in assumption (ii), what is $\mathcal{F}\theta$? What is the difference between $\mathcal{F}\theta$ and $\mathcal{F}(f_P, f_{N_\theta}; Z_P, Z_N)$?
The major limitation is the motivation of this work, as the reviewer mentioned in the weakness section. Another weakness is the lack of technical contribution.
Rebuttal by Authors
We thank Reviewer HAHc for the constructive comments.
- Contribution: Please see the overall response 2 for the significance for the (un)identifiability of the neural component that we focused on, and response 3 for the theoretical contribution of our meta-formulation of identifiable-DGMs in general (applies to hybrid or non-hybrid DGMs). We have added new evidence in the attached pdf to support each of these responses.
- Yes, T denotes the sufficient statistics.
- Both $\mathcal{F_\theta}$ and $\mathcal{F}(f_P, f_{N_\theta}; z_P, z_N)$ denote the hybrid generative/mixing function.
Replying to Rebuttal by Authors
Official Comment by Authors
Dear Reviewer HAHc,
Thank you again for your constructive feedback. We hope that you have had time to go through our response in addressing your previous comments. As the author discussion period is closing soon, we would like to follow up to see if you have additional comments for further discussion.
Best regards,
Authors
Replying to Rebuttal by Authors
Update
Dear Authors,
Many thanks for your reply. The authors' reply has addressed the concern of the reviewer on the motivation of this study (question (1)). I would like to increase my rating from 4 to 5.
Official Review of Submission17125 by Reviewer 5WN4
The paper thoroughly assesses the issue of identifiability of both the neural and physics components in hybrid deep generative models (hybrid-DGMs), and proposes a novel approach to formulate such models using meta-learning. The authors show the performance of the model in comparison to other hybrid-DGMs and the ground truth for three well-known physics examples.
The paper is the first theoretical study of the identifiability of hybrid-DGMs.
Possible limitations of applicability related to specifics of meta-learning.
How resource heavy this method is compared to the other methods used in the comparison? How well does it perform for OOD testing in terms of robustness and generalisation? How well it does predict the behaviour of dynamical systems over longer time intervals (is there a performance degradation over time)? Would meta learning using the Wasserstein distance instead of the KL divergence improve the identification of both the physics and the neural components? As suggested by the authors themselves this work would benefit from in-depth hyperparameters to performance study, particularly taking into account the relationship between data samples and model performance.
Comments:
- as a minor point - there are some references that didn't compile as well as typos in the manuscript.
- another minor point is that the appendix is missing with the default NeurIPS instructions left in place - this can be removed.
The authors didn't provide a separate limitations section but discussed the limitations of their work and future directions in the Discussion/Conclusion section.
Rebuttal by Authors
We thank Reviewer 5WN4 for the supportive and constructive comments.
Resources: On the same device and dataset, Hybrid-VAE takes 250.24s for 50 epochs and Meta-Hybrid-VAE takes 291.68s. In all experiments, Meta-Hybrid-VAE requires approximately 0.5 times more epochs to converge.
Performance over longer time intervals: We thank the reviewer for this suggestion. Please see the overall response 3 and the added results in the attached pdf. The results showed that the presented identifiable hybrid-DGMs are significantly more robust in its performance of predicting over longer time intervals (while the non-identifiable baselines deteriorated rapidly over longer time intervals).
Performance in OoD data: We thank the reviewer for this suggestion. Please see the overall response 3 and the added results in the attached pdf. The results showed that the presented identifiable hybrid-DGMs are significantly more accurate than the non-identifiable baselines in data samples that are out of distribution either due to the physics-component or neural component.
Additional ablation studies: In the manuscript we showed that the meta-formulation is critical to the identification of the neural component. In the added results in the attached pdf (Table 1), we further ablated and showed that the meta-formulation can improve but is not critical to the identification of the physics component. In the added results in Table 3, we empirically ablated the effect of the number of distinct “tasks” (i.e., distinct parameter distributions) on the presented method, through which we empirically verified the theoretical condition for identifiability established in Theorem 1-iv). In the added results in Fig 4, we further added ablation studies to show the effect of the number of parameters to be identified on the presented method.
Wasserstein vs. KL distance: This is an excellent question which we will investigate in our future studies.
Replying to Rebuttal by Authors
Official Comment by Authors
Dear Reviewer 5WN4,
Thank you again for your constructive feedback. We hope that you have had time to go through our response in addressing your previous comments. As the author discussion period is closing soon, we would like to follow up to see if you have additional comments for further discussion.
Best regards,
Authors
Replying to Official Comment by Authors
Official Comment by Reviewer 5WN4
Thank you for looking over my suggestions and concerns, I increased my confidence from 3 to 4.