Early Stopping for Deep Image Prior

Hengkang Wang, Taihui Li, Zhong Zhuang, Tiancong Chen, Hengyue Liang, Ju Sun

Abstract: Deep image prior (DIP) and its variants have shown remarkable potential to solve inverse problems in computational imaging (CI), needing no separate training data. Practical DIP models are often substantially overparameterized. During the learning process, these models first learn the desired visual content and then pick up potential modeling and observational noise, i.e., performing early learning then overfitting. Thus, the practicality of DIP hinges on early stopping (ES) that can capture the transition period. In this regard, most previous DIP works for CI tasks only demonstrate the potential of the models, reporting the peak performance against the ground truth but providing no clue about how to operationally obtain near-peak performance without access to the ground truth. In this paper, we set to break this practicality barrier of DIP, and propose an effective ES strategy that consistently detects near-peak performance across several CI tasks and DIP variants. Simply based on the running variance of DIP intermediate reconstructions, our ES method not only outpaces the existing ones---which only work in very narrow regimes, but also remains effective when combined with methods that try to mitigate overfitting.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=gHsRMCdn6C&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)

Changes Since Last Submission:

We express our gratitude for the valuable suggestions provided by the previous reviewers and the AE! The major changes since our last submission are as follows:

(1) Compared DIP and DIP+ES with the SOTA diffusion-based method for inverse problems---DDNM+ (2023 ICLR oral) in Table 5 and Table 9.
(2) Added remarks on diffusion models for inverse problems at the end of the introduction section and pointed out the overfitting of diffusion-based models in Appendix A.8.
(3) Added one more application---RAW images demosaicing and denoising in Appendix A.7.10.
(4) Added the ablation study for several important hyper-parameters of the DIP models in Section 3.5.
(5) Added more visual comparisons in Figure 11, Figure 27 and Figure 28.
(6) Moved the image super-resolution section to the main body.

Code: https://github.com/sun-umn/Early_Stopping_for_DIP

Assigned Action Editor: ~Wei_Liu3

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1509