Design Principles of Multi-Scale J-Invariant Networks for Self-Supervised Image Denoising

Hayeong Yu, Seungjae Han, Young-Gyu Yoon

Abstract: Recent advancements in image denoising have leveraged neural networks to enhance performance, particularly in scenarios where clean-noisy image pairs are unavailable. In this context, self-supervised image denoising methods have gained prominence, centered around the principle of $\mathcal{J}$-invariance - ensuring that the output pixel is not influenced by its corresponding input pixel. Traditionally, enforcing $\mathcal{J}$-invariance has constrained blind spot network (BSN) designs, requiring even core operations such as upsampling or downsampling to follow complex rules. This limitation has led to the exclusion of efficient multi-resolution architectures such as U-net, increasing computational complexity. To address these constraints, we introduce generalized design principles for multi-scale $\mathcal{J}$-invariant networks that allow for the flexible incorporation of nearly any architectural elements. This approach challenges the prevailing notion that $\mathcal{J}$-invariance must be maintained throughout the entire process. Based on our design principles, we present U-BSN, a novel $\mathcal{J}$-invariant network design that utilizes the versatile U-Net architecture, adapting it to accommodate self-supervised learning effectively. We also propose randomized PD, an advanced technique that enhances denoising of real-world images with structured noise. Experimental results validate that U-BSN surpasses existing BSNs in handling real-world noise scenarios and achieves the lowest computational complexity among comparable networks, thus confirming the effectiveness of our design principles and proposed methodologies.