Learn from orientation prior for radiograph super-resolution: Orientation operator transformer

Yongsong Huang, Tomo Miyazaki, Xiaofeng Liu, Kaiyuan Jiang, Zhengmi Tang, Shinichiro Omachi

Abstract: Highlights•We first propose Orientation Operator for enhancing CNN-based shallow feature extraction modules. This novel operator focuses on the prior knowledge of horizontal and vertical directions and introduces it into local feature extraction. The different orientation prior helps the encoder to capture shallow features for better latent representation, and further benefits the decoder in learning better nonlinear mapping for image reconstruction. To the best of our knowledge, it is the first model focusing on the orientation prior in the radiographic super-resolution task.•We also propose a multi-scale feature fusion strategy for radiographic images. This strategy considers different convolution methods that help to capture shallow features with more diverse local features. The shallow representation with diversity helps the decoder to better reconstruct SR images. Further ablation experiments demonstrate the effectiveness of the feature fusion strategy.•Finally, we propose the end-to-end orientation operator transformer for the superresolution task in radiographic images. This model includes two components: the CNN encoder for shallow feature extraction and the transformer-based decoder to reconstruct the image by connecting global information. Compared with previous approaches, Orientation Operator Transformer focuses more on the non-linear mapping for blurred mapping in radiological image reconstruction and achieves better performance.