MMAN: Multi-Task and Multi-Scale Attention Network for Concurrently Lower Limbs Segmentation and Landmark Detection

Lulu Gai, Zhi Qiao, Lianxi Fan, Xianghong Meng, Shengru Fang, Pei Dong, Zhen Qian

Abstract: Accurate bone segmentation and anatomical landmark detection are vital tasks for the clinical evaluation and treatment planning for patients with lower limbs X-ray films. To leverage the information between the two tasks and deal with the large-scale images, we propose an efficient end-to-end deep network, i.e., multi-task and multi-scale attention network (MMAN), to concurently segment lower limb bones and localize landmarks from large-scale X-ray films in one stage. The results demonstrate that our MMAN outperforms the other state-of-the-art methods for multi-task learning or single-task landmark detection using two separate stages. Our MMAN method has two main technical contributions. First, the local and global encoders are designed to capture multi-scale inputs and provide shared representations including local image details and global contexts, respectively. Second, a global-local attention module is designed to efficiently leverage the global context and learn task-specific information from shared representations under limited computational costs.