SETMIL: Spatial Encoding Transformer-Based Multiple Instance Learning for Pathological Image Analysis

Yu Zhao, Zhenyu Lin, Kai Sun, Yidan Zhang, Junzhou Huang, Liansheng Wang, Jianhua Yao

Abstract: Considering the huge size of the gigapixel whole slide image (WSI), multiple instance learning (MIL) is normally employed to address pathological image analysis tasks, where learning an informative and effective representation of each WSI plays a central role but remains challenging due to the weakly supervised nature of MIL. To this end, we present a novel Spatial Encoding Transformer-based MIL method, SETMIL, which has the following advantages. (1) It is a typical embedded-space MIL method and therefore has the advantage of generating the bag embedding by comprehensively encoding all instances with a fully trainable transformer-based aggregating module. (2) SETMIL leverages spatial-encoding-transformer layers to update the representation of an instance by aggregating both neighbouring instances and globally-correlated instances simultaneously. (3) The joint absolute-relative position encoding design in the aggregating module further improves the context-information-encoding ability of SETMIL. (4) SETMIL designs a transformer-based pyramid multi-scale fusion module to comprehensively encode the information with different granularity using multi-scale receptive fields and make the obtained representation enriched with multi-scale context information. Extensive experiments demonstrated the superior performance of SETMIL in challenging pathological image analysis tasks such as gene mutation and lymph node metastasis prediction.