A prompt-aware knowledge-tuning framework for histopathology subtype classification with scarce annotation

Bo Yu, Jiuman Song, Lele Cong, Xianling Cong, Jouke Dijkstra, Philip S. Yu, Hechang Chen

Abstract: Artificial intelligence can assist pathologists in diagnosing histopathology subtypes, enabling precision medicine and improving survival rates. Many approaches employ multi-scale models or combine knowledge to implement subtype diagnosis. However, they fail to identify explicit features most relevant to subtypes adaptively, resulting model relying heavily on extensive annotation. Moreover, knowledge is qualitatively represented by coarse-grained methods, such as using 0 or 1 to indicate negative or positive samples. However, they cannot be quantitatively described with a fine-grained process, such as with a probability of 0.23 or 0.81. In this paper, we propose a prompt-aware knowledge-tuning model called PAKT for subtype classification, which provides an adaptive feature generation while representing knowledge quantitatively with scarce annotation. Specifically, we design a prompt-aware module that adaptively predicts multi-scale histological probabilities. The pre-trained encoder can leverage vision prompts to obtain explicit features without extensive annotation. Furthermore, a knowledge-tuning module is constructed to provide sensible diagnostic processes. The trainable weight matrix can quantitatively represent diagnosis knowledge, reflecting the influence of different histological probabilities on subtypes. PAKT performs better than state-of-the-art methods in diagnosing subtypes, achieving an average performance improvement of over 10 %, as evidenced by extensive experimentation on both public and in-house datasets, thus validating its effectiveness. Moreover, its complexity is significantly reduced without losing performance compared with baselines. Code: https://github.com/Dennis-YB/PAKT.git

External IDs:dblp:journals/nn/YuSCCDYC26