Learning Universal Sample Difficulty with Pathology Foundation Models in Histopathology Image Analysis

Tianyu Liu, Lijun Wang, Tinglin Huang, Xiao Luo, Wengong Jin, Hongyu Zhao

Keywords: histopathology

TL;DR: We propose a generalized sample difficulty estia

Abstract: The fast scaling speed of histopathology datasets allows researchers to train various foundation models for disease-centered research with applications in classifying disease-state information and predicting gene expression levels. However, it has been shown that current models tend to be overconfident and make classification at a low-calibration level. This case is underexplored for regression-type tasks such as gene expression prediction as well, which could seriously affect the diagnosis and treatment based on the developed models. To resolve this critical issue, we propose a \underline{u}niversal framework\footnote{Full codes can be found here: \url{https://anonymous.4open.science/r/USD-13EB/} (also in supplementary files).} to estimate the \underline{s}ample \underline{d}ifficulty (USD) in both regression and classification tasks. In particular, we fit the data in the embedding space with Gaussian distribution and then utilize prior-informed relative Mahalanobis distance to estimate sample difficulty. Moreover, we incorporate such difficulty as a weight to regularize the model prediction, which can improve model performance by emphasizing challenging samples. Our method can be seamlessly extended to regression tasks by the incorporation of discrete targets. Extensive experiments demonstrate that our proposed USD can improve the disease-state classification accuracy by up to 3.8\% and gene-level correlation by up to 62.2\% compared with the most frequently used approaches. Finally, we provide comprehensive ablation tests to demonstrate the importance of including sample difficulty in the training stage and case studies for the reasonability of assigning samples with different difficulty levels.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 11953