UniNuc – Unified Automatic and Interactive Nucleus Instance Segmentation in Histopathology

Chao Qin, Fahad Shahbaz Khan, Jiale Cao, Huazhu Fu, Salman Khan, Shadab Khan, Rao Muhammad Anwer

Keywords: histopathology, instance segmentation, foundation model

Abstract: Accurate nucleus instance segmentation is necessary for quantitative histopathology, but dense cellularity, stain variability, and limited annotations make purely automatic pipelines prone to systematic errors. Prior efforts for nuclei segmentation, have utilized Segment Anything Model (SAM) or its variants, but often traded-off between having a fully automatic segmentation (without interactivity), or retained interactivity while only achieving limited representational capacity that can break in dense overlapping tissue regions. To address these gaps, we propose UniNuc, a unified prompt-based model that achieves high-quality automatic instance segmentation while providing an optional ability to do interactive refinement. To reduce merge/split errors in crowded tissue, UniNuc introduces a Boundary–Context Fusion Encoder that combines boundary-sensitive convolutional features with long-range transformer context, producing SAM-compatible multi-scale features for high accuracy without requiring scaling to large backbones (e.g., SAM-Huge) and remains efficient. For automatic segmentation, an auxiliary network predicts class-aware box prompts (optionally guided by language priors), which are processed by the same prompt encoder and mask decoder used for user-provided prompts, and enables a consistent human-in-the-loop workflow. Extensive experiments on PanNuke and a benchmark of 14 histopathology datasets show that UniNuc achieves state-of-the-art performance in both automatic and interactive settings, outperforming substantially larger SAM-Huge-based baselines while using fewer parameters and FLOPs.

Primary Subject Area: Segmentation

Secondary Subject Area: Application: Histopathology

Registration Requirement: Yes

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 307