Abstract: Highlights•HIGF-Net captures high-level semantic information and low-level spatial features.•Double-stream structure transmits the shape characteristics of polyps.•Separate Refinement module reinforces the learning of uncertain regions.•Hierarchical Pyramid Fusion module fuses the multiple directions information.•Extensive evaluation of polyp segmentation on five datasets.