Holistic segmentation of minimally-invasive robotic surgical scenes using feature-adaptive localization

Abstract

Holistic surgical scene segmentation is vital for training novice surgeons in robotic minimally invasive surgeries (RMIS). While the SegFormer model shows great potential due to its adaptability to various input image resolutions, the contrast between surgical instruments and anatomical structures poses challenges for segmentation output. In this paper, we propose a F eature- A daptive S patial L ocalization Seg Former ( FASL-Seg ) model, an artificial intelligence-based semantic segmentation model for surgical video analysis. Built on a SegFormer backbone, FASL-Seg introduces feature processing mechanisms to improve holistic segmentation of anatomy and instruments in robot-assisted minimally invasive surgery. FASL-Seg incorporates two feature processing streams: a High-Level Feature Projection (HLFP) targeting low-resolution features, and a Low-Level Feature Projection (LLFP) for high-resolution feature maps. These streams are coupled with a shallow decoder that maximizes information conservation from both stream outputs, together introducing robustness against diverse feature representations of tools and anatomy. Extensive experiments were conducted on four surgical benchmark datasets, where FASL-Seg achieved results that were competitive or superior to thestate-of-the-art, at mean Intersection over Union (mIoU) of 0.73 on Endoscopic Vision 2018 Challenge (EndoVis18), 0.75 on Cholecystectomy Segmentation, 0.86 on EndoVis18 tool segmentation, 0.74 on Endoscopic Vision 2017 (EndoVis17), and 0.9594 on Kvasir-Instrument binary segmentation. These results support the advantages of the proposed architecture for holistic scene segmentation, potentially enabling automated annotations of robotic videos to support RMIS training programs. Code is available at: https://github.com/Muraam-Abdel-Ghani/FASL-Seg .