Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge

Research output: Contribution to journalShort survey/ReviewContributedpeer-review

Contributors

  • Collaborators - (Author)
  • Tobias Rueckert - , Ostbayerische Technische Hochschule Regensburg, AKTORmed Robotic Surgery, Technical University of Munich (Author)
  • David Rauber - , Ostbayerische Technische Hochschule Regensburg (Author)
  • Raphaela Maerkl - , Ostbayerische Technische Hochschule Regensburg (Author)
  • Leonard Klausmann - , Ostbayerische Technische Hochschule Regensburg (Author)
  • Suemeyye R. Yildiran - , Ostbayerische Technische Hochschule Regensburg (Author)
  • Max Gutbrod - , Ostbayerische Technische Hochschule Regensburg (Author)
  • Danilo Weber Nunes - , Ostbayerische Technische Hochschule Regensburg (Author)
  • Alvaro Fernandez Moreno - , Medtronic Ltd., University College London (Author)
  • Imanol Luengo - , Medtronic Ltd. (Author)
  • Danail Stoyanov - , Medtronic Ltd., University College London (Author)
  • Nicolas Toussaint - , Medtronic Ltd. (Author)
  • Enki Cho - , Kyung Hee University (Author)
  • Hyeon Bae Kim - , Kyung Hee University (Author)
  • Oh Sung Choo - , Kyung Hee University (Author)
  • Ka Young Kim - , Kyung Hee University (Author)
  • Seong Tae Kim - , Kyung Hee University (Author)
  • Gonçalo Arantes - , University of Minho (Author)
  • Kehan Song - , Hanglok Tech (Author)
  • Jianjun Zhu - , Hanglok Tech (Author)
  • Junchen Xiong - , Hanglok Tech (Author)
  • Tingyi Lin - , Hanglok Tech (Author)
  • Shunsuke Kikuchi - , Jmees Inc. (Author)
  • Hiroki Matsuzaki - , Jmees Inc. (Author)
  • Atsushi Kouno - , Jmees Inc. (Author)
  • João Renato Ribeiro Manesco - , Universidade Estadual Paulista Júlio de Mesquita Filho (Author)
  • João Paulo Papa - , Universidade Estadual Paulista Júlio de Mesquita Filho (Author)
  • Tae Min Choi - , Korea Institute of Science and Technology (Author)
  • Tae Kyeong Jeong - , Korea Institute of Science and Technology (Author)
  • Juyoun Park - , Korea Institute of Science and Technology (Author)
  • Oluwatosin Alabi - , King's College London (KCL) (Author)
  • Meng Wei - , King's College London (KCL) (Author)
  • Tom Vercauteren - , King's College London (KCL) (Author)
  • Runzhi Wu - , Chinese University of Hong Kong (Author)
  • Mengya Xu - , Chinese University of Hong Kong (Author)
  • An Wang - , Chinese University of Hong Kong (Author)
  • Long Bai - , Chinese University of Hong Kong (Author)
  • Hongliang Ren - , Chinese University of Hong Kong (Author)
  • Amine Yamlahi - , German Cancer Research Center (DKFZ) (Author)
  • Jakob Hennighausen - , German Cancer Research Center (DKFZ) (Author)
  • Lena Maier-Hein - , German Cancer Research Center (DKFZ) (Author)
  • Satoshi Kondo - , Muroran Institute of Technology (Author)
  • Satoshi Kasai - , Niigata University of Health and Welfare (Author)
  • Kousuke Hirasawa - , Konica Minolta Inc (Author)
  • Shu Yang - , Hong Kong University of Science and Technology (Author)
  • Yihui Wang - , Hong Kong University of Science and Technology (Author)
  • Hao Chen - , Hong Kong University of Science and Technology, HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute (Author)
  • Santiago Rodríguez - , Universidad de los Andes Colombia (Author)
  • Nicolás Aparicio - , Universidad de los Andes Colombia (Author)
  • Leonardo Manrique - , Universidad de los Andes Colombia (Author)
  • Stefanie Speidel - , Clusters of Excellence CeTI: Centre for Tactile Internet, National Center for Tumor Diseases Dresden (Author)
  • Christoph Palm - , Ostbayerische Technische Hochschule Regensburg (Author)

Abstract

Reliable recognition and localization of surgical instruments in endoscopic video recordings are foundational for a wide range of applications in computer- and robot-assisted minimally invasive surgery (RAMIS), including surgical training, skill assessment, and autonomous assistance. However, robust performance under real-world conditions remains a significant challenge. Incorporating surgical context – such as the current procedural phase – has emerged as a promising strategy to improve robustness and interpretability. To address these challenges, we organized the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) sub-challenge as part of the Endoscopic Vision (EndoVis) challenge at MICCAI 2024. We introduced a novel, multi-center dataset comprising thirteen full-length laparoscopic cholecystectomy videos collected from three distinct medical institutions, with unified annotations for three interrelated tasks: surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation. Unlike existing datasets, ours enables joint investigation of instrument localization and procedural context within the same data while supporting the integration of temporal information across entire procedures. We report results and findings in accordance with the BIAS guidelines for biomedical image analysis challenges. The PhaKIR sub-challenge advances the field by providing a unique benchmark for developing temporally aware, context-driven methods in RAMIS and offers a high-quality resource to support future research in surgical scene understanding.

Details

Original languageEnglish
Article number103945
JournalMedical Image Analysis
Volume109
Publication statusPublished - Mar 2026
Peer-reviewedYes

Keywords

Keywords

  • Instrument instance segmentation, Instrument keypoint estimation, Robot-assisted surgery, Surgical phase recognition