Force Feedback and Visual Fusion in Minimally Invasive Surgical Robots
DOI:
https://doi.org/10.61173/tszmb480Keywords:
Minimally Invasive Surgical Robot, Force Feedback, Visual Perception, Multi-Modal Fusion, Deep LearningAbstract
The precise operation of minimally invasive surgical (MIS) robots relies on multi-dimensional perception of the surgical scene, and the fusion of force feedback (tactile perception) and visual perception is the core technology to break the bottleneck of the “perception-decision-control” loop. This paper systematically reviews the development of force-visual fusion technology for MIS robots, analyzing from sensor layer innovation, and algorithm layer breakthroughs, to system layer integration. It examines technical adaptability and performance bottlenecks by combining typical clinical scenarios such as tumor resection, vascular suturing and nerve dissection. Finally, addressing key issues including poor generalization in in-vivo environments, difficulties in multi-modal spatiotemporal synchronization, and lack of clinical translation standards, the paper proposes future development paths based on “surgical metaverse,” “hybrid dataset training,” and tele-surgery specifications, providing references for technology R&D and clinical translation in the interdisciplinary field of medicine and engineering.
References
[1] Grand View Research. Minimally Invasive Surgery Robots Market Size Report, 2023-2030 [R]. 2023.
[2] Okamura A M. Haptic feedback in robotic-assisted minimally invasive surgery [J]. Current Opinion in Urology, 2009, 19 (1): 102-107.
[3] Okamura A M. Haptic feedback in robotic-assisted surgery [J]. Annual Review of Control, Robotics, and Autonomous Systems, 2020, 3: 459-482.
[4] Chua Z, Jarc A M, Okamura A M. Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State [C]. 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 12335-12341.
[5] Li X, Wang H, Liu C, et al. Lightweight Vision Transformer for Real-Time Force Estimation in Laparoscopic Surgery[J]. IEEE Transactions on Medical Robotics and Bionics, 2023, 5(4): 892-901.
[6] Nagatomi T, Tanaka Y, Yamamoto H, et al. A flexible tactile sensor to detect stiffness distribution without measuring displacement[J]. Sensors and Actuators A: Physical, 2022, 334: 113062.
[7] Chua Z, Jarc A M, Okamura A M. Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State[C]. 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 12335-12341.
[8] Marban A, Srinivasan V, Samek W, et al. A recurrent convolutional neural network approach for sensorless force estimation in robotic surgery[J]. Biomedical Signal Processing and Control, 2019, 50: 134-150.
[9] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[J]. Journal of Machine Learning Research, 2021, 22(174): 1-87.
[10] De Iturrate Reyzabal M, Chen M, Huang W, et al. DaFoEs: Mixing Datasets towards the generalization of vision-state deep-learning Force Estimation in Minimally Invasive Robotic Surgery[R]. arXiv preprint arXiv:2401.09239, 2024.
[11] Liu W, Pickett A, Huang K, et al. Camera Configuration Models for Machine Vision Based Force Estimation in Robot- Assisted Soft Body Manipulation[C]. 2022 International Symposium on Medical Robotics (ISMR). IEEE, 2022: 1-8.
[12] Zhang L, Chen M, Huang W, et al. Fully 3D Printed Flexible Tactile Sensor with Integrated Biomimetic Structure[J]. Nature Communications Engineering, 2023, 6(1): 1-9.
[13] Sabique P V, Ganesh P, Sivaramakrishnan R. Stereovision based force estimation with stiffness mapping in surgical tool insertion using recurrent neural network[J]. The Journal of Supercomputing, 2022, 78(8): 14648-14679.
[14] Valdastri P, Stoyanov D, de Momi E, et al. Robotic platforms for minimally invasive surgery[J]. Annual Review of Control, Robotics, and Autonomous Systems, 2018, 1: 337-362.
[15] Hannaford B, Rosen J, Friedman D W, et al. Raven- II: an open platform for surgical robotics research[J]. IEEE Transactions on Biomedical Engineering, 2013, 60(4): 954-959.
[16] Patel V, Saikali S, Kavoussi L, et al. Best practices in telesurgery: framework and recommendations from the Society of Robotic Surgery (SRS) for safe and effective implementation[J]. Journal of Robotic Surgery, 2025, 19(3): 370-383.
[17] Tian Z, Wang Y, Li J, et al. Robot-assisted spinal surgery: a systematic review and meta-analysis[J]. Journal of Orthopaedic Translation, 2022, 36: 1-12.
[18] Bahar L, Sharon Y, Nisky I. Surgeon-Centered Analysis of Robot-Assisted Needle Driving Under Different Force Feedback Conditions[J]. Frontiers in Neurorobotics, 2020, 13: 108.
[19] Intuitive Surgical. da Vinci SP Surgical System Technical Dean&Francis ISSN 2959-6157 Specifications[R]. 2023.
[20] Jian ZH, Li JY, Wu KH, Li Y, Li SX, Chen HD, Chen G. Surgical Effects of Resecting Skull Base Tumors Using Pre-operative Multimodal Image Fusion Technology: A Retrospective Study. Front Neurol. 2022 May 12;13:895638.
[21] A. I. Haq, H. Susanti, M. H. Barri, A. Zafira Putri and F. F. Fillah, „Real-time Spinal Needle Position Detection Using YOLOv8,“ 2023 8th International Conference on Instrumentation, Control, and Automation (ICA), Jakarta, Indonesia, 2023, pp. 131-136
[22] Goudeketting SR, Heinen SGH, Ünlü Ç, van den Heuvel DAF, de Vries JPM, van Strijen MJ, Sailer AM. Pros and Cons of 3D Image Fusion in Endovascular Aortic Repair: A Systematic Review and Meta-analysis. J Endovasc Ther. 2017 Aug;24(4):595-603. doi: 10.1177/1526602817708196. Epub 2017 May 9.
[23] Mitani S, Sato E, Kawaguchi N, Sawada S, Sakamoto K, Kitani T, Sanada T, Yamada H, Hato N. Case-specific threedimensional hologram with a mixed reality technique for tumor resection in otolaryngology. Laryngoscope Investig Otolaryngol. 2021 May 19;6(3):432-437.
[24] Y. -E. Lee, H. M. Husin, M. -P. Forte, S. -W. Lee and K. J. Kuchenbecker, „Learning to Estimate Palpation Forces in Robotic Surgery From Visual-Inertial Data,“ in IEEE Transactions on Medical Robotics and Bionics, vol. 5, no. 3, pp. 496-506, Aug. 2023
[25] Tim Miller, Explanation in artificial intelligence: Insights from the social sciences, Artificial Intelligence, Volume 267, 2019, Pages 1-38, ISSN 0004-3702
Downloads
Published
Issue
Section
License
Copyright (c) 2025 by the authors.
This work is licensed under a Creative Commons Attribution 4.0 International License.
