Background : Precise spatial recognition is critical for successful decision-making during open and laparoscopic and robotic liver surgeries. However, real-time visualization of internal anatomy remains challenging. To overcome this, we developed a novel augmented reality (AR) navigation system that integrates spatial computing and machine learning. This system is markerless and compatible with both open, laparoscopic , and robotic hepatectomy procedures, employing a video pass-through head-mounted display (HMD) for enhanced intraoperative visualization and decision support.

Methods : Our system utilizes preoperative CT/MRI data to create detailed 3D models of the liver, tumors, arteries, veins, and bile ducts. These models are dynamically aligned with patient anatomy using an Apple Vision Pro HMD equipped with high-resolution 3D cameras, LiDAR sensors, and positional sensors. To address challenges unique to open surgeries, such as varying lighting and dynamic fields, the system incorporates specialized tracking algorithms and wide-field spatial rendering. Evaluations were conducted in simulated and cadaveric environments with experienced and novice surgeons assessing accuracy, usability, and spatial awareness.

Results : The system demonstrated spatial tracking accuracy within 0–2 mm and video latency of 0.012 seconds, meeting clinical requirements for real-time application. Surgeons reported improved spatial perception, surgical accuracy, and decision-making across open and lap-robotic settings. Its versatility allowed seamless transitions between modalities, highlighting its adaptability and potential for broader surgical applications.

Conclusions : This markerless spatial AR navigation represents a significant leap in HBP surgical support, combining cutting-edge spatial computing and machine learning to enhance precision and safety. Future optimizations and clinical trials aim to establish its use across diverse surgical specialties worldwide.