Respiratory rate is a strong indicator of clinical trajectory and forms the basis of patient care and assessment. However, clinicians often face barriers to easily obtaining a respiratory rate without inefficient methods or costly technology. To remedy this, several phone applications have emerged where clinicians can tap out each breath to calculate a respiratory rate. We aimed to map the available evidence for tap-per-breath applications used in healthcare settings. We searched for articles using multiple databases; including primary research articles which evaluated tap-per-breath apps in healthcare settings. 14 articles were selected for this review, mostly cross-sectional and hospital based. Most applications reported high usability and efficiency, although results of accuracy were mixed across the included literature. Median-based apps were more often an accurate measure of respiratory rate, however more research is required. Articles were commonly limited in generalisability due to poorly defined reference standards, small sample sizes, or using retrospective video recordings for patient assessment. Studies showed favourable usability and efficiency across the literature, with median-based apps demonstrating greater consistency and accuracy of respiratory rate measurements. Though the scope of this review and limited evidence restrict any far-reaching clinical implications until further evidence emerges.

Introduction: Augmented Reality (AR) is increasingly utilized in healthcare, yet its role in outpatient care remains unclear. Patients frequently struggle with understanding medical information. AR could bridge this gap by offering immersive and interactive educational tools. This narrative review assesses AR’s current applications in outpatient settings, categorizes usage types, and highlights limitations and research directions. Methods: PubMed, Embase, Web of Science, and Cochrane Library databases were searched for peer-reviewed articles published between January 2015 and February 2025. Studies using AR in outpatient settings were analyzed and grouped into five clinical categories. Results: Nineteen studies involving 987 participants met inclusion criteria. AR applications included patient education (n=3), cognitive and functional assessment (n=3), device interaction and remote monitoring (n=3), procedural guidance (n=5), and rehabilitation support (n=5). Most were pilot studies (n=6) with small sample sizes (median=28). AR consistently enhanced patient understanding, engagement, and procedural support. Limitations included bulky specialized hardware affecting comfort, reliability concerns, technical challenges, and platform inconsistencies. Conclusion: AR shows promise in outpatient care, particularly in education, assessment, device interaction, procedural guidance, and rehabilitation. Despite usability and clinical advantages, hardware complexity and technical reliability remain challenging. Future research should include large-scale RCTs and integration strategies.

not-yet-known not-yet-known not-yet-known unknown Arthroscopy is a minimally invasive surgical procedure used to diagnose and treat joint problems. The clinical workflow of arthroscopy typically involves inserting an arthroscope into the joint through a small incision, during which surgeons navigate and operate largely by relying on their visual assessment through the arthroscope. However, the arthroscope’s restricted field of view and lack of depth perception pose challenges in navigating complex articular structures and achieving surgical precision during procedures. Aiming at enhancing intraoperative awareness, we present a robust pipeline that incorporates simultaneous localization and mapping, depth estimation, and 3D Gaussian splatting to realistically reconstruct intra-articular structures solely based on monocular arthroscope video. Extending 3D reconstruction to Augmented Reality (AR) applications, our solution offers AR assistance for articular notch measurement and annotation anchoring in a human-in-the-loop manner. Compared to traditional Structure-from-Motion and Neural Radiance Field-based methods, our pipeline achieves dense 3D reconstruction and competitive rendering fidelity with explicit 3D representation in 7 minutes on average. When evaluated on four phantom datasets, our method achieves RMSE = 2.21 mm reconstruction error, PSNR = 32.86 and SSIM = 0.89 on average. Because our pipeline enables AR reconstruction and guidance directly from monocular arthroscopy without any additional data and/or hardware, our solution may hold the potential for enhancing intraoperative awareness and facilitating surgical precision in arthroscopy. Our AR measurement tool achieves accuracy within 1.59 ± 1.81 mm and the AR annotation tool achieves a mIoU of 0.721.

A document by Ngoc Truc Ngan Ho. Click on the document to view its contents.

Despite the benefits of minimally invasive surgery, interventions such as laparoscopic liver surgery present unique challenges, like the significant anatomical differences between preoperative images and intraoperative scenes due to pneumoperitoneum, patient pose, and organ manipulation by surgical instruments. To address these challenges, we propose a method for intraoperative 3D reconstruction of the surgical scene, including vessels and tumors, without altering the surgical workflow. The technique combines Neural Radiance Field (NeRF) reconstructions from tracked laparoscopic videos with ultrasound 3D compounding. We evaluate the accuracy of our reconstructions on a clinical laparoscopic liver ablation dataset, consisting of laparoscope and patient reference poses from optical tracking, laparoscopic and ultrasound videos, as well as preoperative and intraoperative CTs. We propose a solution to compensate for liver deformations due to pressure applied during ultrasound acquisitions, improving the overall accuracy of the 3D reconstructions compared to the ground truth intraoperative CT with pneumoperitoneum. We train a unified NeRF from the ultrasound and laparoscope data, which allows real-time view synthesis providing surgeons with comprehensive intraoperative visual information for laparoscopic liver surgery.

not-yet-known not-yet-known not-yet-known unknown Deep learning models are used to minimize the number of polyps that goes unnoticed by the experts and to accurately segment the detected polyps during interventions. Although state-of-the-art models are proposed, it remains a challenge to define representations that are able to generalize well and that mediate between capturing low-level features and higher-level semantic details without being redundant. Another challenge with these models is that they are computation and memory intensive, which can pose a problem with real-time applications. To address these problems, we propose PlutoNet for polyp segmentation which requires only 9 FLOPs and 2,626,537 parameters, less than 10% of the parameters required by its counterparts. With PlutoNet, we propose a novel decoder consistency training approach that consists of a shared encoder, the modified partial decoder, which is a combination of the partial decoder and full-scale connections that capture salient features at different scales without redundancy, and the auxiliary decoder which focuses on higher-level semantic features. We train the modified partial decoder and the auxiliary decoder with a combined loss to enforce consistency, which helps strengthen learned representations. We perform ablation studies and experiments which show that PlutoNet performs significantly better than the state-of-the-art models, particularly on unseen datasets.

Background: Percutaneous nephrostomy can be an effective means of preventing irreparable renal damage from obstructive renal disease thereby providing patients with more time to access treatment to remove the source of the blockage. In sub-Saharan Africa, where there is limited access to treatments such as dialysis and transplantation, a nephrostomy can be life-saving. Training this procedure in simulation can allow trainees to develop their technical skills without risking patient safety, but still requires an ex-pert observer to provide performative feedback. In this study, we evaluate the feasibility of using video as an accessible method to assess skill in simulated percutaneous nephrostomy. Methods: Six novice urology residents and six expert urologists from Ouakam Military Hospital in Dakar, Senegal performed 4 nephrostomies each using our setup. We computed motion-based metrics for each trial from the predicted bounding boxes of a trained object detection network and compared these metrics between novices and experts. Results: We were able to measure significant differences in both ultrasound and needle handling between novice and expert participants. Additionally, we could measure performance changes within each group over multiple trials. Conclusions: Video-based skill assessment is a feasible and accessible option for providing trainees with quantitative feedback in sub-Saharan Africa.

The Segment Anything Model (SAM) is a powerful vision foundation model that is revolutionizing the traditional paradigm of segmentation. Despite this, reliance on prompting each frame and large computational cost limit its usage in robotically assisted surgery. Applications, such as augmented reality guidance, require little user intervention along with efficient inference to be usable clinically. In this study, we address these limitations by adopting lightweight SAM variants to meet the efficiency requirement and employing fine-tuning techniques to enhance their generalization in surgical scenes. Recent advancements in Tracking Any Point (TAP) have shown promising results in both accuracy and efficiency, particularly when points are occluded or leave the field of view. Inspired by this progress, we present a novel framework that combines an online point tracker with a lightweight SAM model that is fine-tuned for surgical instrument segmentation. Sparse points within the region of interest are tracked and used to prompt SAM throughout the video sequence, providing temporal consistency. The quantitative results surpass the state-of-the-art semi-supervised video segmentation method XMem on EndoVis2015 with 84.8 IoU and 91.0 Dice. In terms of efficiency, we tested our method on a single 4060/4090 GPU, achieving an over 25/90 FPS. Code is available at: https://github.com/wuzijian1997/SIS-PT-SAM

In percutaneous pelvic trauma surgery, accurate placement of Kirschner wires(K-wires) is crucial to ensure effective fracture fixation and avoid complications due to breaching the cortical bone along an unsuitable trajectory. Surgical navigation via mixed reality(MR) can help achieve precise wire placement in a low-profile form factor. Current approaches in this domain are as yet unsuitable for real-world deployment because they fall short of guaranteeing accurate visual feedback due to uncontrolled bending of the wire. To ensure accurate feedback, we introduce StraightTrack, an MR navigation system designed for percutaneous wire placement in complex anatomy. StraightTrack features a marker body equipped with a rigid access cannula that mitigates wire bending due to interactions with soft tissue and a covered bony surface. Integrated with an Optical See-Through Head-Mounted Display (OST HMD capable of tracking the cannula body, StraightTrack offers real-time 3D visualization and guidance without external trackers. In phantom experiments with two experienced orthopedic surgeons, StraightTrack improves wire placement accuracy, achieving the ideal trajectory within 5.26 ± 2.29 mm and 2.88 ±1.49°, compared to over 12.08 mm and 4.07° for comparable methods. As MR navigation systems continue to mature, StraightTrack realizes their potential for internal fracture fixation and other percutaneous orthopedic procedures.

The estimation of the pose of surgical instruments is important in Robot-assisted Minimally Invasive Surgery (RMIS) to assist surgical navigation and enable autonomous robotic task execution. The performance of current instrument pose estimation methods deteriorates significantly in the presence of partial tool visibility, occlusions, and changes in the surgical scene. In this work, a vision-based framework is proposed for markerless estimation of the 6DoF pose of surgical instruments. To deal with partial instrument visibility, a keypoint object representation is used and stable and accurate instrument poses are computed using a PnP solver. To boost the learning process of the model under occlusion, a new mask-based data augmentation approach has been proposed. To validate our model, a dataset for instrument pose estimation with highly accurate ground truth data has been generated using different surgical robotic instruments. The proposed network can achieve submillimeter accuracy and our experimental results verify its generalisability to different shapes of occlusion.

Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder with a strong genetic component. This research work aims to identify key genes associated with ASD using a hybrid deep learning approach To identify influential/key genes associated with ASD, a protein-protein interaction (PPI) network is constructed, and a Graph Convolutional Network (GCN), a deep learning method, is employed. The GCN extracts features from the network structure based on gene interactions. Subsequently, Logistic Regression (LR) leverages these features to classify genes into influential and non-influential categories. The LR is chosen for its effectiveness in binary classification and ability to reveal feature importance, providing valuable insights into the underlying genetic mechanisms of ASD. Additionally, a simulation using the Susceptible-Infected (SI) model is conducted to calculate the infection ability of influential genes. This simulation demonstrates the higher infection ability of the genes identified by the proposed method, highlighting its effectiveness in pinpointing key genetic factors associated with ASD. Also, the result compared with existing centrality methods. The proposed method performs better to identify key genes involve in ASD. Our proposed method outperforms traditional measures in identifying key genes in the network, offering potential applications for identifying biomarkers. This innovative approach aligns with advancements in therapeutic and diagnostic systems, healthcare information systems, and neural engineering, providing a robust framework for future ASD research and potential applications in other neurodevelopmental disorders.

Welcome to this Special Issue of Wiley’s Healthcare Technology Letters (HTL) journal dedicated to the 2023 edition of the Augmented Environments for Computer-Assisted Interventions (AE-CAI), Computer Assisted and Robotic Endoscopy (CARE), and Context-aware Operating Theatres (OR 2.0) joint workshop. We are pleased to present the proceedings of this exciting scientific gathering held in conjunction with the Medical Image Computing and Computer-Assisted Interventions (MICCAI) conference on October 8th, 2023 in Vancouver, British Columbia, Canada. We hope that you will enjoy reading this Special Issue and we look forward to your continuing support and participation in future editions of the AE-CAI, CARE and OR 2.0 workshops. Their continued success demands our ongoing commitment and support, and we hope to welcome you all to the next edition of the workshop at MICCAI 2024 in Marrakesh, Marocco.

The use of head-mounted augmented reality (AR) for surgeries has grown rapidly in recent years. AR aids in intraoperative surgical navigation through overlaying 3D holographic reconstructions of medical data. However, performing AR surgeries on complex areas poses challenges in terms of accuracy and speed. This study explores the feasibility of an AR guidance system for resections of positive tumor margins at the head and neck region. We present an intraoperative solution that enables surgeons to access holographic reconstructions of resected cadaver tissues. The solution involves using 3D scanner to capture detailed scans of the resected tissue, which are uploaded into our software. It then converts these scans into holograms that are viewable through a head-mounted AR display. Surgeons navigates the tumor site by re-aligning these holograms with cadavers using gestures or voice commands. This workflow runs concurrently with frozen section analysis. On average, we achieve an uploading time of 2.98 min, visualization time of 1.05 min and re-alignment time of 4.39 min, within the 20 - 30 min window for frozen section analysis. We achieve a mean re-alignment error of 3.1 mm. Our software provides a foundation for new product development in using AR to navigate complex anatomy in surgery.

Kidney stones require surgical removal when they grow too large to be broken up externally or to pass on their own. Upper tract urothelial carcinoma are also sometimes treated endoscopically in a similar procedure. These surgeries are difficult, particularly for trainees who often miss tumors, stones or stone fragments, requiring re-operation. One cause of difficulty is the high cognitive strain surgeons experience in creating accurate mental models during the endoscopic operation. Furthermore, there are no patient-specific simulators to facilitate training or standardized visualization tools for ureteroscopy despite its high prevalence. We propose ASSIST-U, a system to automatically create realistic ureteroscopy images and videos solely using preoperative CT images to address these unmet needs. We train a 3D UNet model to automatically segment CT images and construct 3D surfaces. These surfaces are then skeletonized for rendering and camera position tracking. Finally, we train a style transfer model using Contrastive Unpaired Translation (CUT) to synthesize realistic ureteroscopy images. Cross validation on the UNet model achieved a Dice score of 0.853 $\pm$ 0.084 for the CT segmentation step. CUT style transfer produced visually plausible images; the Kernel Inception Distance to real ureteroscopy images was reduced from 0.198 (rendered) to 0.089 (synthesized). We also qualitatively demonstrate the entire pipeline from CT to synthesized ureteroscopy. The proposed ASSIST-U system shows promise for aiding surgeons in visualization of kidney ureteroscopy.

The integration of Augmented Reality (AR) into daily surgical practice is withheld by the correct registration of pre-operative data. This includes intelligent 3D model superposition whilst simultaneously handling real and virtual occlusions caused by the AR overlay. Occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. Robotic surgery is particularly suited to tackle these integration challenges in a stepwise approach as the robotic console allows for different inputs to be displayed in parallel to the surgeon. Nevertheless, real-time de-occlusion requires extensive computational resources which further complicates clinical integration. This work tackles the problem of instrument occlusion and presents, to our best knowledge, the first-in-human on edge deployment of a real-time binary segmentation pipeline during three robot-assisted surgeries: partial nephrectomy, migrated endovascular stent removal and liver metastasectomy. To this end, a state-of-the-art real-time segmentation and 3D model pipeline was implemented and presented to the surgeon during live surgery. The pipeline allows real-time binary segmentation of 37 non-organic surgical items, which are never occluded during AR. The application features real-time manual 3D model manipulation for correct soft tissue alignment. The proposed pipeline can contribute towards surgical safety, ergonomics and acceptance of AR in minimally invasive surgery.

Efficient communication and collaboration are essential in the operating room for successful and safe surgery. While many technologies are improving various aspects of surgery, communication between attending surgeons, residents, and surgical teams is still limited to verbal interactions that are prone to misunderstandings. Novel modes of communication can increase speed and accuracy, and transform operating rooms. We present a mixed reality (MR) based gaze sharing application on Microsoft HoloLens 2 headset that can help expert surgeons indicate specific regions, communicate with decreased verbal effort, and guide novices throughout an operation. We test the utility of the application with a user study of endoscopic kidney stone localization completed by urology expert and novice surgeons. We observe improvement in the NASA Task Load Index surveys (up to 25.23%), in the success rate of the task (6.98% increase in localized stone percentage), and in gaze analyses (up to 31.99%). The proposed application shows promise in both operating room applications and surgical training tasks.

Endoscopic renal surgeries have high re-operation rates, particularly for lower volume surgeons. Due to the limited field and depth of view of current endoscopes, mentally mapping preoperative computed tomography (CT) images of patient anatomy to the surgical field is challenging. The inability to completely navigate the intrarenal collecting system leads to missed kidney stones and tumors, subsequently raising recurrence rates. We propose a guidance system to estimate the endoscope positions within the CT to reduce re-operation rates. We use a Structure from Motion algorithm to reconstruct the kidney collecting system from the endoscope videos. In addition, we segment the kidney collecting system from CT scans using 3D U-Net to create a 3D model. We can then register the two collecting system representations to provide information on the relative endoscope position. We demonstrate correct reconstruction and localization of intrarenal anatomy and endoscope position. Furthermore, we create a 3D map supported by the RGB endoscope images to reduce the burden of mental mapping during surgery. The proposed reconstruction pipeline has been validated for guidance. It can reduce the mental burden for surgeons and is a step towards our long-term goal of reducing re-operation rates in kidney stone surgery.

This work present a proof-of-concept of a robotic-driven intra-operative scanner designed for knee cartilage lesion repair, part of a system for direct in vivo bioprinting. The proposed system is based on a photogrammetric pipeline, which reconstructs the cartilage and lesion surfaces from sets of photographs acquired by a robotic-handled endoscope, and produces 3D grafts for further printing path planning. A validation on a synthetic phantom is presented, showing that —despite the cartilage smooth and featureless surface— the current prototype can accurately reconstruct osteochondral lesions and their surroundings with mean error values of 0.199 ± 0.096 mm but with noticeable concentration on areas with poor lighting or low photographic coverage. The system can also accurately generate grafts for bioprinting, although with a slight tendency to underestimate the actual lesion sizes, producing grafts with coverage errors of -12.2 ± 3.7, -7.9 ± 4.9 and -15.2 ± 3.4 % for the medio-lateral, antero-posterior and craneo-caudal directions respectively. Improvements in lighting and acquisition for enhancing reconstruction accuracy are planned as future work, as well as integration into a complete bioprinting pipeline and validation with ex vivo phantoms.