Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery

Augmented reality applications and the future library

“…Below are other examples and future potential: AR can mask the real world and thus can alter the environment for the librarians while getting engaged in video calling with fellow librarians or users in answering queries and thereby helping maintain privacy. Augmented robotic guided surgeries are now carried out with accuracy (De Backer et al , 2023; Franco et al , 2023). Adopting augmented robotics, the library can use robots powered by AR to track inventory and locate displaced materials with precision by applying an instant document tracking system. The AR could elevate the power of a telepresence robot in a library to provide a smart interface and offer deeper insights while guiding users in their academic and research pursuits. Introducing AR solution to library routine operation, the future library improved quality of work with quicker identification of errors and the minimum possibility of oversight. Children’s sections in public libraries can pair storytelling sessions with AR technology and engage intending users in an immersive environment because visuals can be impregnated into the memory easily. Using AR technology, a misplaced document on the shelf can be easily identified because it allows smart devices with AR apps to scan instantly a large number of documents with barcodes and recognize the misplaced one. AR-based and deep learning-assisted fault detection system has been developed (Dalal et al , 2020; Zhen et al , 2021).…”

“…Augmented robotic guided surgeries are now carried out with accuracy (De Backer et al , 2023; Franco et al , 2023). Adopting augmented robotics, the library can use robots powered by AR to track inventory and locate displaced materials with precision by applying an instant document tracking system.…”

First-in-human Realtime AI-assisted Instrument Deocclusion during Augmented Reality Robotic Surgery

“…the orientation of the kidney and tumour with respect to the artery, providing initial support for navigation, as well as confirmation of clamping level with respect to possible earlier bifurcations of the vessel. During this phase both our previous solution [10] and current solution were simultaneously visually compared through separate Tile Pro inputs inside the console. The surgeon (R.D.G) stated a significant improvement in perceived latency, where the latency of the current solution is small enough for surgical adoption.…”

“…As such, previous deocclusion attempts were unsuccessful in detecting all surgical items with sufficient robustness, whilst at the same time having no prior knowledge of the objects' orientations or positions inside the real-time surgical environment. More recent work [10] showed the potential of deep learning binary instrument segmentation for robust de-occlusion during AR surgery. However, the reported latency reached up to 0.5 s and was considered unfeasible for real-time surgical use.…”

“…summarizes the comparison of evaluation metrics of the FPN model versus the DeepLabV3+ model[10] on an identical test set of 1345 images derived from four distinct RAPN procedures. The baseline model for the reported sideby-side improvements is the DeepLabV3+ with FP32 precision, implemented in Pytorch.…”