An autonomous X-ray image acquisition and interpretation system for assisting percutaneous pelvic fracture fixation

Toward perception-based anticipation of cortical breach during K-wire fixation of the pelvis

“…Yet, X-ray imaging of the full human body and from various viewing directions represents an enormous variety of images with markedly different features than visible light or tomographic imaging. The potential applications of a model like FluoroSAM are significant, as exemplified by proliferation of specialized models for X-ray image analysis in chest X-ray diagnosis [3,5,24], dental exams [25], forensics [23], intelligent surgical systems [11][12][13][14][15][16], and AI-driven educational curricula [17,19]. Within this broad spectrum of applications, the benefits of language alignment for X-ray imaging have so far been limited to diagnostic systems, where text descriptions are available as the byproduct of routine clinical workflows [5,10].…”

“…These are placed along the field of view with random location and orientation. We project digitally reconstructed radiographs (DRRs) from the patient CT and tool mesh models using a modified version of the DeepDRR simulator [26], which has been shown to support sim-to-real transfer for X-rays [8,11,15,15]. Using this version, we simultaneously obtain realistic X-ray tansmission images and corresponding projected segmentations for each organ or tool present, enabling synchronous dataset generation of 448×448 images at a rate of ∼ 4 images / s on an RTX 2080 Ti, using less than 4 GB of GPU memory.…”

“…For subsequent prompts, we follow the strategy in the original SAM model, using the previous mask as an additional prompt, followed by up to 8 point embeddings. Following [8], we use strong domain randomization to enable sim-to-real transfer, including coarse dropout, inversion, blurring, Gaussian contrast adjustment [15], random windowing, and CLAHE histogram equalization. At inference time, we window the negative log transformed images to the 99 th percentile and scale to [−1, 1].…”