Rapid Assisted Visual Search

Abstract

Designing useful human-AI interaction for clinical workflows remains challenging despite the impressive performance of recent AI models. One specific difficulty is a lack of successful examples demonstrating how to achieve safe and efficient workflows while mitigating AI imperfections. In this paper, we present an interactive AI-powered visual search tool that supports pathologists in cancer assessments. Our evaluation with six pathologists demonstrates that it can 1) reduce time needed with maintained quality, 2) build user trust progressively, and 3) learn and improve from use. We describe our iterative design process, model development, and key features. Through interviews, design choices are related to the overall user experience. Implications for future human-AI interaction design are discussed with respect to trust, explanations, learning from use, and collaboration strategies. CCS CONCEPTS• Human-centered computing → Empirical studies in interaction design; • Applied computing → Life and medical sciences; • Computing methodologies → Machine learning.

Enabling large-scale screening of Barrett’s esophagus using weakly supervised deep learning in histopathology

“…Lower-risk workflow scenarios can involve manual review of all slides by pathologists where either the H&E or TFF3 BE-TransMIL model outputs (e.g., predictions, attention heatmaps) are provided to pathologists to guide their review and speed-up assessment time. User interfaces have recently been introduced in MLassisted histopathology workflows [29][30][31] , where open questions include how specific visualizations can best assist pathologists' practice to accelerate their visual assessment of slides or aid their diagnostic decision-making. For instance, an overlay of model-generated attention heatmap on the whole-slide image with the ability to adjust opacity could help pathologists focus on the highlighted regions, which could lead to a reduction of overall review time.…”

Surgment: Segmentation-enabled Semantic Search and Creation of Visual Question and Feedback to Support Video-Based Surgery Learning

“…Previous research primarily utilizes texture-level similarity for image retrieval [7,29,48]. Studies like Hegde et al [29] and Cai et al [7] use neural features for pattern matching and image fetching, focusing on pathology images with rich texture features.…”

Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology

“…Within this vast, growing space, our research and design exploration within medical AI imaging (e.g., ophthalmology [7,9], pathology [23,49,50,80]), specifically in radiology [5, 13, 26-28, 97, 132, 136], seeks to better understand -early within AI development processes -if and how specific, anticipated VLM capabilities could be beneficial in assisting clinical workflows.…”