Cross-Modal Data Programming Enables Rapid Medical Machine Learning

Abstract

Labeling training datasets has become a key barrier to building medical machine learning models. One strategy is to generate training labels programmatically, for example by applying natural language processing pipelines to text reports associated with imaging studies. We propose cross-modal data programming, which generalizes this intuitive strategy in a theoretically-grounded way that enables simpler, clinician-driven input, reduces required labeling time, and improves with additional unlabeled data. In this approach, clinicians generate training labels for models defined over a target modality (e.g. images or time series) by writing rules over an auxiliary modality (e.g. text reports). The resulting technical challenge consists of estimating the accuracies and correlations of these rules; we extend a recent unsupervised generative modeling technique to handle this cross-modal setting in a provably consistent way. Across four applications in radiography, computed tomography, and electroencephalography, and using only several hours of clinician time, our approach matches or exceeds the efficacy of physician-months of hand-labeling with statistical significance, demonstrating a fundamentally faster and more flexible way of building machine learning models in medicine.Modern machine learning approaches have achieved impressive empirical successes on diverse clinical tasks that include predicting cancer prognosis from digital pathology, 1, 2 classifying skin lesions from dermatoscopy, 3 characterizing retinopathy from fundus photographs, 4 detecting intracranial hemorrhage through computed tomography, 5, 6 and performing automated interpretation of chest radiographs. 7,8 Remarkably, these applications typically build on standardized reference neural network architectures 9 supported in professionally-maintained open source frameworks, 10, 11 suggesting that model design is no longer a major barrier to entry in medical machine learning. However, each of these application successes was predicated on a not-so-hidden cost: massive hand-labeled training datasets, often produced through years of institutional investment and expert clinician labeling time, at a cost of hundreds of thousands of dollars per task or more. 4,12 In addition to being extremely costly, these training sets are inflexible: given a new classification schema, imaging system, patient population, or other change in the data distribution or modeling task, the training set generally needs to be relabeled from scratch. These factors suggest 1

Leveraging organizational resources to adapt models to new data modalities

“…[Data Modalities] Bridging the Modality Gap. Solutions for similar cross-modal problems (see Section 8) assume that other tasks have already been trained for the target modality [36,45,58], or data of different modalities are directly connected [23]. Examples of direct connections are images paired with captions, 2D projections of 3D point clouds, or clinical notes and lab results.…”

“…We refer to this process of adapting models to new data modalities as cross-modal adaptation (a form of transductive transfer learning [46]). Existing work in cross-modal adaptation assumes points across data modalities are easily or directly linked (e.g., captions directly linked to images, or clinical notes to lab results) to leverage zero-shot learning [45,58] or weak supervision [23]. However, in our environment, such direct connections do not often exist, resulting in a modality gap.…”

“…Even in the absence of custom classifiers for a new modality (e.g., sensitive content detection over video posts), it is still possible to apply existing resources from across an organization (e.g., knowledge-graph querying tools for videos). This holds across large enterprises [3,4], academic labs [39] and medical institutions [23]. As a result, the data management research community has developed systems and algorithms for systematically leveraging these resources for ML tasks [8,33,42,47,68].…”

“…This is difficult in our setting for two reasons. First, specifying predicates over unstructured data like video and images is an active area of research [23,65,67] (e.g., it is difficult to easily specify what a policy-violating image looks like). Second, to evaluate the performance of candidate LFs, we need a labeled development set.…”

A Protein Interaction Information-based Generative Model for Enhancing Gene Clustering

“…Due to the inherent property of Snorkel, it is widely used in various real life problems like surveillance with electronic health records 33 , clinical text classification 34 , web content and event classification 35 . Also, Snorkel is used for improving gene clusteing 36 and medical image training 37 .…”