Systematic Analysis of Challenge-Driven Improvements in Molecular Prognostic Models for Breast Cancer

Abstract

Although molecular prognostics in breast cancer are among the most successful examples of translating genomic analysis to clinical applications, optimal approaches to breast cancer clinical risk prediction remain controversial. The Sage Bionetworks–DREAM Breast Cancer Prognosis Challenge (BCC) is a crowdsourced research study for breast cancer prognostic modeling using genome-scale data. The BCC provided a community of data analysts with a common platform for data access and blinded evaluation of model accuracy in predicting breast cancer survival on the basis of gene expression data, copy number data, and clinical covariates. This approach offered the opportunity to assess whether a crowdsourced community Challenge would generate models of breast cancer prognosis commensurate with or exceeding current best-in-class approaches. The BCC comprised multiple rounds of blinded evaluations on held-out portions of data on 1981 patients, resulting in more than 1400 models submitted as open source code. Participants then retrained their models on the full data set of 1981 samples and submitted up to five models for validation in a newly generated data set of 184 breast cancer patients. Analysis of the BCC results suggests that the best-performing modeling strategy outperformed previously reported methods in blinded evaluations; model performance was consistent across several independent evaluations; and aggregating community-developed models achieved performance on par with the best-performing individual models.

Tumor mutational burden is a determinant of immune-mediated survival in breast cancer

“…Gene expression profiles were generated using the Illumina_Human_WG-v3 array platform 24 and normalized by quantile normalization with linear modeling batch correction, as described elsewhere. 55 Copy number levels were generated on the Affymetrix SNP Array 6.0 and normalized using the supervised normalization of microarrays (SNM) framework 56 as described by 55 and also using DNAcopy 54,57 to define low- and high-level copy number thresholds. A 173-gene exome sequencing panel was used to identify somatic gene mutations and generate measures of mutational burden (gene count).…”

Breast Cancer Prognostic Biomarker Using Attractor Metagenes and the FGD3–SUSD3 Metagene

“…If there was more than one platform provided for each patient, the measurements were combined and renormalized using RMA. The METABRIC dataset was renormalized by Sage Synapse (4). Because the BCAM formula is the linear combination of heterogeneous covariates, we corrected the distribution of genomic assays in each dataset by multiplying the size and the lymph node number with the ratio of the standard deviations of the genomic assays in each dataset to the standard deviation of the genomic assays in the METABRIC dataset.…”

“…A recent crowd-sourced research study, the Sage Bionetworks-DREAM Breast Cancer Prognosis Challenge (BCC; ref. 4) used the METABRIC dataset (5) containing molecular and clinical features from 1,981 patients with breast cancer. The winning model (6,7) as well as all five topscoring models made use of several molecular features, called attractor metagenes (8), as well as the FGD3-SUSD3 metagene defined by the average of the expression levels of the two genes, FGD3 and SUSD3, which are located directly adjacent to each other at Chr9q22.31.…”

Ranking and combining multiple predictors without labeled data

“…A key challenge is to improve decision making by combining these multiple predictions of unknown reliability. Automating this process of combining multiple predictors is an active field of research in decision science (cci.mit.edu/research), medicine (10), business (refs. 11 and 12 and www.kaggle.com/competitions), and government (www.iarpa.gov/Programs/ia/ACE/ace.html and www.goodjudgmentproject.com), as well as in statistics and machine learning.…”