The reusable holdout: Preserving validity in adaptive data analysis

Dwork, Cynthia; Feldman, Vitaly; Hardt, Moritz; Pitassi, Toniann; Reingold, Omer; Roth, Aaron

doi:10.1126/science.aaa9375

Cited by 267 publications

(247 citation statements)

References 18 publications

(18 reference statements)

Supporting

Mentioning

244

Contrasting

Unclassified

Order By: Relevance

“…Nevertheless, we point out that to confirm the stability of our model and in order to ensure the reproducibility of the data, it will be needed to extend the number of observations included in the training set and to collect a further sample of naïve participants to test the model with an out-ofsample procedure [28].…”

Section: Discussionmentioning

confidence: 99%

How Human-Mouse Interaction can Accurately Detect Faked Responses About Identity

Monaro

Fugazza

Gamberini

et al. 2017

Symbiotic Interaction

View full text Add to dashboard Cite

Abstract. Identity verification is nowadays a very sensible issue. In this paper, we proposed a new tool focused on human-mouse interaction to detect fake responses about identity. Experimental results showed that this technique is able to detect fake responses about identities with an accuracy higher than 95%. In addition to a high sensitivity, the described methodology exceeds the limits of the biometric measures currently available for identity verification and the constraints of the traditional lie detection cognitive paradigms. Thanks to the many advantages offered by this technique, its application looks promising especially in field of national and global security as anti-terrorist measure. This paper represents an advancement in the knowledge of symbiotic systems demonstrating that humanmachine interaction may be well integrated into security systems.

show abstract

Section: Discussionmentioning

confidence: 99%

How Human-Mouse Interaction can Accurately Detect Faked Responses About Identity

Monaro

Fugazza

Gamberini

et al. 2017

Symbiotic Interaction

View full text Add to dashboard Cite

show abstract

“…Organizers need to find ways to prevent data leakage and overfitting 83 , such as by limiting the number of submissions to the leaderboard or limiting the information revealed by the leaderboard 84 .…”

mentioning

confidence: 99%

Crowdsourcing biomedical research: leveraging communities as innovation engines

et al. 2016

View full text Add to dashboard Cite

The generation of large-scale biomedical data is creating unprecedented opportunities for basic and translational science. Typically, the data producers perform initial analyses, but it is very likely that the most informative methods may reside with other groups. Crowdsourcing the analysis of complex and massive data has emerged as a framework to find robust methodologies. When the crowdsourcing is done in the form of collaborative scientific competitions, known as Challenges, the validation of the methods is inherently addressed. Challenges also encourage open innovation, create collaborative communities to solve diverse and important biomedical problems, and foster the creation and dissemination of well-curated data repositories.

show abstract

“…They identified stability as the key necessary and sufficient condition for learnability. Recently, algorithmic stability has been also connected to differential privacy40, (robust and perfect) generalization6, adaptive data analysis41, adaptive learning and compression schemes42.…”

mentioning

confidence: 99%

Generating highly accurate prediction hypotheses through collaborative ensemble learning

Arsov

Pavlovski

Basnarkov

et al. 2017

Sci Rep

View full text Add to dashboard Cite

Ensemble generation is a natural and convenient way of achieving better generalization performance of learning algorithms by gathering their predictive capabilities. Here, we nurture the idea of ensemble-based learning by combining bagging and boosting for the purpose of binary classification. Since the former improves stability through variance reduction, while the latter ameliorates overfitting, the outcome of a multi-model that combines both strives toward a comprehensive net-balancing of the bias-variance trade-off. To further improve this, we alter the bagged-boosting scheme by introducing collaboration between the multi-model’s constituent learners at various levels. This novel stability-guided classification scheme is delivered in two flavours: during or after the boosting process. Applied among a crowd of Gentle Boost ensembles, the ability of the two suggested algorithms to generalize is inspected by comparing them against Subbagging and Gentle Boost on various real-world datasets. In both cases, our models obtained a 40% generalization error decrease. But their true ability to capture details in data was revealed through their application for protein detection in texture analysis of gel electrophoresis images. They achieve improved performance of approximately 0.9773 AUROC when compared to the AUROC of 0.9574 obtained by an SVM based on recursive feature elimination.

show abstract

The reusable holdout: Preserving validity in adaptive data analysis

Cited by 267 publications

References 18 publications

How Human-Mouse Interaction can Accurately Detect Faked Responses About Identity

How Human-Mouse Interaction can Accurately Detect Faked Responses About Identity

Crowdsourcing biomedical research: leveraging communities as innovation engines

Generating highly accurate prediction hypotheses through collaborative ensemble learning

Contact Info

Product

Resources

About