Life in the Data Salt Mines: A Tale of Knowledge Gain and Log-Loss

Abstract

Two denizens of wilmott.com document their quest to win a data mining contest that required estimating the probability of a binary response variable given 3,751 training samples described on 1,776 dimensions. The contest's log-loss scoring metric provided strong incentives for high-confidence estimates but also strong penalties for over-confident estimates -a pay-off structure also found in strategic investment decisions. The contest offered a low-risk environment for learning in which the only thing we could lose was our dignity, not our capital. We delved into the combined effects of the curse of dimensionality and small sample sizes on the geometries of subregions in a high-D space. We created methods to test if a given projection or sub-region was statistically useful for classifying a data point. This work illustrated five key lessons and many issues associated with data mining and statistical estimation in the era of "Big Data".