Sampling Bias in Entrepreneurial Experiments

Abstract

Using data from a prominent online platform for launching new digital products, we document that the composition of the platform's 'beta testers' on the day a new product is launched has a systematic and persistent impact on success. Specifically, we use word embedding methods to classify products launched on this platform as more or less focused on the needs of female customers, and show that female-focused products launched on a typical day -when nine-in-ten users on the platform are men -experience 40% less growth and are 5 percentage points less likely to have an any users a year after launch. Using exogenous variation driven by the platform's daily newsletter, we find that that the product gender gap shrinks on days when women are more likely to engage with the platform. Conversely, entrepreneurs who happen to launch a female-focused product on an especially male-dominated day reduce their product development efforts by roughly 30% and are 4 percentage points less likely to raise venture funding. Overall, our findings suggest that sample bias can systematically corrupt signals of a startup's market potential, bias entrepreneurial strategy, and so lead to a dearth of innovations aimed at consumers who are underrepresented among early-users.

Choice Architecture, Privacy Valuations, and Selection Bias in Consumer Data

“…Lastly, our work is also connected to the recent research on how bias in input data creates biased algorithms (Cao et al 2021), biased estimates of public opinions and behaviors (Bradley et al 2021), degraded performance in business analytics (Lin 2022, Neumann et al 2022, and other market outcomes (Johnson et al 2020). Although the sources of input data bias vary, individual differences in privacy concerns is often one of them.…”

“…Note that the target population can be the firm's desired customer database and need not represent the general population. Such a bias can compromise the statistical accuracy of data-driven insights and degrade the value of shared consumer data in the following scenarios: (Cao et al 2021), where startups knew their target customer base but had no idea about the gender (or other demographic) representation of the votes from the platform.…”

“…Biased input data often leads to biased insights and decision-making. For instance, Cao et al (2021) show that gender bias in Product Hunt's product votes leads to severe bias in the predicted appeal of new products, causing female-focused products to experience 45% less growth than male-focused counterparts. As another example, Bradley et al (2021) show that using large but unrepresentative samples leads to overestimates of COVID-19 vaccine uptake by 14-17 percentage points.…”