Estimating the Causal Impact of Recommendation Systems from Observational Data

Abstract

Recommendation systems are an increasingly prominent part of the web, accounting for up to a third of all traffic on several of the world's most popular sites. Nevertheless, little is known about how much activity such systems actually cause over and above activity that would have occurred via other means (e.g., search) if recommendations were absent. Although the ideal way to estimate the causal impact of recommendations is via randomized experiments, such experiments are costly and may inconvenience users. In this paper, therefore, we present a method for estimating causal effects from purely observational data. Specifically, we show that causal identification through an instrumental variable is possible when a product experiences an instantaneous shock in direct traffic and the products recommended next to it do not. We then apply our method to browsing logs containing anonymized activity for 2.1 million users on Amazon.com over a 9 month period and analyze over 4,000 unique products that experience such shocks. We find that although recommendation click-throughs do account for a large fraction of traffic among these products, at least 75% of this activity would likely occur in the absence of recommendations. We conclude with a discussion about the assumptions under which the method is appropriate and caveats around extrapolating results to other products, sites, or settings.

Split-door criterion: Identification of causal effects through auxiliary outcomes

“…The example on the left shows a focal product that receives a large and sudden shock in page visits, while direct visits to its recommended product remains relatively flat. This is reminiscent of the examples analyzed in Carmi, Oestreicher-Singer and Sundararajan [2012] and Sharma, Hofman and Watts [2015]. The example on the right, however, shows more general patterns that are accepted under the split-door criterion but not considered by these previous approaches: although direct visits to both the focal and recommended products vary substantially, they do so independently, and so are still useful in our estimate of the recommender's effect.…”

“…Connections to other methods. The split-door criterion is an example of methods that use empirical independence tests to identify causal effects under certain assumptions [Jensen et al, 2008;Cattaneo, Frandsen and Titiunik, 2015;Sharma, Hofman and Watts, 2015;Grosse-Wentrup et al, 2016]. By searching for subsets of the data where desired independence holds, it also shares some properties with natural experiment methods such as instrumental variables and conditioning methods such as regression.…”

“…Because the vast majority of this data is generated in non-experimental settings, researchers typically must deal with the possibility that any causal effects of interest are complicated by a number of potential confounds. For example, even effects as conceptually simple as the causal impact of recommendations on customer purchases are likely confounded by selection effects [Lewis, Rao and Reiley, 2011], correlated demand [Sharma, Hofman and Watts, 2015], or other shared causes of both exposure and purchase. Figure 1a shows this canonical class of causal inference problems in the form of a causal graphical model [Pearl, 2009], where X is the cause and Y is its effect.…”

How algorithmic confounding in recommendation systems increases homogeneity and decreases utility

“…Liang, et al [40] draw on the language of causal analysis in describing a model of user exposure to items; this is related to distinguishing between user preference and our confidence in an observation [26]. Some work has also been done to understand the causal impact of these systems on behavior by finding natural experiments in observational data [53,55] (approximating expensive controlled experiments [33]), but it is unclear how well these results generalize. Schnabel, et al [52] use propensity weighting techniques to remove users' selection bias for explicit ratings.…”

Evolution of Ego-networks in Social Media with Link Recommendations

“…It is harder to assess whether recommendations foster or limit access to diverse types of content. The academic debate about recommendations being the bane or boon of social media is still very lively [42,10,44], with evidence brought in support of the two views. We aim to provide further evidence to shed light on this point in the context of link recommenders.…”