A Framework for Dataset Benchmarking and Its Application to a New Movie Rating Dataset

Abstract

Rating datasets are of paramount importance in recommender systems research. They serve as input for recommendation algorithms, as simulation data, or for evaluation purposes. In the past, public accessible rating datasets were not abundantly available, leaving researchers no choice but to work with old and static datasets like MovieLens and Netflix. More recently, however, emerging trends as social media and smartphones are found to provide rich data sources which can be turned into valuable research datasets. While dataset availability is growing, a structured way for introducing and comparing new datasets is currently still lacking. In this work, we propose a five-step framework to introduce and benchmark new datasets in the recommender systems domain. We illustrate our framework on a new movie rating dataset -called MovieTweetings -collected from Twitter. Following our framework, we detail the origin of the dataset, provide basic descriptive statistics, investigate external validity, report the results of a number of reproducible benchmarks, and conclude by discussing some interesting advantages and appropriate research use cases.

Improving accountability in recommender systems research through reproducibility

“…More specifically, reproducibility on how the datasets are created is more difficult if not enough data are presented, such as the cleaning process, whether anonymization is applied, etc. (see Dooms et al 2016). On the other hand, reproducibility in statistical testing is necessary if we expect the reported results to be statistically significant, while ensuring some level of accountability and transparency in the process.…”

“…If not taken into consideration, the evaluation of a recommender system can point to an algorithm performing significantly better than it would in a real-world situation simply due to the fact that it will recommend mostly popular items (Zhao et al 2013), i.e., items that the user does not need to get recommended as there is a high probability that she already knows them. These biases that appear naturally create another challenge toward accountability and transparency, if, e.g., changes made to the dataset affecting these biases are not reported, or because a dataset might be more biased than another one (Dooms et al 2016), make it impossible to obtain the same results. The important issue here would be that such biases (or other properties of the evaluation for that matter) are not well understood at the moment, and because of that, they are not properly acknowledged when designing the experiments nor when the settings are reported.…”

Online Doctor Recommendation with Convolutional Neural Network and Sparse Inputs

“…The density metric [ 45 ] in Table 3 , which means that how much elements are rated, is calculated according to the following equation: …”

Introduction to the Special Issue on Recommender System Benchmarking

“…Dooms et al [2016] present an extensive analysis of a new, continuously updated, movie rating dataset. Movie rating datasets, like MovieLens and Netflix, were always of paramount importance in recommender systems research, but they are static and became old.…”