RecSys Challenge 2016

Abstract

The 2016 ACM Recommender Systems Challenge focused on the problem of job recommendations. Given a large dataset from XING that consisted of anonymized user profiles, job postings, and interactions between them, the participating teams had to predict postings that a user will interact with.The challenge ran for four months with 366 registered teams. 119 of those teams actively participated and submitted together 4,232 solutions yielding in an impressive neck-and-neck race that was decided within the last days of the challenge.

ContentWise Impressions

“…Impression datasets have been used by several articles [1,2,7,13,14,20]. They can be classified into two categories: private datasets, collected by the authors of the article but, to the best of our knowledge, not made accessible to the community, and non-redistributable datasets, made accessible only to the participants of a challenge under a non-redistribute clause.…”

“…The impression datasets that have been made available to challenge participants under a non-redistribute clause in recent years are several. Examples are those provided during the RecSys Challenges: 2016 by Xing [1,16], 2017 by Xing [2], and 2019 by Trivago [2]. Those datasets, however, were only accessible to participants of the challenge and have not been made available to the wider research community.…”

“…In previous years, other articles have used private datasets [7,14], which were not released to the community. Others were disclosed under nonredistribution clauses on challenges [1,2,13,20], where only a few researchers have access to them. Furthermore, ContentWise Impressions provides both impressions present in the interactions and without any associated interaction.…”

“…Both LinkedIn PYMK Impressions and LinkedIn Skill Endorsement [14] also present both impressions. On the other hand, other datasets [1,13] only provided impressions present in the interactions.…”

“…While researchers can indeed preprocess and subsample bigger datasets, if needed, this may result in different articles relying on different subsampling, making it more difficult to compare research results and contributing to the reproducibility crisis in our field [9,11]. For instance, Xing RecSys 2017 [1,2] contained around 1.5 users, 1.3 items, 322 interactions, and 314 impressions associated with these interactions. LinkedIn PYMK Impressions, LinkedIn Skill Endorsement Impressions, Tencent SearchAds Impressions [14] had 1.08, 0.19, and 0.15 billion impressions.…”